Premature Software Testing

I've fallen victim to it a few times now and would like to remind myself of the causes and consequences. Don't get me wrong, I love testing and have attempted/done/failed at it ever since CS61B. I've used standardized test frameworks, hackish quickie frameworks from school, and created my own small frameworks for specific projects. Testing is a good thing.

But there is such a thing as testing too early. My most recent mishap happened during my internship. I finished a tool that I thought did what I want, and fully tested it. Well, the plus side was that it was perfectly tested against what it does, and worked beautifully for all possible inputs. The downside was that the code didn't do what it was supposed to do! I misunderstood my own requirements and coded something wrong. Then instead of having tests that could support me as I refactored, I ended up with an extra set of barriers that kept me constrained to wrong solutions. In other words, I believed in my tests and this slowed down my redesign towards a correct solution.

Another time this happened to me was during cs164 while I was writing an Earley parser. This one cut me deep because the solution just happened to work against the cases I specified in my tests. Then about one week before the due date, I noticed that I misread the original paper and implemented a system with different semantics. All the tests that exercised individual methods were scrapped. In the end, all tests were scrapped :(

I lost many hours on both those mistakes, but I have noticed different cases when I'm more successful. Here are some guidelines I think work well:

When to Start Testing

How to Test

There are plenty of good guides out there on 'what' to test, but I think many of them provide examples that are too trivial and brittle against refactoring.

We all knew that testing too late or not testing at all can cause cancer, but there is such a thing as testing too early. It wastes your time, and may restrict you to a suboptimal or incorrect design. Furthermore, testing isn't the same as designing. No amount of random testing will yield a solid design. Rather, an elegant design will naturally encourage you to think about tests, and those tests can then verify your implementation. Testing is like eating your vegetables: it's good for your health in the long run, but eating too much or the wrong kind will give you the runs.