Taking Automated Tests Off The Pedestal

I think that by now we've all learned that automated tests are good. I look at this as one of the big successes of the past ten years, the fact that software teams these days understand the value of tests and often feel downright embarrassed if they don't have them. Yes, there are some holdouts, but there really does seem to be a consensus about this, so it's probably a good time to look at the status quo and examine it a little bit.

Many projects have a very large number of automated tests. And, that's good. It's better than not having them. On the other hand, many teams feel like these tests are a yoke around their necks. Their build time keeps increasing. They spend more and more time dealing with test management, and at the end of the day, they know that things are getting worse. Yes, the tests enable growth in the code base, but it's that kind of reluctant growth that costs a bit more each time it happens.

The typical response when people notice this problem is to optimize. We can make builds and tests faster in a number of ways, from parallelization, build pipelining, and hardware upgrades to the more mundane yet more scalable strategy of rewriting tests so that they are more focused and therefore much much faster.

Again, this is all optimization. It never questions the core assumption, which is that more automated tests are better.

A while back, I used to spend a lot of time helping organizations move to agile development. They all started from different places, but one very common situation was that the developers worked for a period of time, finalized their work, and then handed it off to a QA team. The QA team typically did manual testing. There were often more manual tests than the team could run in the amount of time that they had, and the whole organization sat on pins and needles when QA finally blessed the build. I used to call it the "trailer-hitched QA" anti-pattern.

The remedy is easy, but nerve-wracking. You ramp up automated testing, and pull QA people into the iteration, having them perform whatever testing they need to do as each story completes. Most organizations didn't want to do this in one fell swoop, so you just slowly reduce the amount of time that the QA team has the build after each iteration. The nerve-wracking bit comes from the fact that you know that you are going to end with more bugs in production in the short term, but eventually things get better.

Why do they get better? Well, it's easy to point to the increased emphasis on automated testing. Surely, that makes things get better, but really the reality is much more subtle. Most of the quality benefit that teams get comes from the development team being more careful. And, when you don't have people running around after you spending weeks checking your work, you do end up being more careful. If you aren't, bug reports from production are going to start to spike.

I'm sure that at this point, you might think that I'm making an argument against automated testing. I'm not. I'm merely pointing out that at the end of the day, quality is a development responsibility and it has a lot to do with diligence. To me, the primary benefit of TDD is that we spend time thinking through cases we might not otherwise when we write our code. Quality can't help but go up. It's nice that the tests we write give us a nice regression suite, but is it possible to get too attached to it? I think so.

Automated tests suffer from a political problem. When you have a lot of them, the cost of running them may be high, but the cost of not running them is perceived to be higher. After all, the test that you don't run could be the one which catches some disastrous error. Who wants to take that chance? Frankly, I think that thinking that way is a bit of a trap. It's a trap in the same way that thinking that running all of your manual tests was necessary.

At the end of the day there are all sorts of things which increase quality. It takes a bit of courage to weed through automated tests and say "Yes, I'm sure this one is valid, it would catch a problem, but it's too slow or just doesn't have high enough cost/benefit for us to run any more." Few teams really do this because they are trapped by the political calculus: we could be faulted for not running the test.

Ultimately, tests are a feedback mechanism, and we should make active decisions about what feedback we need and when. My current thought is that it is perfectly reasonable for teams to set a build budget, a maximum amount of time that a build should take in order to provide feedback. If you can make slow tests faster to fit in that budget, fine. If you can't, toss them or rotate them through periodically. Make the value judgement about what tests are most important. We can do a lot of things to avoid that responsibility, but we shouldn't.
Shared publiclyView activity