Small, Medium, and Large test sizes post. Thought it would be small, found it getting large, refactored it back down to medium.
6 plus ones
Shared publicly•View activity
View 8 previous comments
- We picked short, moderate, long, eternal for the timeout values to keep the SML(E) theme. Although we cheated a bit and split the Short category into short and instant.Nov 3, 2011
- When you mention the balance of S/M/L tests as 70/20/10, what exactly do those numbers mean? Are you talking about actual number of test cases?Sep 13, 2012
- That was always left a little ambiguous, and folks debated it endlessly. It could mean the number of test cases, the number of test binaries, or the amount of coverage attributable to each class of tests. The percentage-of-coverage interpretation was the most common, as I recall, and that's what the internal coverage dashboards were designed to show.
The real point was to just get folks thinking about how to increase the number of fast and uncomplicated "small" tests, and by discussing the appropriate balance of small/medium/large tests overall, develop a successful testing strategy that made sense for each team's particular product or system, by whatever standard they decided made the most sense. As long as they made a reasonable case to the Test Certified Mentors and demonstrated progress and commitment, it didn't matter exactly what that standard was, so long as each TC team had a useful standard in effect.Sep 13, 2012
- Yes, I agree that the main point of introducing the ratios is to get those who are concerned about code quality (hopefully everyone!) to think about increasing their test coverage in the right way. My question comes along because I myself am developing an internal dashboard tool to show regression test metrics in a way that promotes the S/M/L ratios (for example, a Sonar plugin that provides a "testing pyramid" visualization), so I need to make some concrete decisions about what these metrics really mean for our team.
It seems weird to me that code coverage would be a good way to determine the ratios. A single Large test will, almost by definition, hit more lines of code than a single Small test. So if I have 2 Large tests that together achieve 80% code coverage, and 10 Small tests with 60% code coverage, what does that mean? And what should my goals be as someone who is writing tests for this part of the codebase? I suppose one possible response is to say, "Well, you should just up your Small-test coverage to >80%", but another equally valid response is to say "Remove one of the Large tests to get to 40% Large-test coverage and then your ratios are satisfied".
Perhaps I'm just stuck on the nature of what's being tested. One of the major use cases I'm thinking of is testing some RESTful services we have. It seems reasonable to me to have 1 Large test where we deploy the webapp and hit the service, 2 Medium tests where we hit some internal API class and talk to the DB, and 7 Small tests that are unit tests against the internal API with mocked dependencies. Here, the ratios make sense to me. One could also imagine a dashboard where each test case is a "block", and if you stacked them on top of one another it would form a pyramid. Then, if you took the three rows in the pyramid and asked, "What is the code coverage for this row?", that could be a secondary aspect of the visualization.
I suppose it's natural that all of the public documentation from Google Testers will be lacking in many implementation details... I was just unsure if, after reading through the How Google Tests Software book and your blog posts, I was missing something critical about the actual implementation of the concepts presented. :) It sounds like I've got the concepts down, and it's just a matter of figuring out what exactly works for our team here.
Thanks!Sep 14, 2012
- Your last sentence nails the entire point. It sounds like you get the S/M/L distinction exactly, and if 80% large test and 60% small test coverage works for your team, don't go through agonizing contortions to artificially factor components apart which really do belong together, or try gaming the system, just to try to drive up your small test coverage numbers. If you have a high degree of confidence that the changes you're making are already well-covered, perhaps keep an eye on your current numbers as a watermark, but don't bother yourself too much about it.
Of course, those of us in the Testing Grouplet--which, I stress, was not part of Test Engineering or Engineering Productivity, despite the synergy of efforts and overlap of membership--took a harder line from 2006-2008 and strongly urged teams to aim for greater small test coverage, because teams most often had only very large, slow, and/or brittle tests, or they neglected to write automated tests at all.
Thanks for the very thoughtful questions, Jessica, and keep 'em coming if you got 'em! Hopefully the collective experience of myself and my close colleagues--several of whom have commented on or +1'd this thread--can be of some service to your own efforts. Do keep me posted--have a link to a blog of your own?Sep 14, 2012
- I will definitely keep you posted! I have a blog at http://jessica.aus10.org , and while I have thus far done a poor job of updating it with work related to QA, I hope to rectify that in the coming months.Sep 15, 2012