CreatingCode

By Robert Greiner on 06 Jan 2012.

Lately we've been noticing fewer and fewer commits getting flagged as risky with each passing week. After doing some analysis we realized, for the most part, we have increased quality awareness across the organization and as a result more people are checking in test code with each commit. As a rule of thumb, we hope to see around 40% of commits flagged as risky. This number seems appropriate for now since the goal is to code review all risky commits and the value of code reviews has diminishing returns.

In our case the issue of having too few commits flagged as risky is a good thing but we still have to do something about it to ensure we are getting the most out of our efforts. Once you get to this point, there are three options that can be taken to move forward:

1. Introduce a new heuristic - If your current set of risk heuristics are giving you the kind of information you want, maybe it's time to add a new one. For instance, measuring cyclomatic complexity can be a nice compliment to an existing code coverage metric.
2. Dial-up one or more existing heuristics - Once the development organization improves to a new level, dial it up a notch and see if you can't get them to improve even more.
3. Make your existing heuristics smarter - One thing we've been toying around with is trying to identify files that get checked in without tests. For instance, if a developer commits three source files, but only updates two different test files, then they likely neglected to add tests for one of the files. This commit should still be flagged as risky, even if overall code coverage increased.

Ratcheting is very much an experimental process that requires constant tweaking to ensure we are tactically striking risky code in areas that provide the most value while minimizing the amount of distractions to other developers.