Realizing quality improvement through test driven development
Reviewed by Jorge Aranda / 2012-01-25
Keywords: Test-Driven Development, Testing
Nagappan2008 Nachiappan Nagappan, E. Michael Maximilien, Thirumalesh Bhat, and Laurie Williams: "Realizing quality improvement through test driven development: results and experiences of four industrial teams". Empirical Software Engineering, 13(3), 2008, 10.1007/s10664-008-9062-z.
Test-driven development (TDD) is a software development practice that has been used sporadically for decades. With this practice, a software engineer cycles minute-by-minute between writing failing unit tests and writing implementation code to pass those tests. Test- driven development has recently re-emerged as a critical enabling practice of agile software development methodologies. However, little empirical evidence supports or refutes the utility of this practice in an industrial context. Case studies were conducted with three development teams at Microsoft and one at IBM that have adopted TDD. The results of the case studies indicate that the pre-release defect density of the four products decreased between 40% and 90% relative to similar projects that did not use the TDD practice. Subjectively, the teams experienced a 15–35% increase in initial development time after adopting TDD.
In the test-driven development (TDD) chapter of Making Software, Turhan et al reported that the evidence for it is mixed: there is moderate support for the claim that it improves quality, and it is not quite clear if this entails a productivity cost. To come up with this conclusion, the authors went through all the papers they could find on the topic—some of them reporting experiments with students, some with practitioners, and with highly varying quality.
In my opinion, one of the stronger papers in their sample was this one, by Nagappan et al. It reports on four teams, one at IBM and three at Microsoft, and it contrasts TDD vs. comparable non-TDD teams post-hoc (so the study did not bias data collection). As the abstract points out, there were far fewer defects in all four products, though managers at all teams reported an increase in development time.
The conservatism in the Making Software chapter is warranted: there is still conflicting empirical evidence with TDD, as with most other practices in software development. But studies like Nagappan et al's show that TDD is likely to be beneficial. Just note that (at least in the Microsoft teams they studied) "there was no enforcement and monitoring of the TDD practice; and decisions to use TDD were made as a group." In other words, developers applied TDD because they wanted to, not because of a decree from their manager. Whether it would've been as effective otherwise is an open question.