It's Not a Bug, It's a Feature: How Misclassification Impacts Bug Prediction

The popular media often gets very excited when a new study overturns an old one. Skeptics seize on this as proof that scientists don't know what they're talking about; what they fail to realize is that this is often actually evidence that science is working as it should. To paraphrase Enrico Fermi, its goal is to leave us confused, but on a higher level, and the insight that comes from realizing that earlier questions were poorly phrased, or their answers mis-understood, is often as valuable as any "Eureka!" moment.

This paper is a prime example of that, and evidence of how quickly empirical software engineering research is maturing. By carefully examining thousands of bug reports from several projects, they have found that many mis-report themselves in ways that will inevitably skew the results of simplistic data mining. From one perspective, it's just a large-scale replication of Aranda and Venolia's "Secret Life of Bugs" paper from ICSE 2009, but there's a lot of hard work hiding in that "just". And while this kind of research may not feel like a big step forward, it's what ensures that the next generation of studies will be more useful. Like the Lewis et al paper on the usability of bug prediction tools described a few days ago, it gives practitioners a more trustworthy foundation for translating insights into projections and decisions.