The Secret Life of Bugs

Twelve years ago, Aranda2009 forced me to rethink a lot of what I thought about software engineering research. Its authors (one of whom later co-founded this site) looked closely at the histories of ten randomly-selected bugs at Microsoft, comparing the electronic records (like version control commits and email messages) with detailed interviews of 26 people. Their conclusion:

The most striking lesson from our cases is the deep unreliability of electronic traces, and particularly of the bug records, which were erroneous or misleading in seven of our ten cases, and incomplete and insufficient in every case. In fact, even considering all of the electronic traces of a bug that we could find (repositories, email conversations, meeting requests, specifications, document revisions, and organizational structure records), in every case but one the histories omitted important details about the bug. [emphases in the original]

I think we can learn a lot about software and the people who create it by mining and analyzing digital data. Since reading this paper, though, one of the first things I check when reviewing a data mining paper is whether the authors thought carefully about what that data didn't capture. If they don't address that issue (which unfortunately many don't), I'm much more skeptical about the paper's claims. If this site ever does turn into the evidence-based introduction to software engineering that I've wanted for over a decade, this paper will be one of the first required readings.