David Log, Nachiappan Nagappan, and Thomas Zimmermann: "How Practitioners Perceive the Relevance of Software Engineering Research". ESEC/FSE'15, August 2015, http://thomas-zimmermann.com/publications/files/lo-esecfse-2015.pdf.
The number of software engineering research papers over the last few years has grown significantly. An important question here is: how relevant is software engineering research to practitioners in the field? To address this question, we conducted a survey at Microsoft where we invited 3,000 industry practitioners to rate the relevance of research ideas contained in 571 ICSE, ESEC/FSE and FSE papers that were published over a five year period. We received 17,913 ratings by 512 practitioners who labelled ideas as essential, worthwhile, unimportant, or unwise. The results from the survey suggest that practitioners are positive towards studies done by the software engineering research community: 71% of all ratings were essential or worthwhile. We found no correlation between the citation counts and the relevance scores of the papers. Through a qualitative analysis of free text responses, we identify several reasons why practitioners considered certain research ideas to be unwise. The survey approach described in this paper is lightweight: on average, a participant spent only 22.5 minutes to respond to the survey. At the same time, the results can provide useful insight to conference organizers, authors, and participating practitioners.
There's lots in this paper, but its most important finding (for me) is buried in the middle of the abstract: "no correlation between citation counts and [practitioner-assigned] relevance scores". In plainer language, academic researchers and practitioners care about different things (which may mean this blog doesn't have much of a future).
So what do the two groups care about? The top studies favored by practitioners were:
while academics' favorites were:
As the authors say several times, this is a preliminary study, and is based on feedback from developers at only one company (albeit a large one). I hope they do another, larger, study, but I hope even more that a decade from now, there will be at least some correlation between what the "two solitudes" of software engineering care about.
Comments powered by Disqus