Happy Software Developers Solve Problems Better: Psychological Measurements in Empirical Software Engineering

I've long harbored a suspicion that most of the technologies that we argue about, (static/dynamic types, TDD, etc.), have little impact on the success of software projects. Their effects, while potentially measurable in controlled experiments, are swamped by the noise of the environments where software is implemented. And so, if it doesn't really matter which technologies developers use, why not just let them choose the ones that make them happiest? But this approaches raises the question: do we actually get better outcomes with happier developers? That's the research question that Graziotin, Wang, and Abrahamsson address in this paper.

Unfortunately, researchers haven't yet discovered how to induce happiness, so a randomized controlled trial isn't an option. Instead, the authors measured affective states (the term psychology researchers use to describe emotional state) using a questionnaire developed by psychology researchers called the SPANE. They divided up the study participants into two equal groups based on their questionnaire responses: a happier group which they called the positive group (POS) and a less happy group which they called the non-positive group (N-POS). The authors looked at how the two groups performed an analytical task and a creative task.

For the analytical task, the authors used the Tower of London test, which is similar to the Towers of Hanoi. They scored performance as a function of how many trials the participant passed and how long it took to solve each trial. For this problem, the POS group did much better, and the results were statistically significant at p<.05. For the creative task, they showed the participants images, and the participants had to invent captions for these images. Neither quantity nor quality measures for this task showed statistically significant differences between the two groups at p=.05, in contradiction with the results of previous studies done by other authors.

Ultimately, what we want to know is how much developer happiness impacts outcomes. The authors do report d values, which I assume is Cohen's d, an effect size measure. Alas, I couldn't find the meaning of d explained in the paper, so I can't be certain. For the analytic task, the authors report a d=0.91. That sounds like an impressive result, given that Cohen considered a d=0.8 to be a "large" effect in psychology research. Unfortunately, we haven't yet come up with conventions in software engineering research to characterize small, medium and large effects. The main takeaway from this study is: happier developers perform much better on analytic tasks.

The authors report d=0.07 and d=0.10 for the creative quality measures, favoring the N-POS group. They report d=0.41 for the creative quantity measure favoring the POS group, which is somewhere between "small' and "medium" according to Cohen's guidelines. Even though the authors didn't get statistically significant results, a positive result would be consistent with one of the other studies cited by the authors (Sowden & Dawson, 2011).

It seemed a bit odd that the tasks were not explicitly software related, making the paper feel like a more traditional psychology study with a software engineering spin. The creativity task did not use images that were related to software development at all. The analytic task was more software-related, but the participants had to solve the problem themselves rather than implement a program to solve the problem. Still, it seems reasonable that these more generic measures of creativity and analytic problem solving would be correlated with performance on software engineering tasks.

One notable thing about this paper is that it's published in PeerJ, an open venue (read: no paywall). The authors also elected to make the peer review comments public, which is wonderful. Hopefully, more authors will follow their lead.