Beyond the Code Itself: How Programmers Really Look at Pull Requests
Reviewed by Greg Wilson / 2021-08-29
Keywords: Bias, Collaborative Development
If I ever teach another full-semester course on software engineering, I'm going to devote at least one lecture to cognitive bias. None of us make decisions as rationally as we'd like to believe, but knowing how our thinking can be skewed is the first step toward thinking more clearly.
Ford2019 used eye-tracking to find out what programmers actually look at when evaluating pull requests on GitHub, and compared those findings with what programmers said they did. Their conclusion? "…after the code snippet, the second place programmers spent their time fixating is on supplemental technical signals, such as previous contributions and popular repositories. Diverging from what participants reported themselves, we also found that programmers fixated on social signals more than recalled."
This study also found that programmers use different strategies to protect their identity on GitHub than the use on Facebook. Those strategies included:
- Stay aware of image presented and how they will be perceived on each platform.
- Make the code stand alone, regardless of the name attached.
- When in doubt, use an anonymous name.
- Complete the online profile to be perceived as trustworthy.
- Create personal rules for sharing content based on the platform's primary function.
As a middle-aged straight white male with no significant physical or mental challenges who is economically secure and fluent in English, I can afford to pretend that "the code speaks for itself". I don't know how often I've made poor decisions because of that blindspot, but I hope teachers will incorporate insights like those in this paper into their courses so that the next generation of programmers will make better ones.
Ford2019 Denae Ford, Mahnaz Behroozi, Alexander Serebrenik, and Chris Parnin: "Beyond the Code Itself: How Programmers Really Look at Pull Requests". 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Society (ICSE-SEIS), 10.1109/icse-seis.2019.00014.
Developers in open source projects must make decisions on contributions from other community members, such as whether or not to accept a pull request. However, secondary factors-beyond the code itself-can influence those decisions. For example, signals from GitHub profiles, such as a number of followers, activity, names, or gender can also be considered when developers make decisions. In this paper, we examine how developers use these signals (or not) when making decisions about code contributions. To evaluate this question, we evaluate how signals related to perceived gender identity and code quality influenced decisions on accepting pull requests. Unlike previous work, we analyze this decision process with data collected from an eye-tracker. We analyzed differences in what signals developers said are important for themselves versus what signals they actually used to make decisions about others. We found that after the code snippet (x=57%), the second place programmers spent their time fixating is on supplemental technical signals (x=32%), such as previous contributions and popular repositories. Diverging from what participants reported themselves, we also found that programmers fixated on social signals more than recalled.