Code and commit metrics of developer productivity: a study on team leaders perceptions

Reviewed by Greg Wilson / 2021-08-21
Keywords: Metrics, Productivity

Oliveira2020's central proposition is straightforward:

  1. Team leaders' perceptions are usually projects managers' primary source of information about development teams.
  2. Those perceptions can be compared to metrics based on the code programmers write or to measures of their commits to the project repository.
  3. It turns out that team leaders' perceptions correlate with the former (code) better than the latter (commits).

It's great to see a careful study of how programmers are evaluated: every team does it explicitly or implicitly, and knowing what factors influence those evaluations can help us make them fairer. The authors are careful to point out that they're comparing metrics to perceived productivity rather than making any claims about programmers' actual productivity. The latter is probably the most difficult question in empirical software engineering: an entire book was recently devoted to the topic (Sadowski2019, which can be downloaded here), and Begel2014 found that individual productivity is the topic programmers want studied least because they're afraid findings will be mis-used.

I enjoyed this paper, but I did come away with a question and a minor concern. The question is how to square this with Rahman2013's finding that process metrics (which roughly align with Oliveira et al's commit metrics) create better defect prediction models than code metrics (see here for a review). The minor concern is with one of the code metrics Oliveira et al used: fancy measures of source code like Halstead's are no better than simply counting lines of code (see ElEmam2001, summarized here), and I think it's time to set them aside.

Oliveira2020 Edson Oliveira, Eduardo Fernandes, Igor Steinmacher, Marco Cristo, Tayana Conte, and Alessandro Garcia: "Code and commit metrics of developer productivity: a study on team leaders perceptions". Empirical Software Engineering, 25(4), 2020, 10.1007/s10664-020-09820-z.

Context: Developer productivity is essential to the success of software development organizations. Team leaders use developer productivity information for managing tasks in a software project. Developer productivity metrics can be computed from software repositories data to support leaders' decisions. We can classify these metrics in code-based metrics, which rely on the amount of produced code, and commit-based metrics, which rely on commit activity. Although metrics can assist a leader, organizations usually neglect their usage and end up sticking to the leaders' subjective perceptions only. Objective: We aim to understand whether productivity metrics can complement the leaders' perceptions. We also aim to capture leaders' impressions about relevance and adoption of productivity metrics in practice. Method: This paper presents a multi-case empirical study performed in two organizations active for more than 18 years. Eight leaders of nine projects have ranked the developers of their teams by productivity. We quantitatively assessed the correlation of leaders' rankings versus metric-based rankings. As a complement, we interviewed leaders for qualitatively understanding the leaders' impressions about relevance and adoption of productivity metrics given the computed correlations. Results: Our quantitative data suggest a greater correlation of the leaders' perceptions with code-based metrics when compared to commit-based metrics. Our qualitative data reveal that leaders have positive impressions of code-based metrics and potentially would adopt them. Conclusions: Data triangulation of productivity metrics and leaders' perceptions can strengthen the organization conviction about productive developers and can reveal productive developers not yet perceived by team leaders and probably underestimated in the organization.