It Will Never Work in Theory

Online vs. Face-to-Face Pedagogical Code Reviews: An Empirical Comparison

Posted Dec 4, 2011 by Greg Wilson

| Controlled Experiments | Education | Quantitative Studies |

Christopher Hundhausen, Pawan Agarwal, and Michael Trevisan: "Online vs. Face-to-Face Pedagogical Code Reviews: An Empirical Comparison." SIGCSE 2011.

Given the increased importance of communication, teamwork, and critical thinking skills in the computing profession, we have been exploring studio-based instructional methods, in which students develop solutions and iteratively refine them through critical review by their peers and instructor. We have developed an adaptation of studio-based instruction for computing education called the pedagogical code review (PCR), which is modeled after the code inspection process used in the software industry. Unfortunately, PCRs are time-intensive, making them difficult to implement within a typical computing course. To address this issue, we have developed an online environment that allows PCRs to take place asynchronously outside of class. We conducted an empirical study that compared a CS 1 course with online PCRs against a CS 1 course with face-to-face PCRs. Our study had three key results: (a) in the course with face-to-face PCRs, student attitudes with respect to self-efficacy and peer learning were significantly higher; (b) in the course with face-to-face PCRs, students identified more substantive issues in their reviews; and (c) in the course with face-to-face PCRs, students were generally more positive about the value of PCRs. In light of our findings, we recommend specific ways online PCRs can be better designed.

Like our previous selection, this paper comes from software engineering education rather than software engineering per se, but has a lot to say about the latter. Code review is now a regular part of most open source projects, thanks in part to online code review tools like ReviewBoard. Here, the authors compare those kinds of reviews with face-to-face reviews, and find that the latter are more effective in several ways: people enjoy them more, they find more issues, and they are more likely to come away believing that reviews are worth doing. It would be fascinating to replicate this study with both junior programmers joining established teams, and developers with more experience who are undertaking reviews systematically for the first time.

Comments powered by Disqus