On Using Stack Overflow Comment-Edit Pairs to Recommend Code Maintenance Changes

Reviewed by Tajkia Rahman Toma / 2021-11-16
Keywords: Maintenance, Stack Overflow, Usability

Have you ever developed any software without the help of Stack Overflow? In just 13 years, Stack Overflow has become an essential platform to get programming help. It has also become a source of data for researchers to solve many software engineering problems. However, Tang2021 are the first to evaluate whether Stack Overflow comments and edits can help solve the software maintenance problems. Their main goal was to find out how useful Stack Overflow comment-edit pairs are in making tools for code maintenance tasks like program repair and code recommender systems.

The authors used SOTorrent to automatically mine edits of code blocks in answers and marked the comment that invoked the edit. The answers were selected from five popular tags---Java, JavaScript, PHP, Python and Android---and the edit-invoker comment was selected based on three conditions:

  1. the comment has been made before the edit;
  2. the comment contains a code term that has been added to or removed from a code snippet in the edit; and
  3. the editor and the commenter are different users.

The authors evaluated the effectiveness of their automated process by comparing it to a manual analysis, and gave priority to finding more actual pairs of comment-edit, rather than having all possible pairs in the dataset. They also explored comment-edit pairs for tangled change, which they defined as a single edit that changes code to resolve multiple issues raised in multiple comments. Tangled changes reduce the usability of the comment-edit pairs as the comment for which a part of the edit occurred becomes untraceable. Their overall findings were:

  • The automated comment-edit mapping algorithm performs considerably well (78% precision).
  • Edits in the comment-edit pairs are rarely tangled (11% are tangled).
  • Nine of the comment categories identified by a previous study are applicable for edit-invoking comments; the most common of these is fixing an error.

Unfortunately, only 27% of comment-edit pairs are useful because many edits add new code blocks or the comments are contextual. (To evaluate usefulness, the authors manually changed the code of 15 GitHub repositories and submitted pull requests, of which 10 were accepted.)

Tang2021 Henry Tang and Sarah Nadi: "On Using Stack Overflow Comment-Edit Pairs to Recommend Code Maintenance Changes". Empirical Software Engineering, 26(4), 2021, 10.1007/s10664-021-09954-8.

Code maintenance data sets typically consist of a before version of the code and an after version that contains the improvement or fix. Such data sets are important for various software engineering support tools related to code maintenance, such as program repair, code recommender systems, or Application Programming Interface (API) misuse detection. Most of the current data sets are typically constructed from mining commit history in versioncontrol systems or issues in issue-tracking systems. In this paper, we investigate whether Stack Overflow can be used as an additional source for building code maintenance data sets. Comments on Stack Overflow provide an effective way for developers to point out problems with existing answers, alternative solutions, or pitfalls. Given its crowd-sourced nature, answers are then updated to incorporate these suggestions. In this paper, we mine commentedit pairs from Stack Overflow and investigate their potential usefulness for constructing the above data sets. These comment-edit pairs have the added benefit of having concrete descriptions/explanations of why the change is needed as well as potentially having less tangled changes to deal with. We first design a technique to extract related comment-edit pairs and then qualitatively and quantitatively investigate the nature of these pairs. We find that the majority of comment-edit pairs are not tangled, but find that only 27% of the studied pairs are potentially useful for the above applications. We categorize the types of mined pairs and find that the highest ratio of useful pairs come from those categorized as Correction, Obsolete, Flaw, and Extension. These categories can provide data for both corrective and preventative maintenance activities. To demonstrate the effectiveness of our extracted pairs, we submitted 15 pull requests to popular GitHub repositories, 10 of which have been accepted to widely used repositories such as Apache Beam (https://beam.apache.org/) and NLTK (https://www.nltk.org/). Our work is the first to investigate Stack Overflow commentedit pairs and opens the door for future work in this direction. Based on our findings and observations, we provide concrete suggestions on how to potentially identify a larger set of useful comment-edit pairs, which can also be facilitated by our shared data.