Compiler Error Messages Considered Unhelpful

Reviewed by Greg Wilson / 2021-09-02
Keywords: Computing Education, Debugging

Becker2019 is 34 pages long and has 219 entries in its bibliography—in other words, researchers know a lot about how the messages produced by today's compilers confuse people and how we could make them better. I would like everyone who is building programming tools (particularly ones aimed at beginners) to read this paper today and incorporate the findings it presents into their work tomorrow. Since I can't have that, I'll summarize the authors' recommendations, all of which are much richer and more specific than these brief headings:

  1. Increase readability (in specific ways that are outlined in the paper).
  2. Reduce cognitive load.
  3. Provide context to the error.
  4. Use a positive tone. (Yes, this actually matters…)
  5. Show examples of similar errors.
  6. Show solutions or hints.
  7. Allow dynamic interaction.
  8. Provide scaffolding for the user.
  9. Use logical argumentation (as explained in Barik2018).
  10. Report errors at the right time.

XKCD comic on error messages

Becker2019 Brett A. Becker, Paul Denny, Raymond Pettit, Durell Bouchard, Dennis J. Bouvier, Brian Harrington, Amir Kamil, Amey Karkare, Chris McDonald, Peter-Michael Osera, Janice L. Pearce, and James Prather: "Compiler Error Messages Considered Unhelpful". Proceedings of the Working Group Reports on Innovation and Technology in Computer Science Education, 10.1145/3344429.3372508.

Diagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for over half of a century. Unfortunately, these messages which include error, warning, and run-time messages, present substantial difficulty and could be more effective, particularly for novices. Recent years have seen an increased number of papers in the area including studies on the effectiveness of these messages, improving or enhancing them, and their usefulness as a part of programming process data that can be used to predict student performance, track student progress, and tailor learning plans. Despite this increased interest, the long history of literature is quite scattered and has not been brought together in any digestible form. In order to help the computing education community (and related communities) to further advance work on programming error messages, we present a comprehensive, historical and state-of-the-art report on research in the area. In addition, we synthesise and present the existing evidence for these messages including the difficulties they present and their effectiveness. We finally present a set of guidelines, curated from the literature, classified on the type of evidence supporting each one (historical, anecdotal, and empirical). This work can serve as a starting point for those who wish to conduct research on compiler error messages, runtime errors, and warnings. We also make the bibtex file of our 300+ reference corpus publicly available. Collectively this report and the bibliography will be useful to those who wish to design better messages or those that aim to measure their effectiveness, more effectively.

Barik2018 Titus Barik, Denae Ford, Emerson Murphy-Hill, and Chris Parnin: "How should compilers explain problems to developers?". Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 10.1145/3236024.3236040.

Compilers primarily give feedback about problems to developers through the use of error messages. Unfortunately, developers routinely find these messages to be confusing and unhelpful. In this paper, we postulate that because error messages present poor explanations, theories of explanation—such as Toulmin's model of argument—can be applied to improve their quality. To understand how compilers should present explanations to developers, we conducted a comparative evaluation with 68 professional software developers and an empirical study of compiler error messages found in Stack Overflow questions across seven different programming languages. Our findings suggest that, given a pair of error messages, developers significantly prefer the error message that employs proper argument structure over a deficient argument structure when neither offers a resolution—but will accept a deficient argument structure if it provides a resolution to the problem. Human-authored explanations on Stack Overflow converge to one of the three argument structures: those that provide a resolution to the error, simple arguments, and extended arguments that provide additional evidence for the problem. Finally, we contribute three practical design principles to inform the design and evaluation of compiler error messages.