Do Developers Read Compiler Error Messages?

Reviewed by Greg Wilson / 2021-09-20
Keywords: Error Messages

Earlier this month we reviewed Becker2019, which found that most of the error messages produced by compilers aren't particularly helpful. One response was, "Well, but nobody reads them anyway." Barik2017 showed that this isn't true: based on an eye-tracking study of 56 students writing Java with Eclipse, they found that:

  1. "participants read error messages, and the difficulty of reading these messages is comparable to the difficulty of reading source code",
  2. "difficulty reading error messages significantly predicts participants' task performance" (i.e., the harder the messages are to read, the longer it takes to fix the problem), and
  3. "participants allocate a substantial portion of their total task to reading error messages (13%-25%)."

13-25% of someone's time roughly a day a week. While the actual gain would be smaller (because programmers do a lot more than write code), reducing this time would certainly make work more enjoyable, and I suspect it would reduce the fault rate in shipped code as well. As we've said before (paraphrasing Dobzhansky), nothing in software engineering makes sense except in light of human psychology; the more attention researchers and developers pay to that, the faster our profession will make real progress.

Barik2017 Titus Barik, Justin Smith, Kevin Lubick, Elisabeth Holmes, Jing Feng, Emerson Murphy-Hill, and Chris Parnin: "Do Developers Read Compiler Error Messages?". 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), 10.1109/icse.2017.59.

In integrated development environments, developers receive compiler error messages through a variety of textual and visual mechanisms, such as popups and wavy red underlines. Although error messages are the primary means of communicating defects to developers, researchers have a limited understanding on how developers actually use these messages to resolve defects. To understand how developers use error messages, we conducted an eye tracking study with 56 participants from undergraduate and graduate software engineering courses at our university. The participants attempted to resolve common, yet problematic defects in a Java code base within the Eclipse development environment. We found that: 1) participants read error messages and the difficulty of reading these messages is comparable to the difficulty of reading source code, 2) difficulty reading error messages significantly predicts participants' task performance, and 3) participants allocate a substantial portion of their total task to reading error messages (13%-25%). The results of our study offer empirical justification for the need to improve compiler error messages for developers.