Demystifying 'Bad' Error Messages in Data Science Libraries

Reviewed by Eddie Antonio Santos / 2021-10-17
Keywords: Data Science, Error Messages

As a computing educator and an occasional software library developer I often feel that a big portion of novice frustration when coding can be resolved by developers writing "better" error messages. Tao2021 provides evidence that dashes my dreams, but also provides actionable suggestions.

The authors mined errors that were raised by six popular Python data science libraries: NumPy, Panadas, SciPy, scikit-learn, TensorFlow, and Gensim. With this large list of possible errors, they found possible fixes by doing what most people I know would do: search that error message on StackOverflow. They categorize each error messages as being either clear, uninformative, or misleading. Note that people still asked for help for resolving "clear" messages under this categorization.

They conclude that some error messages are inevitably misleading or uninformative, due to the libraries being unable to understand user intent or do not have access to understand external errors such as mistakes that occur outside of the library, but are detected once interacting with the library code.

All hope is not lost—the authors provide a few suggestions: For library users:

  • The library does not understand your ultimate goal, and this makes some direct explanations impossible. With this in mind:
  • improve your understating of the problem domain at hand,
  • improve your understanding of the programming language, and
  • read relevant library documentation.

For library developers:

  • Have your library fail fast and add robust error handling to prevent internal errors from creeping out of the library code and confusing users.
  • Error messages should focus on "completeness", such as providing the relevant runtime values that caused an error.
  • Add actionable instructions on how to resolve the error.
  • Instead of stating that some value or state is not allowed, state what is allowed.
  • Remember: messages that are clear to developers are not always clear to end users.

Tao2021 Yida Tao, Zhihui Chen, Yepang Liu, Jifeng Xuan, Zhiwu Xu, and Shengchao Qin: "Demystifying 'bad' error messages in data science libraries". Proc. European Software Engineering Conference/International Symposium on the Foundations of Software Engineering (ESEC/FSE), 2021, 10.1145/3468264.3468560.

Error messages are critical starting points for debugging. Unfortunately, they seem to be notoriously cryptic, confusing, and uninformative. Yet, it still remains a mystery why error messages receive such bad reputations, especially given that they are merely very short pieces of natural language text. In this paper, we empirically demystify the causes and fixes of "bad" error messages, by qualitatively studying 201 Stack Overflow threads and 335 GitHub issues. We specifically focus on error messages encountered in data science development, which is an increasingly important but not well studied domain.

We found that the causes of "bad" error messages are far more complicated than poor phrasing or flawed articulation of error message content. Many error messages are inherently and inevitably misleading or uninformative, since libraries do not know user intentions and cannot "see" external errors. Fixes to error-message-related issues mostly involve source code changes, while exclusive message content updates only take up a small portion. In addition, whether an error message is informative or helpful is not always clear-cut; even error messages that clearly pinpoint faults and resolutions can still cause confusion for certain users. These findings thus call for a more in-depth investigation on how error messages should be evaluated and improved in the future.