Two Papers on Errors

Reviewed by Greg Wilson / 2021-09-05
Keywords: Error Handling

As regular readers will have gathered, I think we can learn a lot of useful things by studying programmers' mistakes and the inadequacies of the tools we have for detecting, reporting, handling, and correcting them. The first of these papers, Barbosa2014, presents an empirical taxonomy of the ways exceptions are mis-handled, the first two entries in which have multiple sub-cases:

  1. Information swallowed
    • Uninformative or wrong error message
    • Swallowed exception: the exception is caught and ignored, often just to silence compiler checks
    • Suppressed exception: the original exception is hidden by another exception raised in the same context
    • Missing log message: the exception is caught and handled but there's no record of it happening
    • Destructive remapping: an exception is caught, mapped to another with information loss, and rethrown
    • Uninformative generic type throws: the code throws a generic (and uninformative) exception rather than a more specific exception
  2. Improper continuation of execution
    • Missing throwing condition: a condition that should have led to an exception being thrown wasn't checked (e.g., it's missing from a multi-part logical test)
    • Wrong context configuration: bad exception handling leaves the system in an inconsistent state (e.g., without access to an external resource)
    • Wrong location of execution resumption: the statements after the catch shouldn't be executed if an exception occurred
    • Missing termination action: actions that should always take place whether there was an exception or not only take place if an exception was thrown
  3. Resource leak
  4. Uncaught exception
  5. Overly-generic catch block
  6. Premature termination
  7. Throwing wrong type
  8. Overly protective try block
  9. Exceptional loop break
  10. Excessive throwing conditions

I don't know any linting tools that check for cases like these, but this list would be a great basis for a code review checklist (or a starting point for exercises in a code review training course).

At 41 pages, Coelho2016 is harder to summarize, but its results are just as valuable. The two main questions its authors set out to answer are:

  1. Can the information available in exception stack traces reveal exception handling bug hazards in both the Android applications and framework?
  2. How do developers deal with the exception handling code in Android apps and what are the developers' perspectives about the main bug hazards found during the mining study?

Unsurprisingly, null pointer exceptions are the most common source of problems. Depressingly, the authors found that runtime exceptions are often not documented; in particular, they found checked exceptions thrown by native methods that were not declared in the interface of the wrapper methods, which would be a debugging nightmare.

One thing I particularly liked in this paper was its shift from analyzing code to surveying and interviewing developers to answer its second research question. Far too many papers scrape GitHub and throw the results into a machine learning package without checking whether the data, the questions, or the findings make any sense to the people the work is supposedly for. Asking programmers "Does this make sense?" and "Is it consistent with your experience?" is a necessary and welcome corrective.

Barbosa2014 Eiji Adachi Barbosa, Alessandro Garcia, and Simone Diniz Junqueira Barbosa: "Categorizing Faults in Exception Handling: A Study of Open Source Projects". 2014 Brazilian Symposium on Software Engineering, 10.1109/sbes.2014.19.

Even though exception handling mechanisms have been proposed as a means to improve software robustness, empirical evidence suggests that exception handling code is still poorly implemented in industrial systems. Moreover, it is often claimed that the poor quality of exception handling code can be a source of faults in a software system. However, there is still a gap in the literature in terms of better understanding exceptional faults, i.e., faults whose causes regard to exception handling. In particular, there is still little empirical knowledge about what are the specific causes of exceptional faults in software systems. In this paper we start to fill this gap by presenting a categorization of the causes of exceptional faults observed in two mainstream open source projects. We observed ten different categories of exceptional faults, most of which were never reported before in the literature. Our results pinpoint that current verification and validation mechanisms for exception handling code are still not properly addressing these categories of exceptional faults.

Coelho2016 Roberta Coelho, Lucas Almeida, Georgios Gousios, Arie van Deursen, and Christoph Treude: "Exception handling bug hazards in Android". Empirical Software Engineering, 22(3), 2016, 10.1007/s10664-016-9443-7.

Adequate handling of exceptions has proven difficult for many software engineers. Mobile app developers in particular, have to cope with compatibility, middleware, memory constraints, and battery restrictions. The goal of this paper is to obtain a thorough understanding of common exception handling bug hazards that app developers face. To that end, we first provide a detailed empirical study of over 6,000 Java exception stack traces we extracted from over 600 open source Android projects. Key insights from this study include common causes for system crashes, and common chains of wrappings between checked and unchecked exceptions. Furthermore, we provide a survey with 71 developers involved in at least one of the projects analyzed. The results corroborate the stack trace findings, and indicate that developers are unaware of frequently occurring undocumented exception handling behavior. Overall, the findings of our study call for tool support to help developers understand their own and third party exception handling and wrapping logic.