Generating Unit Tests for Documentation

Reviewed by Greg Wilson / 2022-04-14
Keywords: Documentation

This paper describes a tool called DScribe, which generates documentation and unit tests simultaneously from templates like this:

  (doc)  If no argument is provided, the default value for the
         arguments of $method$ is $defVal$.
  (test) @Test
         public void test$method$_defaultArgument() {
           assertEquals( $method$() , $method$($defVal$) );

It's a clever extension of things like Python's doctest, but is it actually useful? To find out, the authors studied the inconsistencies between source code, unit tests, and documentation in exceptions in the Apache Commons IO library. They found that:

Among the sampled methods of Commons IO, only 15% of the specifications related to thrown exceptions (ESU) are both tested and documented. The remaining 85% of ESUs represent inconsistent information that would require a non-trivial amount of effort to manually fix. DScribe can generate the tests and documentation for 97% of these ESUs, using only five templates.

They also found that:

Approximately 42% of the unit tests...capture information that would be relevant to document, but that information is documented for only 52% of these tests. Developers can use DScribe to generate documentation from these tests more consistently and efficiently, especially for repetitive specifications.

I don't know if I'll ever use DScribe itself, but I would be surprised if the ideas in it don't show up in other tools in years to come: as more languages edge toward literate programming, putting all the information about a piece of software in a single place becomes more feasible, and I look forward to seeing what other clever ideas emerge.

Nassif2021 Mathieu Nassif, Alexa Hernandez, Ashvitha Sridharan, and Martin P. Robillard. Generating unit tests for documentation. IEEE Trans. Software Engineering, 2021, doi:10.1109/tse.2021.3087087.

Software projects capture redundant information in various kinds of artifacts, as specifications from the source code are also tested and documented. Such redundancy provides an opportunity to reduce development effort by supporting the joint generation of different types of artifact. We introduce a tool-supported technique, called DScribe, that allows developers to combine unit tests and documentation templates, and to invoke those templates to generate documentation and unit tests. DScribe supports the detection and replacement of outdated documentation, and the use of templates can encourage extensive test suites with a consistent style. Our evaluation of 835 specifications revealed that 85% were not tested or correctly documented, and DScribe could be used to automatically generate 97% of the tests and documentation. An additional study revealed that tests generated by DScribe are more focused and readable than those written by human testers or generated by state-of-the-art automated techniques.