Strategies to Improve Continuous Integration

Every time I create or modify a pull request at work, Circle CI runs half a dozen jobs to check that the code and commit message are formatted correctly, that the unit tests pass, that the integration tests pass, and so on. It takes a little over two minutes, but I've worked on systems where continuous integration would run for hours after every change, which was a significant drag on development.

In this paper, Jin and Servant "…perform the first exhaustive comparison of techniques to improve CI, evaluating 14 variants of 10 techniques using selection and prioritization strategies on build and test granularity. We evaluate their strengths and weaknesses with 10 different cost and time-to-feedback saving metrics on 100 real-world projects." The selection and prioritization strategies they use are drawn from existing research, and do things like skip tests that cannot reach changed files based on conservative dependency analysis. To measure effectiveness, they look at build and test time saved, and at the number of builds and number of tests that were skipped. Among other things, they found that there's a tradeoff between saving time by skipping some build steps and missing tests that would have failed if they had been run. While this is intuitively sensible, validating it provides a baseline for future expectations.

Among their other conclusions are:

• Cost-saving techniques focused on skipping passing builds and tests, but did not specifically target those that would provide the highest savings (i.e., slower tests and slower builds), so there is room for improvement.
• Training cost-saving techniques across projects harmed their predictions: in other words, statistical models to predict which tests or builds can be skipped need to be retrained for each project.
• Trying to predict seemingly-safe builds and tests is particularly useful.
• Skipping full builds paid high dividends, since build preparation time takes a large proportion of total build time.
• The best techniques to provide early feedback to developers were test-prioritization techniques.

Jin2021 is both a careful, detailed comparison of existing work and evidence of how mature this research area has become; I hope we'll see some of these ideas become commonplace in production soon.