It Will Never Work in Theory

Short summaries of recent results in empirical software engineering research

2017-11-26: Developer Testing in the IDE
Keywords: Programming Practices
Reviewed by: Greg Wilson

Some depressing but not surprising results about how little developers test.

2017-11-26: Abbreviated vs. Full-Word Identifier Names
Keywords: Programming Languages
Reviewed by: Greg Wilson

Found no difference in development effort or efficiency when using abbreviated identifier names as opposed to full names.

2016-12-01: Hiatus
Keywords: Community
Reviewed by: Greg Wilson

This site is on hiatus until further notice: in the wake of recent political events, our community's energy and attention should be focused on more important things. If you can, please get involved in projects like this.

2016-10-05: Test-Driven Development
Keywords: Testing
Reviewed by: Greg Wilson

This painstaking study is the latest in a long line to find that test-driven development (TDD) has little or no impact on development time or code quality. Here, the authors repeated an earlier study with a couple of new wrinkles, and then blinded their data before giving it to someone else to analyze to remove any possibility of bias. The result: no significant difference between TDD and iterative test-last (ITL) development. …which still surprises me seven years after Turhan et al's meta-analysis in Making Software. reported the same result. I don't program very much any more, but when I do,...

2016-10-02: Revisiting the Anatomy and Physiology of the Grid
Keywords: Software Architecture, Grid Computing
Reviewed by: Greg Wilson

Ernest Rutherford may or may not have once said that all science is either physics or stamp collecting. It's generally assumed that if he did, he meant it disparagingly, but if the physicists of his generation hadn't collected subatomic particles, he probably wouldn't have won the Nobel Prize. Until stamps are collected and organized, theoreticians don't know what patterns and exceptions they need to explain; to put it another way, no finches, no Darwin. This paper makes two significant contributions to the "natural history" side of software engineering. First, it compares 18 grid computing systems to a reference architecture popularized...

2016-10-02: FIDEX: Filtering Spreadsheet Data using Examples
Keywords: Spreadsheets, Programming by Example
Reviewed by: Greg Wilson

Another solid piece of engineering, another reason to be sad that I won't be at SPLASH this year. The authors start by creating a small language to describe the things that people do when filtering spreadsheet data, like starts-with, ends-with, matches, and contains. They then create a sound, complete, and efficient algorithm that learns how to construct filter expressions using these operations along with intersection and subtraction operators. They then show that their algorithm learns both quickly enough and well enough to be useful to real people. And yes, it only works on strings (for now), and cannot yet handle...

2016-10-01: Apex: Automatic Programming Assignment Error Explanation
Keywords: Education, Tools, Program Analysis, Symbolic Execution
Reviewed by: Greg Wilson

In this paper, the authors combine several sophisticated techniques to diagnose the root causes of errors in novice programming assignments even when the novice's code is significantly different from the correct implementation provided by the instructor. They do this by collecting traces of the programs' execution and reasoning about them symbolically to figure out why the final states of variables differ, regardless of the paths taken to create those variables. This may seem like magic, but the methods used are well established. What they aren't is part of the usual undergraduate curriculum. I suspect this is due to the mathematical...

2016-09-30: Purposes, Concepts, Misfits, and a Redesign of Git
Keywords: Programming Tools, Software Design
Reviewed by: Greg Wilson

This paper presents a detailed, well-founded critique of one of the most powerful, but frustrating, tools in widespread use today. A follow-up to earlier work published in 2013, it is distinguished from most other discussion of software design by three things: It clearly describes its design paradigm, which comprises concepts (the major elements of the user's mental model of the system), purposes (which motivate the concepts), and misfits (which are instances where concepts do not satisfy purposes, or contradict one another). It lays out Git's concepts and purposes, analyzes its main features in terms of them, and uses that analysis...

2016-09-21: Paradise Unplugged
Keywords: Diversity
Reviewed by: Greg Wilson

Computing as a whole has a diversity problem, but its online communities are even worse. This careful study explores why by looking at Stack Overflow, and finds five barriers to contribution that are seen as significantly more problematic by women than by men: lack of awareness of site features feeling unqualified to answer questions intimidating community size discomfort interacting with or relying on strangers perception that they shouldn't be "slacking" Surprisingly, "fear of negative feedback" didn't quite make this list, but would have been the next one added if the authors weren't quite so strict about their statistical cutoffs. The...

2016-09-16: Five From ICER'16
Keywords: Education
Reviewed by: Greg Wilson

This was the most thought-provoking paper of the conference, as it shows pretty conclusively that people are seeing evidence for a "geek gene" where none exists. Mark Guzdial has called this belief the biggest myth about teaching computer science; I hope this paper will finally put some nails in its coffin.