Why We Need Evidence
Reviewed by Greg Wilson / 2012-12-30
Keywords: Questions
Our previous post, "Empirical Evidence for the Value of Version Control", generated a lot of comments. Many sought to explain why version control is helpful, but that's not what we were looking for: we were looking for empirical evidence that it is. To see why we need it, take a look at this response from Jordi Cabot [1]. In it, he says:
Quite regularly, I get questions about what empirical evidence supports my "belief" that models are good… Until now, I used to point to the (true, few) scientific empirical studies on the effectiveness of software modeling…but now I have an even anser to give you: "Empirical Evidence of the Value of Version Control".
No, I haven't lost my mind. The point of this link is to show you that there's no proof that version control is better for software development, and yet, I don't think any of you would argue against it.
Same for modeling and model-driven engineering. It would be great to have more proof but the absence of proof alone should not be used against it unless you want to start also abandoning other unproven things like version control.
He's right: if we're willing to accept that version control is valuable, without proof, then we can hardly require advocates of modeling to prove their case. Or advocates of functional programming, or literate programming, or Hungarian notation. Heck, if we don't require proof for our claims, then we're honor-bound to accept that Perl is "intuitive" because its grammar has as many special cases and contradictions as the grammars of natural languages, aren't we? Or that learning Befunge makes you a better programmer (seriously, I've heard that claim too).
At some point, the statement, "If we don't need to prove the value of version control, we don't need to prove the value of X" becomes absurd. However, everyone's threshold of absurdity is different. I personally don't think that modeling adds value for most developers in most situations—I think that if it did, or if its benefits really were as significant as its advocates claim, more developers would have adopted it by now—but I don't know. What I do know is, if we can't demonstrate the value of something that most of us believe in, like version control, what chance do we have of telling whether other practices, like modeling and test-driven development, are worth adopting (or rather, when they're worth adopting and by whom, since I doubt there's a one-size-fits-all answer)?
So here are my requests:
- Tell us what kind of study would convince you that using Befunge didn't make programmers more productive.
- Then tell us what kind of study would convince you that version control didn't either.
If your answer to the second question is is, "Nothing ever could," then version control is an article of faith for you, and there's no point arguing further [2]. If your answer to the second is different from your answer to the first, please tell us why.
[1] Full disclosure: Jordi and I co-authored a study of web-based software project portals. And either way, we hope you have a happy and productive 2013.
[2] This request is inspired by Karl Popper's notion of falsifiability: a claim is only scientific if there is some way to prove it wrong.