It Will Never Work in Theory

Extremely Angry

Posted Apr 26, 2016 by Greg Wilson

| Relevance |

From this thread by Mike Hoye, the Engineering Community Manager for Firefox at Mozilla:

  • Extremely angry with the state of academic CS research right now.
  • We've got a bunch of research papers that have come out of Mozilla's willingness to share a lot of our data. Bugzilla, lots of other stuff.
  • We pay attention to that research. We read it carefully, and our operational decision-making is informed by those results.
  • But we live and work in an evolving world, so the other thing we do as a matter of course is re-check our data to confirm our assumptions.
  • What I want to be able to do is to go back and say, we based this decision in part on the results of paper X. Is that result still valid?
  • In a sane world, that's 3 steps.
    • Clone the author's VCS
    • Run author's [whatever] against an up-to-date dataset.
    • Look at the new graph.
  • Does the data still support the thesis? Great! Full steam ahead! Has something changed? Pause and think! Either way we're winning.
  • But that never happens. Because CS researchers don't publish code or data. They publish LaTeX-templated Word docs as paywalled PDFs.
  • And if you care about sciency stuff like "validity" or "reproducibility", even simpleton stuff like "real-world relevance", well, f*** you.
  • Let's talk about how far behind schedule we are because people don't share what they know, or didn't make what we learned accessible.

All of which makes me ask yet again: what will it take for researchers publishing in closed-access venues to realize how much damage they're doing to themselves by shutting out people like Mike?

Later: coincidentally, the Journal of Open Source Software has just launched. Like the Journal of Open Research Software, its mission is to give people a place and a way to get academic credit for the tools they build. It would be great to see some software engineering research work appearing in both...

Comments powered by Disqus