Extremely Angry

From this thread by Mike Hoye, the Engineering Community Manager for Firefox at Mozilla:

• Extremely angry with the state of academic CS research right now.
• We've got a bunch of research papers that have come out of Mozilla's willingness to share a lot of our data. Bugzilla, lots of other stuff.
• We pay attention to that research. We read it carefully, and our operational decision-making is informed by those results.
• But we live and work in an evolving world, so the other thing we do as a matter of course is re-check our data to confirm our assumptions.
• What I want to be able to do is to go back and say, we based this decision in part on the results of paper X. Is that result still valid?
• In a sane world, that's 3 steps.
  • Clone the author's VCS
  • Run author's [whatever] against an up-to-date dataset.
  • Look at the new graph.
• Does the data still support the thesis? Great! Full steam ahead! Has something changed? Pause and think! Either way we're winning.
• But that never happens. Because CS researchers don't publish code or data. They publish LaTeX-templated Word docs as paywalled PDFs.
• And if you care about sciency stuff like "validity" or "reproducibility", even simpleton stuff like "real-world relevance", well, f*** you.
• Let's talk about how far behind schedule we are because people don't share what they know, or didn't make what we learned accessible.