First Impressions of MSR

Reviewed by Greg Wilson / 2013-06-04
Keywords: Conferences

The MSR program is at http://2013.msrconf.org/ along with links, where possible, to the papers being discussed. Sadly, the IEEE is not in the information-sharing business, but many of the papers can be found linked from here. Prof. Gail Murphy opened the talk with a pretty straightforward question: what is developer productivity? Her slides are here, but they don't really convey how good a talk it was.

The one thing that we as an industry were repeatedly asked for, and this came up a lot—perhaps unsurprisingly, given that this is a group of people whose job is mining software repositories—was this:

You should have instrumentation and logging around your processes.

This isn't pure altruism on their part, of course, but it's not pure self-interest either. Over and over again, the papers being presented drove home the points that there are a surprising number of tool-assisted efficiencies hidden in your software's history, provided that history is around to be analyzed in the first place. Instrumentation grows up to become automation informed by logs that turn into history, and good tooling around a project's history can play an active role in building its future.

Dr. Murphy's talk got a lot more specific and a lot heavier, than that—read the slides, at the very least. But one point that I thought she went through a too fast was: what if it was simple? What if it was easy to drill into comprehensive lifecycle data, see trends, and reflect on my own place and role in those trends? I've got some theories about that, but I'll get back to them shortly.

So, right from the opening, there were results being presented that jumped out at me as being relevant to our work at Mozilla, particularly given our very much inbox-driven development and communication approach. For example, the paper by Mukherjee and Garg about triaging email notifications proposes:

…a machine learning based approach to predict whether a notification will prompt any action from its recipient. Such a prediction can help to suitably mark up notifications and to decide whether a notification needs to be sent out immediately or be bundled in a message digest.

This certainly seems like something we could use, as do the other two papers in that timeslot, describing different approaches to automatically triaging bugs. Later on in the day it was shown that the #1 factor affecting how long it takes a bug to be fixed is fast, accurate triage, so a tool-guided bug triage process seems like a pretty big win, particularly one that learns over the long term.

Over the course of the day, as we moved into the MSR-Mobile and Mining Challenge parts of the conference, even things that weren't directly on-point for Mozilla were still pretty compelling. Textual analysis of Stack Overflow and other help sites that highlights pain points for our community across our products and services is a great idea. Automated approaches to figuring out which parts of an API are brittle or hard to use, which parts of our documentation need help, what new features people are asking for…are we doing that? Not in depth, I don't think, and not to drive decision-making. The Mozilla Developer Network (MDN) is making some moves in that direction, but I think there's some opportunities there we're not taking advantage of.

Another thing that came to light during the Stack Overflow data analysis talks was that the activity from people who are awarded "marathon" badges—500 edits, 500 comments, that sort of thing—drops way off after they've received the badge. I'd like to dig into that a bit more, to try and understand the causal relationship there: is it that people just get the achievement and then stop, is it that people suddenly realize how much time they've spent on Stack Overflow and decide to maybe go outside instead? Still, that discovery should inform some of our badging efforts as we start to ramp up the OpenBadges efforts on Mozillians.

The Analysis Of Bug Reports section of the talk…just read the whole thing. I'd be doing it a disservice to summarize, but certainly in light of the well-documented problems porting defect-prediction methods across software projects (a noble endeavor with a 96% failure rate, if you needed a windmill to tilt at this week) it was pretty great that so many of the papers presented here were built on Mozilla's historical data. Go team! (Incidentally, this is the part of the day that "accurate triage is the most important thing" comes from, courtesy of Mani et al.)

There's a lot more to say about this, but shortly after this the content of the talks went a little bit past my pay grade. I spent more than a few of them feeling like that one guy in GoodBurger: "Hmm, yeah… I know some of these words".

What I want to talk about most is building out tools with an eye towards future machine learning and tool-driven analysis. I've been a sysadmin by trade for a long time, and one of the most important things I've learned there is that if you're in a complex environment with a lot of moving parts, you absolutely must have comprehensive, centralized logging. It's at the core of virtually everything you need to accomplish, whether it's today's sev-1 incident response or next year's capacity planning, and without it you're flying deaf and blind. But if you've got it, you've got notifications, advance warnings, statistical analysis, fault prediction—dozens and maybe hundreds of options and opportunities open up on timescales from preemptive to strategic.

I think we should try to apply the same approach to development tools and developer machines. Disk space isn't super-expensive—not free, I know, but I'm talking about text here—so here's my one-point plan: if you're building a tool that reports back to stdout/stderr, whatever it is, build in an option to send that information (or a structured version of it) to syslog at the same time. If you're part of a team using those tools, centralize those logs. It seems like an odd thing to do—certainly I don't know of anyone else doing that—but while I don't know for sure, I'm willing to bet after a while, the information we'll be able to learn from those logs will surprise us. And even if we don't know what to do with it, the people at MSR who are keen to collaborate with us will.