So my first question is, do you recognize these folks?
You may not so I'll give you some hints.
So this person on the left is probably the more famous of the two - his name's Icarus - he's famous for being associated with human traits like overconfidence, and you may have heard the phrase, don't fly too close to the sun.
But today I'm going to focus on a different character in the story, and that is his father Daedalus.
He's a character I connect with more personally because he's a master craftsman, and you know, come on, like, a master craftsman.
But his inventions often have unintended consequences and I also have inventions that have unintended consequences.
And today I'm going to be talking about some of the unintended consequences of the work we've been doing on mining software build systems, and we're going to use this Icarus and Daedlus metaphor to to help make that explanation clear.
So let's start with our story.
So Icarus and Daedalus are trapped on the island of Crete because Daedalus has created some inventions that have angered the gods.
So they're on this island prison, but Daedalus has a plan.
So Daedalus is famous for using what he can get his hands on to create solutions.
So he's found some feathers and some wax from candles and he's going to try and fashion some wings now just like that.
We do the same thing when we're mining software repositories.
So I'll set this up with an example.
The other day I'm coding - imagine that I'm writing some code - and I wrote some code like this.
And I need an accompanying build system to turn it into an executable format.
So I write a little Maven file and then I turn my attention to executing the build where of course I encounter a build failure.
And I start to question my career choices and start to feel some feelings that perhaps I'm not too proud of, but I decide to keep trundling through.
And I start seeing some more messages that make me feel some other feelings about my career choices but I still decide to keep pushing my way through.
And eventually I start to notice a pattern - I start to notice, it looks like class resolution is not happening properly and it's probably because a package is not being picked up.
And specifically it looks like Log4j is missing.
And when I turn back to my code I notice, you know, I do have Log4j included in my spec, so what could be wrong?
Well it turns out that log4j slightly changed the way that the package is referenced in Maven central.
So after making this change I run my build again and I start to regain confidence in myself.
But then the researcher in me starts waking up - I start wondering to myself, why did I have to fix that by hand?
Why can't my build tool fix this automatically?
And thinking even more broadly, why can't build systems sustain themselves and what what might that mean?
So if we were to have a self-sustaining build system we'd need different pieces of technology to make that happen.
We might need some tools to detect when - when dependencies are incorrectly or underspecified, we might need some tools to detect when build changes are necessary and anticipate those, and we might need some tools to automate the application of fixes to build scripts.
Now these are all different areas that my group at the University of Waterloo is working on, but today I'm going to focus a little bit on automated repair of build scripts, okay, and specifically I'm going to tell you about this wax and feather solution we came up with called Build Medic.
Okay so Build Medic operates on a build failure log and it - at a very high level is broken down into three phases.
So first we try and classify the failure log based on what kind of problem is happening, and then based on that we select a repair strategy we might want to try, and then we'll apply the strategy and evaluate the revision.
Now there are two possible outcomes: we get either the same or another failure, in which case we loop back and try again, or we've resolved the problem and the build is fixed automatically for us.
So we have our solution, and Daedalus and Icarus, their solution is to build these wings and they set off to the sky to get off of the island, right, so the next step is - so they took off and they're flying away from their prison, and now we're at a stage where we're ready to test out our wax and feather solution as well.
And what we're finding when we test our solution is that the Build Medic solution is actually pretty good.
So we tried it out on 84 unseen breakage pairs, so we went mining through open source projects history to find examples of commits where the build was broken and then immediately fixed, and what we found is that a little more than half of the solutions could be immediately fixed by our Build Medic tool, and three quarters of those successes are done in one iteration.
So we didn't need to loop back to try multiple fixes.
Even more than that we're not just making the build pass we're often doing exactly what developers did themselves.
So in those correct solutions about a third of the time we do exactly what the developers did, and another 44 percent of the time we're modifying the same elements that the developers modified but we're changing things in a slightly different way.
So what's the next part of our story?
So in the next part of our story we know that Icarus and Daedalus - there is a limitation on their wings, and that is that the wax from the wings will melt when - when Icarus or Daedalus get too close to the sun.
And just like that our evaluation has some limitations, right, our limitations are that we're relying on historical breakages and we're assuming that those breakages are important - we're assuming that when a build breakage occurs developers are rushing like these firefighters to put out the fire.
But what what happens when we take a closer look at some of this historical build data is that there are a couple of reasons why it's imperfect.
First, there are actively ignored failures occurring, and what we observed is that about 12 percent of passing builds have at least one actively ignored failing job and it's not just one or two jobs it's often up to 87 percent of what's been failing.
We also found there are passively ignored failures, so uh two-thirds of failing builds are actually stale and you can see here there are a variety of different project characteristics - some are fixing builds within a day, some it takes a week, others it takes months or even years.
So we know from the Daedalus and Icarus myth that Icarus ends up in the sea, right, and you might be leaving this talk feeling like something is horribly wrong, and I don't want to leave you with that message, right, so I think this vision of a self-sustaining build is not dead.
Okay, there are two messages that you should take away.
One is, if you're evil - evil Shane might say conclusion number one and angel Shane might say conclusion number two.
So conclusion number one is that our evaluation is correct and your data is broken, okay, if that's how you feel well we have a proposed solution.
We should be looking at how much noise there is in build data and and keeping track of it.
And conclusion number two is that our evaluation is incorrect, in which case I hope that you out there will help us to fix it.
So we need developers and other users to help us evaluate our tools and I hope you'll reach out.
Thank you.