0:00:05.360,0:00:09.520 So my first question is, do  you recognize these folks? 0:00:10.800,0:00:16.480 You may not so I'll give you some hints. So this person on the left is probably the 0:00:16.480,0:00:24.880 more famous of the two - his name's Icarus - he's  famous for being associated with human traits 0:00:24.880,0:00:29.840 like overconfidence, and you may have heard  the phrase, don't fly too close to the sun. 0:00:30.560,0:00:34.240 But today I'm going to focus on a  different character in the story, 0:00:34.880,0:00:39.680 and that is his father Daedalus. He's a character I connect with 0:00:39.680,0:00:46.400 more personally because he's a master craftsman,  and you know, come on, like, a master craftsman. 0:00:46.400,0:00:53.280 But his inventions often have  unintended consequences and I also have 0:00:53.280,0:00:58.800 inventions that have unintended consequences. And today I'm going to be talking about some of 0:00:58.800,0:01:04.320 the unintended consequences of the work we've  been doing on mining software build systems, 0:01:05.680,0:01:11.840 and we're going to use this Icarus and Daedlus  metaphor to to help make that explanation clear. 0:01:12.800,0:01:18.160 So let's start with our story. So Icarus and Daedalus are trapped on the 0:01:18.160,0:01:23.840 island of Crete because Daedalus has created  some inventions that have angered the gods. 0:01:23.840,0:01:39.840 So they're on this island  prison, but Daedalus has a plan. 0:01:41.200,0:01:46.800 So Daedalus is famous for using what he  can get his hands on to create solutions. 0:01:46.800,0:01:50.560 So he's found some feathers  and some wax from candles 0:01:50.560,0:01:54.880 and he's going to try and fashion  some wings now just like that. 0:01:54.880,0:01:58.320 We do the same thing when we're  mining software repositories. 0:01:58.880,0:02:03.840 So I'll set this up with an example. The other day I'm coding - imagine that 0:02:03.840,0:02:08.000 I'm writing some code - and  I wrote some code like this. 0:02:08.000,0:02:12.400 And I need an accompanying build system  to turn it into an executable format. 0:02:12.400,0:02:18.160 So I write a little Maven file and then I  turn my attention to executing the build 0:02:18.800,0:02:25.040 where of course I encounter a build failure. And I start to question my career choices 0:02:25.040,0:02:29.280 and start to feel some feelings  that perhaps I'm not too proud of, 0:02:29.280,0:02:35.520 but I decide to keep trundling through. And I start seeing some more messages that make me 0:02:35.520,0:02:43.280 feel some other feelings about my career choices  but I still decide to keep pushing my way through. 0:02:44.000,0:02:50.320 And eventually I start to notice a pattern - I  start to notice, it looks like class resolution 0:02:50.320,0:02:55.920 is not happening properly and it's probably  because a package is not being picked up. 0:02:57.440,0:03:03.120 And specifically it looks like Log4j is missing. And when I turn back to my code I notice, 0:03:03.680,0:03:08.480 you know, I do have Log4j included  in my spec, so what could be wrong? 0:03:09.680,0:03:14.000 Well it turns out that log4j slightly  changed the way that the package is 0:03:14.000,0:03:18.320 referenced in Maven central. So after making this change 0:03:19.040,0:03:24.000 I run my build again and I start  to regain confidence in myself. 0:03:24.640,0:03:30.480 But then the researcher in me starts  waking up - I start wondering to myself, 0:03:31.040,0:03:37.040 why did I have to fix that by hand? Why can't my build tool fix this automatically? 0:03:39.120,0:03:43.920 And thinking even more broadly,  why can't build systems sustain 0:03:43.920,0:03:50.400 themselves and what what might that mean? So if we were to have a self-sustaining 0:03:50.400,0:03:53.920 build system we'd need different pieces  of technology to make that happen. 0:03:54.640,0:04:02.000 We might need some tools to detect when - when  dependencies are incorrectly or underspecified, 0:04:02.800,0:04:09.200 we might need some tools to detect when build  changes are necessary and anticipate those, 0:04:09.200,0:04:14.080 and we might need some tools to automate  the application of fixes to build scripts. 0:04:15.120,0:04:20.160 Now these are all different areas that my group  at the University of Waterloo is working on, 0:04:20.160,0:04:25.040 but today I'm going to focus a little bit  on automated repair of build scripts, okay, 0:04:25.760,0:04:30.800 and specifically I'm going to tell you  about this wax and feather solution we 0:04:30.800,0:04:35.680 came up with called Build Medic. Okay so Build Medic operates on a 0:04:35.680,0:04:41.840 build failure log and it - at a very high  level is broken down into three phases. 0:04:42.640,0:04:48.160 So first we try and classify the failure log  based on what kind of problem is happening, 0:04:48.880,0:04:52.880 and then based on that we select a  repair strategy we might want to try, 0:04:54.160,0:04:57.920 and then we'll apply the strategy  and evaluate the revision. 0:04:57.920,0:05:04.080 Now there are two possible outcomes: we get either  the same or another failure, in which case we loop 0:05:04.080,0:05:11.680 back and try again, or we've resolved the problem  and the build is fixed automatically for us. 0:05:13.840,0:05:21.280 So we have our solution, and Daedalus and Icarus,  their solution is to build these wings and 0:05:21.280,0:05:41.440 they set off to the sky to get off of the island,  right, so the next step is - so they took off and 0:05:41.440,0:05:47.360 they're flying away from their prison, and now  we're at a stage where we're ready to test out 0:05:47.360,0:05:51.840 our wax and feather solution as well. And what we're finding 0:05:53.200,0:05:58.400 when we test our solution is that the Build  Medic solution is actually pretty good. 0:05:58.400,0:06:04.960 So we tried it out on 84 unseen breakage pairs,  so we went mining through open source projects 0:06:04.960,0:06:11.280 history to find examples of commits where the  build was broken and then immediately fixed, 0:06:12.480,0:06:18.400 and what we found is that a little more than  half of the solutions could be immediately 0:06:18.400,0:06:26.960 fixed by our Build Medic tool, and three quarters  of those successes are done in one iteration. 0:06:26.960,0:06:31.920 So we didn't need to loop  back to try multiple fixes. 0:06:33.200,0:06:37.200 Even more than that we're not just  making the build pass we're often 0:06:37.200,0:06:44.480 doing exactly what developers did themselves. So in those correct solutions about a third of 0:06:44.480,0:06:50.800 the time we do exactly what the developers did,  and another 44 percent of the time we're modifying 0:06:50.800,0:06:56.720 the same elements that the developers modified but  we're changing things in a slightly different way. 0:06:59.120,0:07:01.840 So what's the next part of our story? 0:07:10.880,0:07:15.920 So in the next part of our story we  know that Icarus and Daedalus - there 0:07:15.920,0:07:22.080 is a limitation on their wings, and that  is that the wax from the wings will melt 0:07:22.720,0:07:27.920 when - when Icarus or Daedalus  get too close to the sun. 0:07:28.560,0:07:32.640 And just like that our  evaluation has some limitations, 0:07:33.600,0:07:41.280 right, our limitations are that we're relying on  historical breakages and we're assuming that those 0:07:41.280,0:07:48.960 breakages are important - we're assuming that when  a build breakage occurs developers are rushing 0:07:48.960,0:07:55.120 like these firefighters to put out the fire. But what what happens when we take a closer 0:07:55.120,0:08:00.960 look at some of this historical build data is that  there are a couple of reasons why it's imperfect. 0:08:01.920,0:08:08.000 First, there are actively ignored failures  occurring, and what we observed is that 0:08:08.000,0:08:13.680 about 12 percent of passing builds have  at least one actively ignored failing job 0:08:15.280,0:08:22.560 and it's not just one or two jobs it's often  up to 87 percent of what's been failing. 0:08:26.320,0:08:31.840 We also found there are passively ignored  failures, so uh two-thirds of failing builds 0:08:31.840,0:08:38.400 are actually stale and you can see here there are  a variety of different project characteristics - 0:08:38.400,0:08:44.320 some are fixing builds within a day, some it takes  a week, others it takes months or even years. 0:08:46.560,0:08:51.280 So we know from the Daedalus and Icarus  myth that Icarus ends up in the sea, 0:08:51.280,0:08:56.160 right, and you might be leaving this talk  feeling like something is horribly wrong, 0:09:05.680,0:09:08.960 and I don't want to leave you with that message, 0:09:08.960,0:09:15.120 right, so I think this vision of a  self-sustaining build is not dead. 0:09:15.120,0:09:18.480 Okay, there are two messages  that you should take away. 0:09:19.040,0:09:23.920 One is, if you're evil - evil Shane  might say conclusion number one 0:09:24.480,0:09:30.880 and angel Shane might say conclusion number two. So conclusion number one is that our evaluation is 0:09:30.880,0:09:38.240 correct and your data is broken, okay, if that's  how you feel well we have a proposed solution. 0:09:38.240,0:09:43.680 We should be looking at how much noise there  is in build data and and keeping track of it. 0:09:44.640,0:09:49.040 And conclusion number two is that our  evaluation is incorrect, in which case 0:09:49.040,0:09:56.960 I hope that you out there will help us to fix it. So we need developers and other users to help us 0:09:56.960,0:10:06.400 evaluate our tools and I hope you'll reach out. Thank you.