Build Master

The role Build Master came from the book Continuous Delivery with the aim of empowering someone to keep an eye on the builds in large distributed teams. We have implemented this role in two small scale teams one of which was partially distributed, this article talks about the rules we set up and my findings from the experience.

Why we implemented it

The teams had changed quite a bit and we were needing to upskill people in the system very quickly. Jenkins was in a bit of a bad way, there were jobs which had failed months ago that people had lost sight of, and excuses such as 'we do not need that' or 'that only runs if the moon is aligned with jupiter'. We had many integration points with other suppliers and overnight jobs were failing.

All I cared about was looking at the build radiator and seeing green, and not hearing excuses or blame. This might seem harsh but I believe as a team we could resolve the issues and share the responsibility through this new rotating role.

The initial rules for Build Master

1. Anyone can be the Build Master.
2. It is a rotating role each week.
3. The role is to keep the lights on and investigate failures.
4. It is not an excuse for 'throwing it over the wall'. Developers are still responsible for their code all the way to production.
5. We never accept a failure.
6. No one merges into a red pipeline.
7. A wiki is updated of known issues and tech debt created.
8. Communication is the key, let the team know of problems.
9. Build Master is not to be ignored.

How the role was received and evolved

After some initial teething problems regarding time spent on investigations we had to introduce time-boxing and create a tech debt item for bigger investigations. Those doing the Build Master also had sprint commitments so we made sure to adjust their capacity accordingly (20% of time spent doing Build Master).

We needed to define what the Build Master role was not for, this is where some readers may be thinking general teamwork and a good definition of done will solve all these issues. For us we made sure the developers knew their roles had not changed, they still owned the code and followed it through the pipelines to production. The Build Master was there to fix or remove legacy builds where the pipeline had evolved. If a developer had merged code in, which had broken further down the line and they were not watching the pipeline, then the Build Master would give them a gentle nudge. The Build Master is a facilitator not an excuse.

The wiki hit a surge of activity as common issues were encountered, and before long we had a theme of issues causing failing builds namely to do with third party libraries or integration to external APIs. This lead us to have workshops with other suppliers to gain a better understanding of their tool set and invest time into fixing our build system issues.

The wiki was also used for handover between Build Masters and it even got to the point that people were asking the Build Master how their initial investigation was going. We saw an increase in pride in their work and interactions between different team members.

People from different roles in the team took to being Build Master, 'many eyes solve many problems'. We had Architects, Tech Leads, Developers and Testers all take turns and talk about problems they faced with the build system. As Scrum Master for the team I was even challenged to take to the role for a week.

The team all understood the importance of the role, to put it simple if the pipelines were not green then no stories were going to be merged. By supporting the Build Master teams were able to ship their code, this caused team members to help out the Build Master when they were free in their work. They also focussed on the pipeline when merging their jobs.

Other Benefits

It was noticed by quite a few people that we had large build jobs that tended to fail during the protractor tests. This was causing the entire job to fail and lots of digging to find the root cause. What was proposed was that we broke up our larger jobs into a sequence of smaller jobs for a single pipeline allowing the area of the failure to be identified quicker.

We noticed as a team we wanted to review the process and change aspects. For this we decided to hold a retro every two sprints highlighting any modifications to the rules or investigations into technical debt people believed we needed (this was not the only tech debt review). I was the sponsor of bringing in the Build Master role so would chair these sessions and push these changes through.

TL;DR;

Build Master is a shared rotating responsibility in the team. It allows upskilling of new team members and gets additional eyes looking at shared problems. It may have teething issues but can help you clean up a build radiator with suggestions from the team and prompts the team for a shared knowledge base. The role will require some initial policing but after some team buy in, will grow organically.

Build Master is not a role that all projects need, it depends on the size and distribution of the team. We used it in instances where legacy builds had been left unchecked and in a broken state. Where many points on integration and overnight builds lead to unknown failures not attributed to a single commit and a build of the system. We made sure that the team were aware that the Build Master is not an excuse for 'throwing code over the wall' and that the definition of done remained unchanged.