Using GitHub Copilot to accelerate software engineering

Generative AI has been one of the most talked about and transformative areas within the developer (and wider) communities in recent years. Many tools have been released harnessing the power of Gen-AI. Adopting some of these tools and evolving engineering processes to include Gen-AI has the potential not only to speed up delivery but also to improve the engineering experience.  

In a previous blog, How is AI going to change software engineering? Richard Rothery predicted that using AI will boost engineer satisfaction, improve performance, impact activity levels, make working together easier and make engineers more efficient. Those were bold predictions. In order to test these, a team within Kainos have trialled the use of GitHub Copilot and GitHub Copilot Chat and we have now seen evidence to support all of Richard’s predictions.  

The trial:  

Over a 12-week period, engineers from a Kainos project were provided licenses to begin using Copilot, the first 6 weeks using Copilot and the remaining time having access to Copilot Chat. They were encouraged to spend time researching the tool and the different ways they could interact with it.   

Surveys were filled in weekly, allowing the engineers to provide their feedback. These surveys used the SPACE framework (Satisfaction & Wellbeing, Performance, Activity, Communication & Collaboration, Efficiency & Flow), asking the engineers the same questions asked by GitHub to developers signed up to the Technical Preview of Copilot.  Retrospective sessions happened at the mid-way point of the trial, and at the end.   

The results are extremely positive, and worth noting that no ‘Disagree’ answers were submitted to this survey, suggesting no negative impacts were observed on the SPACE framework.  

A retrospective session also identified that:  

• Repetitive pattern-based tasks had the biggest benefit, e.g. implementing unit tests or mapping between data structures  
• Engineers spent some saved time on reviewing alternative responses and improving Copilot’s suggestions  

• Copilot preserved flow and reduced context-switching by assisting from within the IDE rather than having to jump out to Google/Stack Overflow/etc.  
• Copilot improved confidence when working in areas where the team have less experience  
• Engineers spent more time on higher value tasks: working and collaborating on business problems rather than syntax, and enabling more meaningful conversations  
• Copilot Chat aided learning though conversation style interactions, providing the engineers with explanations of code, suggestions for improvements and clarifying syntax and generally giving assistance in whatever form is asked for by the engineer 
• Copilot does not have a full picture of what an engineer is trying to achieve and therefore should not be considered a replacement for collaborating with others  

Adopting safely 

Like with any tech evaluation we addressed security and privacy concerns. As per the privacy policy, code snippets are sent to GitHub to generate suggestions, but Copilot for Business does not retain data or train their model on customer code. Duplicate detection has also allowed us to exclude suggestions that match public code on GitHub to avoid licensing issues. 

We’ve heard concerns raised around the risk of engineers becoming overly reliant on Copilot, potentially fostering a complacent attitude. However, from the trial, our engineers found that using Copilot required them to critically analyse multiple suggestions returned, to pick the most appropriate one for their scenario. This aided their learning and, in some circumstances, pointed out implementation options the engineer may not have otherwise considered. The trial also found that Copilot and Chat also increased the engineers’ confidence when working in unfamiliar areas due to Chat’s ability to explain, and Copilot suggested patterns and examples. 

Although not directly related to Copilot, ensuring projects have a high standard code quality process mitigates some of these concerns. This should include sufficient tests being added, code reviewed by at least two engineers and all tests passing.  

Getting the most of GitHub Copilot 

• Provide additional context to Copilot to improve the relevance of suggested completions - Copilot can use open IDE tabs and chat history 
• Check all options provided, don’t always choose the first 
• Always review all code provided by Copilot and implement the code as part of a high-standard code quality process 
• Ask follow-up questions to Copilot Chat to utilise the power of context  
• Spend time upfront reading about all the different functionality available with Copilot 
• Give it time, each engineer will take time to find the interaction types that work best for them  

Ryan J. Salva VP of Product at GitHub said “As a product leader, I work hard to ensure my team makes data-informed decisions backed by research. Perhaps because of my own biases, I was impressed with the way Kainos conducted their trial of GitHub Copilot. Their approach was driven by data and conducted in partnership with customers who had full understanding of how their data would be used. In fact, their results align with my team’s findings. We know that GitHub Copilot keeps developers in the flow, reduces the time to complete tasks, and most importantly, increases developer happiness. It’s great to see those results confirmed through the Kainos trial.  

Building on GitHub’s research, I also loved how Kainos took the engineer’s experience into account both in their initial impact analysis, and subsequently in the team roll-out. After all, we each approach our work differently based on experience, expertise, and situation. Thus, it’s crucial that we as engineering leaders take a human approach to when introducing new technologies. I’m excited to see Kainos’ impact as they share their GitHub Copilot experience with the wider community.” 

Due to the successful results of the trial, Kainos intend to adopt GitHub Copilot much more widely with the view to add it as a default tool for our engineers. We will recommend that our customers also adopt GitHub Copilot. As part of adopting Copilot more widely we will continue to share guidance on the use of GenAI, both generally and specifically in regard to GitHub Copilot.  

We have found Github Copilot and Copilot Chat to be an extremely powerful aid for our engineers. We have moved beyond trial to adoption for internal and customer-facing projects, as part of our wider approach to adopting Generative AI for accelerating software delivery. 

We believe any organisation that can use Copilot should use it given the positive effect on productivity and satisfaction, and that it is safe to do so. However, it is another tool in our arsenal and should be used as such. It is not a replacement for engineers and should be used in conjunction with rigorous processes to ensure code quality.