Achievements and challenges from breaking the test silo
One year ago Irek and Steven posted a Kainos blog outlining how testing has changed and how the Kainos test capability planned to adapt to this. Since then we have adopted their ideas in many projects and teams. Within our Digital Services business these changes have led to a higher level of quality. This has improved the quality of both the services we provide and the products we have delivered. But as expected, we've also had challenges that needed to be overcome. This is a look back over the achievements and challenges.
Achievement 1
Being more efficient at adopting the test automation pyramid has led to more efficient test execution, saving time and wasted effort. This is because end-to-end user interface tests are more prone to false positives, slow execution time, and test flakiness. Previously user interface testing had attracted the lion's share of a tester's time writing code and investigating failures.
By adopting the layered approach of the test pyramid, we've increased the number of tests at service and integration levels (while reducing the amount of user interface tests). This has decreased execution time in our builds and reduced the time spent investigating test failures. When builds fail their test automation, the root cause is quicker to identify as the tests are narrower in focus. The increased amount of automation that comes with the test pyramid model has meant our developers write more test code. The outcome is that our automation code has become cleaner and more efficient, and the burden of test creation is spread better across the team.
Achievements 2 and 3
Having more awareness and getting developers involved in testing has meant that they are closer to the test coverage.
Earlier non-functional testing including non-functional considerations into epics and stories has also helped teams to highlight performance issues and security concerns earlier. This has resulted in more time for them to be addressed. We now see performance, accessibility and security implications for features addressed in sprint and are no longer a time-boxed exercise at the tail end of project. This leads to services being built to a high quality standard. Our deployment pipelines for continuous integration now have non-functional test automation included. This is a lot more straightforward for service managers and product owners to work with.
Achievement 4
Shared responsibility for quality exploratory testing plays a large part in achieving quality in our work. But most importantly quality is higher if it is not just performed by QA. By coaching all team members to do exploratory testing and getting them to play a part in execution issues can be found earlier.
As a result of this teams have a higher level of confidence in the software they are delivering. They are happier with what they're delivering to users. The client is more engaged with quality assurance activities and can see the benefits of testing first-hand. Signing off stories in the last available sandbox environment and the stability of our build pipelines means our path to production is a lot faster, more streamlined and robust. All these improvements are a good measure of the success, but there are still some challenges we are working on.
Challenge 1 and 2
Ownership of testing: If QA lead is away when planning or refinement sessions take place, discussions about testing seem to get overlooked or skimmed over. QA activities and mindsets need to be represented in these ceremonies, to ensure the same level of discussions take place as when someone in a QA role is there. We think we can do some education in this area; getting non QA team members comfortable thinking about testing activities, and updating the team's definitions of done to highlight all the facets of testing and that it is a shared responsibility.
Automation as a Role vs. a Responsibility: Perhaps our largest challenge is the uncertainty this approach brings to those in the the role of the automated tester, and what this shared ownership means for their career aspirations. We have members of our testing capability who were hired specifically as test automation engineers, and sometimes they're struggling with the distribution of tasks which were traditionally theirs, and that they're being asked to perform tasks they don't feel proficient in; manual testing, exploratory testing, and non-functional testing. Agile teams and having t-shaped team members requires some 'blurring of lines' between all roles.
Our new QA approach makes them and their skills able to be done by anyone, and their role may be redundant if they are not willing to accept it. As a capability we need to ensure the value their skills add are visible and promoted to teams. Changing the perception of their part in helping the whole team achieve quality. As a team, we need to provide good coaching and guidance in the areas they don't feel proficient in, and bring them into manual testing like we do with developers and business analysts.
Challenge 3
Red Build Apathy: A common challenge teams face is acceptance of red builds. When continuous integration is implemented correctly, change sets are deployed through the pipelines one at at a time. In some of our projects multiple change sets are heading through the pipeline at any one time, and if something fails then multiple developers may be the culprit. This leads to more and more people investigating where the build broke, and an attitude of 'it's probably someone else'. In some cases this tolerance of red builds can stretch on for days. We think adopting the role of the build master can help overcome this, or introducing elements of it to build a good red-response practice, but it is important to remember that it is not the job of the build master to fix every broken build. This should fall on the developers making the change.
Positive steps ahead
As a group we're going to try making more changes, and will continue to reflect and adjust. We're going to do a refresh of our test capability within Digital Services. The strategy has wide reaching impacts on our people and how we do our day to day job, and this needs to be reflected in our internal training, meetings, and documentation and communicated wider in the business. It's in our nature to constantly strive for improvements in the way we work, but it has been good to reflect on what we've achieved and how.
We've made positive steps forward, building on Kainos' vision and setting the foundations for the test capability. We knew it wasn't going to be an overnight change, and we're positive as we look to tackle challenges we know about and those we are yet to be aware of. The test leads (Anna, Sean and I) would like to hear any ways you think we can overcome the challenges we're facing!