Case Study: Data Modernisation for Multinational Insurance Company

Our customer (“the Company”) is a large, multinational insurance company and employee benefits provider that offers expert health and wellbeing support, as well as financial protection through the workplace.

Kainos engaged with the Company to help modernise its legacy ETL jobs which previously utilised a proprietary software product. Kainos began the task for documenting and modernising these jobs with the aim of rewriting into .NET applications running within the AWS cloud.

The problem that Kainos was engaged to solve was around the use of the legacy technology that is utilised by the Company to fulfil their ETL based jobs. The issue centred around the use of on-premises hardware and the limitations it imposed on the software.  There were also general limitations of the software, such as how many concurrent jobs can be run at the same time, which resulted in the Company having to batch their processing.  In the past the Company had to extend capabilities by purchasing more hardware, increasing its capital expenditure around this solution.

The company had since reached the stage of diminishing returns, where the cost of extending the hardware became greater than what could be justified from a commercial standpoint.  In addition, the large volume of ETL jobs which the company had developed reached a management overhead threshold, resulting in increased bottlenecks across several key data processes.

A secondary problem was the lack of proprietary expertise within the Company, increasing the difficulty in maintaining and extending its large catalogue of ETL jobs. This resulted in jobs drifting out of relevance as they could not be modified at a pace sufficient to keep up with ever-changing business needs.  In time, the reliability of the jobs degraded, resulting in ever-increasing end-to-end processing failures, and ever-increasing costs and delays which were impacting business function.

The proposed solution was to move a group of jobs out of the current system and into an AWS solution. This solution is an event-driven design which successfully leverages multiple AWS technologies to handle the processing of data, as shown in the diagram below (simplified for privacy and clarity reasons):

The reason we chose this solution was to allow for greater scalability, enabling the Company to meet their current demand but also allow for growth. The key resources used in the solution were Lambda, ECS and SQS. This approach enabled each of the jobs to be decoupled into smaller applications which can scale independently from each other.

Enhanced Performance: While the ETL process hasn’t experienced a substantial individual-job performance boost as this was not a goal of the engagement, the new cloud-based system offers greater resiliency, monitoring capabilities, and increased throughput. This translates into improved reliability and efficiency, enabling the Company to meet growing demands.

Decoupled Architecture: Our revamped system embraces a decoupled architecture, replacing previously tightly coupled components. This architectural shift enhances maintainability and extensibility, fostering agility in adapting to evolving requirements.

Event-Driven Scalability: Embracing an event-based system architecture enables seamless scaling and improved throughput. The move to using an Event-Driven system on the cloud addressed one of the Company’s key concerns with the existing system, which caused bottlenecks in their ETL processes due to limited on-prem resources.

Improved Visibility: Increased logging mechanisms provide comprehensive visibility into system actions. This heightened transparency enables proactive monitoring and troubleshooting, ensuring optimal system performance at all times.

The outcome of the project has been extremely successful. Kainos has been able to deliver to the Company a solution that enhances systems throughput, scalability, monitoring and transparency at a platform level. Moreover, the solutions include applications that are not only maintainable and extendable but also align with the company’s skillset.  This will be beneficial to them for the long-term evolving business needs of the future.

The average runtime for a job set has reduced from hours to minutes, delivering a roughly 900% speed increase.