Revolutionising NHS Healthcare Research: A Secure Data Environment Success Story

About the customer:

The customer is a collection of UK healthcare organisations within the NHS (National Health Service) based in a large geographical region. The project brings together organisations across the region that use or hold NHS patient data, provide technical expertise in data science and innovation and who work with patients and members of the public. This includes NHS trusts, universities, user representative bodies and local innovation networks. The region covers a diverse population of 6.4 million people.

Challenge:

In February 2021, recognising the potential of health data for research and analysis, the UK government commissioned the Goldacre Review to explore its efficient and safe use. The review concluded that: “Data can drive research. It can be used to discover which treatments work best, in which patients, and which have side effects. It can be used to help monitor and improve the quality, safety and efficiency of health services. It can be used to drive innovation across the life sciences sector.” The primary challenge to realising this enormous potential is joining up data across healthcare organisations and facilitating secure access for research purposes.

This regional project sought to create an innovative Sub-National Secure Data Environment (SNSDE) to overcome the challenge. A Secure Data Environment includes digital spaces to hold NHS data and allow approved researchers to safely access and analyse it for research, while keeping the data protected and under the control of the NHS. Establishing such a solution at a regional level was intended to provide a basis for a broader national initiative to have a world-leading health research infrastructure in place by 2025.

With the right technology in place, future benefits will include improving prediction, prevention, participation, and personalisation of healthcare services, targeting resources effectively, and ensuring services are designed for the population they serve. The immediate IT need was to develop and build a Minimum Viable Product (MVP) target architecture for an SNSDE. The SNSDE needed to be a scalable cloud-native solution with advanced functionality from the outset. It also needed to be designed around the ONS Five Safes principles and capable of onboarding datasets to support initial use-cases (e.g. cardiovascular research) securely.

If the challenge is not addressed, there could be significant healthcare risks, including the inability to target scarce resources effectively, failure to design services for the population's needs, and professionals lacking access to information for informed decisions. This could lead to a continued backlog in treatment, persisting inequalities, and a growing gap in the workforce, all under the pressure of flat spending projections by the UK’s Office for Budget Responsibility. The risk of delaying research results in an inability to accelerate the discovery of new treatments and improve service planning within the NHS. Missed opportunities will result from this failure to generate new openings for research and regional economic growth.

Solution:

Using cutting-edge AWS cloud infrastructure within a tight timeframe, we pioneered the development of a Sub-National Secure Data Environment (SNSDE). Our collaboration with AWS fused our collective expertise, accelerating the co-creation and implementation of a novel ‘first of type’ solution. The strategic alliance between us has yielded a resilient, secure, and high-performance Secure Data Environment that is attuned to NHS goals and regulatory frameworks. It displays our joint commitment to advancing healthcare through innovative cloud-based data solutions.

The Cloud Infrastructure consists of a multi-component architecture that includes an AWS Cloud Infrastructure Landing Zone, Data Platform, and Secure Data Research Environment. Using AWS services extensively, we leveraged the AWS Landing Zone Accelerator for Healthcare alongside AWS CloudFormation, both of which are delivered through AWS CodePipeline using CDK Pipelines to establish secure, compliant, and manageable cloud infrastructure.

Data Management uses Data Review and Transfer Component (DRTC is a new AWS service) alongside AWS Lambda, Step Functions, and other Serverless technologies to create secure Data Ingress and Egress that is airlocked between Research Environments and external data providers. The Data Integration Platform uses AWS services like Amazon Lake Formation, AWS Glue, and EC2 to manage, clean, and transform datasets, providing a secure data lake for researchers.

Research Environment pioneers the AWS Research and Engineering Studio (RES) to provision and manage secure cloud-based research environments. This allows researchers to visualise data and execute applications in a controlled setting. The solution emphasises automation, security, and compliance, with a focus on minimising costs associated with idle times.

Our Identity Management solution manages identities from various academic, health, and commercial users through Amazon Cognito, EntraID, AWS IAM Identity Centre, and managed Microsoft Active Directory. This ensures clear separation of user roles and secure access to resources.

We embedded team collaboration and skills development at the core of our approach, uniting our combined experts with the customer’s project team to guarantee a seamless delivery. Employing agile delivery methods, we refined the solution through bi-weekly sprints and consistent communication, building strong partnerships with the customer’s leadership and technical staff. 

Results and benefits:

The successful delivery of our MVP solution, and value of the research use cases, has demonstrated the significant potential benefits of this world-leading health research infrastructure. This has enabled the customer to gain business case approval and secure funding for its next phase of work.

Key results achieved to date:

• Strengthening public and professional trust: through our data integration and access which facilitates secure access to de-identified NHS data for research. This maximises opportunities for patient and NHS benefit while ensuring data safety and privacy.
• Healthcare Improvement: identified with potential to improve prediction, prevention, and personalisation of healthcare by targeting resources effectively and designing services for specific population needs.
• Operational Efficiency: by providing authorised professionals with easy access to information for informed decision-making, improving disease understanding and treatment responses over time.
• Innovation Acceleration: by supporting the identification of new treatment and drug targets, ensuring patients receive the best treatment faster and improving system delivery pathways.
• Speed to market: Once funded, a project research area can be delivered or uplifted in under three hours.
• Direct Funded Research Environments: Cost clawback due to AWS resource management and tagging policies, allowing for full cost clawback on project specific infrastructure and running costs. This decreases project overheads to the client – functionality that was not previously available.

Our solution and these outcomes are expected to help advance a transformational change in the NHS and social care system, addressing challenges such as UK population health, the management of complex long-term conditions, and healthcare workforce gaps.