From TRE to KARECTL: Our Evolution of The Trusted Research Environment in the Age of AI

Behind the scenes, Lancashire Teaching Hospitals NHS Foundation Trust (LTH) has been pursuing an ambitious goal: to deliver a secure digital infrastructure that can empower both humans and agents to harness vast amounts of data to support lower-cost, higher-quality healthcare services. While many organisations harbour similar ambitions both in and outside of the healthcare space, working towards these outcomes has demanded a mini-revolution on several technical and organisational fronts, which have been quietly playing out under the louder backdrop of the ongoing AI health arms race ^[1]. At LTH, the need to facilitate AI-supported research and working practices at scale is growing at pace. To deliver on our goal, we have spent the last six years actively infrastructuring ^[2] - that is, performing the continuous development of key foundational elements (standardisation of data, automation of information governance, availability of analytical toolchains and reshaping operational practices) so AI research and agentic capabilities for day-to-day work practices can be delivered through more intelligent, automated mechanisms across a secure, scalable, and sustainable localised infrastructure.

In this post, we highlight our recent journey developing trusted research environments (TREs) and the requirements driving our need to deliver next-generation TRE frameworks that possess new characteristics that can mitigate cloud vendor lock-in risks and better exploit emerging AI capabilities to enhance TRE administration processes and harden TRE security against looming AI-enabled threats ^[3]. Alongside development of TRE infrastructure at LTH, critical complementary work is happening to help meet our wider ambition of supporting AI-conformant, secure analytics infrastructure:

Towards AI-Conformant TREs

Recognition of the need to innovate beyond traditional approaches to TRE infrastructure has played a central role in LTH's digital transformation journey. Our initial effort to develop a secure analytics environment emerged from a practical organisational need to deliver secure access to clinical data for research with the aim of reducing costs compared with cloud-vendor research environment frameworks while avoiding long-term vendor lock-in. In addition, we were keen to establish an open source community around any infrastructure developed to ensure long-term support of software, support knowledge exchange within the research software engineering (RSE) community, and to mitigate the risks associated with closed software development projects to which the NHS has often been susceptible (more on our early thinking around this in the secure data environments framework article).

Given these requirements, we began to map the core capabilities and services of traditional TRE implementations and considered how to design an open-source, vendor-agnostic TRE based on Kubernetes (K8s) that could operate across diverse cloud and on-premises environments. Early exploration of a platform-agnostic TRE framework was undertaken through LTH's initial research analytics platform, where we developed a deeper understanding of how a microservices-based approach could provide a more flexible and sustainable foundation for supporting secure data research projects. The lessons learned from this early TRE platform development work directly informed the architectural principles that later shaped the implementation of K8TRE - an open-source Kubernetes-native TRE.

Building on that foundation, and through wider engagement with the UK's health informatics and research software engineering communities, we led K8TRE's technical development within a collaborative infrastructure programme, in collaboration with Lancaster University, the University of Dundee (UoD), and University College London (UCL). The project focused on developing a modular, cloud-agnostic, standards-informed Kubernetes-native TRE that could satisfy a broad range of TRE platform team requirements. A key aspect of K8TRE's design is the agnostics layer, providing key abstractions for infrastructure-dependent capabilities (e.g. secret providers, networking, identity providers), enabling TRE application services (e.g. secure workspaces, observability, project governance) to operate regardless of the underlying cloud or on-premises infrastructure.

Through the adoption of open cloud-native technologies and modern CI/CD practices (including GitOps), K8TRE has provided a flexible, portable, and sustainable TRE foundation as an alternative to cloud vendor frameworks. At the same time, demand at LTH is evolving: researchers, clinicians, developers, and healthcare administrators increasingly need TRE environments that can support AI-driven research and agent-based workflows. In response, we have begun to extend our focus beyond vendor-agnostic design to explore AI's role in trusted research infrastructure, including the practical opportunities and challenges of building AI-conformant TREs. In this context, we see AI-conformance as an important characteristic for next-generation TREs, meaning AI research workloads and agentic workflows are supported under the same governance, security, and audit standards that apply to human-led research that relies on access to sensitive datasets.

In practical terms, we have begun to translate this shift in focus into practical areas of implementation. In particular, building on K8TRE we have begun to (i) investigate where AI can be safely embedded into day-to-day research environment operations, (ii) address technical gaps to support AI research & agentic workflows, and (iii) strengthen platform resilience.

AI research workloads from TRE projects

LTH is establishing new, on-premises GPU-enabled compute hardware to serve AI training and inference workloads from trusted research projects governed and managed through our secure analytics framework (KARECTL) deployed within a local Kubernetes cluster.

TRE agent development and secure deployment runtime

To support secure in-house development and deployment of agents that depend on LTH's sensitive data assets (e.g. electronic health record data) we have begun to implement a runtime environment within our Kubernetes TRE that includes localised model provisioning supported by vLLM, allowing agents to exploit privacy-preserving large language model (LLM) inference. In addition, this runtime integrates workflow orchestration and model lifecycle tooling, secure agent observability for monitoring, and governance auditing activities with optional privacy controls to mask sensitive information from traces.

TRE AI-assisted workspaces

We are piloting secure research workspaces that embed AI-assisted notebook capabilities integrated with locally provisioned LLMs to better support analytical and coding research activities.

TRE infrastructure maintenance agents

AI-assisted platform remediation is now live, supporting TRE platform engineers and administrators. Agents with Site Reliability Engineering (SRE) skillsets can detect issues within the TRE through contextual signals provided via KARECTL's observability stack and propose fixes for engineers to approve or apply pre-approved routine remediations based on a real-time understanding of the TRE state through a platform operations agent framework for Kubernetes. We believe such capabilities will become important to ensure TRE engineer resource requirements do not scale with increased research and agent usage.

TRE cybersecurity agents

We are investigating a collection of cybersecurity-focused agents for TREs that aim to support continuous security testing, vulnerability discovery, and defensive validation of a TRE's controls. For example, we are developing red-team capture-the-flag (CTF) agents for Kubernetes-native TREs that can be directed to undertake adversarial behaviours (e.g. unauthorised lateral movement across controlled boundaries) within the target environment and emulating actions that could be performed by different TRE stakeholders (e.g. researchers, platform engineers) to highlight security vulnerabilities.

KARECTL Architecture

KARECTL, LTH's extension of K8TRE, has been designed as a cloud-agnostic, scalable, AI-conformant secure analytics platform to support AI research, AI-assisted research, and AI-enabled platform engineering.

We have extended KARECTL with the following technical capabilities. Operational AI applications and AI-assisted platform remediation are live. Agentic research tools and AI orchestration of federated analysis are in development:

Local model provisioning and routing (LLM gateway, vLLM)

KARECTL's agent gateway service provides a locally governed LLM gateway that allows agents operating within the TRE to connect to a collection of model inference backends through a single endpoint. In particular, the agent gateway integrates with KARECTL's platform foundation services, including Keycloak, for supporting JSON Web Token (JWT) based access control. This allows internal agents to perform authenticated API requests that can be routed to a variety of model inference pools. The gateway implements a dynamic content-based routing method that inspects the agent's request, extracts the model name requested and attempts to route to an available pool serving the specified model. A range of LLM models are deployed through vLLM that provides an efficient local inference engine for lower-latency model serving.

Agentic application runtime

The agentic runtime in KARECTL comprises a collection of services that provide an execution environment where agent-based applications can leverage capabilities that support the end-to-end lifecycle of an agent. These include:

• LLM observability to capture prompts and trace execution paths to support debugging, quality assurance and auditability.
• LLM user interfaces provide a secure interaction layer for researchers and platform engineers to use to communicate with LLM models and support human-in-the-loop agentic workflows. Model inference (vLLM) provides a private model inference and serving engine that integrates with KARECTL's agent gateway service.
• Workflow orchestration provides workflow automation, and model lifecycle tooling manages machine learning models.

TRE platform operations agent framework

KARECTL implements a platform operations agent framework as an application-level service to enable routine TRE platform tasks typically performed by an infrastructure engineer. It allows approved agents to interpret TRE platform context, assist with diagnostics, and propose fixes for engineers to approve or apply pre-approved routine ones. Their actions are logged.

Emerging Challenges

Ensuring Data Security & Privacy

Supporting AI-conformance materially increases the TRE attack surface. Kubernetes control planes, model-serving endpoints, vector stores, agent runtimes and workflow engines all introduce potential security vulnerabilities.

Privacy risk also evolves beyond traditional user access patterns. This is particularly the case where sensitive NHS datasets are fed into local inference models, captured in traces or cached as context to support model inference optimisation. For TRE platform teams, this means extending existing governance controls so they remain effective for AI-assisted and agent-driven workflows.

Governing Agent Behaviour

Agentic systems introduce a different class of governance challenge because they can plan, decide, and execute multi-step actions with limited human intervention. Within a TRE, this raises practical questions around what an agent is permitted to do, what it is explicitly prevented from doing, and how those boundaries are enforced at runtime. From a platform team perspective, this requires policy-driven guardrails.

Managing Costs

One of the central opportunities of AI-conformant TREs is to use automation to improve operational efficiency. If implemented safely, infrastructure and support agents can reduce repetitive administration and issue triage effort, allowing platform teams to focus on higher-value engineering and service improvement. Reduced operational costs are a natural consequence of this increased efficiency, but they should not be the primary objective.

As AI workloads become more common, empowering researchers and TRE infrastructure engineers to develop and deploy agentic workloads also requires new cost tracking and management capabilities. Cost management therefore has to be treated as a core architectural concern for AI-conformant TREs. Practical controls include local model routing, workload scheduling and quotas, and model right-sizing.

Conclusion

The shift in how we think about TREs has evolved significantly over the last six years.

Historically, TREs have been designed primarily as secure environments for researchers to access sensitive data, however, we believe that they are increasingly becoming platforms that should support a diverse ecosystem of participants, including researchers, analysts, clinicians, applications, AI models, and approved AI agents.

This evolution requires a new set of architectural characteristics:

• Cloud-agnostic deployment
• AI-conformant capabilities
• New forms of governance
• Observability & auditing
• Agent-ready runtimes

Significant challenges remain. Cybersecurity threats continue to evolve, governance models must adapt to autonomous systems, and organisations need to develop new expertise spanning cloud-native infrastructure, platform engineering, AI engineering, and security operations.

By building AI-conformant Trusted Research Environments, NHS organisations can create a common foundation capable of supporting research, innovation, operational intelligence, and future agentic healthcare systems. In doing so, the TRE evolves from a secure data access platform into a secure execution environment for both human and machine intelligence.

References

1. Ahmed, M. M., & Othman, Z. K. (2026). The AI Health Arms Race: A Critical Perspective on Big Tech and the Widening Global Health Equity Gap. Journal of Multidisciplinary Healthcare, 19, 610208. https://doi.org/10.2147/JMDH.S610208
2. Pipek, V. and Wulf, V. (2009) "Infrastructuring: Toward an Integrated Perspective on the Design and Use of Information Infrastructures," Journal of the Association for Information Systems, 10(5). DOI: 10.17705/1jais.00195
3. Guembe, B., Azeta, A., Misra, S., Osamor, V. C., Fernandez-Sanz, L., & Pospelova, V. (2022). The Emerging Threat of Ai-driven Cyber Attacks: A Review. Applied Artificial Intelligence, 36(1). https://doi.org/10.1080/08839514.2022.2037254