Why Healthcare AI Safety Is a Stack, Not a Feature

Why Healthcare AI Safety Is a Stack, Not a Feature

Safety in healthcare AI is not something you switch on at the model. It is a property of the whole system, built in layers, and the patient-data layer underneath carries most of the risk that people attribute to the model on top. When a model hallucinates, leaks, or discriminates, it is usually exposing a weakness lower in the stack, in how data was collected, governed, secured, and validated. That cannot be patched at the output.

This matters because the industry tends to debate safety at the wrong altitude. The headline fear is a model giving dangerous clinical advice. The quieter and more common failure is a data layer that was never sound to begin with. If you want to deploy AI in a hospital responsibly, the useful question is not "is the model safe," it is "is the stack safe."

The four layers of healthcare AI safety

It helps to name the layers explicitly, because each fails differently and each needs different controls.

The first is data governance and provenance. Where did the data come from, who consented to its use, how is it labeled, and can you trace any given training example back to its origin. This is the foundation, and weakness here propagates upward invisibly.

The second is security and privacy. Whether patient data is encrypted, access-controlled, and protected across every system and vendor that touches it. This is where breaches happen.

The third is model behavior, meaning accuracy, calibration, and bias. Whether the model is right, whether it knows when it is uncertain, and whether it performs evenly across patient groups.

The fourth is human oversight and accountability. Whether a clinician can see, question, and override the system, and whether someone is answerable when it fails.

Most safety conversations jump straight to the third layer. The first two decide whether the third layer ever had a chance.

Why the data layer is load-bearing

Start with what the data layer actually exposes when it fails. 2025 was the worst year on record for large healthcare data breaches by count. More than 770 breaches affecting 500 or more individuals were reported to the US Office for Civil Rights, exposing the protected health information of roughly 140 million people, the third-highest annual total ever recorded (HIPAA Journal, 2025 breach data). This is strong evidence, drawn from a mandatory federal reporting portal, though the tally keeps rising as investigations close.

Two patterns stand out. Hacking and IT incidents account for the overwhelming majority of exposed records, and a large share of breaches originate at business associates rather than at the providers themselves (HIPAA Journal statistics). The lesson for anyone building or buying healthcare AI is that the attack surface is the whole vendor chain, not just your own four walls. Every system that touches patient data inherits the obligation to protect it.

De-identification is often treated as the safety valve that makes patient data usable for AI. It helps, but it is not a guarantee. Researchers have repeatedly shown that supposedly anonymous health data can be re-identified. In one widely cited episode, Australia released de-identified medical records for 10 percent of the population and researchers re-identified individuals within weeks (documented re-identification cases). A 2025 analysis in the Journal of the American Medical Informatics Association found that common privacy-preserving patient-matching schemes can carry re-identification rates above 45 percent when tokens are shared alongside basic demographics (JAMIA, 2025).

The honest reading of the evidence is that re-identification risk is real but governable. A 2025 review of de-identified clinical free text concluded that when data is properly de-identified and held inside a secure data environment, residual re-identification risk is very low (AI and Ethics, 2025). The risk is not the de-identification step alone. It is the combination of weak de-identification with loose access and linkable external data. That is a data-layer architecture decision, not a model problem.

Provenance closes the loop. Bias and model drift almost always trace back to what the data captured and what it missed. A model trained on data that underrepresents a population will underperform for that population, and no amount of model-layer tuning fully corrects for a foundation that was skewed from the start.

What the model layer can and cannot promise

The model layer has earned genuine optimism and genuine caution, and both are supported by evidence.

On the optimistic side, large language models now pass medical licensing examinations and approach specialist-level performance on structured benchmarks (benchmarking study, 2025). On the cautious side, the same models hallucinate, meaning they produce confident, plausible, and incorrect output. A study from Mount Sinai published in Communications Medicine found that leading models were highly vulnerable to adversarial hallucination, generating false clinical details about 66 percent of the time under a default prompt, falling to about 44 percent with a mitigation prompt (Communications Medicine, 2025).

Grade that carefully. It is a single peer-reviewed study run under adversarial conditions, where false details were deliberately planted, so it is not a measure of everyday error rates. But the direction is unambiguous and consistent across the medical hallucination literature. Even strong models fail in ways that look authoritative, and prompting alone does not eliminate the problem. That is precisely why the fourth layer, human oversight, is not optional. A model is a clinical aid, not a clinical authority.

What regulation now requires

The regulatory picture has firmed up considerably, and it reinforces the same layered logic.

The strongest signal is the EU AI Act, which is binding law. Most clinical AI, including systems for diagnosis, decision support, triage, and patient monitoring, is classified as high-risk by design (EU AI Act guidance for healthcare). The core obligations for high-risk systems became applicable in August 2026, with an extended transition to August 2027 for AI embedded in medical devices that require Notified Body assessment under the existing device regulations (implementation analysis). Those obligations map almost one to one onto the safety stack. They mandate data governance, transparency, risk management across the lifecycle, post-market monitoring, and meaningful human oversight. Regulators have effectively encoded the idea that safety is layered.

In the United States, HIPAA remains the binding floor for patient data, and the Office for Civil Rights spent 2025 focused on the risk-analysis provision of the Security Rule, the most commonly cited failure (HIPAA Journal, 2025 report). Risk analysis is a data-layer and security-layer discipline.

The World Health Organization sits one tier down in enforceability but high in credibility. Its 2024 guidance on large multi-modal models offers more than 40 recommendations for governments, developers, and providers, with particular emphasis on equity and on closing the governance gap between high-income and lower-income countries (WHO LMM guidance). This is emerging best practice rather than law, but it is the most authoritative global direction available.

The view from a low-resource setting

For health systems in much of Africa, the stack framing matters more, not less. Records are often fragmented across paper, disconnected registries, and incompatible systems, and formal data-governance infrastructure is thinner than in the markets where most AI regulation is written. The WHO has been explicit that the governance gap between developing and developed economies is one of the central risks of this moment.

The temptation in that context is to treat safety as a feature to add later, once the AI is working. The evidence points the other way. A weak data foundation is harder to retrofit than to build well the first time, and the populations served by these systems are exactly the ones most exposed when safety is bolted on as an afterthought. Building the data layer properly is not a luxury reserved for well-resourced systems. It is the part that determines whether the model layer is safe to stand up at all.

The takeaway

Safety in healthcare AI is an architecture decision, and it is made early. The model on top can only be as trustworthy as the data, security, and governance beneath it. Treat the patient-data layer as load-bearing, because it is. Get that right, and the model has something safe to stand on. Get it wrong, and no feature added later will hold the weight.