Prompt Injection Emerges as a Patient Safety Threat in Clinical AI

Artificial intelligence in medicine has spent the past decade under scrutiny for the failures everyone can see: a model that misreads a scan, a chatbot that produces a confident but wrong diagnosis, an algorithm that performs worse for one patient population than another. A new letter in the Annals of Biomedical Engineering argues that the most dangerous failure mode may be the one nobody notices, because it does not look like a failure at all. Ethan Waisberg of the University of Cambridge and Joseph W. Guarnieri of the Blue Marble Space Institute of Science contend that prompt injection, the technique of smuggling hidden instructions into the text a language model reads, should be classified as a patient safety hazard with a formal threat model, not dismissed as a technical curiosity confined to security conferences.

The core of their argument rests on a property of current large language model architectures that is easy to state and hard to fix. A model such as GPT-4 or Llama receives an undifferentiated stream of tokens. The words typed by a clinician, the text extracted from a referral letter, the contents of a patient’s portal message, and any instructions embedded by a system designer all arrive as one continuous sequence. The architecture has no mechanism for distinguishing content that carries authority from content that does not. When a model follows an instruction that neither the clinician wrote nor can see, it is not malfunctioning in the usual sense. It is performing exactly as instructed, by an instruction that entered the context window from the outside world.

This is what separates prompt injection from the failure modes that dominate clinical AI safety discussions. Accuracy, bias, and hallucination all describe a model failing at its assigned task, and each has spawned a mature evaluation culture of benchmarks, audits, and regulatory checkpoints. Prompt injection describes the opposite condition: the model succeeding at a task nobody assigned to it. That inversion matters because the entire apparatus of clinical validation is built to detect poor performance, not to detect a model that has been quietly redirected. A safety evaluation that tests whether an AI summarizer produces faithful summaries tells you nothing about what happens when a malicious string inside a scanned document tells the model to omit a critical allergy or to append a reassuring but false statement.

Medicine is particularly exposed to this attack surface because of how clinical records are assembled. The letter emphasizes that the medical record is not a curated, institution-controlled corpus. It is a confluence of material originating outside the hospital walls: referral correspondence from other practices, patient-entered messages, external imaging reports, scanned documents of uncertain provenance, and records acquired elsewhere and pasted into the chart. Every one of those channels is an entry point for text that an adversary can craft. A referral letter could contain a sentence invisible to human readers but legible to a model. A patient message could carry a benign-looking request that doubles as an instruction to the summarization system processing the inbox. Scanned paperwork, where optical character recognition converts pixels into tokens, offers an especially wide aperture because no human will ever read the raw text the model consumes.

The empirical evidence that this vulnerability is real rather than theoretical has accumulated rapidly. A 2025 study in JAMA Network Open demonstrated that large language models providing medical advice can be manipulated through prompt injection to produce harmful recommendations. In the same year, researchers reported in Nature Communications that adversarial prompts and fine-tuning attacks threaten medical large language models systematically, and a separate Nature Communications study showed prompt injection attacks succeeding against vision language models in oncology, the multimodal systems designed to interpret pathology slides and clinical images alongside text. A Nature Medicine paper added a further dimension, showing that medical large language models are vulnerable to data-poisoning attacks, in which corruption is introduced during training rather than at inference time. Together, these findings suggest that the injection problem spans text, images, and the training pipeline itself.

The threat grows sharper as medicine moves from language models that answer questions to agentic AI systems that take actions. Waisberg and Guarnieri, who have previously written about delegation and supervision in clinical AI agents, point out that an agent designed to draft orders, schedule appointments, triage messages, or query the electronic health record does not merely generate text; it converts text into consequences. A prompt injection that redirects a chatbot might produce a wrong answer. A prompt injection that redirects an agent with write access to a record system, a prescribing interface, or a lab-ordering pipeline can produce a wrong action, executed automatically and possibly at scale. The authors’ earlier work on correlated failure is relevant here: when many clinical decisions are delegated to systems built on the same foundation models, a single effective injection technique can propagate across institutions that believed themselves independent.

Why can’t the problem simply be prompted away? The letter is direct on this point: improved prompting and input filtering do not address the vulnerability. Telling a model to ignore hidden instructions relies on the same mechanism that makes it vulnerable in the first place, namely treating all text as instructions to weigh. Input filtering fares little better, because the boundary between legitimate clinical content and malicious payload is semantic rather than syntactic; a sentence that says ‘the patient reports that all previous instructions should be disregarded’ is grammatically indistinguishable from ordinary patient prose. The OWASP Foundation has recognized the severity of the issue by listing prompt injection among the top risks for large language model applications in its 2026 guidance, but awareness alone does not restore an authority signal that the architecture never had.

What would a workable response look like? The authors outline three pillars. The first is provenance-aware context handling: systems should track where every token in the context window came from and treat externally sourced text as untrusted data rather than as instruction-bearing content, engineering the authority distinction that the architecture lacks. The second is restricted privileges for irreversible actions. An AI system should never hold, in a single context, both the ability to be influenced by untrusted input and the authority to perform actions that cannot be undone, such as ordering medication or altering a legal record; human confirmation gates must sit between untrusted text and irreversible consequence. The third is adversarial testing before deployment, treating prompt injection the way aviation treats engine failure, as a scenario to be probed systematically before patients are exposed, rather than discovered in the field.

The regulatory dimension is already taking shape. The European Union’s Artificial Intelligence Act, Regulation 2024/1689, establishes harmonized rules for high-risk AI systems, and clinical decision-support tools fall squarely within that category. The letter’s contribution to that conversation is conceptual: prompt injection should be treated as a patient safety hazard in the same register as medication error or device malfunction, which means it belongs in risk assessments, incident reporting systems, and pre-market validation requirements, not only in cybersecurity frameworks. Whether hospitals, vendors, and regulators converge on that framing in time for the wave of agentic clinical deployments now in development may determine whether the first large-scale prompt injection incident in medicine is a contained near-miss or a patient harm that reshapes the field.

Subject of Research: Prompt injection attacks on large language models and agentic AI in clinical medicine

Article Title: Prompt Injection in Clinical Artificial Intelligence Systems: The Emerging Security Challenge of Large Language Models and Agentic AI

Article References: Waisberg, E., & Guarnieri, J. W. (2026). Prompt Injection in Clinical Artificial Intelligence Systems: The Emerging Security Challenge of Large Language Models and Agentic AI. Annals of Biomedical Engineering. https://doi.org/10.1007/s10439-026-04376-3

Image Credits: AI Generated

DOI: 10.1007/s10439-026-04376-3

Keywords: prompt injection, large language models, clinical AI, patient safety, agentic AI, AI security, adversarial attacks, electronic health records, medical decision support, AI regulation, provenance, hallucination