Large language models are moving rapidly from research laboratories into hospitals, clinics and medical offices, promising to transform how healthcare professionals document cases, search medical knowledge and make clinical decisions. Yet a new review published in Nature warns that the systems powering tools such as ChatGPT, Claude and specialized medical assistants are advancing faster than the safeguards needed to control them. Researchers from the Else Kröner Fresenius Center for Digital Health at TU Dresden University of Technology and University Hospital Dresden, working with colleagues across Germany and the United States, have assembled evidence from medical artificial intelligence, cybersecurity, regulatory science, ethics and behavioral psychology to map the risks of using large language models, or LLMs, in healthcare. Their conclusion is both optimistic and urgent: these systems could meaningfully improve clinical workflows, but their benefits cannot be separated from rigorous oversight.
LLMs are artificial-intelligence systems trained on enormous collections of text and other data to predict and generate sequences of language. In a clinical environment, that ability can be used to summarize patient records, draft medical notes, explain complex information, retrieve relevant research and assist with diagnostic reasoning. Some models are also being connected to imaging systems, laboratory data and electronic health records. This integration gives them access to highly sensitive information and places their outputs closer to real medical decisions. The review notes that many healthcare workers are already using such tools informally, sometimes without institutional approval, technical safeguards or clear instructions about responsibility. This unofficial “shadow use” makes it difficult for hospitals to know which systems are being used, what information is being entered and whether the resulting recommendations influence patient care.
The authors emphasize that risk is not confined to the moment when a clinician receives an AI-generated answer. It can emerge throughout the entire lifecycle of a system, beginning with model design and continuing through data collection, training, deployment, software updates and everyday use. A model may inherit weaknesses from its training data, including incomplete representation of populations, outdated medical knowledge or systematic biases. Data used to train or adapt a model can also be deliberately manipulated in an attack known as data poisoning. By inserting carefully crafted examples into a training set, an attacker may influence how the model behaves later. Because modern LLMs are complex and difficult to interpret, identifying the precise cause of a dangerous response can be challenging even after an incident has occurred.
Another cybersecurity threat involves prompt injection. In this type of attack, hidden or misleading instructions are embedded in text supplied to the model. The instructions can redirect the model away from its intended task, cause it to disclose information or make it ignore clinically relevant evidence. In a medical setting, a compromised prompt might cause an AI system to overlook an abnormality in a pathology report or fail to identify a tumor that is visible in a tissue sample. Prompt injection can be especially difficult to prevent when an LLM processes material from multiple sources, such as clinician instructions, patient records, web pages and laboratory reports. If the system treats all text as equally authoritative, malicious content may be mistaken for a legitimate command. Beyond the model itself, weaknesses in hospital networks, application programming interfaces and cloud infrastructure could expose patient data or interrupt essential clinical services.
The most familiar model-inherent hazard is the phenomenon commonly called hallucination: the generation of information that sounds convincing but is false, unsupported or invented. LLMs do not retrieve truth in the human sense. They generate statistically plausible sequences based on patterns learned during training, unless they are connected to carefully controlled databases or retrieval systems. Even retrieval-augmented systems can produce errors if the source material is incomplete, poorly indexed or misinterpreted. In medicine, a fabricated citation, incorrect drug interaction or inaccurate diagnostic explanation could have immediate consequences. The danger is amplified by the fluent and confident style of many models, which can make a wrong answer appear more reliable than a cautious human response.
The review also examines how human behavior can turn technical imperfections into clinical harm. Clinicians may develop automation bias, placing too much trust in an AI recommendation simply because it was produced by a sophisticated system. This can reduce the likelihood that an output will be independently checked, particularly in busy environments where staff face time pressure and heavy workloads. The opposite problem can occur when an AI system is used to confirm an existing belief. If a clinician suspects a particular diagnosis, a model may be prompted in a way that encourages it to support that assumption, reinforcing confirmation bias rather than challenging it. LLMs can also adapt their language to the apparent expectations of the user, producing answers that are more agreeable than accurate. Long or complicated conversations create further risks because errors can accumulate across multiple exchanges and become difficult to trace.
Privacy and accountability add another layer of complexity. Many commercially available AI services are hosted outside the healthcare institution, meaning that hospitals may have limited control over where patient information is processed, how long it is retained and whether it is used for further system development. Even when providers promise data protection, healthcare organizations must understand the technical and legal arrangements governing access, storage and deletion. De-identification can reduce privacy risks, but it is not a universal solution: detailed clinical narratives may contain enough unusual information to make individuals identifiable when combined with other data. At the same time, responsibility can become blurred when a recommendation is produced by a model, reviewed by a clinician and embedded in software supplied by a third party. The authors argue that clinical accountability must remain explicit rather than being dispersed across an opaque technical chain.
To reduce these dangers, the researchers call for security to be built into AI development from the beginning rather than added after deployment. This includes carefully curating training and fine-tuning data, testing models against adversarial attacks, documenting known limitations and evaluating performance on clinically representative cases. Hospitals should monitor systems continuously instead of treating approval as a one-time event. Monitoring can include tracking error patterns, unexpected changes in performance, suspicious user activity and the effects of software updates. The authors recommend clear institutional rules defining which tools may be used, what data may be entered, when human review is mandatory and who is responsible for responding to incidents. They also propose local teams dedicated to supervising AI in clinical practice, supported by centralized AI Security Operations Centers, or SOCs, that could identify threats across institutions and coordinate responses to attacks or failures.
The review further argues that regulation must evolve alongside rapidly changing AI technology. Only a small proportion of AI systems are formally approved as medical devices, while many general-purpose models can influence healthcare without fitting neatly into existing regulatory categories. Traditional frameworks were largely designed for products that remain stable after approval. LLMs, by contrast, may change through updated training data, modified instructions, new connected tools or silent software revisions. A system that performs well during an initial evaluation may behave differently months later or when used in a new hospital with a different patient population. The authors therefore support regulatory approaches based on continuous evaluation, post-deployment surveillance and transparent reporting of significant changes. Stephen Gilbert, Professor of Medical Device Regulatory Science at TU Dresden, says that oversight and technological development must be more closely integrated if patients and health systems are to benefit safely from these tools.
The researchers stress that safer medical AI will require cooperation among engineers, clinicians, cybersecurity specialists, regulators, ethicists and patients. Human oversight remains essential, but oversight must be meaningful: clinicians need sufficient training, enough time to review AI outputs and access to information about how systems were evaluated. Institutions should also make responsible use easier than unofficial use by providing approved tools, practical guidance and channels for reporting errors without fear of punishment. “AI is already being used in healthcare, often without formal oversight,” says Jakob N. Kather, Professor of Clinical AI at the Else Kröner Fresenius Center. “The key question is how to implement these systems in a way that is transparent, robust, and aligned with clinical responsibility.” The review’s central message is not that LLMs should be kept out of medicine, but that their deployment must be treated as an ongoing safety process. Powerful language models may become valuable clinical partners, but only if hospitals continuously test, monitor and govern them with the same seriousness applied to other technologies that can affect human life.
Subject of Research: Safety and security risks of large language models in healthcare
Article Title: Safety and security of large language models in healthcare
News Publication Date: 19 August 2026
Web References: https://doi.org/10.1038/s41586-026-10687-1
References: Clusmann J, Freyer O, Ostermann M, Ferber D, Ghaffari Laleh N, Hilgers L, Kolbinger FR, Schneider CV, Downing A, Wekenborg MK, Gilbert S, Foersch S, Truhn D, Wiest IC, Kather JN. “Safety and security of large language models in healthcare.” Nature. 2026. DOI: 10.1038/s41586-026-10687-1
Keywords: Large language models; artificial intelligence; healthcare; clinical AI; patient safety; cybersecurity; data privacy; prompt injection; data poisoning; hallucinations; automation bias; AI regulation; medical devices; AI governance
