LLMs are artificial-intelligence systems trained on enormous collections of text and other data to predict and generate sequences of language. In a clinical environment, that ability can be used to summarize patient records, draft medical notes, explain complex information, retrieve relevant research and assist with diagnostic reasoning. Some models are also being connected to imaging systems, laboratory data and electronic health records. This integration gives them access to highly sensitive information and places their outputs closer to real medical decisions. The review notes that many healthcare workers are already using such tools informally, sometimes without institutional approval, technical safeguards or clear instructions about responsibility. This unofficial “shadow use” makes it difficult for hospitals to know which systems are being used, what information is being entered and whether the resulting recommendations influence patient care.

The authors emphasize that risk is not confined to the moment when a clinician receives an AI-generated answer. It can emerge throughout the entire lifecycle of a system, beginning with model design and continuing through data collection, training, deployment, software updates and everyday use. A model may inherit weaknesses from its training data, including incomplete representation of populations, outdated medical knowledge or systematic biases. Data used to train or adapt a model can also be deliberately manipulated in an attack known as data poisoning. By inserting carefully crafted examples into a training set, an attacker may influence how the model behaves later. Because modern LLMs are complex and difficult to interpret, identifying the precise cause of a dangerous response can be challenging even after an incident has occurred.

Another cybersecurity threat involves prompt injection. In this type of attack, hidden or misleading instructions are embedded in text supplied to the model. The instructions can redirect the model away from its intended task, cause it to disclose information or make it ignore clinically relevant evidence. In a medical setting, a compromised prompt might cause an AI system to overlook an abnormality in a pathology report or fail to identify a tumor that is visible in a tissue sample. Prompt injection can be especially difficult to prevent when an LLM processes material from multiple sources, such as clinician instructions, patient records, web pages and laboratory reports. If the system treats all text as equally authoritative, malicious content may be mistaken for a legitimate command. Beyond the model itself, weaknesses in hospital networks, application programming interfaces and cloud infrastructure could expose patient data or interrupt essential clinical services.

The most familiar model-inherent hazard is the phenomenon commonly called hallucination: the generation of information that sounds convincing but is false, unsupported or invented. LLMs do not retrieve truth in the human sense. They generate statistically plausible sequences based on patterns learned during training, unless they are connected to carefully controlled databases or retrieval systems. Even retrieval-augmented systems can produce errors if the source material is incomplete, poorly indexed or misinterpreted. In medicine, a fabricated citation, incorrect drug interaction or inaccurate diagnostic explanation could have immediate consequences. The danger is amplified by the fluent and confident style of many models, which can make a wrong answer appear more reliable than a cautious human response.

The review also examines how human behavior can turn technical imperfections into clinical harm. Clinicians may develop automation bias, placing too much trust in an AI recommendation simply because it was produced by a sophisticated system. This can reduce the likelihood that an output will be independently checked, particularly in busy environments where staff face time pressure and heavy workloads. The opposite problem can occur when an AI system is used to confirm an existing belief. If a clinician suspects a particular diagnosis, a model may be prompted in a way that encourages it to support that assumption, reinforcing confirmation bias rather than challenging it. LLMs can also adapt their language to the apparent expectations of the user, producing answers that are more agreeable than accurate. Long or complicated conversations create further risks because errors can accumulate across multiple exchanges and become difficult to trace.

Privacy and accountability add another layer of complexity. Many commercially available AI services are hosted outside the healthcare institution, meaning that hospitals may have limited control over where patient information is processed, how long it is retained and whether it is used for further system development. Even when providers promise data protection, healthcare organizations must understand the technical and legal arrangements governing access, storage and deletion. De-identification can reduce privacy risks, but it is not a universal solution: detailed clinical narratives may contain enough unusual information to make individuals identifiable when combined with other data. At the same time, responsibility can become blurred when a recommendation is produced by a model, reviewed by a clinician and embedded in software supplied by a third party. The authors argue that clinical accountability must remain explicit rather than being dispersed across an opaque technical chain.

To reduce these dangers, the researchers call for security to be built into AI development from the beginning rather than added after deployment. This includes carefully curating training and fine-tuning data, testing models against adversarial attacks, documenting known limitations and evaluating performance on clinically representative cases. Hospitals should monitor systems continuously instead of treating approval as a one-time event. Monitoring can include tracking error patterns, unexpected changes in performance, suspicious user activity and the effects of software updates. The authors recommend clear institutional rules defining which tools may be used, what data may be entered, when human review is mandatory and who is responsible for responding to incidents. They also propose local teams dedicated to supervising AI in clinical practice, supported by centralized AI Security Operations Centers, or SOCs, that could identify threats across institutions and coordinate responses to attacks or failures.

The review further argues that regulation must evolve alongside rapidly changing AI technology. Only a small proportion of AI systems are formally approved as medical devices, while many general-purpose models can influence healthcare without fitting neatly into existing regulatory categories. Traditional frameworks were largely designed for products that remain stable after approval. LLMs, by contrast, may change through updated training data, modified instructions, new connected tools or silent software revisions. A system that performs well during an initial evaluation may behave differently months later or when used in a new hospital with a different patient population. The authors therefore support regulatory approaches based on continuous evaluation, post-deployment surveillance and transparent reporting of significant changes. Stephen Gilbert, Professor of Medical Device Regulatory Science at TU Dresden, says that oversight and technological development must be more closely integrated if patients and health systems are to benefit safely from these tools.

The researchers stress that safer medical AI will require cooperation among engineers, clinicians, cybersecurity specialists, regulators, ethicists and patients. Human oversight remains essential, but oversight must be meaningful: clinicians need sufficient training, enough time to review AI outputs and access to information about how systems were evaluated. Institutions should also make responsible use easier than unofficial use by providing approved tools, practical guidance and channels for reporting errors without fear of punishment. “AI is already being used in healthcare, often without formal oversight,” says Jakob N. Kather, Professor of Clinical AI at the Else Kröner Fresenius Center. “The key question is how to implement these systems in a way that is transparent, robust, and aligned with clinical responsibility.” The review’s central message is not that LLMs should be kept out of medicine, but that their deployment must be treated as an ongoing safety process. Powerful language models may become valuable clinical partners, but only if hospitals continuously test, monitor and govern them with the same seriousness applied to other technologies that can affect human life.

Subject of Research: Safety and security risks of large language models in healthcare

Article Title: Safety and security of large language models in healthcare

News Publication Date: 19 August 2026

Web References: https://doi.org/10.1038/s41586-026-10687-1

References: Clusmann J, Freyer O, Ostermann M, Ferber D, Ghaffari Laleh N, Hilgers L, Kolbinger FR, Schneider CV, Downing A, Wekenborg MK, Gilbert S, Foersch S, Truhn D, Wiest IC, Kather JN. “Safety and security of large language models in healthcare.” Nature. 2026. DOI: 10.1038/s41586-026-10687-1

Keywords: Large language models; artificial intelligence; healthcare; clinical AI; patient safety; cybersecurity; data privacy; prompt injection; data poisoning; hallucinations; automation bias; AI regulation; medical devices; AI governance

The concept of social license describes the informal level of public acceptance that allows an emerging technology or organization to operate with society’s approval, even when that approval is not established through laws or formal regulations. In the context of artificial intelligence, social license can determine whether patients are willing to accept algorithm-assisted diagnoses, automated administrative systems, predictive tools, or AI-supported clinical decision-making. The study indicates that acceptance depends on more than whether an algorithm is technically accurate. It also depends on whether people believe the technology fits within the values and relationships that define good health care.

The researchers found that structural factors form one of the central foundations of public trust. These factors include the policies, governance systems, institutional safeguards, and accountability mechanisms surrounding an AI tool. Patients may ask who developed the system, what data were used to train it, how personal information is protected, and who is responsible if the system makes a harmful recommendation. Such questions are especially important because modern AI systems can process vast volumes of medical records, imaging data, laboratory results, and other sensitive information, often through complex computational models that are difficult for non-specialists to interpret.

Performance factors also strongly influence whether consumers continue to support AI in clinical settings. An AI system may be designed to recognize patterns in medical images, estimate a patient’s risk of developing a disease, or help clinicians identify the most appropriate treatment. However, technical performance measured in a laboratory or validation study may not fully predict how the system works in real-world care. Accuracy, reliability, consistency, transparency, and the ability to perform across different patient populations all contribute to public confidence. A tool that performs well for one demographic group but produces less reliable results for another could quickly undermine trust and raise concerns about algorithmic bias.

The study further emphasizes that technical capability alone cannot replace human judgment. Health care decisions frequently involve uncertainty, personal priorities, cultural context, and emotional distress—factors that may not be captured in structured data. Consumers appear to expect AI to support clinicians rather than remove the human relationship from care. This distinction is crucial: an algorithm may identify a potential diagnosis or flag a concerning trend, but patients may still want a qualified professional to explain the result, discuss alternatives, respond to questions, and understand the circumstances that make each person’s situation unique.

Relational factors therefore emerged as another important component of social license. These factors concern the relationships among patients, clinicians, health care institutions, and the developers of AI systems. Patients may be more willing to accept AI when clinicians remain visibly involved and can explain how a tool contributed to a decision. Conversely, acceptance may weaken if people feel that an automated system is being used to replace meaningful communication or to accelerate appointments at the expense of empathy. The findings suggest that AI implementation is not simply a technical upgrade; it is also a change to the social and emotional structure of medical care.

Because social license is dynamic, public acceptance can shift as people encounter AI in different circumstances. A system that is welcomed for reducing paperwork may be viewed differently when it influences a diagnosis or treatment recommendation. Similarly, a positive initial experience could be reversed by a privacy breach, an unexplained error, or evidence that a tool performs unevenly across communities. This means health care organizations may need to monitor trust continuously rather than treating public consultation as a one-time stage completed before deployment.

The study provides evidence-based recommendations for stakeholders designing and implementing clinical AI. Developers are encouraged to build tools around the needs of both consumers and clinicians, with clear explanations of what the system does, what its limitations are, and when its recommendations should be questioned. Health care organizations may need strong oversight procedures, transparent evaluation, mechanisms for reporting errors, and clear lines of responsibility. Clinicians, meanwhile, will require training that covers not only how to use AI systems but also how to communicate their role to patients and recognize situations in which automated outputs may be unreliable.

The researchers argue that the most socially acceptable AI will be technology that strengthens personalized, empathetic, and responsive care. This could include systems that help clinicians spend less time on repetitive documentation and more time speaking with patients, provided that efficiency gains do not become a justification for reducing human contact. The public’s willingness to accept AI may ultimately depend on whether it is experienced as an instrument for better care or as a barrier between patients and the professionals they trust.

The findings arrive as health systems worldwide explore increasingly sophisticated forms of artificial intelligence, from machine-learning models that analyze medical images to generative systems capable of summarizing records and producing clinical text. The study suggests that the future of health care AI will be shaped not only by computing power or benchmark scores but also by sustained public judgment. Social license will have to be earned repeatedly through responsible governance, dependable performance, open communication, and visible respect for the human relationships at the center of medicine.

Subject of Research: Consumer perspectives on artificial intelligence in health care and the conditions shaping public acceptance, or “social license,” for clinical AI.

News Publication Date: Not provided.

Web References: https://doi.org/10.1001/jamanetworkopen.2026.26916

References: Duong T et al. Qualitative study of consumer perspectives on AI in health care. JAMA Network Open. DOI: 10.1001/jamanetworkopen.2026.26916.

Keywords: Artificial intelligence, health care, clinical AI, social license, patient trust, algorithmic bias, digital health, health care delivery, machine learning, patient-centered care