AI Chatbot for Parkinson’s Disease Shows Hidden Safety Risks in First Real-World Trial

A pioneering German chatbot built to answer questions about Parkinson’s disease has become the first patient-facing medical AI system to undergo prospective, conversation-level safety surveillance after public launch, and the results reveal both the promise and the hidden dangers of deploying generative artificial intelligence to vulnerable patients. During its first 129 days of live operation, the chatbot, known as jAImes, handled 2,035 conversations containing 6,146 messages from real users. An independent clinical evaluation of that complete record, published in The Lancet Regional Health – Europe, found that while the vast majority of exchanges were rated adequate by automated triage, a small number of clinically critical failures slipped through every automated safety net—including one interaction in which a user expressed suicidal thoughts shortly after a Parkinson’s diagnosis and the system’s emergency protocol never activated.

The study, led by researchers at University Hospital Würzburg together with the system’s developer and the Parkinson Stiftung, the non-profit German foundation that commissioned and operates the service, is being hailed as a template for what regulators have long demanded but rarely seen: structured, post-deployment monitoring of an autonomous, unsupervised, publicly accessible medical AI. Unlike earlier evaluations that relied on physician supervision of every exchange or on user surveys, this analysis covered every conversation from the first day of routine operation, with no recruitment, incentives, or modification of user behaviour. Users asked genuinely unprompted questions about symptoms, medications, daily coping, and therapies—precisely the conditions under which the failure modes that matter most become visible.

jAImes was deliberately engineered as a counter-design to the general-purpose chatbots that have fared poorly in recent audits. Rather than allowing a large language model to answer freely, the system separates a user-facing advisory agent, built on Claude Sonnet 4.5, from a database agent running Mistral Small 3.2, which synthesises answers strictly from a curated knowledge base of 221 documents comprising peer-reviewed literature, expert lectures, and validated patient-education materials. A retrieval-augmented pipeline with hybrid search, fusion, and reranking grounds the answers in these sources, while system-level prompts explicitly prohibit diagnostic interpretation, medication dosing, and therapy modification, and define emergency triggers that are supposed to activate a crisis protocol focused on supportive language and signposting to emergency services. The tool is explicitly scoped as a non-diagnostic, non-therapeutic information system, free to use in Germany without registration or advertising.

To evaluate the deployment, the team introduced a new quality-assurance framework called CARE-LLM, short for Conversation-level AI Real-world Evaluation, described for the first time in this study. It combines four components: comprehensive automated triage of every conversation, structured human expert review of flagged cases, sampling-based validation of conversations the triage rated as adequate, and a feedback loop that routes confirmed failures to class-specific remediation. Automated screening classified 88.6 percent of conversations as good, 11.0 percent as partially adequate, and just 0.4 percent as inadequate, with knowledge-base gaps rather than unsafe answers accounting for most partial ratings. Triage flagged 212 conversations, and together with 28 negative-feedback exchanges, 224 unique conversations entered structured review; 45 warranted detailed specialist assessment, and five were confirmed critical by board-certified neurologists with more than a decade of movement-disorder experience, all of whom were structurally independent of the developer and the foundation.

Those confirmed events fell into three distinct failure classes that the authors propose as an empirically grounded taxonomy. Knowledge boundary failures occur when user queries exceed the system’s retrieval coverage and the generative component fills the gap with confident output instead of signalling uncertainty—one such case involved factually incorrect information about an ongoing clinical trial and a misstated investigator affiliation. Robustness failures reflect susceptibility to manipulative or adversarial prompting, illustrated by a conversation interrupted by the underlying content-safety filter in a pattern consistent with a jailbreak-style attack. Escalation failures are breakdowns in predefined emergency responses: of three conversations containing explicit suicidal ideation, two surfaced emergency contacts but were judged insufficiently aligned with the intended protocol, and one—the exchange following a fresh Parkinson’s diagnosis—triggered no crisis response at all.

Perhaps the most consequential finding came from the framework’s third component, which exists precisely to expose the blind spots of automated triage. When the researchers randomly sampled 100 of the 1,803 conversations classified as adequate and subjected them to independent clinical review, four contained clinically critical errors—a conditional false-negative rate of 4 percent within the good stratum. These were not exotic failures but classic knowledge boundary errors: an inappropriate recommendation of memantine for Parkinson’s disease dementia, a pharmacologically inaccurate statement about the duration of prolonged-release levodopa, a non-first-line suggestion of amantadine for tremor, and a medication misidentification in which a levodopa/carbidopa preparation was labelled as selegiline. The authors stress that this figure is a single-window estimate with a wide confidence interval, spanning roughly one missed critical event per 60 to one per 10 good-rated conversations, and that it cannot be extrapolated to the full corpus. But its message is unambiguous: favourable automated quality metrics can coexist with a clinically meaningful residual rate of critical failures invisible to those metrics.

The usage data themselves offer a portrait of who turns to such systems and what they want to know. Usage was continuous, averaging 47.3 messages per day, with a median response latency of 33 seconds and knowledge-base retrieval triggered in 91.9 percent of conversations. Where users disclosed their role, 75 percent were patients, 16 percent relatives or caregivers, and the remainder physicians and nursing staff; the most common age band was 60 to 69 years, matching the intended population. The dominant themes were symptoms, daily life and coping, medications, and therapies. Explicit knowledge gaps appeared in 12.3 percent of conversations, most often concerning region-specific contacts such as local self-help groups and specialist clinics, newer medications, current studies, and procedures like high-intensity focused ultrasound. These gaps rarely produced unsafe answers but limited completeness and local usefulness, underscoring that content maintenance is itself a safety function.

The authors are candid that the evaluation is an operator-led post-market assessment embedded in an imperfect deployment. jAImes entered public use without a formal pre-deployment validation study; the informal six-month expert-testing phase, which included adversarial jailbreak-style probing, was formative rather than evaluative, and none of the critical failures reported here was caught before launch. The team explicitly rejects the notion that deployment-with-surveillance can substitute for pre-deployment validation—the missed suicidality escalation occurred in a live system, and no retrospective monitoring result can remove that exposure. They also acknowledge that anonymity, while lowering the threshold for disclosing stigmatised concerns and enforcing GDPR data minimisation, made individual follow-up impossible and ruled out any real-time clinical monitoring or emergency intervention during deployment. The crisis-response protocol was revised at the first scheduled maintenance after the analysis identified the missed escalation, and re-testing confirmed the specific failure was remedied, though the system configuration has since changed in other respects as well.

Notably, an external legal assessment concluded that jAImes does not qualify as a medical device under the EU Medical Device Regulation, because its purpose is confined to general disease information, nor as a high-risk AI system under the EU AI Act—yet the surveillance reported here was adopted voluntarily, echoing the life-cycle monitoring principles of the AI Act’s Article 72 and FDA postmarket guidance. The study’s regulatory argument is pointed: clinically consequential failures arise in patient-facing information tools regardless of device status, and a conventional usability or satisfaction study would have detected none of the critical events documented. The authors argue that prospective, conversation-level evaluation with independent clinical adjudication should be treated as a design requirement for patient-facing medical large language models, and that periodic sampling-based validation should complement flag-driven review as routine deployment-level quality assurance.

The broader significance extends well beyond Parkinson’s disease. Recent randomised trials of patient-facing large language models in digital psychotherapy and primary-to-specialist care transitions measured clinical efficacy, not post-deployment safety, and audits of unscoped consumer chatbots have rated roughly half of health-related responses as problematic, with models hallucinating citations. jAImes shows that tightly scoped, retrieval-augmented design can dramatically improve on that baseline while still harbouring residual failure modes that only real-world, clinician-adjudicated surveillance can reveal. The team cautions that the framework’s transferability to other domains and systems requires independent replication, and that fairness auditing—particularly for older, non-German-speaking, or digitally excluded populations—remains a priority. But the central lesson stands: safety properties of medical AI must be specified, monitored, and governed across the entire operational life cycle, because even the most carefully designed safeguards are themselves objects that demand ongoing evaluation.

Subject of Research: Post-deployment safety surveillance of a publicly deployed generative AI chatbot providing Parkinson's disease information to patients and caregivers

Article Title: Real-world use and evaluation of a generative AI chatbot for Parkinson's disease information: a prospective observational study

Article References: Lange, F., Mardi, S., Binder, T., Reich, M. M., Odorfer, T., & Volkmann, J. (2026). Real-world use and evaluation of a generative AI chatbot for Parkinson's disease information: a prospective observational study. The Lancet Regional Health – Europe, 70, Article 101866. https://doi.org/10.1016/j.lanepe.2026.101866

Image Credits: AI Generated

DOI: 10.1016/j.lanepe.2026.101866

Keywords: Parkinson's disease, generative AI chatbot, large language models, post-market surveillance, patient safety, retrieval-augmented generation, clinical adjudication, CARE-LLM framework, digital health, medical misinformation, suicidality escalation, EU AI Act