Expert analysis of LLM hantavirus outbreak responses finds 11.1% potentially misleading

An expert-rated analysis of 900 responses generated by three publicly accessible large language models found that 63.1% were completely correct and sufficient, while 11.1% were partially correct or potentially misleading about hantavirus risk communication during the 2026 Andes virus outbreak. The comparative, cross-sectional study assessed outputs from ChatGPT, Gemini and Claude rather than patient outcomes, disease transmission or the models’ effects in a clinical setting. Across the evaluated dataset, no response was rated completely incorrect or overtly harmful, but the findings indicate that model-generated outbreak information still requires scrutiny before public use.

The researchers submitted 150 predefined hantavirus-related questions to the non-Pro, publicly accessible versions of the three models. The questions covered three areas: information based on official health-authority or guideline material, expert and clinical-practice issues, and frequently asked public questions. Each question was submitted twice, using either a standard zero-shot prompt or a prompt directed toward public-health risk communication. The two prompt formats generated 900 responses altogether, providing a dataset for comparing model outputs and assessing whether specialized risk-communication framing changed the expert-rated distribution of answers.

Three independent experts evaluated the responses with a predefined rubric addressing accuracy, completeness, consistency, safety and actionability. Ratings were summarized descriptively, while inter-rater reliability was assessed with percentage agreement and Fleiss’ kappa. Of the 900 responses, 568 were classified as completely correct and sufficient. A further 232, or 25.8%, were rated mostly correct but incomplete, and 100, or 11.1%, were considered partially correct or potentially misleading. Thus, 88.9% received one of the two higher ratings, even though nearly one in nine answers fell into the category that allowed potentially misleading information.

ChatGPT had the numerically highest proportion of completely correct responses, with 198 of 300 answers, or 66.0%, and the lowest proportion in the potentially misleading category, with 28 of 300, or 9.3%. Gemini and Claude had completely correct rates of 62.7% and 60.7%, respectively. The investigators interpreted these model-level differences descriptively and did not present them as evidence that one system was superior. The aggregate distribution of categorical ratings also did not change with the public-health risk communication prompt compared with the standard zero-shot prompt, indicating no measured change in the overall rating pattern under the two tested formats.

The study supports a qualified role for publicly accessible language models in infectious-disease and outbreak-related communication, but its results do not establish that the systems can serve as stand-alone sources of clinically complex or rapidly changing guidance. The analysis was confined to 150 predefined questions, three public model systems, two prompt formats and an expert scoring framework, and its results were reported descriptively rather than through statistical claims of model superiority. The researchers concluded that these tools may support risk communication when their outputs are grounded in official information and subjected to expert oversight. During a hantavirus outbreak, that distinction remains important because a majority of correct or mostly correct answers does not eliminate the possibility of incomplete or potentially misleading content in individual responses.

Subject of Research: Expert comparison of large language model responses to hantavirus risk-communication questions during the 2026 Andes virus outbreak

Article Title: Large language models for hantavirus-related infectious disease risk communication during the 2026 Andes virus outbreak: an expert-rated comparative content analysis

Article References: Polat, A. O., Tunç Karaman, S., & Basat, O. (2026). Large language models for hantavirus-related infectious disease risk communication during the 2026 Andes virus outbreak: an expert-rated comparative content analysis. BMC Infectious Diseases. https://doi.org/10.1186/s12879-026-14386-0

Image Credits: AI Generated

DOI: 10.1186/s12879-026-14386-0

Keywords: large language models, hantavirus, Andes virus, outbreak communication, public health, ChatGPT, Gemini, Claude, content analysis, risk communication, Large, language