The original study, led by Ludovic Semenzato and colleagues and published in JAMA Network Open, drew on the French National Healthcare Data System, known as SNDS, which records healthcare reimbursements for nearly the entire population. The researchers followed more than 33 million adults aged 18 to 59 from November 2021 onward and compared all-cause death rates between those who had received an mRNA vaccine and those who had not. Over 39 months of follow-up, the vaccinated group showed a substantially lower annual mortality rate than the unvaccinated group, and this difference has been widely cited as evidence that the French vaccination campaign saved lives.

Günther and his colleagues did not dispute the raw numbers. Instead, they asked a deceptively simple question: do the mortality rates in the study’s two groups match what is known about mortality in the French population as a whole? Using official demographic and mortality data from the French statistics institute Insee, they calculated that the nationwide all-cause mortality rate for 18 to 59 year olds in 2022 was approximately 0.001715 per person per year. The unvaccinated group in the study, with a mean annual rate of 0.001694, sat almost exactly at this national average. The vaccinated group, however, showed a rate of just 0.001330—far below what the national figures would predict.

The discrepancy, the authors argue, points to a systematic distortion introduced at the very start of the study. Approximately 4 million people aged 18 to 59 who were vaccinated before May 2021 were excluded from the analysis, along with smaller groups who were vaccinated later or who received non-mRNA vaccines. These excluded individuals, according to the age distribution reported in the original study, were on average older than those who remained. Because mortality risk rises steeply with age even within this relatively young adult range, removing the oldest and presumably most vulnerable vaccinated individuals from the analyzed cohort would mechanically lower the death rate observed among the vaccinated group.

To quantify this effect, the team performed a conservation calculation. They reasoned that the roughly 33.9 million people actually analyzed should be representative of the full French population of 35.3 million in that age bracket. If the unvaccinated and vaccinated groups in the study had mortality rates of 0.001694 and 0.001330 respectively, then the excluded group—about 5.16 million people who were vaccinated early, vaccinated during a six-month lag period, or received non-mRNA vaccines—must have had a mortality rate roughly twice the national average to make the total number of deaths add up. This is not, the authors stress, a claim that vaccination doubled mortality risk in that group; rather, it reflects that the excluded individuals were older and more vulnerable on average, which is precisely why they were vaccinated first.

The authors also tested whether their conclusion was sensitive to assumptions about people who moved between groups during follow-up. About 642,000 individuals initially classified as unvaccinated received their first vaccine dose after October 2021 and thus crossed into the vaccinated category during the study period. When the researchers adjusted for this movement by ignoring the first three months of follow-up, the unvaccinated group’s mortality rate rose to 0.001832, and the implied mortality rate of the excluded vaccinated group remained approximately 1.83 times the national average. The overall conclusion held under this alternative accounting.

When the excluded vaccinated individuals are folded back into the vaccinated group, the apparent mortality advantage largely disappears. The combined mortality rate of all vaccinated people, including those excluded from the original analysis, would be close to or above the national average rather than substantially below it. The hazard ratio that the original study reported—a measure of relative risk between vaccinated and unvaccinated groups—would shift from the favorable range of roughly 0.73 to 0.79 toward values consistent with no meaningful difference in all-cause mortality between the two groups.

This pattern is a variant of what epidemiologists call the healthy vaccinee effect, a well-documented bias in observational vaccine research. People who choose to get vaccinated, or who are prioritized for vaccination in ways that exclude the frail, tend to differ systematically from those who do not. In this case, the bias was not primarily a matter of individual health-seeking behavior but of study design: the decision to exclude everyone vaccinated before May 2021 removed a disproportionate share of older and sicker individuals from the vaccinated group being analyzed. The comparison group was therefore not comparable in the way the study’s conclusions assumed.

The authors acknowledge limitations in their approach. The original study’s published data do not include detailed information about how the age profile of the unvaccinated group shifted over time as younger people got vaccinated and left that group, which introduces some uncertainty into the corrected estimates. Nevertheless, they argue that the direction and approximate magnitude of the distortion are clear enough to undermine the central claim that vaccination was associated with a 25 percent reduction in all-cause mortality in this population.

The critique arrives at a moment when claims about vaccine benefits and risks continue to generate intense public debate. The authors emphasize that their analysis concerns a specific methodological question about one observational study, not a judgment about whether COVID-19 vaccines prevent severe disease or death from infection—outcomes that randomized trials and other evidence have addressed separately. But they contend that when observational studies are held up as evidence of population-level mortality benefits, the composition of the groups being compared must be scrutinized as carefully as the headline numbers. In this case, they conclude, the apparent benefit dissolves once the full vaccinated population is accounted for, and the study’s design, rather than the vaccine itself, produced the celebrated result.

Subject of Research: Healthy vaccinee bias in observational studies of COVID-19 vaccination and all-cause mortality in France

Article Title: Healthy vaccinee effect by study design

Article References: Günther, M., Walach, H., Rockenfeller, R., Reitzner, M., & Meyer, C. (2026). Healthy vaccinee effect by study design. Frontiers in Epidemiology, 6, Article 1837295. https://doi.org/10.3389/fepid.2026.1837295

Image Credits: AI Generated

DOI: 10.3389/fepid.2026.1837295

Keywords: healthy vaccinee effect, COVID-19 vaccination, all-cause mortality, observational study bias, France, SNDS database, mRNA vaccines, epidemiological methods, hazard ratio, study design, Frontiers in Epidemiology, Semenzato study

The core problem the framework addresses is familiar to anyone who has followed the AI governance debate: deployed language models are opaque, and the most sophisticated tools for interpreting them — mechanistic interpretability, feature attribution methods such as SHAP — are expensive, fragile, and in some cases provably limited. Impossibility theorems for feature attribution mean that no technique can reliably reveal why a model produced a given output in every case. Meanwhile, AI incidents continue to accumulate in databases cataloging real-world failures, and audits of large language models remain labor-intensive, point-in-time exercises rather than continuous monitoring. What is missing, the paper argues, is a measurement layer: a way to turn messy, free-form conversations between professionals and AI systems into structured, comparable data that institutions can actually monitor.

The proposed solution is a measurement standardization framework built around a grammar of eight interaction fields. Three of these are input–output fields — mission, conclusion, and justification — that capture what a professional asked the model, what the model recommended, and why. The paper illustrates how these fields transfer across domains: in a clinical setting, the mission might be recommending a diagnostic workup for a persistent cough in a heavy smoker, with the conclusion being urgent imaging justified by red-flag features for malignancy; in lending, the mission might be assessing a mortgage application with a low credit score, ending in a rejection justified by underwriting thresholds; in law, evaluating a settlement offer against comparable precedent awards. Because the fields carry the same structure regardless of domain, scores produced in one sector can in principle be compared with those in another.

The remaining fields feed a scoring engine. A large language model acting as a judge — the now well-established “LLM-as-a-judge” paradigm — evaluates each standardized interaction along dimensions such as risk level, policy alignment, and evidential alignment. Policy alignment measures whether the AI’s recommendation conforms to applicable guidelines and regulations; evidential alignment measures whether the justification is actually supported by the evidence cited. Crucially, the judge does not need access to the model’s internals. It works entirely from the observable interaction, which means the framework can be applied to any deployed system, including proprietary models whose weights are hidden even from the institutions using them.

Of course, using one black-box model to grade the outputs of another black-box model introduces what the paper candidly names structural circularity. LLM judges are known to suffer from systematic biases: sycophancy, the tendency to agree with whatever a user asserts; self-preference, the tendency to favor outputs resembling their own generations; and verbosity bias, the tendency to reward longer answers regardless of quality. There is also the problem of non-determinism — even supposedly deterministic settings can produce different judgments across runs, and small prompt changes can have butterfly-effect consequences for model performance. The framework confronts these weaknesses head-on rather than pretending they do not exist.

Its answer is a set of bounded conditions designed to reduce measurement inconsistency: explicit scoring rubrics, structured chain-of-thought reasoning before each verdict, reference documents that anchor judgments to authoritative guidelines, and low-temperature generation to suppress randomness. On top of these, the paper specifies a reliability verification procedure to detect and quantify the residual biases. The statistical machinery is equally explicit: paired bootstrap inference for the main comparisons, DeLong’s test for paired areas under the receiver-operating-characteristic curve as a sensitivity check, a pre-specified one-sided non-inferiority margin of 0.05, and Holm–Bonferroni correction to control for multiple testing. Agreement statistics drawn from the reliability literature, such as intraclass correlation coefficients and the Landis–Koch benchmarks for observer agreement, provide the yardsticks for judging whether the automated judge is consistent enough to be trusted.

To demonstrate feasibility, the paper applies the protocol in a minimal form to a published corpus of expert–AI interactions: the Rheum2Guide vignette study, in which ChatGPT’s treatment recommendations for rheumatic patients were compared against specialist decisions across nineteen clinical cases covering conditions from rheumatoid arthritis to giant cell arteritis. The judge scored each interaction for risk, policy alignment, and evidential alignment, using EULAR guideline documents as the reference corpus. The key finding is narrow but important: the judge reproduced its policy and evidential alignment scores across two independent runs under the specified conditions. In other words, the measurement instrument was stable. The author is careful to stress what this does not show — reliability at scale, across domains and models, remains a question for future empirical work, and the population-level claims the framework is designed to support are the subject of a staged research program, not results claimed in this paper.

If that research program succeeds, the payoff comes in three stages. The first is a reliability claim: under bounded conditions, language models can produce dependable, standardized assessments of expert–AI interactions. The second is a governance claim: alignment scores give experts an immediate signal during deployment — a warning light that flashes when a model’s advice drifts from policy or evidence — while giving institutions a basis for monitoring alignment patterns across mission types, models, and domains. The third, and most ambitious, is the outcome validation claim: once measurement is standardized, aggregate alignment scores could be correlated with downstream outcomes in regulated professional settings, exactly as cholesterol measurements were standardized and then linked to heart disease risk in the Framingham study, and as the lipid standardization programs of the CDC and National Heart, Lung and Blood Institute made population-scale cardiovascular epidemiology possible.

This is where the epidemiological analogy does its heaviest lifting. John Snow did not need to identify the cholera vibrio to map deaths around the Broad Street pump; Doll and Hill built the case against tobacco on correlated variables long before the molecular mechanisms of smoking-induced carcinogenesis were worked out. Hill’s famous 1965 criteria for distinguishing association from causation — strength, consistency, specificity, temporality, and the rest — were tools for reasoning rigorously in the absence of mechanistic certainty. The paper proposes importing precisely this style of reasoning into AI safety: instead of waiting for interpretability research to make model internals transparent, collect standardized measurements at the interface, look for statistical associations with harm, and intervene on the patterns that emerge. It is risk detection by surveillance rather than by dissection.

The framework is honest about its own limits. The demonstration involved a single judge, a single domain, and a small corpus; the circularity of black-box judging is mitigated, not eliminated; and unfaithful chain-of-thought reasoning — language models that do not always say what they think — remains a live concern even for structured scoring pipelines. But the conceptual move is significant. AI governance has long oscillated between two poles: demanding impossible transparency from opaque systems, or resigning itself to anecdotal incident reports. A standardized measurement layer offers a third path, one that medicine took a century to build and that could give institutions a shared language for describing, comparing, and ultimately predicting AI risk. Whether AI epidemiology can graduate from concept paper to working surveillance system will depend on the empirical studies the author has now carefully specified — but the blueprint for the experiment is on the table.

Subject of Research: A measurement standardization framework for detecting risks in deployed AI systems using epidemiological methods

Article Title: Toward AI epidemiology: a measurement standardization framework for prospective risk detection

Article References: Tempest-Walters, K. (2026). Toward AI epidemiology: a measurement standardization framework for prospective risk detection. AI & SOCIETY. https://doi.org/10.1007/s00146-026-03276-3

Image Credits: AI Generated

DOI: 10.1007/s00146-026-03276-3

Keywords: AI epidemiology, measurement standardization, LLM-as-a-judge, AI governance, risk detection, alignment, large language models, epidemiological methods, AI auditing, sycophancy, reliability, AI & Society