Negative Controls Reveal Hidden Biases in Real-World Vaccine Effectiveness Studies

When a vaccine is rolled out at scale, randomized controlled trials are rarely able to answer every question about how well it performs in the messy circumstances of everyday life. Health systems instead turn to observational studies, mining electronic health records to compare people who received the vaccine with people who did not. But such comparisons are notoriously vulnerable to residual bias, the lingering distortion that survives even after careful statistical adjustment. A new study published in BMC Medicine by Bingyu Zhang, Qiong Wu and colleagues, led by senior author Yong Chen of the University of Pennsylvania, takes a hard quantitative look at this problem, using data from a network of eight major United States children’s hospitals to determine when a popular bias-detection tool, the negative control outcome, can be trusted and when it silently fails.

The central premise of the study is deceptively simple. A negative control outcome is a condition that shares the patterns of healthcare use, testing and diagnosis with the outcome of interest but is not causally affected by the exposure being studied. In vaccine effectiveness research, if vaccinated and unvaccinated people truly differ only in their vaccination status, a condition that the vaccine cannot prevent should show no association with vaccination. Any apparent protective effect on such a condition signals that unmeasured confounding, selection bias or temporal artifacts are contaminating the analysis. Researchers can then use the size of this spurious effect to recalibrate, or empirically adjust, their estimates for the real outcome. The catch, as the new work demonstrates, is that the validity of the negative control is an assumption, and assumptions about control outcomes deserve the same scrutiny as assumptions about the exposure itself.

In COVID-19 research, a widely proposed negative control has been the documented SARS-CoV-2 infection occurring within the first days after vaccination. The reasoning goes that biological protection cannot yet have developed in that window, so any difference in infection rates between newly vaccinated and unvaccinated individuals must reflect bias rather than vaccine effect. That logic, however, embeds a second assumption: that the biases operating in the immediate post-vaccination window resemble those operating later in follow-up. The team behind the new study suspected that the immediate window carries its own idiosyncratic distortions, driven by behavior changes around the vaccination visit itself, and that a control validated in that window may be grossly miscalibrated for later time periods.

To test this, the investigators designed what epidemiologists call a target trial emulation, an attempt to mirror the design of a hypothetical randomized trial using observational data. They drew on PEDSnet, a pediatric clinical data network spanning institutions including Children’s Hospital of Philadelphia, Cincinnati Children’s Hospital Medical Center, Children’s Hospital Colorado, Lurie Children’s Hospital of Chicago, Nationwide Children’s Hospital, Nemours Children’s Hospital, Seattle Children’s Hospital and Stanford Children’s Health. The study window ran from January 1 to November 16, 2022, and the population comprised adolescents aged 12 to 20 years who had no prior documented SARS-CoV-2 infection and no prior COVID-19 vaccination. Vaccinated patients were matched one to one to unvaccinated patients using propensity scores, a technique that balances measured characteristics such as demographics, comorbidities and healthcare utilization between the two groups.

The key analytical move was to compare two outcomes side by side across two distinct time windows. The first outcome was documented SARS-CoV-2 infection, the outcome of genuine interest for vaccine effectiveness. The second was infection with other respiratory pathogens, principally influenza and respiratory syncytial virus, or RSV. These other respiratory infections share a great deal with COVID-19: similar clinical presentations, similar testing practices and similar patterns of seeking care when a child falls ill. Crucially, however, COVID-19 vaccination has no biological mechanism to prevent them. That makes them theoretically ideal negative controls, since any observed association with vaccination should be pure bias. The researchers estimated hazard ratios separately for days 1 through 14 after vaccination and for day 15 onward, and then confronted these estimates with an empirical null distribution constructed from a set of prespecified health events that vaccination cannot plausibly influence.

The results were striking and, in places, cautionary. In the first 14 days after vaccination, the hazard ratio for documented SARS-CoV-2 infection was 0.33, with a 95 percent confidence interval of 0.24 to 0.47, suggesting that vaccinated adolescents appeared 67 percent less likely to be diagnosed with COVID-19 than their matched unvaccinated peers. That might look like remarkably rapid vaccine protection. But the negative control told a different story. Over the same window, the hazard ratio for other respiratory infections was 0.30, with a confidence interval of 0.14 to 0.63. Since the vaccine cannot protect against influenza or RSV, an apparent 70 percent reduction in those infections could only be bias. Both estimates deviated significantly from the empirical null distribution, with p values below 0.001, indicating that the immediate post-vaccination period is saturated with spurious protective associations.

The picture changed substantially after day 15. The hazard ratio for documented SARS-CoV-2 infection settled at 0.32, with a confidence interval of 0.27 to 0.36, a figure broadly consistent with the genuine protection expected from vaccination during that period of the pandemic. Meanwhile the hazard ratio for other respiratory infections rose to 0.68, with a confidence interval of 0.60 to 0.77. Although still numerically below one, this estimate no longer differed significantly from the empirical null distribution, with a p value of 0.142. In other words, once the immediate post-vaccination window was excluded, the biases that had inflated apparent protection largely dissipated, and the negative control behaved the way theory says a valid control should. The same control outcome that flagged massive distortion in days 1 through 14 was reasonably well calibrated for later follow-up.

The mechanistic interpretation offered by the authors centers on behavioral and temporal artifacts surrounding the vaccination encounter itself. A vaccination visit is not a neutral event. People who feel ill tend to postpone vaccination, creating a healthy-vaccinee effect in the days immediately after the shot. Vaccination may prompt contact with the health system in ways that alter testing for coincidental infections, or it may coincide with periods when families are unusually cautious about exposing children to respiratory pathogens. Conversely, the act of attending a clinic can itself lead to exposure or to diagnostic workups that would not otherwise occur. These time-dependent distortions are strongest immediately after vaccination and fade with longer follow-up, which is exactly the pattern the empirical calibration revealed. A negative control borrowed from the early window therefore imports the very biases it is supposed to measure, miscalibrating the analysis for later periods.

The implications extend well beyond COVID-19. Vaccine effectiveness studies for influenza, RSV and future pandemic vaccines routinely rely on negative controls to detect and correct residual bias, and many adopt short post-vaccination windows as convenient controls without systematically validating them. The study shows that negative control validity is not a fixed property of an outcome but depends jointly on the choice of outcome and the time at risk. A control that is valid for a 60-day hazard ratio may be badly invalid for a 14-day hazard ratio, and vice versa. The authors argue that researchers should evaluate control outcomes within each time-at-risk window used in the analysis, comparing estimates against an empirical null built from multiple unrelated health events rather than relying on a single assumed-null outcome. Where a control deviates from the null, empirical calibration can rescale the confidence interval of the main estimate, yielding inference that is honest about the uncertainty introduced by unmeasured confounding.

The work also illustrates a broader shift in how large-scale observational evidence is being made trustworthy. Rather than treating bias detection as an afterthought, the framework embeds diagnostics into the analysis pipeline itself, leveraging the scale of networks like PEDSnet to construct empirical null distributions from real data. With funding from the National Institutes of Health and the Patient-Centered Outcomes Research Institute, and institutional review board approval from the University of Pennsylvania, the study exemplifies a rigorous, open-access approach to a problem that shapes public confidence in vaccines. As real-world evidence increasingly informs regulatory and clinical decisions, the message is clear: the tools that guard against misleading results must themselves be tested, because an unvalidated negative control can be just as misleading as no control at all.

Subject of Research: Validation of negative control outcomes for detecting and calibrating residual bias in real-world COVID-19 vaccine effectiveness studies among adolescents

Article Title: Assessing residual bias in real-world vaccine effectiveness studies: validating negative controls to avoid misleading results

Article References: Zhang, B., Wu, Q., Zhou, T., Zhang, D., Tong, J., Lei, Y., Chu, H., Wang, L., Ostropolets, A., Qiu, Y., Lu, Y., Ryan, P. B., Morris, J. S., Schuemie, M. J., Forrest, C. B., Hripcsak, G., & Chen, Y. (2026). Assessing residual bias in real-world vaccine effectiveness studies: validating negative controls to avoid misleading results. BMC Medicine. https://doi.org/10.1186/s12916-026-05260-6

Image Credits: AI Generated

DOI: 10.1186/s12916-026-05260-6

Keywords: vaccine effectiveness, negative controls, residual bias, target trial emulation, electronic health records, PEDSnet, empirical calibration, SARS-CoV-2, COVID-19 vaccination, observational studies, confounding, adolescents