The authors trace the intellectual lineage of experimentation back to Ronald Fisher’s The Design of Experiments and Jerzy Neyman’s work on agricultural trials, noting that the same statistical machinery now powers decisions in economics, political science, public health, education and digital product development. Randomized evaluations involving millions of observations have transformed social science, generating causal evidence at a pace and scale once unimaginable. Yet as companies and institutions run thousands of experiments each year, often across decentralized teams with competing incentives, the assumptions that made textbook methods reliable no longer hold automatically. The Perspective argues that the research community must re-engage with these practical frictions, drawing on sustained collaboration between academics and the practitioners who operate experimentation platforms at scale.

The first challenge concerns organizational incentives and experimental governance. In large companies, experiments are not conducted by neutral statisticians but by teams whose careers depend on the results. Researchers have begun modeling this as a principal-agent problem, where the people commissioning a test may have strategic reasons to select metrics, stopping rules or reporting practices that favor a preferred outcome. Recent work on principal-agent hypothesis testing and screening for experiments formalizes how such misaligned incentives can distort what gets run and what gets reported. The authors also draw on Friedrich Hayek’s insight about the use of knowledge in society, noting that decentralized organizations generate enormous experimental activity, but coordinating that activity requires governance structures that few firms have built carefully. Democratizing experimentation, they suggest, can accelerate learning, but only if paired with guardrails that prevent local optimization from damaging global objectives.

Privacy, fairness and ethics form the second area of concern. Online experiments involve human subjects who rarely know they are being studied, a situation that sits uneasily with the ethical principles articulated in the Belmont Report. The authors review recent advances in differential privacy, including federated experiment designs that allow companies to run randomized controlled trials while limiting how much any individual’s data can leak into published results. Synthetic data generation offers another path, with new methods producing privacy-conscious datasets suitable for causal effect estimation. Fairness raises its own difficulties: treatments that improve average outcomes may harm specific subgroups, and protected attributes are often unobserved, complicating any assessment of disparate impact. New bounding techniques can estimate the fraction of a population negatively affected by a treatment even when individual-level harm cannot be directly measured, giving practitioners a way to quantify risk rather than ignore it.

The third challenge is estimating long-term impact. Most experiments run for days or weeks, but the decisions they inform concern effects that unfold over months or years. The literature on surrogate end points, from clinical trials in medicine to proxy metrics in technology companies, shows how easily short-term indicators can mislead. A famous cautionary example comes from cardiac medicine, where drugs that suppressed arrhythmias ultimately increased mortality, a lesson the authors invoke to highlight the danger of optimizing the wrong outcome. Newer approaches include the surrogate index, which combines short-term proxies to estimate long-term treatment effects, and methods that pool information across many weak experiments to learn the covariance of treatment effects. Evaluations at Netflix, drawing on hundreds of A/B tests, suggest these tools can improve decision-making, but imperfect surrogates remain an unsolved problem, and sensitivity analysis techniques for unmeasured confounding are increasingly seen as essential.

Fourth, the Perspective examines time-adaptive experimental studies, in which the design of the experiment itself changes as data accumulates. Classical sequential testing, pioneered by Abraham Wald, addressed the problem of peeking at results, but modern platforms monitor experiments continuously by default. Always-valid inference methods and confidence sequences now allow researchers to check results at any time without inflating false positive rates, and these techniques are being deployed in enterprise A/B testing platforms. Beyond monitoring, adaptive assignment schemes such as multi-armed bandits reallocate traffic toward better-performing treatments, trading statistical rigor for user benefit. Phased release strategies using batched bandits balance risk and reward during rollouts, and switchback experiment designs allow causal inference in settings like marketplaces where treatments must alternate over time. Inference after adaptive experiments remains technically demanding, but recent work on demystifying such inference and on conformal methods for distribution shifts is narrowing the gap.

Heterogeneous treatment effects constitute the fifth area. An average treatment effect can conceal enormous variation: a feature that helps most users may actively harm a vulnerable minority, and a policy that works in one region may fail in another. Machine learning methods, including metalearners, causal random forests and recursive partitioning, now allow researchers to estimate how effects differ across individuals, while calibration techniques help discover stable, interpretable subgroups. The challenge is statistical as much as computational: hunting for subgroups multiplies hypothesis tests and invites false discoveries, requiring multiple testing corrections and careful validation. New approaches also bridge prediction and causal targeting, distinguishing the question of who will benefit from the question of who is likely to respond, a distinction that matters enormously when experiments inform personalized product decisions. Interpretable personalized experimentation is emerging as a practical goal, letting non-experts understand which groups an experiment affects and why.

The final and perhaps most timely challenge comes from generative artificial intelligence. Large language models are being proposed both as tools within experiments and as simulated participants, raising fundamental questions about what a digital-era experiment should be. Research on simulated economic agents, sometimes described as Homo Silicus, and on using language models to replicate human subject studies suggests these systems can mimic certain human response patterns, but their biases, training data provenance and tendencies toward sycophancy remain poorly understood. Warnings about the illusions of understanding that AI can create in scientific research loom large. The authors also note that AI agents themselves are becoming subjects of experimentation, with large-scale autonomous negotiation competitions already underway. Whether generative AI will serve as a substitute for human experiments, an augmentation of them, or a new experimental subject entirely is one of the field’s most consequential open questions.

Underlying all six areas is a shared theme: the infrastructure of experimentation has outpaced its statistical and ethical scaffolding. P-hacking, publication bias and the winner’s curse in estimating effects across many experiments are old problems given new urgency by sheer volume. Post-selection inference, always-valid confidence sequences and empirical Bayes methods offer partial remedies, but the Perspective is explicit that many open problems remain unsolved and that progress will require methodological innovation grounded in real operational constraints. The authors, who include researchers from Columbia Business School, Harvard Business School, Stanford University, Cornell University and MIT alongside their industry collaborators, describe their goal as surfacing practical challenges that merit greater attention from the research community.

The stakes extend well beyond technology companies. As governments experiment with digital public services, health systems test behavioral interventions and educators evaluate online learning at scale, the methods developed for platform experimentation are migrating into domains with far less tolerance for error. The authors hope the Perspective will act as a research agenda, encouraging statisticians, economists, computer scientists and social scientists to work directly with the practitioners running experiments on millions of people. In the digital era, they argue, the future of large-scale experimentation depends less on any single technical breakthrough than on building institutions, methods and norms capable of keeping rigorous causal inference aligned with human welfare at unprecedented scale.

Subject of Research: Methodological and organizational challenges facing large-scale randomized experiments in the digital era

Article Title: The future of large-scale experiments and their challenges in the digital era

Article References: Holtz, D., Bojinov, I., Johari, R., Kallus, N., Lal, A., Anand, S., Carlson, K., Cohn, B., Cunningham, T., Deng, A., Dimakopoulou, M., Gandhi, A., Kostyuk, V., Kumar, M., Loh, S.-M., Machmouchi, W., Mao, J., McQueen, J., Meakin, J., … Tingley, M. (2026). The future of large-scale experiments and their challenges in the digital era. Nature Human Behaviour. https://doi.org/10.1038/s41562-026-02582-6

Image Credits: AI Generated

DOI: 10.1038/s41562-026-02582-6

Keywords: randomized experiments, A/B testing, causal inference, experimental design, differential privacy, fairness, surrogate metrics, multi-armed bandits, anytime-valid inference, heterogeneous treatment effects, generative AI, large language models