The authors trace the intellectual lineage of experimentation back to Ronald Fisher’s The Design of Experiments and Jerzy Neyman’s work on agricultural trials, noting that the same statistical machinery now powers decisions in economics, political science, public health, education and digital product development. Randomized evaluations involving millions of observations have transformed social science, generating causal evidence at a pace and scale once unimaginable. Yet as companies and institutions run thousands of experiments each year, often across decentralized teams with competing incentives, the assumptions that made textbook methods reliable no longer hold automatically. The Perspective argues that the research community must re-engage with these practical frictions, drawing on sustained collaboration between academics and the practitioners who operate experimentation platforms at scale.

The first challenge concerns organizational incentives and experimental governance. In large companies, experiments are not conducted by neutral statisticians but by teams whose careers depend on the results. Researchers have begun modeling this as a principal-agent problem, where the people commissioning a test may have strategic reasons to select metrics, stopping rules or reporting practices that favor a preferred outcome. Recent work on principal-agent hypothesis testing and screening for experiments formalizes how such misaligned incentives can distort what gets run and what gets reported. The authors also draw on Friedrich Hayek’s insight about the use of knowledge in society, noting that decentralized organizations generate enormous experimental activity, but coordinating that activity requires governance structures that few firms have built carefully. Democratizing experimentation, they suggest, can accelerate learning, but only if paired with guardrails that prevent local optimization from damaging global objectives.

Privacy, fairness and ethics form the second area of concern. Online experiments involve human subjects who rarely know they are being studied, a situation that sits uneasily with the ethical principles articulated in the Belmont Report. The authors review recent advances in differential privacy, including federated experiment designs that allow companies to run randomized controlled trials while limiting how much any individual’s data can leak into published results. Synthetic data generation offers another path, with new methods producing privacy-conscious datasets suitable for causal effect estimation. Fairness raises its own difficulties: treatments that improve average outcomes may harm specific subgroups, and protected attributes are often unobserved, complicating any assessment of disparate impact. New bounding techniques can estimate the fraction of a population negatively affected by a treatment even when individual-level harm cannot be directly measured, giving practitioners a way to quantify risk rather than ignore it.

The third challenge is estimating long-term impact. Most experiments run for days or weeks, but the decisions they inform concern effects that unfold over months or years. The literature on surrogate end points, from clinical trials in medicine to proxy metrics in technology companies, shows how easily short-term indicators can mislead. A famous cautionary example comes from cardiac medicine, where drugs that suppressed arrhythmias ultimately increased mortality, a lesson the authors invoke to highlight the danger of optimizing the wrong outcome. Newer approaches include the surrogate index, which combines short-term proxies to estimate long-term treatment effects, and methods that pool information across many weak experiments to learn the covariance of treatment effects. Evaluations at Netflix, drawing on hundreds of A/B tests, suggest these tools can improve decision-making, but imperfect surrogates remain an unsolved problem, and sensitivity analysis techniques for unmeasured confounding are increasingly seen as essential.

Fourth, the Perspective examines time-adaptive experimental studies, in which the design of the experiment itself changes as data accumulates. Classical sequential testing, pioneered by Abraham Wald, addressed the problem of peeking at results, but modern platforms monitor experiments continuously by default. Always-valid inference methods and confidence sequences now allow researchers to check results at any time without inflating false positive rates, and these techniques are being deployed in enterprise A/B testing platforms. Beyond monitoring, adaptive assignment schemes such as multi-armed bandits reallocate traffic toward better-performing treatments, trading statistical rigor for user benefit. Phased release strategies using batched bandits balance risk and reward during rollouts, and switchback experiment designs allow causal inference in settings like marketplaces where treatments must alternate over time. Inference after adaptive experiments remains technically demanding, but recent work on demystifying such inference and on conformal methods for distribution shifts is narrowing the gap.

Heterogeneous treatment effects constitute the fifth area. An average treatment effect can conceal enormous variation: a feature that helps most users may actively harm a vulnerable minority, and a policy that works in one region may fail in another. Machine learning methods, including metalearners, causal random forests and recursive partitioning, now allow researchers to estimate how effects differ across individuals, while calibration techniques help discover stable, interpretable subgroups. The challenge is statistical as much as computational: hunting for subgroups multiplies hypothesis tests and invites false discoveries, requiring multiple testing corrections and careful validation. New approaches also bridge prediction and causal targeting, distinguishing the question of who will benefit from the question of who is likely to respond, a distinction that matters enormously when experiments inform personalized product decisions. Interpretable personalized experimentation is emerging as a practical goal, letting non-experts understand which groups an experiment affects and why.

The final and perhaps most timely challenge comes from generative artificial intelligence. Large language models are being proposed both as tools within experiments and as simulated participants, raising fundamental questions about what a digital-era experiment should be. Research on simulated economic agents, sometimes described as Homo Silicus, and on using language models to replicate human subject studies suggests these systems can mimic certain human response patterns, but their biases, training data provenance and tendencies toward sycophancy remain poorly understood. Warnings about the illusions of understanding that AI can create in scientific research loom large. The authors also note that AI agents themselves are becoming subjects of experimentation, with large-scale autonomous negotiation competitions already underway. Whether generative AI will serve as a substitute for human experiments, an augmentation of them, or a new experimental subject entirely is one of the field’s most consequential open questions.

Underlying all six areas is a shared theme: the infrastructure of experimentation has outpaced its statistical and ethical scaffolding. P-hacking, publication bias and the winner’s curse in estimating effects across many experiments are old problems given new urgency by sheer volume. Post-selection inference, always-valid confidence sequences and empirical Bayes methods offer partial remedies, but the Perspective is explicit that many open problems remain unsolved and that progress will require methodological innovation grounded in real operational constraints. The authors, who include researchers from Columbia Business School, Harvard Business School, Stanford University, Cornell University and MIT alongside their industry collaborators, describe their goal as surfacing practical challenges that merit greater attention from the research community.

The stakes extend well beyond technology companies. As governments experiment with digital public services, health systems test behavioral interventions and educators evaluate online learning at scale, the methods developed for platform experimentation are migrating into domains with far less tolerance for error. The authors hope the Perspective will act as a research agenda, encouraging statisticians, economists, computer scientists and social scientists to work directly with the practitioners running experiments on millions of people. In the digital era, they argue, the future of large-scale experimentation depends less on any single technical breakthrough than on building institutions, methods and norms capable of keeping rigorous causal inference aligned with human welfare at unprecedented scale.

Subject of Research: Methodological and organizational challenges facing large-scale randomized experiments in the digital era

Article Title: The future of large-scale experiments and their challenges in the digital era

Article References: Holtz, D., Bojinov, I., Johari, R., Kallus, N., Lal, A., Anand, S., Carlson, K., Cohn, B., Cunningham, T., Deng, A., Dimakopoulou, M., Gandhi, A., Kostyuk, V., Kumar, M., Loh, S.-M., Machmouchi, W., Mao, J., McQueen, J., Meakin, J., … Tingley, M. (2026). The future of large-scale experiments and their challenges in the digital era. Nature Human Behaviour. https://doi.org/10.1038/s41562-026-02582-6

Image Credits: AI Generated

DOI: 10.1038/s41562-026-02582-6

Keywords: randomized experiments, A/B testing, causal inference, experimental design, differential privacy, fairness, surrogate metrics, multi-armed bandits, anytime-valid inference, heterogeneous treatment effects, generative AI, large language models

The findings, published in the peer-reviewed journal Explore by Marina Weiler, PhD, Damon Abraham, David J.P. Acunzo and Niffe Hermansson, do not constitute evidence for out-of-body experiences, the investigators are careful to stress. The participants’ remarks about the researchers’ positions and activities were not part of the trial’s predefined outcome measures, and in the strict architecture of experimental science, unplanned observations cannot be treated as confirmations of a hypothesis. What they can do, however, is reshape the questions researchers ask next. Weiler and her collaborators argue that the unexpected observations point toward a potentially important but overlooked variable in this line of research: the nature of the targets themselves, and whether the objects chosen for such trials are the right things to test at all.

The structure of the trial was straightforward in principle. Twenty-one participants took part, each asked to identify a random object displayed on a laptop screen located in another room, physically separated from the participant and shielded from ordinary perception. Ten of the participants reported having an out-of-body experience during the session — the subjective sensation of leaving one’s physical body and perceiving the world from an external vantage point. Thirteen participants provided descriptions of the target, a group that included not only those who reported leaving their bodies but also individuals who said they could see images on an internal mental screen or something similar, without the full sensation of bodily separation. When the researchers scored the descriptions against the actual targets, the results fell squarely within the range expected from random guessing.

For decades, reports of out-of-body experiences have posed a stubborn challenge to conventional accounts of perception and consciousness. Such experiences, in which people describe floating above their bodies, traveling to distant locations, or observing scenes they should have no sensory access to, are well documented in the clinical and research literature, most often in association with neurological conditions, near-death episodes, and contemplative practices. Mainstream neuroscience generally interprets them as disturbances of bodily self-representation, generated by the brain when the integration of sensory and vestibular signals falters. But a subset of researchers, including those at the University of Virginia’s Division of Perceptual Studies, have long pursued a more radical possibility: that in at least some cases, something about the person’s awareness might genuinely extend beyond the physical body, acquiring information that could not have arrived through the known senses.

Testing that possibility rigorously is extraordinarily difficult, and the Brazilian trial illustrates why. The primary measure — accuracy in describing a hidden target object — rests on the assumption that if remote perception occurs, it should function something like ordinary vision, delivering a recognizable image of a designated object. The results of this trial, like many before it, offered no support for that assumption. But the post-session interviews suggested a different picture. One participant described how a researcher in another room, behind closed doors, had been seated on the left looking at a laptop screen, while another researcher sat farther back reading a book. Another participant described a researcher seated on the right taking notes, with a second researcher sitting on the left at the computer. Both descriptions matched the trial records precisely, and neither scene was visible or inferable from the participants’ location.

Weiler emphasizes that these observations must be treated with caution. Because they were collected informally, after the fact, and outside the trial’s registered outcome measures, they are vulnerable to every pitfall that plagues anecdotal claims: imperfect memory, unconscious inference, and the possibility of unrecorded cues. What elevates them above mere anecdote is their specificity and the documentary confirmation preserved in the trial records. Still, the researchers themselves insist that the finding is a hypothesis generator, not a demonstration. “I hope this study encourages researchers to think creatively about how we test these experiences,” Weiler said. “The unexpected findings give us a potentially useful new direction, but they also need to be tested prospectively and under rigorous conditions before we can know what they mean.”

The direction the team proposes is a reframing of what such trials should measure. It is possible, they suggest, that the targets conventionally used in out-of-body experiments — random objects on screens — are poorly suited to the phenomenon, if the phenomenon exists at all. Memorable, distinctive, or emotionally engaging targets may be easier for participants to visualize or register than arbitrary images, much as people remember vivid scenes better than neutral ones in ordinary memory experiments. More provocatively, the two accurate descriptions of the researchers suggest that human activity and social scenes may register more readily than inanimate objects. The team proposes that future trials supplement traditional targets with other unseen details that participants might notice: for example, having a researcher wear a distinctive or colorful item of clothing while behind closed doors. If participants could later recall such items without ever seeing or being told about them, under controlled and pre-registered conditions, that would substantially bolster the case for genuine remote perception.

The proposal carries real methodological appeal even for skeptics. Pre-specifying such incidental details as outcome measures would convert a class of observations that science currently discards — the unprompted, unexplained remarks made after formal testing ends — into quantifiable data collected under blind conditions. Independent judges could score participants’ statements against documented room configurations without knowing which statements came from which sessions, and chance baselines could be established in advance. In this way, a trial that fails on its primary endpoint could still advance the field by testing a refined hypothesis with sharper instruments. The researchers have no financial interest in the work, which was supported by a gift from an undisclosed donor, and the study’s publication in a peer-reviewed journal subjects its reasoning to external scrutiny.

For Weiler, the implications reach beyond experimental design into foundational questions about consciousness and reality. “I hope these findings encourage us to think more deeply about what they might mean for our understanding of consciousness and, ultimately, the nature of reality,” she said. “If studies demonstrate that people can obtain accurate information that they could not have accessed through their ordinary senses, we would have to reconsider some of our assumptions about the relationship between consciousness, perception and the physical world. At its deepest level, this research is not only asking whether out-of-body experiences are real. It is asking what we mean by ‘real’ in the first place, and whether our current understanding of reality is broad enough to account for everything human consciousness can experience.” Whether the unexpected observations from Brazil survive prospective testing remains to be seen, but the trial has already accomplished something rare: it turned a null result into a specific, testable idea about where the evidence, if it exists, might be hiding.

The University of Virginia’s Division of Perceptual Studies, established in 1967 under psychiatrist Ian Stevenson, remains the most productive university-based research group in the world dedicated to phenomena that challenge conventional scientific paradigms of consciousness, and this trial reflects its long-standing strategy of pairing skepticism with persistence. The division’s researchers operate a state-of-the-art neuroimaging laboratory alongside their parapsychological studies, and they frame their mission as the rigorous empirical evaluation of exceptional human experiences rather than advocacy for any particular interpretation. The Brazilian trial embodies that dual commitment: a negative primary result reported honestly, an anomaly documented rather than suppressed, and a concrete proposal for how the next generation of experiments might be built. As the team itself concludes, unexpected findings can sometimes help scientists ask better questions — and in a field as contested as consciousness research, better questions may be the most valuable outcome a single study can deliver.

Subject of Research: A controlled trial testing whether people reporting out-of-body experiences can perceive hidden targets, which produced unexpected unprompted observations of researchers' concealed activities.

Article Title: Trial testing out-of-body experiences produces unexpected twist

Article References: Trial testing out-of-body experiences produces unexpected twist. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: out-of-body experiences, consciousness, parapsychology, remote perception, University of Virginia, Division of Perceptual Studies, Marina Weiler, Explore journal, experimental design, anomalous cognition, perception, trial results

The core idea is a conceptual shift in how scientists might use these models. In the conventional picture, a large language model is a text engine: it predicts the next word, or a sequence of words, based on patterns learned from enormous corpora that include textbooks, patents, journal articles and technical documentation. But from an optimization standpoint, that same next-token distribution can be read as something more interesting, a probability distribution over scientific hypotheses, reaction conditions, parameter settings or candidate materials. When a researcher asks a model to suggest a catalyst composition or a synthesis route, the model is not retrieving a stored answer; it is sampling from a learned distribution over plausible answers. That distribution, the authors argue, is precisely the raw material an optimizer needs.

Optimization is the mathematical heartbeat of experimental discovery. Whether the goal is to maximize the efficiency of a solar cell, minimize the toxicity of a drug candidate, or find the processing conditions that yield a stronger alloy, the experimenter is navigating a search space that is too large to enumerate. Classical approaches, from design of experiments to Bayesian optimization, have long formalized this problem: build a surrogate model of the system, quantify which regions of the search space remain uncertain, and choose the next experiment to balance exploiting what is already known against exploring what is not. The celebrated exploration–exploitation trade-off sits at the center of this framework, and it is impossible to navigate without a trustworthy measure of uncertainty. A surrogate model that claims to know everything will keep re-testing the same region; one that trusts nothing will wander aimlessly.

This is where the new contribution sharpens. Large language models, used naively, are poorly calibrated optimizers. They are trained to produce fluent, plausible text, not to report honest error bars. Empirical studies across domains have repeatedly shown that such models can express high confidence in statements that are false, and hedged language in cases where they are in fact reliable, a mismatch between linguistic presentation and statistical reality. For a chatbot, an overconfident hallucination is an embarrassment. For an autonomous laboratory that delegates its next synthesis to the model, an overconfident hallucination costs reagents, instrument time and, in worst cases, safety. The article’s prescription is therefore to treat calibration not as an optional refinement but as the defining requirement that turns a language model from a suggestion generator into a scientific optimizer.

The technical toolkit for achieving this is drawn largely from the broader machine-learning literature on predictive uncertainty. Ensemble methods, in which several model variants are queried and their disagreement used as an uncertainty signal, offer one route. Prompt-level perturbations, where the same question is paraphrased or re-framed many times and the spread of answers measured, provide a lightweight proxy for epistemic uncertainty. Token-level probabilities, which are a native output of autoregressive models, can be aggregated into confidence scores over structured answers such as numerical property predictions or ranked candidate lists. More formally, conformal prediction techniques can wrap the model’s outputs in distribution-free guarantees, ensuring that, under stated assumptions, a claimed coverage level actually holds. Each of these approaches has limitations when applied to text-generating systems, and the article’s argument is that the field must now systematically evaluate which of them survives contact with real experimental workflows.

The significance of getting this right extends well beyond a single laboratory. Over the past several years, self-driving laboratories, facilities in which robotic platforms execute, measure and iterate on experiments with minimal human intervention, have moved from demonstrations to early deployment in fields such as photocatalysis, thin-film deposition and protein engineering. In these systems, the decision about which experiment to run next is made by software, and the quality of that decision compounds over hundreds of iterations. A poorly calibrated optimizer quietly degrades the entire campaign, steering the robot toward regions of parameter space that look promising in text but are barren in reality. A well-calibrated one behaves like a seasoned experimentalist: it proposes bold moves when the evidence supports them and retreats to careful, incremental measurements when its knowledge runs out.

Language models bring something to this setting that traditional optimizers lack. Bayesian optimization typically begins with almost no prior knowledge, requiring dozens of experiments before its surrogate model becomes informative. A large language model, by contrast, starts with a compressed representation of the accumulated scientific literature. It knows, in some statistical sense, that perovskite compositions follow certain structural rules, that particular functional groups tend to destabilize a molecule, or that certain reaction classes fail under specific solvents. Used as a warm start, this knowledge can dramatically shorten the cold-start phase of an optimization campaign, focusing early experiments on physically plausible regions of the search space. The danger, equally clear, is that the model’s literature-derived priors may be stale, biased toward published positive results, or simply wrong for the specific system at hand, which is exactly why the uncertainty layer is indispensable: it allows the optimizer to lean on prior knowledge where that knowledge is reliable and to abandon it where measurement contradicts it.

The article also speaks to a growing tension in how the scientific community evaluates AI for research use. Benchmark results, in which a model answers exam-style questions or reproduces known findings, are easy to report and have fueled enormous enthusiasm. But experimental discovery is an interactive, sequential process, and a model’s offline accuracy on static benchmarks is a weak predictor of its value inside a closed loop with a robot and a spectrometer. What matters in the loop is the joint behavior of proposal quality and calibration: does the system know the difference between a prediction it can defend and one it is extrapolating? The authors’ framing suggests that the next generation of evaluations for scientific AI should measure calibration curves, regret over the course of an optimization campaign and the ability of the system to detect its own failures, metrics that the machine-learning community has refined over decades but that have only recently been applied to generative models in laboratory settings.

For working scientists, the practical message is a set of concrete questions to ask before handing experimental authority to a language model. How is the model’s confidence estimated, and has that confidence been validated against held-out experiments? Does the system distinguish between knowledge that comes from the literature and knowledge inferred for the specific experimental platform at hand? What happens when the model’s uncertainty is high, does the workflow defer to a human, fall back on a classical design of experiments, or simply run a cheap screening test? And crucially, is the full loop, proposal, calibration, decision, execution, measurement, updating, instrumented so that its performance can be audited after the fact? The Nature Machine Intelligence perspective does not claim that language models will replace Bayesian optimization or the judgment of experimentalists. Its more durable contribution may be to define the standard that any AI-driven discovery system must meet: not fluency, and not even accuracy in the aggregate, but honest, quantified awareness of the boundary between what the model knows and what only the next experiment can reveal. If the wave of AI-augmented laboratories now being built around the world adopts that standard early, the combination of literature-scale priors and rigorous uncertainty quantification could compress discovery timelines in ways that neither classical optimization nor generative AI can achieve alone.

Subject of Research: Use of uncertainty-calibrated large language models as optimizers for autonomous experimental scientific discovery

Article Title: Large language models as uncertainty-calibrated optimizers for experimental discovery

Article References: Large language models as uncertainty-calibrated optimizers for experimental discovery. (n.d.). https://doi.org/10.1038/s42256-026-01283-z

Image Credits: AI Generated

DOI: 10.1038/s42256-026-01283-z

Keywords: large language models, uncertainty quantification, Bayesian optimization, self-driving laboratories, experimental design, calibration, autonomous experimentation, scientific discovery, conformal prediction, machine learning, generative AI, Nature Machine Intelligence