In a striking demonstration of artificial intelligence’s encroachment into the scientific method itself, a massive new study has shown that large language models can forecast the outcomes of social science experiments with an accuracy rivaling that of pooled human experts. The research, published in Nature, marshalled an archive of 70 preregistered, nationally representative survey experiments encompassing 469 distinct treatment effects and responses from nearly 120,000 American participants. By prompting GPT-4 to simulate how representative samples of individuals would react to experimental stimuli, the team was able to infer treatment effects that were strongly correlated with the real-world results, even when the experiments had not been published before the model’s training data cutoff.
The core methodology involved feeding the language model the exact text of survey vignettes, question wording, and condition assignments, then asking it to generate responses as if it were a randomly selected member of the target population. Critically, the model was not fine-tuned on the experimental outcomes; it relied solely on the parametric knowledge embedded during pretraining. The researchers then applied standard causal inference techniques—comparing the simulated responses across treatment and control arms—to compute predicted effect sizes. This approach transforms the LLM from a mere text generator into a computational laboratory for virtual experimentation.
The headline finding is that predictions derived from GPT-4 achieved a correlation with actual treatment effects that was nearly indistinguishable from the accuracy of pooled human forecasts collected from social scientists. The correlation remained robust for studies that were not publicly available by the model’s training-data cutoff, and it held for prominent open-weight models as well, ruling out simple memorization of published results. This suggests that the models are capturing something fundamental about the structure of human attitudes, beliefs, and behavioral responses, rather than regurgitating memorized outcomes.
Yet the predictions were not flawless. The analysis revealed a systematic tendency for the LLM to overestimate effect sizes, meaning that the simulated participants often showed larger differences between conditions than real humans did. This inflation of treatment effects is a crucial caveat, as it could lead researchers to overstate the practical significance of an intervention if they relied blindly on AI-generated forecasts. The source of this bias remains an open question; it may stem from the model’s training on texts that emphasize dramatic, clear-cut causal narratives, or from inherent limitations in how the model aggregates probabilistic knowledge across diverse contexts.
To further stress-test the approach, the team compiled a secondary archive of 15 megastudies containing 606 experimental effects. These megastudies are large-scale projects that simultaneously test many different interventions on the same outcome, providing a demanding benchmark. Here, the correlations between LLM predictions and actual effects were lower than in the primary archive, but they remained comparable to those achieved by pooled expert forecasters. This pattern suggests that the model’s predictive power is sensitive to the complexity and heterogeneity of the experimental landscape, but it still offers a meaningful signal above chance.
The implications for scientific practice are profound. The researchers surveyed 460 social scientists to gauge perceived uses and risks, then used the archives to simulate several real-world applications. They found that LLM predictions could serve as a rapid pilot-testing tool, allowing researchers to prune unpromising experimental conditions before committing to costly data collection. The models also showed promise in aiding intervention selection, helping practitioners identify which of several possible treatments might yield the largest effect, and in flagging published effects that might be too good to be true and thus warrant replication. However, the surveyed scientists also voiced concerns about potential misuse, such as replacing human participants entirely with simulated data, which could erode the external validity of findings and amplify biases already present in the training corpora.
The risks of bias and misuse are not hypothetical. If an LLM’s internal representations systematically underrepresent minority perspectives or overemphasize culturally dominant narratives, predictions could steer research away from understudied populations or produce misleadingly homogeneous results. The authors caution that the technology should be viewed as an augmentation of traditional methods, not a replacement, and that any deployment must be accompanied by careful validation against human data. The study’s architecture—preregistered analyses, transparent reporting of predictions, and direct comparison with expert forecasts—provides a template for how the field might responsibly integrate these tools into the scientific workflow.
Ultimately, the research demonstrates that the boundary between artificial intelligence and human judgment in the social sciences is becoming increasingly porous. LLMs can now serve as silicon-based surrogates that, while imperfect, capture enough of the regularities in human social behavior to meaningfully guide experimental design. As the models continue to improve, their ability to anticipate how people will react to carefully crafted interventions could accelerate the pace of discovery and help policymakers design more effective programs. The challenge ahead lies in harnessing this predictive power without losing the indispensable grounding in real human experience that makes social science a mirror of society itself.
Subject of Research: Predicting the outcomes of social science experiments using large language models.
Article Title: Large language models can predict the results of social science experiments.
Article References:
Ashokkumar, A., Hewitt, L., Ghezae, I. et al. Large language models can predict the results of social science experiments.
Nature (2026). https://doi.org/10.1038/s41586-026-10742-x
Image Credits: AI Generated
DOI: https://doi.org/10.1038/s41586-026-10742-x
Keywords: large language models, social science experiments, prediction, treatment effects, GPT-4, survey experiments, human forecasts, effect size overestimation, megastudies, bias, replication, computational social science.
