Tuning Trumps Technique: Hyperparameters Decide Whether AI Gets Causality Right

Machine learning has quietly become the workhorse of modern causal inference. When researchers want to know whether a drug, a job-training program, or a policy intervention actually caused an outcome—as opposed to merely correlating with it—they increasingly hand the problem to algorithms that can sift through tangles of observational data. But a large-scale new study in the journal Machine Learning delivers an uncomfortable message: the choice of algorithm may matter far less than the humble hyperparameters that govern how those algorithms are trained. In fact, tuning settings that most practitioners leave at their defaults can dramatically change whether an estimator reaches state-of-the-art performance, throwing doubt on how causal methods have been benchmarked for years.

The study, conducted by Damian Machlanski of the University of Edinburgh, Spyridon Samothrakis of the University of Essex, and Paul Clarke of the University of Essex, tackles a puzzle at the heart of what statisticians call treatment effect estimation. Using Rubin’s potential outcomes framework, the goal is to estimate the difference between what happened to people who received a treatment and what would have happened to those same people had they not received it. The average of these differences across a population is the Average Treatment Effect, or ATE, while the conditional version, CATE, describes how the effect varies across subgroups. The catch is that one of the two potential outcomes for every individual is always counterfactual—it never happened and can never be observed. Unlike predicting house prices or classifying images, where a validation set reveals exactly how wrong a model is, causal models are graded on answers that are fundamentally hidden.

This hiding is what makes hyperparameter tuning so treacherous in the causal setting. In standard supervised machine learning, everyone agrees on the metric: minimize mean squared error on held-out data. The best-fitting model wins. In causal inference, what researchers actually care about is the error on the difference between potential outcomes, quantified by metrics like the Precision in Estimation of Heterogeneous Effects (PEHE). But because counterfactuals are unobservable, only the observable Mean Squared Error on factual outcomes is available in real datasets. The team formalizes this tension by distinguishing the observable metric, which can be computed, from the potential metric, which is ideal but inaccessible. When treatment assignment is random, as in a randomized controlled trial, the observable error is a decent stand-in for the potential one. Observational data, however, is riddled with covariate shift—the characteristics of treated and untreated individuals differ systematically—and the proxy can fail spectacularly.

To demonstrate the problem vividly, the authors construct a deceptively simple toy example: a one-dimensional input, a sinusoidal data-generating process, and a treated group with missing regions of data. Three candidate models emerge. The first fits the control arm beautifully but misses the trend in the treated arm, achieving the best observable MSE while producing terrible causal estimates. The second captures the true trend in both arms despite the data gaps—the solution anyone would want—but its MSE is worse, so conventional model selection would reject it. The third fits the data poorly overall yet, through leakage of benchmark test information, manages to top the PEHE leaderboard. All three illustrate the same disturbing conclusion: in causal estimation, the metric used to pick a model can be actively misleading, and a solution that looks best on paper may be the wrong one.

The centerpiece of the paper is an exhaustive empirical study built on the four classic causal benchmarks: IHDP, derived from the Infant Health and Development Program; Jobs, combining experimental and observational employment data; Twins, based on twin births in the United States between 1989 and 1991; and News, a high-dimensional dataset of simulated user responses to articles. Crucially, these benchmarks contain ground-truth counterfactuals, which allowed the researchers to compute the otherwise inaccessible potential metrics—not to tune models, which would violate good practice, but to grade how well different observable metrics perform as tuning proxies. The experimental grid spans seven widely used causal estimators, including S-Learners, T-Learners, X-Learners, Doubly Robust estimators, Double Machine Learning, Inverse Propensity Score Weighting, and Causal Forests, crossed with nine base learners ranging from Lasso and Ridge regression to gradient-boosted trees and neural networks.

The headline result is stark. Using an oracle that selects hyperparameters with the ideal potential metrics—the best any data-driven tuning could ever achieve—the probability of reaching state-of-the-art performance jumped from 65 percent to 81 percent for average treatment effect estimation, and from 50 percent to 57 percent for individualized estimation. The comparison held across nearly every estimator and base learner, with typical gains of 8 to 12 percent, and even larger improvements of 10 to 33 percent for learners like Lasso, Decision Trees, Kernel Ridge, and CatBoost. Only the ensemble methods, including Random Forests and Causal Forests, showed robust defaults—a likely explanation for the strong out-of-the-box performance of Causal Forests, whose ensembling structure echoes the resilience of the other ensemble learners.

The implications ripple outward. If swapping hyperparameters alone can push an estimator from the bottom of the league table to state-of-the-art territory, then published benchmarks that compare causal methods may be measuring tuning quality as much as algorithmic merit. The authors point out that different studies use different metrics for hyperparameter optimization, making cross-paper comparisons of estimators inherently fragile. Their conclusion is deliberately provocative: a practitioner who tunes a single model carefully may outperform a rival who searches over many models without tuning. In other words, the casual assumption that newer, more sophisticated neural estimators inherently beat older regression-based approaches may simply reflect differences in how diligently their knobs were turned.

Yet the study also offers a measure of hope. Not every observable metric fails equally. When the team evaluated several practical tuning proxies—including factual MSE, plugin validation that synthesizes pseudo-ground-truth effects with a second estimator, k-nearest-neighbour matching, and the R-Score derived from the R-Learner framework—they found that some came remarkably close to oracle performance. Matching-based plug-in metrics in particular showed low variability, with a 95 percent confidence interval of roughly 0.01 on the aggregated probability scale, suggesting that well-chosen proxies can put excellent hyperparameters within reach. Others fared badly: the R-Score behaved erratically, diverging from recent findings, and plain MSE, while on average close to the oracle, produced inconsistent results across scenarios. The choice of metric, the authors conclude, can seriously distort final estimation performance, and there is currently no metric uniformly capable of identifying the best causal model.

The researchers are candid about the limits of their work. The analysis rests on the standard assumptions of causal inference—no interference between units, no hidden confounders, and positive probability of treatment for everyone—and the search spaces, while broad, had to be trimmed for computational feasibility. Still, their practical recommendation is strikingly modest: until more robust observable metrics are developed, practitioners may be better served by highly interpretable models such as linear regression or decision trees, whose hyperparameters domain experts can tune manually and thereby embed genuine prior knowledge. Perhaps most provocatively, they argue that the field does not need new estimators at all. Meaningful progress, they contend, will come from a deeper understanding of causal model selection and hyperparameter tuning—the same realization that transformed general machine learning a decade ago—potentially aided by standardized public benchmarking platforms that tune every causal estimator equally thoroughly, so that claimed breakthroughs can finally be credited to the methods themselves rather than to lucky settings.

Subject of Research: Hyperparameter tuning and model selection metrics for machine learning-based causal effect estimation

Article Title: The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation

Article References: Machlanski, D., Samothrakis, S., & Clarke, P. (2026). The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation. Machine Learning, 115(11), Article 249. https://doi.org/10.1007/s10994-026-07140-9

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07140-9

Keywords: causal inference, hyperparameter tuning, treatment effect estimation, CATE, model selection, machine learning, benchmarks, evaluation metrics, double machine learning, observational data, counterfactuals, Challenges