Machine Learning Audit Reveals Composite Resilience Scores Fail to Predict Real Grain Yield Losses Under Climate Shocks

A rigorous new audit of machine learning models applied to China’s grain production systems has delivered an uncomfortable verdict: the composite resilience indices that many studies use to rank regions by their capacity to withstand climate stress show almost no measurable relationship with how production actually performs when climate shocks strike. The study, published in Frontiers in Sustainable Food Systems, examined prefecture-level grain production, socioeconomic, and climate data across China and found that high predictive accuracy for a capacity-based resilience index evaporates under strict validation, and that the index itself fails to align with realized resistance to climate shocks. The findings strike at the heart of a growing literature that uses machine learning to score agricultural resilience, warning that accuracy alone can be deeply misleading when the measurement target is poorly defined.

The research team, led by Han Yan of Nankai University Business School with colleagues from Tianjin University of Finance and Economics, North University of China, and Hebei University of Economics and Business, built their analysis around a distinction drawn from earlier resilience scholarship: the difference between potential resilience, meaning the resources a system has available before a shock, and revealed resilience, meaning how the system actually performs during one. Capacity-based composite indices summarize the former, combining indicators such as grain-sown area, irrigation rates, agricultural machinery power, rural incomes, science and technology investment, output volatility, disaster-affected area rates, and fiscal support into a single weighted score. Outcome-based resistance, by contrast, measures whether grain yields stay near their expected trajectory during a specific climate stress event. The question the authors posed was deceptively simple: does a higher capacity score actually correspond to better production performance during a shock?

To answer it, the researchers constructed a provincial composite index for 13 major grain-producing provinces over 2011 to 2024, using the entropy-weight method across resistance, adaptation, and recovery dimensions, yielding 182 province-year observations. They then built a separate outcome measure for cities: for each city and target year, a local linear Huber trend fitted to the five most recent valid yield observations produced an expected yield benchmark based strictly on past information, and the deviation of actual yield from that benchmark during climate stress events defined realized resistance. Climate stress events were identified using daily NASA POWER weather data, standardized within each city against a 2008 to 2018 baseline, with events flagged when a composite stress score combining heat, cold, drought, and excess precipitation measures crossed a fixed 80th-percentile threshold.

The first striking result concerns how easily the composite index can be predicted, and how fragile that predictability is. Under a random 80/20 row split, a Gradient Boosting model with Huber loss achieved an R-squared of 0.957 on held-out province-year observations, a figure that would look impressive in any paper’s results table. But when the evaluation was redesigned to prevent information leakage, with 2011 to 2018 for training, 2019 to 2020 for validation, and 2021 to 2024 for testing, the selected Ridge model managed an R-squared of only 0.805, and a simple province-specific linear trend benchmark actually outperformed it at 0.869, while a persistence benchmark that carried each province’s 2020 value forward scored 0.809. Leave-one-province-out validation, testing generalization to entirely unseen provinces, dropped the pooled R-squared to 0.531. In other words, the composite index contains substantial learnable structure, but that structure is dominated by persistent regional differences and smooth temporal evolution rather than anything a machine learning model uniquely captures.

Far more consequential is the construct alignment test. The researchers matched provincial composite scores and their three component scores to the mean realized resistance among cities experiencing P80 climate stress events in the same province and year, producing 120 matched province-year observations. The Spearman rank correlation between the composite resilience score and realized resistance was just 0.022, with a 95 percent confidence interval of negative 0.102 to 0.162 from 2,000 bootstrap replications clustered by province. The resistance, adaptation, and recovery components fared no better, with correlations of 0.038, negative 0.042, and 0.095 respectively, and all confidence intervals including zero. Association models controlling for province and year fixed effects found no statistically detectable relationship. Sensitivity tests using area-weighted resistance aggregation and one-year-lagged capacity scores produced the same near-zero pattern, confirming that the capacity index and realized shock performance carry genuinely different empirical information.

The second half of the study turned to whether machine learning could predict realized resistance itself, using a full city-level model with 26 predictors spanning contemporaneous climate variables, yield-history variables generated solely from past information, and lagged capacity and production structure variables. The answer, again, was sobering. Across 350 P80 events in 160 cities during the strict 2021 to 2023 test period, the continuous model achieved an R-squared of negative 0.081, meaning it provided no positive explanatory gain relative to simply using the test-sample mean. Its Spearman correlation with observed deviations was 0.172, and predictions clustered in a narrow range while large positive and negative deviations went uncaptured. For below-trend classification, the area under the receiver operating characteristic curve reached 0.621, modest discrimination, but at the fixed classification threshold of 0.585 the model identified only 48 of 204 below-trend events, with sensitivity of 0.235 against specificity of 0.904. It was better at ruling out failures than catching them.

Information-set ablation revealed where the model’s limited signal actually came from. Removing only the lagged yield anomaly barely changed below-trend AUC, from 0.621 to 0.617. But removing all yield-history information, including expected yield, collapsed AUC to 0.495, essentially chance. Climate variables alone or capacity and structure variables alone each produced AUCs of roughly 0.477, and the same pattern held in nationwide replication. A simple Logit model using only lagged yield and the lagged yield anomaly achieved a test AUC of 0.629, matching the full 26-variable model, and adding expected yield pushed it to 0.768. The authors are careful to note that expected yield enters the definition of the outcome itself, so this comparison partly audits dependence on outcome construction, but the broader message stands: grain production history supplies most of the identifiable ranking information, and adding climate and capacity variables yields no consistent improvement.

Spatial transfer and nationwide replication added nuance without changing the picture. Applying the model finalized in the major producing areas directly to 117 external cities, drawn from 18 provincial-level regions with no overlap in model development, produced a below-trend AUC of 0.613 across 237 events, close to the 0.621 at home, but continuous R-squared fell to negative 0.152 and probability calibration deteriorated sharply, with a Brier score of 0.305 that was worse than the 0.247 achieved by a constant prediction fixed at development-sample prevalence. Repeating the full workflow across 297 cities nationwide yielded an AUC of 0.618. Beneath these similar aggregates, annual AUCs swung widely, from 0.775 in 2021 down to 0.371 in 2023 in the major producing areas, and calibration slopes fell below one in all three samples, indicating systematic underprediction of risk probabilities. Spatial diagnostics compounded the concern: Global Moran’s I for prediction residuals was significantly positive in all three samples, with p-values of 0.002 from 499 permutations, revealing regional prediction biases that aggregate metrics conceal.

The authors frame their results as a methodological audit rather than a rejection of machine learning. Their argument is that predictive performance does not establish construct validity, a distinction long recognized in the statistical literature between prediction and explanation. When a resilience prediction target is itself constructed from variables that reappear as model inputs, high accuracy may reflect structural relationships among variables rather than any identified capacity to withstand shocks. Random train-test splits on data with temporal persistence and spatial dependence place closely related observations in both training and evaluation sets, producing overly favorable estimates for applications to unseen years or regions. The study therefore prescribes an ordered sequence for future work: specify the system, function, shock, and response dimension; distinguish capacity indicators from realized outcomes; use temporal and spatial holdouts matched to the intended application; compare against simple trend, persistence, and prevalence benchmarks; and establish calibration before interpreting feature importance.

The practical implications reach into agricultural risk governance. The results do not support ranking mechanization, research expenditure, or fiscal policy priorities by feature importance in models of composite indices, since the study identifies no causal effects. Instead, the authors argue that monitoring systems should track capacity dashboards while evaluation systems separately test whether high-capacity regions actually maintain core functions during shocks, and that repeated divergence between the two may reveal measurement gaps or bottlenecks in how resources convert into responses. Data priorities include crop-specific yields and phenology, cropland-weighted climate measures, irrigation and soil moisture information, observed disaster impacts, and varieties, together with prospective validation in later years and independent regions. The study’s scope remains bounded: aggregate prefecture-level yields mix crops with different climate sensitivities, city-centroid weather measures miss cropland variation, and the final test covers only three years. But its central lesson is durable and uncomfortable for the field: a resilience score that a machine can predict with near-perfect accuracy may still tell you almost nothing about whether the grain actually comes in when the heat and drought arrive.

Subject of Research: Construct validity and machine learning validation of grain production resilience measures under climate stress in China

Article Title: Assessing grain production resilience under climate stress: temporal holdouts and cross-regional validation

Article References: Yan, H., Li, D., Ma, S., & Li, Y. (2026). Assessing grain production resilience under climate stress: temporal holdouts and cross-regional validation. Frontiers in Sustainable Food Systems, 10, Article 1939695. https://doi.org/10.3389/fsufs.2026.1939695

Image Credits: AI Generated

DOI: 10.3389/fsufs.2026.1939695

Keywords: grain production, climate shocks, resilience, machine learning, construct validity, external validation, yield prediction, China, food security, composite indices, probability calibration, spatial diagnostics