The ten systems examined were Argentina, Chile, Chinese Taipei, Finland, Hungary, Italy, Japan, Korea, the Philippines, and the United States. The authors stress that this was a purposive comparative sample, chosen to span geographic regions, OECD membership status, and a wide range of average performance, from a weighted mean mathematics score of 352.57 in the Philippines to 531.14 in Chinese Taipei. The design supports comparisons across these ten systems but does not license claims about all countries or continents. Within the sample, 2,617 schools contributed students, with country sample sizes ranging from 4,838 in the United States to 11,975 in Argentina.

What distinguishes this study from much of the educational data-mining literature is its fidelity to PISA’s assessment architecture. Mathematics achievement in PISA is not a single number but a set of ten plausible values, multiple imputed estimates that represent uncertainty in each student’s latent proficiency. The researchers fitted and evaluated every model across all ten plausible values rather than relying on just the first. They also incorporated the final student weight, W_FSTUWT, in descriptive statistics, model fitting, and test-set evaluation, and used the 80 student replicate weights to estimate sampling variability. Total variance for each performance metric combined the mean replicate-weight sampling variance with the between-plausible-value variance, following the formula U-bar plus (1 + 1/M) times B, so that confidence intervals reflected both complex sampling uncertainty and achievement-scaling uncertainty. Few machine-learning studies of PISA data go to these lengths.

The predictor side of the analysis was equally disciplined. The authors constructed a transparent primary candidate pool of 33 predictors from PISA 2018 student questionnaire variables and OECD-derived indices, organized into six substantive domains: student demographics, family socioeconomic status and home resources, engagement and learning time, peer climate, belonging and parent support, and classroom climate. Rather than assuming all 33 variables mattered everywhere, they performed country-specific stability feature selection using only training schools. Three complementary methods were applied: mutual information, which captures general dependence; weighted elastic net regression, a regularized linear method that handles correlated predictors through combined L1 and L2 penalties; and weighted random forest ranking, which captures nonlinear and interaction-based structure. Feature rankings were repeated across ten plausible values and five internal school-level folds, yielding 50 selection runs per country. A predictor was deemed stable if selected in at least half of those runs. The resulting stable sets contained between 11 and 15 predictors, averaging 14.0 per country.

Four models were then compared under country-specific school-level holdout evaluation, with roughly 70 percent of students in training and 30 percent in test sets. Splitting by school rather than by student reduced leakage from students in the same school appearing on both sides. The contenders were weighted linear regression, weighted linear regression augmented with pairwise interactions, random forest, and CatBoost, a gradient-boosted tree ensemble designed for structured tabular data. Averaged across countries, CatBoost was the clear winner, achieving a mean weighted R-squared of 0.358 and a mean mean absolute error of 57.29 score points. Random forest followed with a mean weighted R-squared of 0.313 and an MAE of 59.30. The interaction-augmented linear model barely improved on the additive linear benchmark, with mean R-squared values of 0.290 versus 0.286. CatBoost posted the lowest weighted MAE in all ten systems, and its advantage over random forest was consistent but moderate: roughly 2.01 MAE points and 0.045 in R-squared on average. Its country-level R-squared ranged from 0.210 in Italy to 0.443 in Hungary.

The interpretive core of the study used SHAP, or Shapley additive explanations, a technique that decomposes each model prediction into additive contributions from individual features. The authors were careful to frame SHAP summaries as model-based explanations of predictive associations, not causal effects. Because CatBoost performed best, interpretation focused on its SHAP values, computed on held-out test schools and averaged across all ten plausible values to produce PV-robust stability summaries. The headline finding is that books at home was the most stable predictor of all, ranking among the top ten SHAP predictors in all ten countries on average and among the top five in 8.7 countries on average. Highest parental occupational status was nearly as stable, appearing in the top ten in 9.2 countries and the top five in 7.1. Socioeconomic status itself appeared in all ten country-specific models and ranked in the top ten in 8.3 countries on average, though its average top-five count was lower at 4.3.

Beyond the socioeconomic core, the picture became more heterogeneous. Grade placement entered the stable feature sets of seven countries and ranked in the top ten in all seven, but it was not selected in Chinese Taipei, Japan, or Korea. Mathematics learning time appeared in six countries and ranked in the top ten in all six, with an average top-five count of 5.5, and it ranked especially highly in Chinese Taipei, Japan, and the United States. Test effort was retained in only four systems but was consistently influential there, ranking in the top five in all four and even first in Finland, Italy, and Korea in the diagnostic visualization. Directed instruction and disciplinary climate also showed cross-national reach, ranking in the top ten in 7.0 and 5.8 countries on average respectively. The authors argue that this mix of stability and country-specificity is precisely why a single pooled importance ranking would be misleading.

Sensitivity analyses reinforced the robustness of these conclusions. Comparing SHAP summaries based on the first plausible value with summaries averaged across all ten showed only small differences, with maximum absolute gaps of 0.8 countries for top-ten counts, 1.2 countries for top-five counts, and 1.25 rank positions for mean rank, confirming that the main pattern was not an artifact of PV1MATH. A second sensitivity analysis removed grade placement from the stable feature sets where present and refitted the CatBoost models. Performance dropped modestly, with a mean R-squared change of -0.028 and a mean MAE increase of 1.17 score points, the largest decline occurring in Chile, where R-squared fell by 0.091 and MAE rose by 3.92 points. Crucially, the SHAP stability pattern remained broadly similar without grade: books at home, parental occupational status, learning time, socioeconomic status, test effort, directed instruction, and disciplinary climate all stayed among the most stable predictors.

The authors are explicit about the limits of their contribution. They do not claim to have discovered new determinants of mathematics achievement; domains such as socioeconomic resources, home literacy environments, learning time, and classroom climate are already well established. The value lies in showing how these familiar predictors behave under a rigorous, country-specific, survey-weighted, plausible-value-aware explainable machine-learning framework, and in separating three questions that are often conflated: which models predict best, which predictors are stable across systems, and which are context-specific. They also caution that the analysis is predictive rather than causal, that the ten-system sample is not statistically representative of global education, that the 33-predictor pool cannot exhaust all relevant influences such as school policies or teacher characteristics, and that cross-national questionnaire comparisons may be affected by response styles, translation, and measurement comparability.

The implications reach well beyond this dataset. The framework, with its train-only feature selection, school-level holdout evaluation, replicate-weight variance estimation, and all-plausible-value pooling, offers a reproducible template that could be extended to more PISA systems, to other assessments such as TIMSS and PIRLS, to school-level and system-level predictors, and to fairness analyses examining whether prediction errors or SHAP patterns differ across gender, socioeconomic, immigrant, or language groups. Repeated-cycle analyses could test whether cross-national stability patterns persist as PISA evolves. For a field increasingly drawn to black-box prediction, the study makes a compelling case that explainability and methodological rigor are not optional extras but the foundation of responsible machine learning in comparative education.

Subject of Research: Explainable machine learning applied to predicting and interpreting mathematics achievement in PISA 2018 across ten education systems.

Article Title: Explainable AI for predicting and interpreting mathematics achievement: a cross-national analysis of PISA 2018

Article References: Liu, L., & Dai, R. (2026). Explainable AI for predicting and interpreting mathematics achievement: a cross-national analysis of PISA 2018. Large-scale Assessments in Education, 14(1), Article 46. https://doi.org/10.1186/s40536-026-00320-y

Image Credits: AI Generated

DOI: 10.1186/s40536-026-00320-y

Keywords: PISA 2018, mathematics achievement, explainable artificial intelligence, SHAP, CatBoost, machine learning, survey weights, plausible values, large-scale assessment, socioeconomic status, comparative education, feature selection

The study, led by Sukanya Chaemchoy of Chulalongkorn University together with Thanakrit Supsin, analyzed data from 24,273 students nested in 1,049 schools across Chile, Cyprus, Denmark, Greece, Korea, Norway, Romania, the Slovak Republic, Slovenia, Sweden, Chinese Taipei, and Uruguay. These were the 12 systems that administered ICILS 2023’s optional questionnaire on generative AI, which asked principals how likely they believed ChatGPT and similar tools were to help or harm students’ learning in their schools. The researchers linked these leadership reports with achievement data, principal questionnaires, and surveys completed by schools’ ICT coordinators, constructing an unusually rich picture of the organizational digital climate surrounding each student.

The five focal conditions were carefully distinguished. Positive ChatGPT expectations measured principals’ anticipated benefits, such as greater student interest in learning, better written work, and improved critical evaluation of information. Negative ChatGPT expectations captured anticipated harms, including shallow conceptual understanding, submission of work that is not the student’s own, and dependence on AI tools. Instructional ICT-use expectations reflected whether teachers were expected to integrate digital technology into teaching, assessment, and monitoring of progress, while ICT collaboration expectations concerned professional communication and collaboration via technology. Finally, pedagogical ICT hindrances, reported by ICT coordinators, captured constraints such as insufficient teacher skills, limited preparation time, weak pedagogical support, and restrictive policies.

Methodologically, the study is a masterclass in caution. The authors estimated separate weighted two-level models for each education system, treating students as nested within schools and adjusting for student sex, home internet access, computer experience, within-school socioeconomic background, and school socioeconomic composition. Because CIL was measured using five plausible values, each model was run five times and the results formally combined. The family of 60 primary tests, five school conditions across 12 systems, was then subjected to Benjamini-Hochberg false discovery rate control, a correction that dramatically raises the bar for what counts as a credible finding in large-scale educational research.

The outcome was unambiguous at the top line: not a single school condition produced a false-discovery-rate-retained association with CIL. Three coefficients did carry raw p values below 0.05. In Korea, higher ICT collaboration expectations were associated with roughly 9.55 points lower CIL, and pedagogical ICT hindrances were associated with about 4.11 points lower CIL. In the Slovak Republic, the direction reversed dramatically: ICT collaboration expectations were associated with 7.82 points higher CIL. Yet all three signals carried an FDR-adjusted q value of 0.730, meaning they are best read as nominal, system-specific hints for future replication rather than established associations.

The Korea-Slovak Republic contrast is perhaps the most intriguing descriptive finding. The same survey instrument, measuring the same construct, produced estimates with opposite signs and non-overlapping confidence intervals in the two systems. The authors are careful not to overclaim: no formal test of slope heterogeneity was conducted, and the study did not measure the national policies, governance arrangements, curricula, or implementation histories that might explain the divergence. But the pattern echoes earlier ICILS research from 2013, which found that the school-level conditions linked to teachers’ ICT use differed across Australia, the Czech Republic, Germany, and Norway, suggesting that identical school-scale scores may be embedded in fundamentally different institutional realities.

A secondary analysis reinforced this caution about pooled summaries. When the researchers constrained the focal slopes to be equal across systems, using equal total weights for each education system, all five common-slope confidence intervals included zero. The near-zero pooled estimate for ICT collaboration expectations simply cannot represent both Korea’s negative and the Slovak Republic’s positive coefficient. As the authors note, adjusting for system mean differences through country indicators does not demonstrate that school-level relationships are homogeneous, a point long emphasized in methodological work on multilevel modeling of country effects.

Six prespecified families of sensitivity analyses, covering 350 focal comparisons, tested whether the conclusions depended on weighting choices, socioeconomic decomposition, coding decisions, complete-case selection, influential schools, or survey-design variance estimation. Every sensitivity confidence interval overlapped its primary counterpart, and no comparison met the prespecified material-sensitivity criterion. The three nominal signals kept their direction in every available comparison, but they never escaped the multiplicity adjustment. In short, the null finding for school digital conditions is not a fragile artifact of one particular model specification.

What did matter, consistently, was student background. Within-school socioeconomic background showed positive adjusted associations with CIL in all 12 systems, with coefficients ranging from 7.78 to 26.52 CIL points per index point. School socioeconomic composition was also positive everywhere, at 30.57 to 64.49 points per index point, though the authors treat it cautiously because the aggregated school mean had reliability below 0.90 in seven systems. Computer experience was positively associated with CIL in every system, and female students outperformed male students in adjusted comparisons across the board. These results align with prior meta-analytic evidence of a positive, if modest, relationship between socioeconomic status and ICT literacy.

The implications reach beyond academia. As governments pour resources into digital infrastructure and school leaders form opinions about generative AI, this study warns against conflating leadership expectations with classroom reality. A principal who expects ChatGPT to boost learning is not necessarily leading a school where students actually develop stronger digital competencies, and none of the measured school conditions reliably distinguished high-CIL schools from low-CIL schools once socioeconomic and experiential factors were accounted for. The indirect pathway from leadership vision through organizational conditions, teacher practice, and student learning opportunities remains largely unmeasured, and this analysis explicitly did not test whether expectations influenced implementation or whether teacher practices mediated any association.

The authors also flag important limitations. The design is cross-sectional, so no causal or temporal claims are possible, and the 12 systems were defined by participation in the optional ChatGPT questionnaire rather than representative sampling of countries. The focal measures were principal and ICT coordinator reports rather than observations of actual teaching or student AI use, and included students had higher weighted mean CIL than excluded students in every system. Several systems, notably Chile, Norway, and Denmark, contributed fewer than 50 schools, widening school-level confidence intervals. These constraints mean the findings generalize only to the participating systems and samples analyzed.

Still, the study’s core message is a timely corrective to technological optimism. At a moment when generative AI is reshaping debates about homework, assessment, and information literacy, the largest multilevel evidence base yet assembled on principals’ AI expectations and students’ actual digital skills finds no common direct link between the two. What predicts assessed computer and information literacy, in these data, is not what school leaders expect of ChatGPT or their teachers, but the socioeconomic circumstances and accumulated computer experience that students bring with them. Until future research connects leadership expectations to real implementation, teacher enactment, and students’ digital activities, the study suggests, expectations about AI in schools should be read as organizational commentary, not as predictors of learning.

Subject of Research: Associations between school digital conditions and student computer and information literacy across 12 ICILS 2023 education systems

Article Title: School digital conditions and student computer and information literacy across 12 education systems: system-specific multilevel evidence from ICILS 2023

Article References: Chaemchoy, S., & Supsin, T. (2026). School digital conditions and student computer and information literacy across 12 education systems: system-specific multilevel evidence from ICILS 2023. Large-scale Assessments in Education, 14(1), Article 42. https://doi.org/10.1186/s40536-026-00315-9

Image Credits: AI Generated

DOI: 10.1186/s40536-026-00315-9

Keywords: ICILS 2023, computer and information literacy, ChatGPT expectations, school digital conditions, ICT in education, generative AI, multilevel modeling, socioeconomic status, digital inequality, international assessment, plausible values, false discovery rate