Old-School Trees Beat Attention Models in Student Performance Prediction, Rigorous Benchmark Finds

In an era when artificial intelligence headlines are dominated by ever-larger neural networks, a new study delivers a refreshingly contrarian message: when it comes to predicting how students will perform, the humble decision-tree ensemble still reigns supreme. Researchers Kadir Kesgin of Bandırma Onyedi Eylül University and Erdoğan Usta of Tokat Gaziosmanpaşa University in Turkey have published one of the most methodologically careful comparisons to date of machine learning models for educational prediction, and their verdict is a reproducible negative result. Feature self-attention, the mechanism that powers modern language models, offered no statistically significant advantage over a matched neural network without attention, and neither came close to Gradient Boosting across three public tabular datasets.

The study, published in Discover Artificial Intelligence, arrives at a moment when educational institutions are under growing pressure to adopt predictive analytics. Universities and online learning platforms increasingly want early-warning systems that flag students at risk of failing or dropping out, so that advisors can intervene before it is too late. But the field of educational data mining has a persistent problem: many published benchmarks are built on a single dataset, tuned in ways that leak information from the future into the training process, and evaluated without statistical rigor. The result is a literature littered with claims of breakthrough accuracy that collapse when tested honestly.

Kesgin and Usta set out to close that gap with an unusually disciplined benchmark. They assembled three structurally distinct datasets: the UCI Dropout dataset, a Mendeley AI dataset of 560 observations spread across ten small grade classes, and a Kaggle 2024 student performance dataset. Across these, they evaluated seven model families, ranging from classical baselines such as Logistic Regression and Ordinal Logistic Regression to tree ensembles like Random Forest, Gradient Boosting, and CatBoost, and neural architectures including FT-Transformer and TabNet, alongside a custom self-attention model. Every model faced the same outer-fold validation protocol, and all preprocessing and hyperparameter tuning were confined strictly to training partitions, never touching the held-out test data.

The leakage controls deserve particular attention, because they strike at one of the most common ways educational AI studies overstate their results. In the Kaggle 2024 dataset, the researchers removed a variable called ExamScore before any preprocessing began, recognizing it as a target proxy, a feature so tightly correlated with the outcome that including it would amount to letting the model peek at the answer key. They also discovered that identical feature groups appeared repeatedly in the data, so they adopted group-aware splitting to keep those duplicates from straddling the boundary between training and test sets. Student identifiers were dropped everywhere, and scalers were fitted only on training folds. These may sound like housekeeping details, but they are precisely the omissions that inflate reported accuracy in much of the applied machine learning literature.

The scale of the evaluation was also notable: 840 model-fold evaluations in total. The UCI Dropout and Kaggle 2024 datasets each went through ten repetitions of five-fold cross-validation, producing 50 outer folds apiece, while the small Mendeley AI dataset used ten repetitions of two-fold validation, yielding 20 folds, because its 560 observations were too thinly spread across ten grade categories to support more splits. Within each training partition, the team ran Optuna hyperparameter searches with ten trials, giving every model a fair and equal optimization budget. Neural networks were trained with the AdamW optimizer, early stopping, and a fixed random seed to guarantee reproducibility.

The headline finding was unambiguous. Gradient Boosting ranked first on all three datasets, with Macro-F1 scores ranging from 0.6988 on the UCI Dropout data to 0.9185 on Kaggle 2024 and 0.8642 on Mendeley AI. On the Mendeley dataset, the gap was dramatic: Gradient Boosting reached 97.6 percent accuracy while the neural models, including the attention variant and its ablation, managed only between 0.3354 and 0.7009 Macro-F1. On UCI Dropout, the attention model did edge slightly past Random Forest and CatBoost, but still trailed Gradient Boosting by a margin of just 0.0076 Macro-F1, a difference the statistical tests could not distinguish from noise.

That statistical rigor came from Nadeau–Bengio corrected resampled t-tests, a technique designed for the reality that cross-validation folds are not independent samples, paired with Holm correction to control the family-wise error rate across all comparisons. Under this corrected paired testing, the self-attention model failed to significantly outperform its capacity-matched ablation, a feedforward network with the same parameter count and the same tuning budget, on any dataset. The attention-versus-ablation differences of plus 0.0098 on UCI Dropout, minus 0.0461 on Mendeley AI, and plus 0.0535 on Kaggle 2024 all returned Holm-adjusted p-values of 1.0000. In plain terms, whatever benefit attention provided in some folds was indistinguishable from chance once the comparison was made fair.

This controlled ablation is the study’s methodological centerpiece and its most transferable lesson. Many papers celebrating attention-based tabular models compare them against weak baselines or against neural backbones that were not given equal tuning effort, making it impossible to tell whether attention itself, or simply extra capacity and care, drove the improvement. By jointly tuning two networks that differed only in the presence of the attention layer, Kesgin and Usta isolated the causal contribution of the mechanism itself. The answer, for these educational tabular tasks, was essentially zero. The result aligns with a growing body of evidence, including influential analyses by Grinsztajn and colleagues and by Shwartz-Ziv and Armon, showing that tree ensembles remain remarkably hard to beat on structured data with modest sample sizes, heterogeneous features, and noisy interactions, exactly the conditions that characterize educational records.

The study went beyond raw accuracy to probe dimensions that matter when predictive models touch real students’ lives. The researchers evaluated probability calibration using Expected Calibration Error, Brier Score, and negative log-likelihood, examined ordinal metrics such as Quadratic Weighted Kappa for the graded Mendeley data, and ran a bootstrap-based fairness audit on the UCI Dropout dataset using demographic parity, equal opportunity, and disparate impact across gender groups. They also compared SHAP feature attributions between the attention model and its ablation. The authors are candid about the limits of these analyses: the fairness audit covered only binary gender, the Mendeley dataset retained two columns that might be target-derived, and computational costs such as training time and memory were not recorded. That transparency about residual uncertainty is itself a model for the field.

Perhaps the most valuable contribution of this work is its reframing of what a negative result can accomplish. Rather than adding another leaderboard entry, the study provides a reproducible template: remove target proxies, respect grouped data, match model capacity, tune everyone equally, and apply corrected statistical inference before declaring a winner. The authors suggest that attention architectures may yet find roles in education, not as standalone classifiers but as diagnostic probes, feature-interaction auditors, or components in knowledge-distillation pipelines where a strong tree ensemble teaches a neural student. They also point toward imbalance-aware loss functions for skewed grade distributions and post-hoc calibration fitted strictly within development partitions. For institutions weighing whether to invest in fashionable deep learning for their early-warning systems, the message is clear and sobering: under honest validation, gradient-boosted trees remain the benchmark to beat, and any claim that a newer architecture surpasses them deserves the same scrutiny this study brought to attention.

Subject of Research: Leakage-aware benchmarking of attention-based and ensemble machine learning models for student performance prediction on tabular educational datasets

Article Title: Leakage-aware benchmarking and controlled ablation of attention models for student performance prediction across multiple tabular datasets

Article References: Kesgin, K., & Usta, E. (2026). Leakage-aware benchmarking and controlled ablation of attention models for student performance prediction across multiple tabular datasets. Discover Artificial Intelligence, 6(1), Article 1426. https://doi.org/10.1007/s44163-026-02398-3

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02398-3

Keywords: student performance prediction, educational data mining, tabular deep learning, feature self-attention, gradient boosting, target leakage, controlled ablation, cross-validation, machine learning benchmarking, learning analytics, reproducibility, CatBoost