New SHAP-McNemar feature selection method boosts machine learning credit risk models

The Algorithm That Explains Itself: Researchers Fuse Game Theory With a 1955 Statistics Test to Make AI Loan Decisions Trustworthy

Every day, machine learning models embedded in the world’s banking systems quietly decide who receives credit and who is turned away, yet even their creators often struggle to say which piece of information tipped the scales. A newly published study offers a striking remedy: a framework that forces predictive models to justify, with formal statistical proof, exactly which variables deserve a place in their reasoning. The method, called SHAP-McNemar stepwise feature selection, was developed by Passawish Gonlachanvit, Angsumalin Senjuntichai, and Teerapong Senjuntichai and is described in the journal Machine Learning with Applications. It fuses SHAP, the leading explainable AI technique rooted in cooperative game theory, with the McNemar test, a hypothesis test first published in 1955, to determine which features genuinely drive predictions of loan default and which are merely noise inflating the model’s complexity. The stakes reach far beyond statistical housekeeping: inaccurate default predictions can snowball into waves of non-performing loans capable of destabilizing entire economies.

The researchers anchor their motivation in the subprime crisis of 2007 through 2010, when U.S. mortgages were aggressively originated, securitized, and sold worldwide, embedding bad credit risk throughout a highly leveraged and tightly interconnected financial system. As defaults rose, complex securitized products lost value, funding markets froze, and the collapse in credit and confidence triggered a global recession. World Bank data show that non-performing loans at U.S. banks climbed from under 2 percent of total loans before the crisis to roughly 5 percent in 2009, staying elevated into the early 2010s. The authors stress the ensuing vicious cycle: when non-performing loan ratios rise, banks grow cautious, credit growth slows, and heightened risk aversion can depress demand, push businesses into insolvency, and generate still more defaults. Because macroeconomic conditions demonstrably shape loan performance, the team argues, default models must weave national economic indicators together with borrower-level characteristics rather than judging applicants in an economic vacuum.

At the technical core of the study lies a problem that has haunted machine learning for decades: feature selection. Modern datasets arrive stuffed with columns, many of them irrelevant, redundant, or noise-heavy, and keeping them all invites overfitting, higher computational cost, and opaque models. Finding the optimal subset is formally an NP-hard problem: as the number of features grows, the space of possible combinations explodes at a rate of two to the power of N, making exhaustive search hopeless. Researchers therefore lean on heuristics that fall into three families. Filter methods rank features with statistical measures such as information gain or correlation coefficients; they are fast and model-independent but blind to how features interact with the classifier. Embedded methods such as LASSO fold selection into model training by applying an L1 penalty that drives some regression coefficients exactly to zero, yet they generalize poorly on high-dimensional data. Wrapper methods, which judge candidate subsets by retraining the model, capture feature dependencies and achieve higher accuracy, but they are computationally expensive, prone to overfitting, and governed by heuristic stopping rules that carry no statistical inference about the underlying population.

The framework’s first active ingredient comes from explainable AI, the field devoted to prying open black-box models. Its centerpiece is SHAP, short for SHapley Additive exPlanations, introduced by Lundberg and Lee in 2017 and built on Shapley values from cooperative game theory dating to 1953. The idea: treat each prediction as a game in which every feature is a player, and divide the model’s output among the players according to their marginal contributions. A feature’s SHAP value is a weighted average of how much the prediction changes when that feature joins an existing coalition of variables, with weights that guarantee fair attribution. Because the contributions are additive, they yield a unified, theoretically grounded measure of feature importance no matter what model sits underneath. The authors highlight a crucial caveat, however: recent work on feature-based drift detection shows that feature rankings can shift over time even when a model’s decision boundary stays stable, and other research formally treats Shapley values as random variables subject to uncertainty. Feature importance is not a fixed quantity to be read off a chart but an estimate that demands testing.

That insight is where the second ingredient enters. Rather than making irreversible removal decisions from raw SHAP point estimates, the framework treats every stepwise decision as a formal hypothesis. The procedure alternates between backward removal and forward re-addition, a floating scheme designed to escape the nesting trap of greedy methods, in which a feature discarded early can never return even if it becomes valuable later. At the start of each iteration, a full model is fitted on the current feature set, and its predictions on a held-out validation subset form the baseline. SHAP values are then computed and averaged across observations, features are ranked by their mean absolute contribution to the output, and the candidate with the lowest SHAP value is nominated for elimination. A reduced model is fitted without that feature, and here comes the pivotal move: the full and reduced models are compared with the McNemar test, which determines whether their difference in classification performance is statistically real or merely random variation. Only changes that pass the test survive the cut.

The McNemar test, introduced in 1955 and promoted for comparing classifiers by Thomas Dietterich in 1998, is built for exactly this paired comparison. Both models are evaluated on the same validation examples, and their errors are tallied in a two-by-two contingency table that counts examples misclassified by both models, by only the first, by only the second, and by neither. Only the two discordant cells, where the models disagree, carry information about whether their performance truly differs. Under the null hypothesis of no difference, a statistic built from those discordant counts, with Yates’ continuity correction applied, follows approximately a chi-squared distribution with one degree of freedom. The authors adapt the test to a one-sided question: whether the reduced model performs no worse than the full model. Because the classical chi-squared approximation becomes unreliable when fewer than ten discordant pairs exist, a situation the researchers encountered in practice, they employ the McNemar mid-p test, which remains valid in small-sample settings. The result is a stopping rule grounded not in heuristic convenience but in formal statistical evidence.

To put the framework through its paces, the team assembled a dataset with real-world teeth. The core records, sourced from Kaggle, capture credit histories of U.S. residents from 2007 to 2015, the years spanning the subprime crisis and its aftermath. From this pool the researchers extracted small-business financing loans: 4,765 records containing 1,390 non-performing loans. Fourteen borrower-level variables were retained, including interest rate, funded amount, annual income, debt-to-income ratio, delinquencies of thirty or more days in the past two years, credit inquiries in the previous six months, revolving credit utilization and balance, and the counts of open and total credit lines, alongside categorical descriptors such as employment length, home ownership status, and borrower region. Crucially, each loan was merged with concurrent national indicators using a one-period lag to reflect the timing of official data releases. Eight economy-wide features joined the mix: the consumer price index, real gross domestic product, gross national income, total public debt, the current account balance, inflation, unemployment, and real household consumption expenditure, drawn primarily from the Federal Reserve Economic Data service.

Two contrasting classifiers anchor the modeling effort. The first is penalized logistic regression, the traditional workhorse of credit scoring, prized for interpretability. The version employed here blends L1 and L2 penalties in the elastic-net style: the L1 term drives some coefficients exactly to zero for built-in selection, while the L2 term shrinks the rest toward zero to stabilize estimates against multicollinearity. The second is XGBoost, the extreme gradient boosting algorithm that builds decision trees sequentially, each new tree trained to correct the residual errors of its predecessors using both first- and second-order gradient information, while a regularized objective penalizes overly complex trees. Because boosted trees tend to produce distorted, sigmoid-shaped probability outputs pushed away from zero and one, the researchers apply Platt scaling, a sigmoid transformation whose parameters are estimated by maximizing likelihood, to recalibrate the probabilities. Hyperparameters are tuned with Bayesian optimization, which models the objective function probabilistically and selects configurations expected to yield the greatest improvement, while class imbalance is handled with K-Means SMOTE, which clusters the minority class and then synthesizes new samples from within those clusters.

Even the decision threshold receives economic scrutiny. Classification models output probabilities, and a cutoff must separate predicted defaulters from predicted non-defaulters. The team optimizes that cutoff with the F-beta measure, a generalization of the F1 score whose beta parameter controls the relative weight of recall versus precision. Their choice of beta, 0.2, is calibrated to the economics of lending: surveys of the credit literature indicate that misclassifying a bad borrower as good is roughly five times more costly than the reverse error, and the parameter is tuned to reflect that imbalance. The pipeline is likewise engineered to prevent data leakage, the subtle error in which test-set information contaminates training. After an initial stratified 70-30 train-test split, the training portion is subdivided again: 50 percent for hyperparameter tuning, 20 percent for probability threshold calibration, and 30 percent as the validation set on which feature-selection decisions are made. Outliers are flagged with the interquartile-range rule, missing values are imputed with five-nearest-neighbor averaging, numerical features are standardized to zero mean and unit variance, and categorical variables are one-hot encoded, all fitted on training data alone.

The significance of the work stretches well beyond the mechanics of a single pipeline. By fusing explainable AI with classical statistical inference, the framework confronts a tension that has grown acute as machine learning spreads through sensitive industries: regulators and the public demand transparency, yet the most accurate models are often the least interpretable. A selection procedure in which every deletion and re-admission is certified by a hypothesis test hands banks a defensible, auditable rationale for the inputs their models consume, potentially lowering data-acquisition costs while preserving predictive power and satisfying oversight requirements. The framing also carries a philosophical message: feature importance should be treated as an inherently uncertain, statistically testable quantity rather than a fixed ranking handed down by an attribution algorithm. And although the study focuses on credit risk, the same logic, rank candidates with SHAP and validate every stepwise change with the McNemar test, could travel to healthcare, manufacturing, or any domain where models must justify which variables they rely on. In an era when a single miscalibrated model can ripple through economies, a statistical test from 1955 may prove one of the freshest tools for taming twenty-first-century artificial intelligence.