The standard approach to testing the liberation hypothesis has been regression-based. Researchers build statistical models of sentencing outcomes, include measures of case severity, and test whether the effect of race or sex on punishment grows stronger as severity weakens, typically through interaction terms. If the coefficient linking defendant race to sentence length is larger in low-severity cases than in high-severity cases, the finding is taken as evidence of liberation. Holmes and Arrigo do not dispute that these coefficients answer a real question. They simply point out that it is not the question Kalven and Zeisel asked. The original theory was about what explains disagreement, not merely about how large a particular effect happens to be in one setting compared with another.

The distinction the authors draw is between effects and contributions. A regression coefficient can tell us, for example, that the odds of a judge-jury disagreement are twice as high when a defendant appears sympathetic than when the defendant does not. What it cannot tell us is how much of the overall variation in disagreements is actually accounted for by sympathy compared with the strength of the evidence. Two variables can have identical coefficients yet contribute very different shares of explained variance, depending on how much they vary and how they correlate with everything else in the model. Effect size and explanatory power are related but distinct quantities, and a theory about the sources of judicial discretion, the authors contend, is fundamentally a theory about explanatory power.

To capture that second quantity, Holmes and Arrigo turn to variance decomposition, a family of techniques borrowed from economics, psychology, and statistics that partition a model’s explained variance among its predictors. The study employs two such methods. Dominance analysis, adapted for logistic regression, determines the relative importance of predictors in the decision to incarcerate by comparing each variable’s contribution across all possible reorderings of the model. Shapley value decomposition, rooted in cooperative game theory, assigns each predictor its average marginal contribution to the model’s R-squared across every possible subset of predictors. These approaches are computationally demanding. With 26 variables, a complete Shapley decomposition would require estimating more than 67 million submodels, a burden the authors managed by computing individual contributions for key independent variables while grouping control variables.

The empirical setting is the federal sentencing system, using data from the United States Sentencing Commission. The authors exploit a structural feature of the federal guidelines: the sentencing table’s zones, labeled A through D, which impose progressively tighter constraints on judicial discretion as case severity rises. Zone A cases, the least serious, permit probation and give judges wide latitude, with incarceration imposed roughly 43 percent of the time. Zone D cases are so severe that imprisonment is essentially mandatory, with incarceration rates near 96 percent. By estimating separate models within each zone, the researchers create a natural severity spectrum across which to compare both the effects and the contributions of extralegal factors, sidestepping the interpretive distortions that arise when predicted probabilities are transformed through the nonlinear sigmoid function at very different base rates.

The results deliver the study’s central punchline. When the authors ran conventional regression tests, asking whether the effect of race, ethnicity, sex, or citizenship on sentencing outcomes was moderated by case severity, the pattern expected under the liberation hypothesis was largely absent. But when they decomposed the variance, asking whether the contribution of extralegal factors to explained variation in sentencing grew as severity declined, evidence consistent with liberation emerged. In other words, the two inquiries, which many researchers have treated as interchangeable, produce different answers, and only the variance-based inquiry finds support for the theory. A variable’s coefficient can stay flat across the severity spectrum even as its share of explained variance shifts substantially, because contributions depend on the full covariance structure of the model, not on any single slope.

The technical choices behind this finding deserve attention in their own right. The authors restricted their sentence-length models to cases receiving prison time, noting that coding non-prison sentences as zero months would create heterogeneous masses at zero across zones, conflating the incarceration decision with the length decision and corrupting comparisons between zones. They also declined to include a Heckman selection correction in the length models because the correction term was substantively collinear with legally relevant variables such as pre-sentence detention, a problem documented in prior methodological work. All models were interpreted on a common log-odds scale rather than as predicted probabilities, precisely because the sigmoid transformation would compress marginal effects differently in low-incarceration Zone A than in near-certain-incarceration Zone D, undermining interpretive consistency.

What does this mean for our understanding of American courts? If the liberation hypothesis survives only under the contribution framing, then the mechanism Kalven and Zeisel described, in which doubts about evidence free decision-makers to follow sentiment, is best understood as a claim about what drives variation in punishment, not about how much any demographic characteristic changes the average sentence. That reframing has practical consequences. A small coefficient for race in a low-severity zone could still reflect a meaningful share of the discretion exercised in those cases, while a statistically significant interaction term might reflect little explained variance at all. Policymakers seeking to rein in unwarranted disparity would therefore want to know not just whether disparities exist in lenient cases, but how much of the discretionary variation those disparities actually occupy.

The study also carries a broader methodological message for criminology and the social sciences generally. Regression coefficients have long been the default currency of quantitative testing, and interaction terms the default tool for moderation. Holmes and Arrigo join a growing literature, including recent work applying variance decomposition to inequality of opportunity, arguing that relative importance analysis should supplement rather than replace conventional regression. Their demonstration that a classic criminological theory changes its empirical verdict depending on which question is asked is a vivid case study in why the choice of statistical lens is itself a theoretical commitment. As sentencing researchers continue to debate when and whether race, sex, and citizenship shape punishment, the liberation hypothesis may need to be re-litigated, this time in the currency of explained variance.

Subject of Research: Testing the liberation hypothesis in federal sentencing using variance decomposition methods

Article Title: Using Variance Decomposition to Test Criminal Justice Theory: A Redeliberation of the Liberation Hypothesis

Article References: Holmes, B., & Arrigo, S. R. (2026). Using Variance Decomposition to Test Criminal Justice Theory: A Redeliberation of the Liberation Hypothesis. American Journal of Criminal Justice. https://doi.org/10.1007/s12103-026-09950-y

Image Credits: AI Generated

DOI: 10.1007/s12103-026-09950-y

Keywords: liberation hypothesis, sentencing, variance decomposition, Shapley values, dominance analysis, federal courts, racial disparity, judicial discretion, criminology, United States Sentencing Commission, case severity, extralegal factors

The research, led by Xinfang Zhang, Xiaoxiao Chang, and Bingzhen Sun of Xidian University’s School of Economics and Management, together with clinical collaborator Xiaoxia Liu and Xiaoli Chu of the State Key Laboratory of Dampness Syndrome of Chinese Medicine at Guangzhou University of Chinese Medicine, tackles a problem that has dogged integrative medicine for decades. Western medicine produces structured, numeric, database-ready data. Traditional Chinese Medicine, by contrast, generates largely unstructured, experience-based textual records — clinical narratives steeped in specialized terminology that standard data-processing pipelines simply discard. The result is a systematic loss of information whenever the two traditions are combined, and the researchers argue that this loss is precisely where the predictive power of integrated diagnosis leaks away.

The team’s solution unfolds in two stages. First, they constructed a semantic framework for Traditional Chinese Medicine drawn from classical literature, creating a structured vocabulary that allows the idiosyncratic language of TCM diagnosis to be systematically parsed. They then deployed SnowNLP, a natural language processing library optimized for Chinese text, to convert free-form textual diagnostic information into numerical scores. In essence, the qualitative judgments of a TCM practitioner — descriptions of a patient’s constitution, symptom patterns, and syndrome differentiations — become quantified features that can sit alongside laboratory measurements in the same dataset without one tradition drowning out the other.

The second and more mathematically ambitious stage is where the study makes its central technical contribution. The researchers developed what they call a Weighted Neighborhood Probability Rough Set model, abbreviated WNPRS-MSIHIS, designed from the ground up to handle heterogeneous, incomplete, and multi-source medical data. Rough set theory, pioneered by the Polish mathematician Zdzisław Pawlak in 1982, is a framework for reasoning about vague and uncertain information: rather than forcing every patient into crisp categories, it defines lower and upper approximations of concepts, acknowledging that some cases are clearly inside a category, some clearly outside, and some genuinely borderline. Neighborhood rough sets extend this idea to continuous data by grouping patients who are close to one another in feature space, while the probabilistic variant introduces tolerance for the statistical noise that real clinical data inevitably carries.

What distinguishes the new model from its predecessors is its explicit accounting for complementarity — the recognition that different attributes do not merely add up their individual contributions but interact with one another in complex ways. To capture these interactions, the researchers incorporated the Choquet integral, a powerful aggregation operator dating to 1954 that generalizes the ordinary weighted average by allowing the weight assigned to one attribute to depend on which other attributes are present. Alongside it, they employed generalized Shapley values, a concept borrowed from cooperative game theory, to measure each attribute’s importance not in isolation but in the context of every coalition of features it might join. A laboratory marker that seems mediocre on its own, for example, might become highly informative when combined with a particular TCM syndrome score — and Shapley-based importance is exactly the kind of tool designed to surface that hidden synergy.

A further layer of sophistication comes from granularity weights, which the researchers introduced to maintain knowledge consistency across information sources. Because the dataset draws from multiple hospitals and diagnostic traditions, the same underlying feature may be recorded with different precision or reliability depending on its origin. By weighting the information granules accordingly, the model prevents a well-documented data source from being diluted by noisier ones, and prevents redundant attributes — measurements that effectively repeat the same information — from masquerading as additional evidence. The feature selection process thus prunes the dataset down to a compact subset that preserves the critical diagnostic signal while discarding the noise.

To test the framework, the team assembled a substantial clinical cohort of 3,008 chronic renal failure cases, each carrying both Western clinical indicators and Traditional Chinese Medicine diagnostic information. The feature selection procedure was then evaluated by measuring how well the retained features supported downstream prediction across multiple machine learning classifiers. The results were striking on two fronts. The framework consistently removed redundant attributes while preserving the diagnostic information that mattered most, and the resulting reduced feature sets improved predictive performance across the board rather than benefiting only one particular algorithm — a sign that the selected features capture genuine structure in the disease rather than artifacts of any single model.

The headline numbers are impressive for a task of this complexity. The approach achieved a recall of 0.9115, meaning it correctly identified more than 91 percent of true chronic renal failure cases — a critical property in a screening context, where missed cases carry the gravest consequences. The F1-score, which balances recall against precision and penalizes false alarms as well as misses, reached 0.8542. Beyond raw accuracy, the researchers emphasize that the method remains interpretable: because the feature selection is grounded in rough set theory and explicit importance measures, clinicians can trace which attributes — whether a laboratory value or a quantified TCM syndrome indicator — drove a given prediction. In an era when black-box medical AI is facing mounting skepticism, that transparency is not a luxury but a requirement.

The implications extend well beyond nephrology. Machine learning approaches to chronic kidney disease prediction have proliferated in recent years, and systematic reviews of the field note both their promise and their persistent limitations, including problems with data quality, heterogeneity, and generalizability. The Xidian-led study suggests that part of the answer may lie not in bigger models but in richer, better-fused data — specifically, in recovering the diagnostic information that integrative clinical practice already generates but that conventional pipelines throw away. If the framework generalizes, the same complementarity-aware machinery could be applied to other conditions where qualitative and quantitative diagnostic traditions coexist, from cardiovascular disease to metabolic disorders.

Challenges remain, as the authors and the broader literature acknowledge. The training data used in the study is confidential, which complicates independent replication, and the semantic framework for Traditional Chinese Medicine must be carefully maintained as clinical language evolves. Integrating text-derived features also raises questions about how reliably SnowNLP-style quantification captures the nuance of an experienced physician’s narrative. Still, the study — funded in part by the National Natural Science Foundation of China and the Shaanxi National Funds for Distinguished Young Scientists — represents a concrete, technically rigorous step toward a long-sought goal: a single analytical framework in which a pulse reading and a creatinine measurement are not rivals but collaborators. For millions of patients at risk of losing kidney function, that collaboration could translate into earlier warnings, better decisions, and time — the resource that chronic renal failure takes away most quietly.

Subject of Research: A machine learning feature selection method combining traditional Chinese and Western medicine data for chronic renal failure prediction

Article Title: Complementarity-aware feature selection via weighted neighborhood probability rough sets for integrated traditional Chinese and Western medicine data in chronic renal failure prediction

Article References: Zhang, X., Chang, X., Liu, X., Sun, B., & Chu, X. (2026). Complementarity-aware feature selection via weighted neighborhood probability rough sets for integrated traditional Chinese and Western medicine data in chronic renal failure prediction. Applied Intelligence, 56(15), Article 447. https://doi.org/10.1007/s10489-026-07482-w

Image Credits: AI Generated

DOI: 10.1007/s10489-026-07482-w

Keywords: Traditional Chinese Medicine, chronic renal failure, rough set theory, feature selection, Choquet integral, Shapley values, machine learning, medical diagnosis, data fusion, health informatics, predictive medicine, natural language processing

The study’s central innovation lies not in inventing new algorithms but in applying them to a problem that has resisted conventional clinical workflows for decades. The researchers drew their training data from the National Epidemiologic Survey on Alcohol and Related Conditions-III, a nationally representative US survey that includes both detailed diagnostic interviews and a rich array of background variables resembling the kinds of information captured in real-world health records. This design allowed the team to create an EHR-like dataset in which every person had a ground-truth diagnostic status, something almost never available in actual medical systems, where definitive diagnostic data on substance use is sparse and often contaminated by treatment-seeking behavior.

Because the survey contained standardized DSM-5 diagnostic assessments, the researchers could label each respondent as having or not having an untreated alcohol use disorder, opioid use disorder, or cocaine use disorder. The models were trained to predict these labels using only features that plausibly map onto data already present in health records: demographic characteristics, socioeconomic indicators, healthcare utilization patterns, and other routinely collected variables. Crucially, the team performed much of the modeling in two configurations, one excluding and one including psychiatric comorbidity information, so that they could assess how much predictive signal persists when mental health diagnostic data is unavailable, a common situation in primary care settings.

The technical pipeline reflects careful attention to the pitfalls that have undermined many machine learning studies in medicine. The authors benchmarked several algorithm families, including logistic regression, balanced random forests, extreme gradient boosting, and Gaussian naive Bayes, and tuned hyperparameters through grid search with nested cross-validation to prevent optimistic bias. They avoided relying on area under the receiver operating characteristic curve as the sole performance metric, citing well-documented critiques that this measure can be misleading for imbalanced classification problems like rare disorder detection. Instead, they optimized models for precision and recall, the quantities that matter most clinically: how many flagged patients actually have an untreated disorder, and how many true cases are successfully caught.

The headline result is striking. When models were optimized to balance precision and recall, the number of people who would need to be screened to identify one untreated case of alcohol use disorder fell dramatically relative to universal screening. In several scenarios, the reduction exceeded an order of magnitude, meaning that a healthcare system using the model as a triage tool could find the same number of hidden cases by conducting structured assessments on a small fraction of the population. For a public health problem as large as untreated alcohol use disorder, which affects tens of millions of Americans, such efficiency gains translate into enormous savings of clinician time, screening costs, and patient burden.

Interpretability was another pillar of the study. Rather than treating the models as opaque black boxes, the researchers used Shapley additive explanations, a game-theoretic attribution method that decomposes each prediction into the contributions of individual features. This approach, rooted in the Shapley value framework from cooperative game theory, allows clinicians to see exactly why a particular patient was flagged: which combination of demographic factors, utilization patterns, and other variables pushed the risk estimate upward. In an era when healthcare algorithms face legitimate scrutiny over transparency and accountability, building explanation into the tool from the ground up is a deliberate design choice with practical consequences for clinical trust and adoption.

The findings for opioid and cocaine use disorders were more preliminary but still encouraging. Because these disorders are less prevalent in the general population than alcohol use disorder, the number of positive cases available for training was small, limiting the statistical power of the exploratory models. The authors are candid about this constraint: the opioid and cocaine models showed promise but were limited by sample size. This honesty matters, because the field of clinical machine learning has repeatedly been criticized for overstating the readiness of tools trained on thin data. The study’s framing of these results as exploratory signals rather than validated instruments reflects a more mature approach to model development.

The broader context of this research includes a growing literature applying machine learning to addiction science. Previous studies have used neuroimaging data to classify nicotine dependence, resting-state functional connectivity to predict smoking status, and behavioral markers such as impulsivity dimensions to identify cocaine dependence. Others have built predictive models for opioid use disorder, including the Veterans Health Administration’s Stratification Tool for Opioid Risk Mitigation, which was developed to improve opioid safety and prevent overdose and suicide. What distinguishes the new work is its focus on untreated disease in a general population sample and its grounding in data types that real health systems actually collect at scale, rather than specialized research measurements.

The clinical deployment pathway the authors envision is clinical decision support. Integrated into electronic health record systems, such a model could continuously evaluate incoming patient data and flag individuals who warrant a structured substance use assessment, in the way that suicide risk prediction models have been trialed in large health systems. The literature on computerized decision support suggests these tools can produce absolute improvements in care when they are well designed and properly implemented, though the track record is mixed and depends heavily on workflow integration, alert fatigue, and clinician trust. A screening nudge that reduces the number of assessments needed per case found by more than tenfold is precisely the kind of efficiency that could make universal detection ambitions feasible in under-resourced primary care.

The study also engages seriously with the ethical dimensions of psychiatric prediction. Screening for stigmatized conditions raises questions about privacy, labeling, and the possibility that algorithmic flags could lead to discrimination if mishandled. The authors acknowledge concerns about bias in artificial intelligence applications for mental health, noting the field-wide calls to assess and mitigate algorithmic bias, and they emphasize that interpretability is essential for accountability when automated tools influence clinical decisions. Questions about who takes responsibility for algorithmic recommendations, how patients consent to being flagged, and how flagged individuals are approached with sensitivity to stigma all remain live issues that must be resolved before deployment. The treatment gap this technology targets is enormous and costly: untreated substance use disorders impose substantial burdens on individuals, families, and health systems, and most people who meet diagnostic criteria never seek care. By showing that routinely collected, EHR-like data carries enough signal to identify these hidden cases efficiently and transparently, the study offers a concrete technical foundation for closing a gap that has persisted through decades of conventional screening efforts, while making clear that careful validation in real health systems will be the decisive next step.

Subject of Research: Machine learning detection of untreated substance use disorders from electronic health record-like data

Article Title: Supervised machine learning enables identification of people with untreated substance use disorders using EHR-like data

Article References: Supervised machine learning enables identification of people with untreated substance use disorders using EHR-like data. (n.d.). https://doi.org/10.1038/s44220-026-00717-2

Image Credits: AI Generated

DOI: 10.1038/s44220-026-00717-2

Keywords: machine learning, substance use disorders, electronic health records, alcohol use disorder, opioid use disorder, cocaine use disorder, clinical screening, clinical decision support, interpretability, Shapley values, NESARC-III, early detection

The attention mechanism is the defining innovation of the transformer architecture. When a transformer processes a sentence or an image, it breaks the input into tokens, words in the case of language and small patches in the case of pictures. For every token, the model computes attention weights that describe how much that token should look at every other token when building its internal representation. These weights form a rich, model-internal map of token influence, and because they are computed explicitly during every forward pass, they come with a seemingly natural interpretation: high attention suggests that one token matters to another. This interpretability has fueled a long-running debate. Some researchers have argued that attention weights reveal genuine linguistic and visual structure, with studies of models like BERT showing heads that track syntactic relations or fixed positional patterns. Others, most famously in the paper titled Attention is not Explanation, have demonstrated that attention weights often fail to correlate with the features that actually drive a model’s output.

The Norwegian team, led by Marte Eggen together with Jacob Lysnæs-Larsen and Inga Strümke, sidesteps this binary argument with a pragmatic insight. Attention weights, they note, do not directly determine model outputs, but they do encode explanatory information about how tokens interact. The question is not whether attention alone explains a model, but whether attention can meaningfully improve existing explanation methods when integrated into them. To answer this, the researchers developed two novel methods, each targeting a fundamentally different aspect of model behavior. The first produces local explanations, attributions that justify a single prediction. The second produces global explanations, revealing which high-level human-understandable concepts the model has learned to represent.

The local method is grounded in cooperative game theory. The Shapley value, introduced by mathematician Lloyd Shapley in 1953, is the canonical solution for fairly dividing a game’s total payoff among its players. In machine learning, the players are input features and the payoff is the model’s prediction, so Shapley values tell you how much each token contributed to a particular classification. The catch is computational: exact Shapley values require evaluating the model on every possible coalition of features, a number that grows exponentially with input length. The researchers’ breakthrough was to redefine the game itself. Instead of measuring model predictions under masked inputs, which demands many forward passes, they construct a characteristic function directly from attention weights combined with their gradients, using a first-order Taylor approximation to relate internal attention dynamics to the output. They explored three formulations, one based on attention to the classification token, one on mutual attention between token pairs, and one using a non-linear maximum over pairwise interactions.

This design yields a striking efficiency advantage. Computing the attention-based Shapley attributions requires only a single forward pass to extract attention weights from all layers and heads, followed by one backward pass to obtain the gradients. Standard SHAP, by contrast, must mask input features over many iterations and perform numerous forward passes for each explanation. In evaluations on three standard text classification benchmarks, the Stanford Sentiment Treebank, IMDb movie reviews, and the Ag News topic corpus, the method held its own. The strongest variant, built on attention interactions with the classification token, performed comparably to the widely used SHAP framework when accounting for statistical uncertainty, while the kernel-based approximation of the attention-based game showed the best results among the sampled approaches. Pure attention weights without gradient information, included for completeness, performed notably worse, reinforcing the authors’ point that attention is most valuable as an ingredient within a principled framework rather than as a standalone explanation.

The second method tackles global explainability through concept activation vectors, an approach introduced by Kim and colleagues in their influential TCAV framework. The idea is elegant: train a simple linear classifier to distinguish the model’s internal activations for examples containing a concept, such as stripes, from examples that do not, then use the resulting direction in activation space to measure how sensitive the model’s output is to that concept. The researchers adapted this to transformers at the token level, exploiting the architecture’s so-called residual stream, a shared high-dimensional space through which all layers communicate. Because token representations live in this common space, a concept direction can be compared directly with each token’s representation, yielding a per-token concept sensitivity score. The novel step is weighting these scores by the attention that the classification token pays to each token, producing what the team calls the T-TCAV score.

The motivation for this weighting comes from a fascinating asymmetry the authors observed in vision transformers. In early layers, the classification token attends roughly uniformly across all image patches, while concept sensitivity varies widely from token to token. In deeper layers the pattern reverses: attention becomes sharply concentrated on a subset of tokens, but concept sensitivity spreads out, apparently because self-attention shares concept information among all tokens. By combining the two signals, the method lets attention provide the selectivity that concept sensitivity loses with depth, and lets concept sensitivity supply the semantic grounding that raw attention lacks. Experiments on ImageNet classes paired with concepts from the Broden dataset showed that the concept most semantically aligned with each target class, such as stripes for zebras or dots for dalmatians, tended to dominate in deeper layers, exactly as theory predicts. Heatmap visualizations of the token-level scores localized concept-relevant image regions, though the authors candidly report inconsistencies, with some concept-relevant regions receiving negative sensitivity scores in certain images of the same class.

The study also confronts an uncomfortable truth about how explanation methods are judged. The team evaluated their local methods using three quantitative metrics: F1 score, comprehensiveness, and sufficiency, all of which measure whether an explanation can reconstruct or invert the model’s prediction by masking tokens. These output-focused metrics, the authors argue, may not fully capture the value of explanations that reflect internal model dynamics rather than direct input-output influence. There is currently no widely accepted metric for how well an explanation mirrors a model’s internal representational machinery, and the researchers explain why simply perturbing attention weights to test this is not feasible: attention weights are learned jointly with all other parameters, and disrupting them in isolation produces a model state that corresponds to no valid trained solution. This observation points to a broader gap in the explainability field that the authors hope future work will address, possibly through structured human evaluations of whether attention-enriched explanations align with human reasoning.

The implications extend beyond any single benchmark. Because both methods make no modality-specific assumptions, they are in principle directly applicable to text and images alike, though practical challenges remain. For images, the exponential cost of exact Shapley computation means only sampled approximations are feasible with the many patches of a vision transformer. For text, concept-based methods require carefully curated concept datasets, which can be resource-intensive to build. The authors also suggest that other attribution frameworks, such as Integrated Gradients, could be adapted to incorporate attention weights in future efforts. What the study ultimately delivers is a demonstration of versatility: attention weights, long dismissed by critics and oversold by enthusiasts, emerge as a flexible raw material that can enrich game-theoretic attributions and concept-based global explanations alike. As transformers continue to mediate high-stakes decisions in medicine, finance, and law, tools that balance theoretical rigor with practical efficiency, and that illuminate not just what a model decided but how its internal machinery arrived there, are becoming less of an academic luxury and more of a societal necessity.

Subject of Research: Integrating attention weights into explainable AI frameworks for transformer models in natural language processing and computer vision

Article Title: Integrating attention into explanation frameworks for language and vision transformers

Article References: Eggen, M., Lysnæs-Larsen, J., & Strümke, I. (2026). Integrating attention into explanation frameworks for language and vision transformers. Neural Computing and Applications, 38(18), Article 744. https://doi.org/10.1007/s00521-026-12257-y

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12257-y

Keywords: explainable AI, transformers, attention mechanism, Shapley values, concept activation vectors, TCAV, BERT, Vision Transformer, natural language processing, computer vision, interpretability, machine learning

The new study, published in the International Journal of Data Science and Analytics by Akilu Rilwan Muhammad, Ana Aguiar, and João Mendes-Moreira, presents a class-aware classification framework called Class-subspace. Its central innovation lies in borrowing an idea from cooperative game theory: the Shapley value. Originally developed to fairly divide a payout among players in a coalition, Shapley values have become one of the most trusted tools for explaining machine learning predictions through the SHAP method, short for SHapley Additive exPlanations. Rather than using SHAP merely to explain a model after the fact, the Porto team flips the workflow, using SHAP attributions to actively construct the model itself.

The reasoning is deceptively simple. Conventional transportation mode detection pipelines apply a single, globally selected feature set across every travel mode, optimising one subset of variables for the entire multi-class problem. But the researchers found that the features most relevant for distinguishing a bus journey were often sub-optimal for identifying a car trip. A single global feature set, they argue, necessarily compromises performance for certain classes. Class-subspace instead trains an initial classifier, computes mean absolute SHAP values for each feature and each class, ranks the features by their contribution to each mode’s prediction, and retains the minimal set of features that explains a cumulative threshold of 60 percent of each class’s relevance. Each transportation mode thus acquires its own compact feature subspace, and a specialised base learner is trained for every mode before their votes are aggregated into a final decision.

The technical pipeline behind the framework is substantial. The team worked with two real-world GPS trajectory datasets: SenseMyFEUP, collected from 227 participants in Porto, Portugal, using an Android application that sampled location roughly once per second, and the well-known GeoLife dataset from Microsoft Research Asia, gathered from 182 participants in Beijing over nearly five years. Rigorous preprocessing was essential. The researchers chained GPS points into trips using a 30-minute gap threshold, filtered out anomalous sessions that continued recording without movement, applied a spatial filter to keep only city-scale journeys, and capped implausible walking trips at eight kilometres. Segments shorter than 50 metres were discarded, removing nearly 39 percent of segments in the Porto data and about 14 percent in GeoLife.

Feature engineering drew on both the time and frequency domains. For each 100-step instance, the team computed statistics of speed, acceleration, jerk, and bearing rate, capturing the kinematic character of motion. They then applied the Fast Fourier Transform to convert these signals into the frequency domain, hypothesising that cyclic human movements such as walking or pedalling would produce distinctive high-frequency components, while smooth mechanical travel by car or bus would concentrate energy at low frequencies. In total, 80 features per instance were generated, with frequency-domain information encoded through the indices of the top ten spectral components rather than their magnitudes, capturing which frequency bands are active for each mode rather than how much energy they carry.

Evaluation was deliberately conservative. Instead of the random data splits common in the field, which can leak future patterns into training and inflate accuracy, the team used temporal splits, training on earlier data and testing on later periods. Against baselines including Sequential Forward Floating Selection, Mutual Information filtering, and the Boruta algorithm, Class-subspace achieved weighted ROC-AUC scores of up to 78.8 percent on the Porto dataset and 89.5 percent on GeoLife using the XGBoost classifier, performing comparably to established global feature selection methods. A formal statistical comparison using the Friedman test and Bonferroni-Dunn post hoc analysis confirmed that no baseline differed significantly from the proposed method.

What the framework sacrifices in raw accuracy gains, it returns in interpretability. An analysis of feature overlap across class-specific subspaces, measured with the Jaccard similarity index, revealed moderate overlap and a small universal core of speed-based statistics, including maximum, mean, standard deviation, and 85th-percentile speed, that appeared in every configuration. Beyond that shared core, each mode carried its own signature. Walking was captured almost entirely by speed statistics, consistent with its narrow and stable range. Cycling added bearing rate features reflecting frequent directional changes. Bus detection drew on jerk and bearing rate, echoing stop-and-go dynamics. Metro uniquely selected bearing rate distributions and acceleration frequency components, matching the confined, scheduled vibration patterns of rail transit. On the Porto data with random forest, only 7 of 32 distinct features were common to all classes, while 9 served exclusively as metro signatures.

The authors are candid about limitations. Bus and metro classes remained difficult on the Porto dataset, a challenge they attribute to the joint effect of class imbalance and class overlap rather than to feature selection itself, pointing to their earlier work characterising these issues. The computational cost of Shapley values is a known constraint, and the sensitivity of performance to the contribution threshold varied between datasets, with GeoLife remarkably stable across thresholds while the Porto data peaked sharply at 0.60. Future work, they suggest, could explore adaptive threshold selection through cross-validated grid search, Bayesian optimisation, or information-theoretic criteria, and extend the class-subspace concept to other multi-class domains.

The broader implications reach well beyond transportation research. As cities worldwide lean on crowdsensed mobility data for planning, emissions modelling, and infrastructure investment, the trustworthiness of the underlying classification models matters enormously. Class-subspace demonstrates that per-class interpretability need not come at the price of predictive power: a model can match the performance of globally optimised baselines while explicitly revealing which evidence supports each decision. In a field where reported accuracies above 90 percent sometimes rest on leaky evaluation protocols, the Porto team’s insistence on temporal splits and statistical rigor sets a standard. Their message to the machine learning community is clear: when classes are fundamentally different, stop forcing them to share the same lens, and let game theory hand each one its own.

Subject of Research: SHAP-driven class-aware machine learning for transportation mode detection from GPS trajectory data

Article Title: Class-subspace learning: a SHAP-driven framework for class-aware transportation mode detection

Article References: Muhammad, A. R., Aguiar, A., & Mendes-Moreira, J. (2026). Class-subspace learning: a SHAP-driven framework for class-aware transportation mode detection. International Journal of Data Science and Analytics, 22(1), Article 302. https://doi.org/10.1007/s41060-026-01263-x

Image Credits: AI Generated

DOI: 10.1007/s41060-026-01263-x

Keywords: transportation mode detection, SHAP values, machine learning, GPS trajectories, feature selection, Shapley values, urban mobility, smart mobility, interpretability, XGBoost, Fast Fourier Transform, class-aware classification