Attention Weights Turned Into Powerful New Explanations for AI Transformers

Transformer models now sit at the heart of the most consequential artificial intelligence systems in the world, translating languages, classifying medical images, and powering the chatbots that millions of people consult daily. Yet for all their capability, these networks remain largely opaque, their decisions buried in billions of learned numerical parameters that no human can directly inspect. A new study from researchers at the Norwegian University of Science and Technology, published in Neural Computing and Applications, offers a fresh attack on this transparency problem by turning the transformer’s own attention mechanism into a rigorous engine for explanation. Rather than treating attention weights as a curiosity or a contested shortcut to interpretability, the team shows how they can be systematically woven into two of the most respected frameworks in explainable artificial intelligence, producing methods that work for both text and images.

The attention mechanism is the defining innovation of the transformer architecture. When a transformer processes a sentence or an image, it breaks the input into tokens, words in the case of language and small patches in the case of pictures. For every token, the model computes attention weights that describe how much that token should look at every other token when building its internal representation. These weights form a rich, model-internal map of token influence, and because they are computed explicitly during every forward pass, they come with a seemingly natural interpretation: high attention suggests that one token matters to another. This interpretability has fueled a long-running debate. Some researchers have argued that attention weights reveal genuine linguistic and visual structure, with studies of models like BERT showing heads that track syntactic relations or fixed positional patterns. Others, most famously in the paper titled Attention is not Explanation, have demonstrated that attention weights often fail to correlate with the features that actually drive a model’s output.

The Norwegian team, led by Marte Eggen together with Jacob Lysnæs-Larsen and Inga Strümke, sidesteps this binary argument with a pragmatic insight. Attention weights, they note, do not directly determine model outputs, but they do encode explanatory information about how tokens interact. The question is not whether attention alone explains a model, but whether attention can meaningfully improve existing explanation methods when integrated into them. To answer this, the researchers developed two novel methods, each targeting a fundamentally different aspect of model behavior. The first produces local explanations, attributions that justify a single prediction. The second produces global explanations, revealing which high-level human-understandable concepts the model has learned to represent.

The local method is grounded in cooperative game theory. The Shapley value, introduced by mathematician Lloyd Shapley in 1953, is the canonical solution for fairly dividing a game’s total payoff among its players. In machine learning, the players are input features and the payoff is the model’s prediction, so Shapley values tell you how much each token contributed to a particular classification. The catch is computational: exact Shapley values require evaluating the model on every possible coalition of features, a number that grows exponentially with input length. The researchers’ breakthrough was to redefine the game itself. Instead of measuring model predictions under masked inputs, which demands many forward passes, they construct a characteristic function directly from attention weights combined with their gradients, using a first-order Taylor approximation to relate internal attention dynamics to the output. They explored three formulations, one based on attention to the classification token, one on mutual attention between token pairs, and one using a non-linear maximum over pairwise interactions.

This design yields a striking efficiency advantage. Computing the attention-based Shapley attributions requires only a single forward pass to extract attention weights from all layers and heads, followed by one backward pass to obtain the gradients. Standard SHAP, by contrast, must mask input features over many iterations and perform numerous forward passes for each explanation. In evaluations on three standard text classification benchmarks, the Stanford Sentiment Treebank, IMDb movie reviews, and the Ag News topic corpus, the method held its own. The strongest variant, built on attention interactions with the classification token, performed comparably to the widely used SHAP framework when accounting for statistical uncertainty, while the kernel-based approximation of the attention-based game showed the best results among the sampled approaches. Pure attention weights without gradient information, included for completeness, performed notably worse, reinforcing the authors’ point that attention is most valuable as an ingredient within a principled framework rather than as a standalone explanation.

The second method tackles global explainability through concept activation vectors, an approach introduced by Kim and colleagues in their influential TCAV framework. The idea is elegant: train a simple linear classifier to distinguish the model’s internal activations for examples containing a concept, such as stripes, from examples that do not, then use the resulting direction in activation space to measure how sensitive the model’s output is to that concept. The researchers adapted this to transformers at the token level, exploiting the architecture’s so-called residual stream, a shared high-dimensional space through which all layers communicate. Because token representations live in this common space, a concept direction can be compared directly with each token’s representation, yielding a per-token concept sensitivity score. The novel step is weighting these scores by the attention that the classification token pays to each token, producing what the team calls the T-TCAV score.

The motivation for this weighting comes from a fascinating asymmetry the authors observed in vision transformers. In early layers, the classification token attends roughly uniformly across all image patches, while concept sensitivity varies widely from token to token. In deeper layers the pattern reverses: attention becomes sharply concentrated on a subset of tokens, but concept sensitivity spreads out, apparently because self-attention shares concept information among all tokens. By combining the two signals, the method lets attention provide the selectivity that concept sensitivity loses with depth, and lets concept sensitivity supply the semantic grounding that raw attention lacks. Experiments on ImageNet classes paired with concepts from the Broden dataset showed that the concept most semantically aligned with each target class, such as stripes for zebras or dots for dalmatians, tended to dominate in deeper layers, exactly as theory predicts. Heatmap visualizations of the token-level scores localized concept-relevant image regions, though the authors candidly report inconsistencies, with some concept-relevant regions receiving negative sensitivity scores in certain images of the same class.

The study also confronts an uncomfortable truth about how explanation methods are judged. The team evaluated their local methods using three quantitative metrics: F1 score, comprehensiveness, and sufficiency, all of which measure whether an explanation can reconstruct or invert the model’s prediction by masking tokens. These output-focused metrics, the authors argue, may not fully capture the value of explanations that reflect internal model dynamics rather than direct input-output influence. There is currently no widely accepted metric for how well an explanation mirrors a model’s internal representational machinery, and the researchers explain why simply perturbing attention weights to test this is not feasible: attention weights are learned jointly with all other parameters, and disrupting them in isolation produces a model state that corresponds to no valid trained solution. This observation points to a broader gap in the explainability field that the authors hope future work will address, possibly through structured human evaluations of whether attention-enriched explanations align with human reasoning.

The implications extend beyond any single benchmark. Because both methods make no modality-specific assumptions, they are in principle directly applicable to text and images alike, though practical challenges remain. For images, the exponential cost of exact Shapley computation means only sampled approximations are feasible with the many patches of a vision transformer. For text, concept-based methods require carefully curated concept datasets, which can be resource-intensive to build. The authors also suggest that other attribution frameworks, such as Integrated Gradients, could be adapted to incorporate attention weights in future efforts. What the study ultimately delivers is a demonstration of versatility: attention weights, long dismissed by critics and oversold by enthusiasts, emerge as a flexible raw material that can enrich game-theoretic attributions and concept-based global explanations alike. As transformers continue to mediate high-stakes decisions in medicine, finance, and law, tools that balance theoretical rigor with practical efficiency, and that illuminate not just what a model decided but how its internal machinery arrived there, are becoming less of an academic luxury and more of a societal necessity.

Subject of Research: Integrating attention weights into explainable AI frameworks for transformer models in natural language processing and computer vision

Article Title: Integrating attention into explanation frameworks for language and vision transformers

Article References: Eggen, M., Lysnæs-Larsen, J., & Strümke, I. (2026). Integrating attention into explanation frameworks for language and vision transformers. Neural Computing and Applications, 38(18), Article 744. https://doi.org/10.1007/s00521-026-12257-y

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12257-y

Keywords: explainable AI, transformers, attention mechanism, Shapley values, concept activation vectors, TCAV, BERT, Vision Transformer, natural language processing, computer vision, interpretability, machine learning