Transformers Can Reuse Hidden Token Representations Without Retraining

Large language models spend enormous computational effort building a separate contextualized representation for every token in a sequence, yet much of that work may be redundant. A new study published in Complex & Intelligent Systems introduces token representation reuse (TRR), an inference-time framework that finds highly similar hidden states inside a transformer and replaces selected ones with earlier anchor representations, all without retraining the model or touching a single parameter.

The approach, developed by Xinpeng Xu and colleagues, targets redundancy within the intermediate hidden-state spaces of transformer-based models, an area the authors describe as insufficiently explored as an inference-time representation property. Unlike token pruning or merging, which shorten or combine sequences, TRR preserves the original sequence length and positional structure. The model continues to process the same number of tokens; some of them simply carry representations borrowed from earlier positions that were judged sufficiently similar.

The mechanics are straightforward in principle. As the model processes an input, TRR identifies contextual token representations that are highly similar to one another and swaps selected hidden states for preceding anchor representations, operating under explicit similarity thresholds and a reuse budget that caps how much replacement can occur. The team also provides an end-to-end complexity analysis of the representation discovery step and characterizes the local conditions under which identical hidden states can, in principle, permit projection reuse.

That theoretical analysis comes with an important caveat the authors make explicit: equality between hidden states does not generally persist across subsequent transformer layers. Two tokens that share nearly identical representations at one layer can diverge at the next, which means the framework must treat reuse as a controlled, layer-local intervention rather than a global shortcut through the network’s computation.

To test the idea, the researchers evaluated TRR on four tasks—BoolQ, AGNews, SST-2, and TruthfulQA—using two models: Qwen2.5-1.5B-Instruct and Llama-3-8B. The results show representation reuse ratios of up to 12.40 percent, meaning roughly one in eight hidden states could be replaced with a reused anchor while downstream predictive performance was largely preserved.

The statistical picture is more nuanced than a simple

Subject of Research: Token representation reuse as an inference-time method for exploiting redundancy in transformer hidden states

Article Title: Exploiting representational redundancy in transformer hidden states through token representation reuse

Article References: Xu, X. (2026). Exploiting representational redundancy in transformer hidden states through token representation reuse. Complex & Intelligent Systems. https://doi.org/10.1007/s40747-026-02564-1

Image Credits: AI Generated

DOI: 10.1007/s40747-026-02564-1

Keywords: transformers, large language models, token representation reuse, hidden states, inference-time optimization, representational redundancy, Qwen2.5-1.5B-Instruct, Llama-3-8B, BoolQ, TruthfulQA, complexity analysis, model efficiency