Distilled sparsity-adaptive Mamba model boosts sequential recommendation performance

Sequential recommendation systems—the algorithms that predict what a user will click, buy, or watch next based on the history of their past interactions—have long been dominated by stacked Transformer architectures. But a new study argues that these attention-heavy models carry an invisible tax: when user data is sparse, as it almost always is in real-world settings, the deep layers of a Transformer-based recommender tend to homogenize their representations, blurring the very distinctions between users and items that make personalization possible. A team of researchers at Chongqing University of Technology in China has now proposed a remedy that combines two ideas rarely brought together in this domain: the linear-complexity Mamba state space architecture and a layer-wise self-distillation scheme designed to keep deep features semantically consistent. The resulting model, called DiSAM4Rec (Distilled and Sparsity Adaptive Mamba for Sequential Recommendation), is described in a paper published in the Journal of Intelligent Information Systems, and its authors report consistent gains over strong baselines across four benchmark datasets while consuming less computational power.

The problem the researchers set out to solve is one that practitioners know intimately: data sparsity. In any large-scale recommender, the vast majority of items are interacted with only a handful of times, and most users leave only short, fragmented traces of behavior. This “long tail” phenomenon means that the training signal available to a deep model is unevenly distributed, and the layers nearest the output—those responsible for the final ranking decisions—often receive gradients dominated by a small set of popular items. Over the course of training, the hidden representations produced by successive layers can drift toward one another and collapse into near-identical patterns, a degradation that the paper’s authors characterize as deep-layer feature homogenization. The consequence is twofold: the model’s expressive capability is impaired, because distinct user intents can no longer be separated in the embedding space, and its optimization efficiency suffers, because the extra depth that should refine features instead adds parameters without adding discriminative information.

One standard countermeasure is data augmentation—artificially expanding sparse interaction sequences through masking, cropping, reordering, or generative synthesis. Augmentation does help, but the DiSAM4Rec team points out a subtler failure mode: augmentation applied independently at different layers can break semantic consistency across the network’s depth. If layer three sees one augmented view of a user’s history and layer six sees a different one, the model is effectively being asked to align features that describe different objects. This can undermine robustness rather than improve it. Moreover, the researchers note, most existing architectures adopt a fixed structural complexity regardless of whether the dataset at hand is densely populated or extremely sparse, even though sparse datasets arguably demand different inductive biases than dense ones.

DiSAM4Rec’s first major component addresses this structural rigidity directly. Leveraging Mamba—a selective state space model introduced in late 2023 as a linear-time alternative to self-attention—the authors designed a Mamba-based Sparsity-Aware Dual-Branch module, abbreviated MSADB. The module contains two parallel processing paths: a dense branch optimized for extracting rich, fine-grained features when interaction data is plentiful, and a sparse branch tuned to the noisier, shorter, less informative sequences characteristic of low-density regimes. A dynamic switching mechanism decides, at the level of feature extraction, which branch (or how to weight their combination) should dominate for a given input context. Crucially, because Mamba processes sequences with a recurrence-like state update rather than an all-pairs attention matrix, its computational cost grows linearly with sequence length rather than quadratically. This means the dual-branch design can be deployed without the punishing overhead that a comparable Transformer-based dual architecture would incur, particularly on long user histories.

The second component targets representation degradation head-on. The authors introduce a Layer-wise Semantic Alignment Self-Distillation mechanism, or LSA-SD. In classical knowledge distillation, a compact “student” model learns from a larger “teacher.” In self-distillation, the model is its own teacher: shallower or intermediate layers are trained to mimic the outputs of deeper, more abstract layers. LSA-SD applies this principle layer by layer, aligning intermediate features with high-level abstract semantic spaces. The intuition is that the deepest layers of a well-trained recommender encode the most refined notion of user intent; by pulling intermediate layers toward that same semantic target during training, the mechanism prevents the intermediate representations from collapsing into homogeneous patterns. Deep layers remain consistent with one another, the gradient signal is enriched throughout the network, and the final embedding space retains the diversity of user interests that ranking accuracy depends on.

The two components are complementary in an appealing way. MSADB decides how to extract features under varying data conditions; LSA-SD ensures that whatever is extracted survives the journey through depth without degrading. The sparsity adaptation handles the external environment—the dataset—while self-distillation handles the internal dynamics of optimization. Together, they form what the authors present as a coherent answer to the twin challenges of sparsity-induced degradation and fixed architectural complexity.

To validate the design, the researchers conducted extensive experiments on four real-world benchmark datasets drawn from different domains: ML-1M (the MovieLens movie rating dataset), and Beauty, Fashion, and Yelp, representing Amazon product reviews and local business reviews respectively. These datasets span a range of sparsity levels and interaction-length distributions, providing a meaningful test of whether the sparsity-adaptive machinery actually pays off. The results, as reported in the paper, show DiSAM4Rec outperforming a slate of competitive baseline models with average improvements of 4.12 percent on HIT@10, 9.24 percent on NDCG@10, and 9.97 percent on MRR@10. These metrics capture complementary aspects of recommendation quality: HIT@10 measures whether a relevant item appears anywhere in the top ten suggestions, NDCG@10 rewards placing relevant items near the top of that list, and MRR@10 (mean reciprocal rank) emphasizes how early the first relevant item appears. Gains of nearly ten percent on reciprocal rank suggest that DiSAM4Rec is not merely finding more relevant items but finding them sooner—precisely what matters in latency-sensitive production systems.

Equally notable, the authors emphasize, is that these accuracy improvements come with lower computational overhead rather than higher. This inverts the familiar trade-off in which better recommendation quality is purchased with bigger models and longer training times. The linear complexity of the underlying Mamba backbone, combined with a distillation scheme that requires no additional teacher network at inference time, keeps the model compact. The researchers also report ablation studies dissecting the contribution of each component, confirming that removing either the dual-branch sparsity adaptation or the layer-wise semantic alignment degrades performance, and analyses showing how the model’s behavior changes across datasets with different interaction densities.

The work sits within a rapidly growing research current. Since the original Mamba architecture was proposed by Albert Gu and Tri Dao as a selective state space model for sequence modeling, variants have swept through vision, time-series analysis, and recommendation research. Prior efforts such as Mamba4Rec demonstrated that selective state space models could serve as efficient sequential recommenders, and follow-ups have explored bidirectional state spaces and selective gating. DiSAM4Rec distinguishes itself, the authors argue, by being the first to combine the Mamba backbone with explicit sparsity-adaptive structural switching and self-distillation, treating sparsity not as a nuisance to be augmented away but as a property the architecture itself should respond to. The self-distillation component, meanwhile, draws on a line of theoretical work suggesting that self-distillation can act as a form of implicit ensemble learning, improving generalization without extra parameters.

The broader implications reach beyond academic benchmarks. Sequential recommenders power the feeds of e-commerce platforms, streaming services, and social networks, and their behavior on sparse data shapes what the long tail of items—and the long tail of users—ever sees. A model whose deep layers homogenize under sparsity tends to fall back on popularity-driven suggestions, reinforcing feedback loops that concentrate exposure on already-popular content. Architectures that preserve representational diversity under sparse conditions could, in principle, produce more varied and individually tailored recommendations while also reducing the energy and hardware cost of serving them. The DiSAM4Rec authors note that their code and preprocessing scripts are publicly available on GitHub, and that the raw datasets they used are freely accessible from the MovieLens, Amazon, and Yelp repositories, lowering the barrier for other groups to reproduce and extend the results.

Limitations and open questions remain, as they do for any single paper. The sparsity-aware switching mechanism introduces hyperparameters governing the balance between dense and sparse branches, and the paper’s evaluation, while broad, is confined to four datasets with particular demographic and domain characteristics. Whether the same mechanisms transfer to multi-modal recommendation settings—where text, images, or audio accompany item identifiers—or to extremely long session histories at industrial scale are questions for future work. The authors themselves situate the contribution within a broader agenda of making recommender systems adaptive to the statistical realities of the data they consume rather than assuming uniform conditions.

Still, the study offers a concrete demonstration that the post-Transformer wave of sequence architectures is beginning to mature from raw efficiency claims into task-specific innovations. By pairing Mamba’s linear-time sequence processing with a training signal explicitly designed to resist representation collapse, DiSAM4Rec shows that architectural adaptivity and optimization hygiene can be engineered together—and that the combination can deliver measurable accuracy gains at reduced cost. For a field where a few percentage points of NDCG translate into substantial real-world impact, that is a result likely to attract attention.