SparseRec Optimizes Recommender Systems for Edge Devices

Researchers have introduced a new method called SparseRec that significantly reduces the memory requirements for training recommender systems on resource-constrained devices. Published in the journal Data Science and Engineering, the study presents a lightweight embedding technique that maintains high recommendation accuracy while overcoming the computational inefficiencies of traditional sparse training paradigms. The method is particularly relevant for on-device applications, such as mobile phones, where large embedding tables typically consume excessive memory and slow down local model updates.

Recommender systems are ubiquitous in online platforms, including e-commerce and social media, where they predict user preferences by mapping users and items to dense vector representations. As the number of users and items grows, these dense embedding tables become a major bottleneck for memory consumption and computational speed. This challenge is acute in federated learning scenarios, where model parameters are updated locally on edge devices with limited resources before being aggregated on a central server. Existing solutions often struggle to balance memory efficiency with the need for effective parameter updates during the training process.

The study identifies three critical limitations in current embedding-pruning methods based on the Dynamic Sparse Training (DST) paradigm. First, despite using sparse embeddings during forward passes, these methods still compute dense gradients during backward passes, which undermines the efficiency gains of sparsity. Second, the weight exploration mechanism in traditional DST tends to prioritize users or items from the most recent batch, potentially reactivating parameters that do not significantly enhance overall model performance. Third, random uniform initialization of mask matrices creates unstructured connections that can impede post-training performance by ignoring the structural properties of the user-item interaction graph.

To address these issues, SparseRec introduces a novel approach that ensures sparsity throughout both forward and backward passes. The method begins by initializing the binary mask matrix using Nonnegative Matrix Factorization (NMF), a data-driven technique that produces sparse representations where important parameters are likely to remain nonzero. This initialization strategy allows the mask matrix to be set to a local optimum, effectively avoiding the inefficiencies associated with random uniform initialization. By leveraging the sparse adjacency matrix of user-item interactions, NMF creates a structured foundation for the embedding table that respects the underlying data distribution.

A key innovation of SparseRec is its use of sparse gradients during the backward pass. Instead of computing gradients for all parameters, the method selectively samples a subset of embedding vectors based on user and item frequencies. Gradients are calculated only for the active parameters and this sampled subset, ensuring that computational resources are focused on the most relevant parts of the model. This frequency-based sampling strategy is computationally efficient because probabilities are computed once and stored, avoiding iterative updates. The stochastic nature of the sampling, combined with a Softmax function, ensures that infrequent yet critical users or items are still adequately considered, promoting a balanced activation of parameters.

Furthermore, SparseRec improves the parameter regrowth process by accumulating gradients over several consecutive batches rather than relying on instantaneous gradients from a single batch. In recommender systems, each training sample only updates the embeddings for the specific user and item involved, meaning that instantaneous gradients may not reflect the true importance of a parameter for the overall model. By summing sparse gradients over a window of steps, SparseRec can better identify inactive parameters that, when reactivated, contribute meaningfully to model performance. This cumulative approach provides a more reliable signal for deciding which pruned parameters should be restored during the weight exploration phase.

The researchers evaluated SparseRec on three benchmark datasets: Gowalla, Yelp2018, and Movielens 10M, which cover different domains and data densities. The method was integrated with three base recommenders: XSimGCL, MGDCF, and LightGCN, to demonstrate its generalizability across different GNN-based architectures. The experiments compared SparseRec against several state-of-the-art lightweight embedding methods, including PEP, DSL, CIESS, BET, CERP, and DHE. Performance was measured using Recall@20 and NDCG@20, standard metrics for evaluating recommendation quality in top-k retrieval tasks.

Experimental results showed that SparseRec achieved state-of-the-art performance in 25 out of 27 experimental settings. The largest performance gain over the second-best baseline was 11.79%, observed on the Yelp dataset with a low density ratio of 0.0625 and the LightGCN base recommender. Statistical tests confirmed that these improvements were significant, with p-values less than 0.02 in the settings where SparseRec outperformed the best baseline. The method demonstrated particular robustness in low-density settings, where efficient parameter utilization is crucial for minimizing performance degradation in memory-constrained environments.

Analysis of the model components revealed that both NMF-based initialization and cumulative gradient regrowth contributed to the overall performance gains. When compared to random uniform initialization, NMF consistently improved the recommendation accuracy of the sparse network. Similarly, using cumulative gradients for regrowth resulted in better performance than using instantaneous gradients. The study also found a positive correlation between user/item frequency and the assigned embedding size, indicating that the method successfully allocates more parameters to frequent entities while still accommodating critical tail users and items.

Memory complexity analysis demonstrated that SparseRec uses less memory than conventional DST methods like DSL when the sampling ratio is set appropriately. The total memory usage for SparseRec is proportional to the sum of the sparse embedding table, the sparse gradients, and the cumulative gradient table. By keeping the sampling ratio below a specific threshold relative to the density, SparseRec ensures that the memory footprint remains lower than that of methods relying on dense gradients. This efficiency makes the method suitable for deployment on edge devices with heterogeneous memory budgets, where the sampling ratio can be tuned to match available resources without catastrophic performance degradation.

The study highlights the importance of maintaining sparsity throughout the entire training process, not just during inference. Traditional methods often fail to account for the computational cost of dense gradients, which can slow down on-device updates and increase energy consumption. By addressing this gap, SparseRec provides a more comprehensive solution for lightweight recommendation systems. The method’s ability to maintain performance at low parameter densities suggests its potential for broader application in resource-constrained settings, including mobile and WoT devices.

Despite its advantages, the study notes that the performance of lightweight embedding algorithms can vary depending on the dataset and base model. For instance, on the Movielens 10M dataset, increasing parameter density yielded only marginal performance improvements for some models, indicating that efficient parameter allocation is more critical than simply increasing the number of parameters. The researchers also acknowledge that while SparseRec improves memory efficiency, future work is needed to quantify its impact on latency and energy efficiency through hardware-level evaluations. This will provide a more comprehensive assessment of its suitability for real-world on-device deployment.

In conclusion, SparseRec offers a significant advancement in the field of lightweight recommender systems by integrating sparse initialization, sparse gradients, and cumulative regrowth into a unified framework. By overcoming the limitations of traditional DST methods, it enables more efficient training and inference on memory-constrained devices. The study’s findings underscore the potential of sparse training techniques to enhance the scalability and privacy of recommender systems, paving the way for more personalized and efficient user experiences on edge devices.

Beyond high-level performance metrics, the authors detailed specific optimization mechanics that governed the training procedure. The framework was trained for up to 500 epochs using the Adam optimizer with a weight decay of 0.0001 and Bayesian Personalized Ranking loss. Early stopping was initiated after 300 epochs, evaluating validation metrics every five epochs with a five-epoch patience threshold. Training leveraged large batch sizes of 8,000 interactions for Gowalla and Yelp2018, and 20,000 for the denser Movielens 10M, which accelerated training runtimes while maintaining recommendation precision. The optimization started with an initial learning rate of 0.01, decaying each epoch by 0.99 for LightGCN on Yelp and 0.995 in all other configurations until reaching a lower bound of 0.0005. To manage the prune-regrow schedule across training, a cosine annealing schedule modulated the pruning rate over time, and all newly restored parameters were initialized to zero so that reactivation would not destabilize the base recommender’s predictions. After training concluded, the sparse embedding table was quantized into signed 8-bit integers for deployment, then converted back to floating-point representations during evaluation.

The researchers also conducted granular hyperparameter investigations to establish practical operational guidelines for weight exploration and gradient sampling. Furthermore, evaluating the sampling ratio across fractional tiers of the upper theoretical limit indicated that recommendation quality remained stable when halving the sampling ratio from the boundary threshold to half that value. However, accuracy declined sharply once the sampling ratio dropped below a quarter of the threshold, confirming that maintaining at least half the boundary provides an optimal balance between peak memory reduction and recommendation accuracy.

Subject of Research: Sparse Gradient Training for Recommender Systems

Article Title: Sparse Gradient Training for Recommender Systems

Article References: Qu, Y., Qu, L., Chen, T., Zhao, X., Li, J., & Yin, H. (2026). Sparse Gradient Training for Recommender Systems. Data Science and Engineering, 11(1), 230-247. https://doi.org/10.1007/s41019-025-00327-5

Image Credits: AI Generated

DOI: 10.1007/s41019-025-00327-5

Keywords: Sparse, Gradient, Training, Recommender, Systems, scientific research