Deep reinforcement learning improves simulated short-video session duration and diversity

An offline simulation study found that a deep reinforcement learning recommender produced longer simulated viewing sessions and more diverse recommendation lists than matrix-factorization and supervised deep-neural-network baselines. The study evaluated sequential short-video recommendation using 12.53 million anonymized interaction records from the KuaiRec dataset; it did not test the system with live platform traffic or real users. The proposed framework used a Deep Q-Network, or DQN, to learn a sequential recommendation policy rather than optimize each recommendation solely for an immediate click.

In the reported model comparison, the DQN agent produced a simulated average session duration of 345.9 seconds, compared with 281.7 seconds for the supervised DNN and 254.1 seconds for matrix factorization. The DQN–DNN difference was reported as more than 22%. Immediate response was slightly lower for the reinforcement-learning agent.

Its click-through rate was 8.31%, compared with 8.52% for the supervised DNN. The DQN nevertheless recorded a higher intra-list diversity score, which measures dissimilarity among recommended items: 0.72 for the DQN, 0.65 for matrix factorization and 0.61 for the DNN. These results concern simulated recommendations and do not establish how the system would perform in a live platform environment.

The researchers formulated recommendation as a Markov Decision Process. A user state represented historical interactions, current-session activity, contextual information and multimodal candidate-video features; an action selected a video; and the reward combined immediate and delayed engagement signals. The tested reward incorporated clicks, watch ratio and social interactions.

Evaluation separately measured click-through rate, session duration and intra-list diversity so that short-term response could be compared with longer-term and diversity-oriented outcomes. User and video information was represented at several levels. A coarse-grained profile summarized long-term preferences and demographic data, while a session-level representation emphasized recent activity.

Video representations combined visual, textual and audio information. The framework also used a pretrained LLaMA-2 7B model to generate reasoning about a user’s five most recent interactions; a pretrained Sentence-BERT encoder then converted that text into a numerical state representation. Candidate-video and contextual features were combined with the user-state information.

To handle an action space containing 10,728 videos, the implementation used two stages. A fast dual encoder retrieved the top 100 candidates, after which the DQN re-ranked them. Invalid actions were masked, and an epsilon-greedy strategy supported exploration during training.

The paper’s broader architecture also describes data and feature engineering, reinforcement-learning policy optimization and an online recommendation and interaction layer, although the reported experiments were conducted offline rather than with live recommendation traffic. For the offline evaluation, the researchers split the KuaiRec interactions chronologically, allocating 80% to training, 10% to validation and 10% to testing to avoid future-behavior information leakage. A three-layer multilayer perceptron served as the user simulator, predicting feedback such as clicks and watch duration after a recommendation, then generating a reward and the next state.

The DQN used a 256-128-64 multilayer perceptron with ReLU activations and Adam optimization. This setup permitted repeated policy evaluation without exposing people to experimental recommendations, but errors in the simulator could bias the results. Ablation experiments assessed both the reward design and the language-model-derived reasoning features.

A DQN rewarded only for clicks achieved an 8.45% click-through rate and a 289.5-second session duration. A watch-time-only variant reached 330.1 seconds but had a 7.95% click-through rate. The combined-reward agent recorded the strongest reported balance between the two measures, with 8.31% click-through and 345.9 seconds of session duration.

When chain-of-thought features were masked from the state, average session duration fell by approximately 12% within the simulation. These comparisons support the contribution of the tested components in this experimental setup, rather than establishing a general causal mechanism. A separate binary model classified recommendations as likely to produce high or low engagement.

It achieved 90.5% accuracy, with 4,250 true positives and 4,800 true negatives. For the high-engagement class, precision was 0.955 and recall was 0.850; the class-specific F1 scores were 0.900 and 0.909. This module supplied an auxiliary signal for reward construction, and its classification performance was evaluated separately from the recommender’s click, duration and diversity results.

The paper’s user-session example was hypothetical and illustrated a projected content allocation rather than an observed user outcome. The principal limitation was reliance on an offline simulator calibrated from historical KuaiRec interactions, which may not reproduce behavior under live conditions. The authors therefore called for online A/B testing before deployment.

They also identified the computational latency of language-model-based reasoning as a challenge for real-time recommendation and proposed model distillation as a direction for future work. Additional research was described as necessary for more advanced reinforcement-learning algorithms, broader use of multimodal features and signals such as explicit “not interested” feedback and multi-day retention.

Subject of Research: Offline simulation of a deep reinforcement learning framework for long-term short-video recommendation

Article Title: Long-term engagement-oriented short video recommendation via deep reinforcement learning

Article References: Di, J. (2026). Long-term engagement-oriented short video recommendation via deep reinforcement learning. Discover Artificial Intelligence, 6(1), Article 1269. https://doi.org/10.1007/s44163-026-02098-y

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02098-y

Keywords: short-video recommendation, deep reinforcement learning, Deep Q-Network, KuaiRec, session duration, recommendation diversity, click-through rate, offline simulation, user modeling, Long-term, engagement-oriented, short