Source-linked AI summary
Reinforcement Learning to Optimize Long-term User Engagement in Recommender Systems
Lixin Zou, Long Xia, Zhuoye Ding, Jiaxing Song, Weidong Liu, Dawei Yin
TL;DR
Feed-streaming recommenders need to optimize long-term engagement beyond instant metrics, but complex behaviors and unstable offline policy learning make this difficult. FeedRec addresses these challenges with a hierarchical-LSTM Q-Network and an environment-simulating S-Network. Experiments on synthetic and real-world data show effectiveness over state-of-the-art baselines.
Problem
Long-term engagement combines difficult-to-model instant and delayed behaviors, while effective offline policy learning remains challenging for recommender systems.
Method
FeedRec formalizes feed recommendation as an MDP, models complex user behavior with a hierarchical-LSTM Q-Network, and uses an S-Network to simulate environments during offline learning.
Results
FeedRec outperforms state-of-the-art baselines for long-term engagement optimization on synthetic and real-world e-commerce data.
Takeaways & Limitations
The framework directly targets long-term user engagement while handling versatile behavior modeling and offline policy-learning stability.
Takeaways & Limitations
The simulation assumes that leave, stay, or revisit feedback is determined by recommendation-list mean entropy and follows diversity-based assumptions.
Abstract
from arXiv · showhide
Recommender systems play a crucial role in our daily lives. Feed streaming mechanism has been widely used in the recommender system, especially on the mobile Apps. The feed streaming setting provides users the interactive manner of recommendation in never-ending feeds. In such an interactive manner, a good recommender system should pay more attention to user stickiness, which is far beyond classical instant metrics, and typically measured by {\bf long-term user engagement}. Directly optimizing the long-term user engagement is a non-trivial problem, as the learning target is usually not available for conventional supervised learning methods. Though reinforcement learning~(RL) naturally fits the problem of maximizing the long term rewards, applying RL to optimize long-term user engagement is still facing challenges: user behaviors are versatile and difficult to model, which typically consists of both instant feedback~(e.g. clicks, ordering) and delayed feedback~(e.g. dwell time, revisit); in addition, performing effective off-policy learning is still immature, especially when combining bootstrapping and function approximation. To address these issues, in this work, we introduce a reinforcement learning framework --- FeedRec to optimize the long-term user engagement. FeedRec includes two components: 1)~a Q-Network which designed in hierarchical LSTM takes charge of modeling complex user behaviors, and 2)~an S-Network, which simulates the environment, assists the Q-Network and voids the instability of convergence in policy learning. Extensive experiments on synthetic data and a real-world large scale data show that FeedRec effectively optimizes the long-term user engagement and outperforms state-of-the-arts.
1 INTRODUCTION
FeedRec addresses long-term engagement optimization in feed-streaming recommenders, where delayed behaviors are difficult to model and offline RL can be unstable. It combines hierarchical-LSTM behavior modeling with an environment-simulating S-Network and outperforms state-of-the-art baselines.
- Feed-streaming recommenders must optimize long-term engagement, including delayed metrics such as dwell time, browsing depth, and revisits, beyond instant clicks or conversions.
- RL is suitable for maximizing both instant and long-term rewards, but offline recommendation learning faces complex behaviors, costly online exploration, and limitations of MC and TD methods.
- FeedRec formalizes feed recommendation as an MDP and uses a Q-Network with hierarchical LSTM to model fine-grained, imbalanced, and long-term user behaviors.
- An S-Network simulates environments and generates user experiences to assist offline Q-learning and avoid convergence instability.
- FeedRec is evaluated on synthetic and real-world e-commerce data, with results showing effectiveness over state-of-the-art baselines for optimizing long-term engagement.
2 RELATED WORK
Prior recommender research emphasizes instant feedback, diversity, contextual bandits, and MDP-based interaction models, but lacks a systematic solution for delayed engagement metrics.
- Existing systems commonly optimize instant signals such as clicks, ratings, and dwell time, while diversity-based methods target satisfaction indirectly.
- Contextual bandits address online exploration and exploitation but assume relatively stable or smoothly drifting interests, which may not fit feed streaming.
- Prior work on delayed revisiting time does not provide a systematic approach to optimizing delayed metrics for overall user engagement.
3 PROBLEM FORMULATION
The paper formulates feed streaming recommendation as an MDP whose sequential feedback and reward design represent both instant and delayed user engagement. Rewards combine metrics such as clicks, browsing depth, and return time, while cumulative optimization targets future engagement.
- 3.1 Feed Streaming Recommendation: Feed streaming recommendation presents items sequentially, receives feedback such as clicks, purchases, skips, or leaving, and seeks to maximize long-term engagement.
- 3.1 Feed Streaming Recommendation: The interaction history contains the user, recommended items, feedback, and dwell times, and determines the next recommendation.
- 3.2 MDP Formulation of Feed Streams: The MDP uses states, actions, transitions, rewards, and a discount factor; its state is the browsing sequence and its action is the next recommended item.
- 3.2 MDP Formulation of Feed Streams: Transition uncertainty arises from user feedback, while available actions are initialized from recalled items and shrink as recommendations are shown.
- 3.3 User Engagement and Reward Function: RL reward design can directly combine instant and delayed engagement metrics, rather than relying only on immediate feedback.
- 3.3 User Engagement and Reward Function: Instant metrics are triggered by the current action and include clicks or purchases; click reward counts clicks in the current feedback.
- 3.3 User Engagement and Reward Function: Delayed metrics include browsing depth, system dwell time, and revisits, reflecting dependencies on previous behaviors and longer-term engagement.
- 3.3 User Engagement and Reward Function: Return time uses the interval between visits, and cumulative rewards optimize future browsing depth and visiting frequency.
4 POLICY LEARNING FOR RECOMMENDER SYSTEMS
The policy-learning framework estimates long-term engagement with a Q-Network and uses an S-Network simulator to support offline learning. The Q-Network models diverse behaviors with hierarchical LSTMs, while the S-Network generates simulated user responses and is trained alongside policy updates.
- Q-Network: Q-values estimate expected long-term engagement and guide recommendation selection through the maximum-valued item.The Q-Network approximates the action-value function because explicitly estimating every state-action pair is infeasible.
- Q-Network: The Q-Network represents user state from item history, user feedback, and dwell time before optimizing recommendations.Behavior feedback includes clicking, purchasing, skipping, leaving, and dwell time; item embeddings are transformed into feedback-dependent spaces.
- Q-Network: Hierarchical behavior layers assign separate LSTM pipelines to major behaviors such as clicks, skips, and purchases.Separate pipelines reduce dominance by frequent actions and preserve characteristics of sparse behaviors.
- Off-Policy Learning: Offline Q-Learning is difficult because logged data and function approximation can produce instability and divergence.The paper identifies the Deadly Triad as a source of danger in off-policy learning.
- Off-Policy Learning: The S-Network simulates feedback, dwell time, revisits, and leaving behavior to assist offline policy learning.It is a multi-head network with shared state-action features and task-specific output layers, trained from logged data with importance weighting.
- Off-Policy Learning: FeedRec alternates S-Network and Q-Network updates in an offline training procedure using logged and simulated experience.The procedure pretrains the simulator, samples experiences, updates the Q-Network, and then updates the S-Network adaptively.
5 SIMULATION STUDY
The simulation study tests whether FeedRec can directly optimize delayed engagement under linear and quadratic relationships between recommendation diversity and user behavior. Across training, FeedRec converges toward the engagement-optimal diversity pattern under both assumptions.
- Simulation Setup: The synthetic study uses known user-interest mechanisms to test whether FeedRec learns policies maximizing delayed engagement.The simulation evaluates browsing depth and return time as delayed engagement metrics.
- Diversity Assumptions: Under the linear assumption, higher entropy increases satisfaction, browsing depth, and system-use frequency.Return intervals are modeled as decreasing with recommendation-list entropy.
- Simulation Results: The figures encode training progress with blue-to-red interaction iterations and show average browsing depth or return time as dots.Dashed lines represent the corresponding entropy relationship, linear in panel (a) and quadratic in panel (b).
- Diversity Assumptions: Under the quadratic assumption, moderate entropy produces the highest user satisfaction.Both higher and lower entropy can reduce engagement relative to the preferred level.
- Simulation Setup: The simulator contains 1,000 users, 5,000 items, and 2M training episodes.User and item topic vectors have dimension 10, and interactions generate click, skip, leave, stay, and revisit feedback.
- Simulation Results: FeedRec converges to the best diversity under both linear and quadratic diversity assumptions by directly optimizing delayed metrics.The reported trajectories cover browsing depth and return time as training progresses.
6 EXPERIMENTS ON REAL-WORLD E-COMMERCE DATASET
On the JD e-commerce dataset, FeedRec was evaluated offline against supervised and reinforcement-learning baselines using multiple engagement metrics. It outperformed all baselines significantly, remained stable during off-policy training, and linked direct engagement optimization with greater recommendation diversity.
- Evaluation Setting: Offline policy evaluation used step-wise NCIS with capped importance ratios, training on the first 15 days and testing on the final 2 days.The clipping value c was set to 5 to control variance and bias.
- Comparison against baselines: FeedRec outperformed all baseline methods on all three evaluation metrics, with statistically significant improvements at p < 0.01.The comparison used supervised-learning and reinforcement-learning baselines on the real-world dataset.
- The influence of weight ω: Moderate weights on depth and return-time metrics improved cumulative clicks, while excessive weights overwhelmed click importance.The best cumulative-click performance occurred when ω was set to 0.005.
- The effect of S-Network: FeedRec remained stable at high performance as training continued, whereas DQN and DDPG-KNN peaked near 40 iterations before degrading rapidly.The comparison examined three engagement metrics under the same configuration.
- The relationship between user engagement and diversity: FeedRec’s policy favored recommending more diverse items while directly optimizing user engagement.The analysis sampled 300 state-action pairs with importance ratio ρ̄ > 0.01 and plotted diversity against engagement measures.
7 CONCLUSION
FeedRec addresses long-term engagement optimization in feed streaming recommendation by modeling complex user behaviors and supporting offline policy learning with an environment simulator.
- FeedRec targets long-term user engagement in feed streaming recommendation, where user feedback includes clicks, dwell time, and revisits.
- Its hierarchical RNN Q-Network models complex user behaviors for engagement optimization.
- Its S-Network simulates the environment and assists the Q-Network to address instability during policy learning.
- Experiments on synthetic datasets and a real-world e-commerce dataset demonstrate FeedRec’s effectiveness for feed streaming recommendation.