Source-linked AI summary
Deep Reinforcement Learning based Recommendation with Explicit User-Item Interactions Modeling
Feng Liu, Ruiming Tang, Xutao Li, Weinan Zhang, Yunming Ye, Haokun Chen, Huifeng Guo, Yuzhou Zhang
TL;DR
Many recommendation methods treat preferences as static and focus on immediate feedback, leaving dynamic interaction and long-term rewards unmodeled. DRR uses an Actor-Critic framework and explicit user-item interaction representations to make recommendations sequentially. Experiments on four real-world datasets report that DRR outperforms state-of-the-art competitors.
Problem
Many prior recommendation methods treat recommendations as static and optimize immediate feedback, neglecting dynamic user interaction and long-term rewards.
Source: §I · p. 1Method
DRR models recommendation as sequential decision making with an Actor-Critic framework and a state representation module for explicit user-item interactions.
Source: §I · p. 2Results
DRR outperforms state-of-the-art competitors in experiments on four real-world datasets.
Source: §I · p. 2, §VI · p. 10Takeaways & Limitations
The paper concludes that sequential Actor-Critic recommendation with explicit user-item interaction modeling can achieve superior performance to state-of-the-art methods.
Source: §VI · p. 10
Abstract
from arXiv · showhide
Recommendation is crucial in both academia and industry, and various techniques are proposed such as content-based collaborative filtering, matrix factorization, logistic regression, factorization machines, neural networks and multi-armed bandits. However, most of the previous studies suffer from two limitations: (1) considering the recommendation as a static procedure and ignoring the dynamic interactive nature between users and the recommender systems, (2) focusing on the immediate feedback of recommended items and neglecting the long-term rewards. To address the two limitations, in this paper we propose a novel recommendation framework based on deep reinforcement learning, called DRR. The DRR framework treats recommendation as a sequential decision making procedure and adopts an "Actor-Critic" reinforcement learning scheme to model the interactions between the users and recommender systems, which can consider both the dynamic adaptation and long-term rewards. Furthermore, a state representation module is incorporated into DRR, which can explicitly capture the interactions between items and users. Three instantiation structures are developed. Extensive experiments on four real-world datasets are conducted under both the offline and online evaluation settings. The experimental results demonstrate the proposed DRR method indeed outperforms the state-of-the-art competitors.
I. INTRODUCTION
The introduction argues that recommendation should account for users’ changing preferences and the long-term rewards of items, not only static preferences and immediate feedback. It presents DRR, an Actor-Critic framework with explicit user-item interaction modeling, and reports that it outperforms state-of-the-art competitors on four real-world datasets.
I. INTRODUCTION
- Prior methods often assume user preferences are static, although previous items can affect users’ choices of subsequent items.The introduction describes rating patterns after consecutive positive or negative feedback as evidence of changing preferences.Source: §I · p. 1
- Recommendation methods that maximize immediate feedback can overlook items with small immediate rewards but larger long-term benefits.The introduction contrasts news items that may earn equal immediate engagement but differ in what users may read afterward.Source: §I · pp. 1–2
- DRR treats recommendation as sequential decision making and uses an Actor-Critic scheme to account for immediate and long-term rewards.The framework includes a state representation module and three network structures for explicitly modeling user-item interactions.Source: §I · p. 2
- DRR outperforms state-of-the-art competitors in experiments on four real-world datasets.The introduction reports extensive experiments but does not give specific metrics here.Source: §I · p. 2
A. Non-RL based Recommendation Techniques
- Non-RL recommendation methods include content-based filtering, collaborative filtering, matrix factorization, logistic regression, factorization machines, deep learning, and contextual bandits.The cited overview identifies static preference assumptions and optimizing immediate rewards while ignoring long-term benefits as limitations of these methods.Source: A. Non-RL based Recommendation… · pp. 2–3
B. RL based Recommendation Techniques
DRR recommendation is framed as a sequential MDP interaction, with user state, recommender action, and feedback linked across timesteps. The section also contrasts prior RL approaches and their limitations.
B. RL based Recommendation Techniques
- Model-free recommendation methods are policy-based or value-based, while model-based methods are described as too costly for large candidate-item spaces.Policy-based methods may face a gap between continuous action spaces and discrete items; value-based methods evaluate Q-values across all actions.Source: B. RL based Recommendation Techniques · p. 3
- DRR explicitly models user-item interactions to learn state representations for large-scale recommendation.The framework is proposed against the state-representation limitations of earlier RL methods.Source: B. RL based Recommendation Techniques · p. 3
III. PRELIMINARIES
- The MDP represents recommendation as a sequence of user states, continuous parameter-vector actions, and feedback-based rewards.The action scores candidate items for ranking, and user feedback updates the state and supplies rewards.Source: §III · pp. 3–4
- The discount rate γ controls the present value of long-term rewards, from ignoring them at γ = 0 to weighting them equally with immediate rewards at γ = 1.The recommender seeks cumulative rewards across the recommendation procedure.Source: §III · pp. 3–4
IV. THE PROPOSED DRR FRAMEWORK
DRR combines an Actor-Critic scheme with an explicit state representation module. Its Actor summarizes a user’s recent positive-item history into a state, then produces a ranking action.
IV. THE PROPOSED DRR FRAMEWORK
- DRR uses an Actor-Critic framework and an explicit state representation module to model user-item interactions.The framework is presented to address the lack of sequential modeling or long-term rewards in conventional recommendation techniques.Source: §IV · p. 4
- A recommendation episode that ends before T timesteps uses its actual length.Source: §IV · p. 4
1) The Actor network:
- The Actor takes embeddings of a user’s latest positively interacted items and applies the state representation module to produce a summarized user state.If a recommended item receives positive feedback, it enters the history used for the next state; otherwise, that history remains unchanged.Source: 1) The Actor network · p. 4
- The Actor transforms the user state into a continuous parameter-vector action used to rank items, and recommends the top-ranked item.The framework also adopts ε-greedy exploration.Source: 1) The Actor network · p. 4
2) The Critic network:
The Critic estimates the value of user-state/action pairs, while the state representation module supplies user-item interaction features through three alternative structures. DRR-p models item-item pairs, DRR-u adds user-item pairs, and DRR-ave pools item embeddings before modeling their interaction with the user.
2) The Critic network:
- The Critic approximates the state-action value Qπ(s, a) from the user state and action, outputs a scalar Q-value, and guides Actor updates toward higher values.Its parameters are updated using temporal-difference learning, with target Actor and Critic networks also adopted.Source: 2) The Critic network · p. 5
3) The State Representation Module:
- DRR-p concatenates cloned item representations with n(n −1)/2 element-wise product features to encode pairwise item dependencies.A learned weight indicates each item's importance; p_a,b models interactions between items i_a and i_b.Source: 3) The State Representation Module · p. 5
- DRR-u combines item-item interactions with pairwise user-item interactions by incorporating the user embedding into the state representation.This addresses DRR-p's omission of user-item interactions.Source: 3) The State Representation Module · p. 5
- DRR-ave removes item-position effects by weighted-average-pooling item embeddings, modeling the pooled vector's interaction with the user, and concatenating both with the user embedding.Its state has dimensionality 3k.Source: 3) The State Representation Module · p. 6
B. Training Procedure of the DRR Framework
DRR is trained with DDPG using user interaction histories as training data. Training alternates between generating transitions from recommendations and updating the Actor, Critic, and target networks.
B. Training Procedure of the DRR Framework
- DDPG trains DRR from users' interaction histories through recommendation actions, user feedback rewards, and updated user states.The recommender follows its current policy after observing the state, then updates that policy based on feedback.Source: B. Training Procedure of the DRR… · p. 6
- During transition generation, the recommender selects an action with ε-greedy exploration, recommends an item, calculates its feedback reward, updates the user state, and stores the transition.The transition stored in replay buffer D is (s_t, a_t, r_t, s_t+1).Source: B. Training Procedure of the DRR… · p. 6
- During model updating, prioritized replay samples a minibatch for Actor and Critic updates, followed by soft replacement of target-network parameters.The sampling is described as an importance-sampling strategy.Source: B. Training Procedure of the DRR… · p. 6
3 Initialize replay buffer D
The training algorithm builds transitions from recommendations and feedback, then uses prioritized replay to update the Actor and Critic. Target-network parameters are softly replaced.
3 Initialize replay buffer D
- A positive reward shifts the history window to include the recommended item, while a non-positive reward leaves the history unchanged.The updated transition is stored in replay buffer D.Source: §3 · p. 6
- Prioritized experience replay samples a minibatch of N transitions from D for model updating.The algorithm specifies transitions (s_i, a_i, r_i, s_i+1).Source: §3 · p. 6
- The algorithm updates the Critic and Actor from sampled transitions, then softly replaces target-network parameters.The target update uses τθ + (1 −τ)θ′.Source: §3 · p. 6
C. Evaluation
The evaluation uses public offline datasets because direct online experiments pose commercial risk and deployment costs. Offline evaluation reranks items in each logged session using the learned policy without updating model parameters.
C. Evaluation
- Because direct recommender-user experiments carry commercial risk and costly deployment, evaluation uses public offline datasets with offline and online evaluation procedures.The passage describes these as two evaluation approaches conducted on public offline datasets.Source: C. Evaluation · p. 7
1) Offline evaluation:
- Offline evaluation iteratively reranks only items present in each logged session, selecting them according to actions from the learned policy and using recorded user feedback as rewards.The candidate set is restricted because logged feedback is available only for items in the session.Source: 1) Offline evaluation · p. 7
- After recommending an item, offline evaluation removes it from the session candidate set and updates the user state, while keeping model parameters fixed.The remaining candidates are then available for subsequent recommendations in the session.Source: 1) Offline evaluation · p. 7
2) Online evaluation with environment simulator:
The simulated online evaluation uses a pretrained PMF model to predict feedback for unrated items. Model parameters update during evaluation, with learned policies reset before each recommendation session for fair comparison.
2) Online evaluation with environment simulator:
- A pretrained PMF model simulates user feedback for recommended items that users have not rated.The policy parameters update during simulated evaluation, then reset to the training-stage policy before each session.Source: 2) Online evaluation with environment… · p. 7
A. Datasets and Evaluation Metrics
The evaluation uses four real-world rating datasets, offline ranking metrics, and simulated-online accumulated rewards. It compares the proposed methods with conventional recommenders, a DRR variant, and online bandit baselines.
A. Datasets and Evaluation Metrics
- Four datasets cover movie, music, and joke ratings, ranging from 100,000 MovieLens ratings to over 1.7 million Jester ratings.Yahoo! Music contains over 0.36 million ratings; MovieLens (1M) contains one million.Source: A. Datasets and Evaluation Metrics · p. 7
- Offline performance is measured by Precision@k and NDCG@k, while simulated-online performance uses total accumulated rewards.Ratings are discrete from 1 to 5 except for Jester, whose ratings range from -10.0 to +10.0.Source: A. Datasets and Evaluation Metrics · p. 7
B. Compared Methods
- Offline baselines include Popularity, PMF, SVD++, and DRR-n; online baselines include LinUCB, HLinUCB, and DRR-n.DRR-n concatenates item embeddings to represent the user state and serves as a comparison for the proposed state representation module.Source: B. Compared Methods · p. 8
C. Experimental Settings
The experiments tune and evaluate DRR across offline and simulated-online settings. The proposed interaction structures outperform baselines in most offline cases and deliver higher simulated-online rewards than the online baselines.
C. Experimental Settings
- Training uses 80% of each user's interactions, five latest positive items, 100-dimensional pretrained embeddings, and a discount rate γ of 0.9.Repeated items are excluded within an episode; Adam with L2 regularization is used, and normalized ratings provide rewards from interaction logs or simulator predictions.Source: C. Experimental Settings · p. 8
1) Offline Evaluation Results and Analysis:
- The proposed DRR-p, DRR-u, and DRR-ave structures outperform baselines in most cases; DRR-n, which concatenates item embeddings to represent the state, performs worse than those proposed methods.Source: 1) Offline Evaluation Results and… · p. 8
2) Simulated online evaluation results and analysis:
- In simulated online evaluation, the proposed methods deliver higher rewards than LinUCB, HLinUCB, and DRR-n; DRR-ave performs best among the proposed structures.The authors attribute the comparison to dynamic adaptation and long-term rewards, and to the proposed state representations outperforming DRR-n's fully connected network.Source: 2) Simulated online evaluation results… · p. 9
E. Parameter Study
DRR combines sequential Actor-Critic recommendation with explicit user-item interaction modeling, and experiments report advantages over state-of-the-art competitors. Episode length and recommendation behavior are also examined in parameter and case studies.
E. Parameter Study
- Performance peaks at T = 10 on MovieLens and T = 20 on Yahoo! Music, so episode length should be set to balance exploration and exploitation.The study presents DRR-ave; DRR-p and DRR-u are reported to show similar observations.Source: E. Parameter Study · p. 9, Figure 7 · p. 9
F. Case Study
- After two negative recommendations, DRR-ave recommends a risk-free movie matching the genre of an earlier positively rated item, while LinUCB keeps exploring.The case study fixes the first three recommendations for one MovieLens user before comparing subsequent recommendations.Source: F. Case Study · pp. 9–10, Table VII · p. 10
VI. CONCLUSION
- Experiments on four real-world datasets show DRR outperforms state-of-the-art competitors in offline and online evaluation.The framework uses an Actor-Critic scheme and three state-representation structures to model user-item interactions.Source: §VI · p. 10