Source-linked AI summary
Reinforcement learning based recommender systems: A survey
M. Mehdi Afsar, Trafford Crump, Behrouz Far
TL;DR
The paper surveys reinforcement learning based recommender systems and classifies algorithms according to the field’s shift toward deep reinforcement learning. It reports that RL can recommend better video content and identifies instability risks and evaluation gaps.
Problem
Recommendation becomes more difficult as the available information and item lists grow.
Method
The paper presents a comprehensive survey of RLRSs, emphasizing DRL and classifying algorithms into RL- and DRL-based methods.
Results
RL can be employed to recommend better video content to YouTube users.
Takeaways & Limitations
The survey organizes RLRS research around the changing role of DRL and highlights important trends and challenges in the field.
Takeaways & Limitations
Combining function approximation, bootstrapping, and off-policy learning can cause instability and divergence in RL.
Abstract
from arXiv · showhide
Recommender systems (RSs) have become an inseparable part of our everyday lives. They help us find our favorite items to purchase, our friends on social networks, and our favorite movies to watch. Traditionally, the recommendation problem was considered to be a classification or prediction problem, but it is now widely agreed that formulating it as a sequential decision problem can better reflect the user-system interaction. Therefore, it can be formulated as a Markov decision process (MDP) and be solved by reinforcement learning (RL) algorithms. Unlike traditional recommendation methods, including collaborative filtering and content-based filtering, RL is able to handle the sequential, dynamic user-system interaction and to take into account the long-term user engagement. Although the idea of using RL for recommendation is not new and has been around for about two decades, it was not very practical, mainly because of scalability problems of traditional RL algorithms. However, a new trend has emerged in the field since the introduction of deep reinforcement learning (DRL), which made it possible to apply RL to the recommendation problem with large state and action spaces. In this paper, a survey on reinforcement learning based recommender systems (RLRSs) is presented. Our aim is to present an outlook on the field and to provide the reader with a fairly complete knowledge of key concepts of the field. We first recognize and illustrate that RLRSs can be generally classified into RL- and DRL-based methods. Then, we propose an RLRS framework with four components, i.e., state representation, policy optimization, reward formulation, and environment building, and survey RLRS algorithms accordingly. We highlight emerging topics and depict important trends using various graphs and tables. Finally, we discuss important aspects and challenges that can be addressed in the future.
1 INTRODUCTION
Recommender systems face information overload and limitations in traditional and deep-learning approaches, while reinforcement learning models recommendation as sequential user-system interaction. This survey reviews RLRS research, organizes methods through a four-component framework, and identifies trends and future directions.
- Traditional recommendation methods face cold start, limited novelty and diversity, scalability issues, low-quality recommendations, and high computational expense.
- Deep learning can model complex user-item relationships but is typically non-interpretable, data hungry, and computationally expensive.
- RL models recommendation as interaction with an environment, enabling reward-based learning for sequential decisions and long-term user engagement.
- DRL made RL applicable to recommendation problems with enormous state and action spaces, contributing to a changing research direction in RLRSs.
- The proposed RLRS framework comprises state representation, policy optimization, reward formulation, and environment building, and supports comparison, trend analysis, and future research directions.
- The survey reviews 97 RLRS articles collected through a multi-level search and screening process, while acknowledging that some relevant papers may have been missed.
2 PRELIMINARIES
This section introduces recommender systems and reinforcement learning, then formulates recommendation as an MDP and presents the paper’s RLRS framework.
- Recommender Systems: Recommender systems suggest items that may interest users, using collaborative filtering, content-based filtering, or hybrid methods.Collaborative filtering can suffer from sparsity, scalability, and gray-sheep problems; content-based filtering has limited content analysis, serendipity, and new-user shortcomings.
- Reinforcement Learning: RL models recommendation as a closed-loop interaction in which an agent observes states, takes actions, and receives rewards from the environment.RL problems involve no tutor, and actions influence both short-term and long-term outcomes.
- MDP Formulation: An RL problem is typically represented as an MDP tuple (S, A, R, P, γ), with a policy maximizing expected discounted cumulative reward.States, actions, rewards, transitions, and the discount factor define the formulation; in recommendation, an action recommends an item and a state represents user preferences and history.
- RL and DRL Algorithms: RL algorithms include tabular and approximate methods, while DRL uses deep learning for function approximation in large state spaces.The survey classifies RLRS algorithms into RL-based and DRL-based methods to reflect the field’s recent trend.
- RL and DRL Algorithms: RL includes dynamic programming, Monte Carlo, temporal-difference, Q-learning, Sarsa, and related methods, with policy and value iteration as dynamic-programming examples.Policy iteration repeatedly evaluates and improves policies, whereas value iteration stops policy evaluation after one sweep; policy evaluation can be computationally prohibitive.
- Challenges and RLRS Framework: The survey identifies instability from the Deadly Triad, sample inefficiency in model-free RL, and four common RLRS components.The four components are state representation, policy optimization, reward formulation, and environment building.
3 REINFORCEMENT LEARNING BASED RECOMMENDER SYSTEMS ALGORITHMS
This section classifies RLRS algorithms into RL-based and DRL-based methods and surveys them according to the proposed framework.
- Algorithm Classification: RLRS algorithms are first divided into RL-based and DRL-based methods, then surveyed with respect to the proposed RLRS framework.The classification and framework organize the algorithm review in this section.
3.1 RL-based RSs
RL-based recommender systems represent users, items, context, or interaction histories as states and optimize policies with tabular or approximate RL methods. Their main challenge is scalability as state and action spaces grow.
- Scope: RL-based RSs are surveyed as methods that do not use deep learning for policy optimization.The survey organizes these methods separately from DRL-based recommender systems.
- Policy Optimization: Tabular temporal-difference and dynamic-programming methods become impractical as state or item spaces grow, motivating approximation and state-space management.Fitted Q can scale to high-dimensional spaces, but increasing transition tuples can create substantial computational and memory overhead.
- State Representation: State representations range from recent interaction windows and item sequences to user, item, context, and latent-interest features.Examples include sliding windows, feature variables, song-vector concatenations, and belief states.
3.2 DRL-based RSs
DRL-based recommender systems use deep models to optimize policies over large recommendation spaces, with dense state representations and several policy-optimization families. The survey also finds strong reliance on offline and simulated environments.
- State Representation: SR3 is the dominant state-representation scheme for DRL-based RSs, while SR2 accounts for around 20% of these systems.Deep models are trained more effectively on dense, low-dimensional vectors.
- Policy Optimization: DRL policy-optimization algorithms are generally divided into value-based, policy-gradient, and actor-critic methods.DQN and its extensions are ruling value-based methods, while Q-network architectures must address large action spaces.
- Environment Building: 60% of DRL-based RSs belong to R2, and more than half use offline environment building; almost 40% use simulators and only 10% use online studies.The survey reports that simulation has become more popular while online evaluation is difficult or costly.
4 EMERGING TOPICS
Emerging RLRS research extends reinforcement learning to multi-agent, hierarchical, conversational, and knowledge-graph-based recommendation settings. These approaches decompose decisions, coordinate agents, or add structured information and explanations.
- Multi-agent RL: MARL applies multiple agents to cooperative, competitive, and mixed recommendation tasks, often using centralized training with decentralized execution.Applications include collaborator, Twitter mention, page-wise, whole-chain, and charging-spot recommendation.
- Hierarchical and meta-controller RL: Hierarchical RL separates recommendation into high-level goals and low-level actions, supporting conversational control, profile revision, channel selection, and large action spaces.DHCRS selects item categories at the high level and individual items within categories at the low level.
- Knowledge graph based RLRSs: Knowledge-graph-based RLRSs use graph information for recommendation accuracy and explainability, including reasoning paths that show why items were recommended.KGRE-Rec prunes graph actions using a user-conditional scoring function, while graph size creates scalability concerns.
5 OPEN RESEARCH DIRECTIONS
Open directions include slate recommendation, explainability, RL algorithm design, and stronger evaluation and reproducibility practices. The survey emphasizes that current datasets, metrics, environments, and reporting remain limited.
- Slate Recommendation: Slate recommendation remains challenging because RL methods must select lists of items rather than single actions, while existing approaches have important optimality or scalability limits.Slate-MDP cannot guarantee optimality, and SlateQ is restricted to two-stage systems with large-action-space limitations.
- Explainability: RLRS explainability is still underdeveloped despite its potential to improve user experience, trust, decision-making, and system debugging.Existing methods include intrinsic graph-path explanations and model-agnostic agents that generate and assess explanations.
- Environment and Evaluation: RLRS evaluation lacks dedicated metrics, public datasets, and unified simulators, while online studies are difficult and costly and offline data are static and biased.MovieLens is the most popular dataset, and the survey calls for a general-purpose simulator comparable to OpenAI Gym.
- Reproducibility: Only about 16% of RLRS researchers have shared implementation code, underscoring the need for broader code, dataset, parameter, seed, and significance-test reporting.The survey links these practices to accelerating progress in the field.
6 CONCLUSION
The survey organizes state-of-the-art RL-based recommender systems into RL- and DRL-based methods and analyzes them through a four-component framework. It concludes that RLRS research remains in its infancy, with substantial advancements still needed.
- The survey classifies RLRS algorithms into RL-based and DRL-based groups.
- Its framework comprises state representation, policy optimization, reward formulation, and environment building.
- The authors survey algorithms according to these framework components.
- RLRS research is still in its infancy and needs substantial further advancement.
- The survey aims to help researchers understand RLRS concepts and advance the field.