Source-linked AI summary
Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learning
Sijin Zhou, Xinyi Dai, Haokun Chen, Weinan Zhang, Kan Ren, Ruiming Tang, Xiuqiang He, Yong Yu
TL;DR
RL-based interactive recommendation must learn dynamic preferences from sparse feedback over large action spaces, making sample efficiency a central challenge. KGQR extends Q-learning with KG-guided candidate selection, enriched representations, and preference propagation, and it improves performance across two real-world datasets while using fewer interactions. Its demonstrated scope is limited to the evaluated datasets, with richer news or image domains left for future work.
Problem
RL-based IRS requires many interactions because user feedback is sparse and recommendation action spaces are large, while online exploration can harm user experience.
Method
KGQR extends DQN by using KG correlations for candidate selection, item and state representation, and preference propagation across correlated items.
Results
KGQR consistently achieves the best performance across tested settings on Book-Crossing and MovieLens-20M, improving Reward over strongest baselines by 3.2% and 5.3%, respectively.
Takeaways & Limitations
KGQR demonstrates that incorporating KG information can improve sample efficiency and recommendation performance in RL-based IRS.
Takeaways & Limitations
Experiments use two typical datasets; evaluation on richer domains such as news or images is left for future work.
Abstract
from arXiv · showhide
Interactive recommender system (IRS) has drawn huge attention because of its flexible recommendation strategy and the consideration of optimal long-term user experiences. To deal with the dynamic user preference and optimize accumulative utilities, researchers have introduced reinforcement learning (RL) into IRS. However, RL methods share a common issue of sample efficiency, i.e., huge amount of interaction data is required to train an effective recommendation policy, which is caused by the sparse user responses and the large action space consisting of a large number of candidate items. Moreover, it is infeasible to collect much data with explorative policies in online environments, which will probably harm user experience. In this work, we investigate the potential of leveraging knowledge graph (KG) in dealing with these issues of RL methods for IRS, which provides rich side information for recommendation decision making. Instead of learning RL policies from scratch, we make use of the prior knowledge of the item correlation learned from KG to (i) guide the candidate selection for better candidate item retrieval, (ii) enrich the representation of items and user states, and (iii) propagate user preferences among the correlated items over KG to deal with the sparsity of user feedback. Comprehensive experiments have been conducted on two real-world datasets, which demonstrate the superiority of our approach with significant improvements against state-of-the-arts.
1 INTRODUCTION
Interactive recommender systems model recommendation as sequential decision-making, but RL methods face dynamic preferences, sparse feedback, large action spaces, and poor sample efficiency. KGQR incorporates item correlations to improve candidate selection, representations, preference propagation, and empirical performance with fewer interactions.
- IRS differs from traditional recommendation by delivering items sequentially and using user feedback to determine subsequent decisions.
- MAB approaches often assume unchanged user preferences, whereas IRS seeks to model dynamic preference transitions and optimize long-term utility.
- DRL training is sample-inefficient because recommendation involves large item-state search spaces and trial-and-error interaction.
- KG represents item relations and provides side information that can support recommendation decisions under sparse feedback and limited interactions.
- KGQR integrates graph learning with sequential decision making, propagates feedback among correlated items, enriches representations, and narrows candidate selection.
- KGQR improves RL-based IRS sample efficiency, with experiments on two real-world datasets outperforming state-of-the-art methods using fewer user-item interactions.
2 RELATED WORK
Prior recommender research includes KG-enhanced one-step models and RL-based interactive methods, but existing approaches have difficulty modeling long-term interaction dynamics or learning efficiently from limited history. KGQR combines KG semantic and structural information with IRS to address these limitations.
- Traditional KG Enhanced Recommendation: Traditional KG-enhanced recommenders use path-based, embedding-based, or hybrid methods, but path-based performance depends heavily on manually designed meta-paths.
- Traditional KG Enhanced Recommendation: Most traditional KG-enhanced methods perform one-step prediction and greedily optimize immediate feedback rather than long-term user utility.
- Reinforcement Learning in IRS: RL-based recommendation methods formulate interaction as an MDP and include model-based policy iteration alongside model-free PG, DQN, and DDPG approaches.
- Reinforcement Learning in IRS: Existing RL-based recommendation models suffer from low sample efficiency, require historical pretraining, and do not handle cold-start recommendation well.
- Reinforcement Learning in IRS: KGQR combines KG semantic and structural information with IRS to address sample-efficiency and cold-start limitations in prior RL-based models.
3 PROBLEM FORMULATION
IRS is formulated as an MDP in which an agent recommends items from interaction history and seeks maximum cumulative utility. Deep Q-learning estimates action values, while KG supplies item connections that can make Q-function learning more efficient.
- In feed-streaming recommendation, the agent repeatedly recommends an item, receives feedback, updates its observations, and continues until the user leaves.
- The recommendation process is modeled as an MDP, with a policy mapping states to items to maximize cumulative utility.
- Traditional methods optimize immediate reward, whereas DRL optimizes discounted long-term rewards across future interaction steps.
- Q-values evaluate recommending an item in a state, and the optimal Q-function follows the Bellman equation for maximum expected reward.
- Because state and action spaces are enormous, deep neural networks parameterized by θQ approximate the optimal Q-function.
- KG provides latent item connections and prior environment knowledge that can make Q-function learning more efficient than learning from scratch.
4 KGQR METHODOLOGY
KGQR integrates graph convolution, recurrent state modeling, KG-based candidate selection, and dueling double-Q learning into an interactive recommendation framework. KG information enriches item and state representations, narrows candidate actions, and supports Q-value-based policy learning.
- Framework overview: KGQR combines graph convolution, state representation, candidate selection, and Q-learning modules for sequential interactive recommendation.The framework updates recommendations from user feedback at each timestep.
- KG-enhanced state representation: Graph convolution recursively propagates information through connected KG entities to learn dense item and entity representations.The layer aggregates neighboring representations and integrates them with the target entity representation.
- KG-enhanced state representation: A GRU aggregates historical item embeddings into the current user-state representation used by the Q-network.The item embeddings supplied to the GRU are learned through graph convolution in the KG.
- Neighbor-based candidate selection: KG-based candidate selection samples k-hop neighbors of historically interacted items to dynamically restrict the action space.The selected candidate items are embedded through graph convolutional layers before Q-value prediction.
- Q-learning network: The Q-network uses dueling value and advantage functions together with double-Q learning to estimate long-term satisfaction and stabilize training.The value and advantage outputs are combined into the Q-value, while a delayed target network reduces overestimation.
- Framework scope: KGQR is presented with DQN as a running example, but its KG integration can also be incorporated into policy-gradient and DDPG models.The paper emphasizes KG integration into deep reinforcement learning rather than a single RL architecture.
5 EXPERIMENT
The experiments evaluate KGQR on two real-world datasets against interactive recommendation methods and examine sample efficiency and component contributions. The evaluation specifically tests comparative performance, interaction efficiency, and the effects of KG-enhanced modules.
- Research questions: Experiments on two real-world datasets compare KGQR with state-of-the-art interactive recommendation methods.This addresses the first research question.
- Research questions: The evaluation tests whether KGQR improves sample efficiency with fewer user-item interactions.This is the second stated research question.
- Research questions: Ablation-style analysis examines KG-enhanced state representation, GCN-based task-specific representation learning, and neighbor-based candidate selection.These components define the third research question.
5.1 Experimental Settings
The study evaluates KGQR in simulated interactive environments built from two KG-linked rating datasets under cold-start conditions. It measures episode rewards and cumulative precision and recall against traditional, bandit, and deep reinforcement learning baselines.
- Datasets: Book-Crossing and MovieLens-20M provide the two KG-linked real-world benchmark datasets used for evaluation.Book-Crossing ratings range from 0 to 10, while MovieLens-20M contains 20 million ratings ranging from 1 to 5.
- Datasets: The experiments leave evaluation on richer domains such as news or images for future work.The selected datasets reflect the paper’s focus on incorporating KG into RL-based interactive recommendation.
- Simulator: The simulator combines normalized predicted ratings with sequential feedback based on consecutive positive and negative responses.The trade-off parameter η is selected from {0.0, 0.1, 0.2}.
- Experimental protocol: Users are split 80% for training and 20% for testing, creating a cold-start setting without test-user click history at initialization.The simulator begins interaction by recommending a popular training-set item to each test user.
- Metrics: Evaluation uses average reward, average cumulative precision, and average cumulative recall over the interaction horizon.The study also applies the Wilcoxon signed-rank test to compare KGQR with baselines.
- Baselines: KGQR is compared with traditional recommendation, multi-armed bandit, and deep reinforcement learning baselines.The seven baselines include GreedySVD, GRU4Rec, LinearUCB, HLinearUCB, DDPGKNN, DDPGR, and DQNR.
- Baselines: Traditional KG-enhanced recommenders are excluded because the tested online cold-start setting requires modeling preferences during interaction.The paper states that these models cannot handle this cold-start scenario.
- Implementation settings: KGQR uses a maximal KG hop number of k = 2 because larger hops increase computational cost exponentially with limited performance improvement.Item embeddings are fixed at dimension 50 across models, while KGQR initializes KG embeddings with TransE.
5.2 Overall Performance (RQ1)
KGQR achieves the strongest recommendation performance across both datasets and all environment settings, outperforming RL-based baselines on Reward and Precision@32.
- KGQR consistently achieves the best performance across all environment settings on both datasets.
- 3.2% and 5.3% Reward improvements over the strongest baselines occur on Book-Crossing and MovieLens-20M, respectively.
- 0.5% and 1.9% Precision@32 improvements are reported on Book-Crossing and MovieLens-20M, respectively.
- Non-RL methods generally underperform RL-based methods because they model sequential preference and future rewards less effectively.
5.3 Sample Efficiency (RQ2)
KGQR requires substantially fewer interactions than other RL-based methods to reach the same reward, supporting improved sample efficiency through KG information.
- KGQR reaches the same performance as other RL-based methods using the fewest interactions.
- 17.3% and 13.6% of the second-efficient baseline’s interactions are sufficient for KGQR to reach test reward 2.0 on the two datasets.
- Sample efficiency is evaluated by comparing interactions needed to achieve rewards 0.5, 1.0, 1.5, and 2.0 for each dataset.
5.4 Analysis (RQ3)
Ablation analyses attribute KGQR’s gains to KG-enhanced representations, GCN preference propagation, and neighbor-based candidate selection, while candidate size has a non-monotonic effect.
- Component analysis: KGQR evaluates three KG components: enhanced item representation, GCN propagation in state representation, and neighbor-based candidate selection.
- Item representation: KG information contributes almost as much as historical interaction data, suggesting utility for cold-start scenarios without historical interactions.
- State representation: GCN propagation improves representation suitability by using recommendation signals to update KG embeddings for the current task.
- Candidate selection: Neighbor-based candidate selection filters irrelevant items through local KG structure, improving final recommendation performance.
- Candidate size: Recommendation performance first increases and then decreases as candidate size grows from 1000 to 10000.
- Sample efficiency: KG use, task-specific representation learning, and candidate selection each improve sample efficiency in the analyzed comparisons.
6 CONCLUSION
KGQR uses knowledge-graph structure to improve interactive recommendation efficiency and performance, while future work extends evaluation to additional tasks, frameworks, and deployment settings.
- KGQR narrows the recommendation action space and propagates user preferences among correlated items to address large action spaces and sparse feedback.
- Experiments in a carefully designed simulation using two real-world datasets show significantly better performance and higher sample efficiency than state-of-the-art methods.
- Future work will investigate KGQR on news and image recommendation tasks with policy-gradient and DDPG frameworks, and consider online commercial deployment.
- The authors also plan to model user-preference dynamics more complexly by incorporating propensity toward relations indicated by click history.