PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Reinforcement Learning for Recommender Systems

Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M. Jose

arXiv:2006.05779v2cs.LGcs.AI

TL;DR

Sequential recommendation needs to account for long-term engagement and multiple feedback types, while offline RL is hindered by pure off-policy data and missing negative feedback. The paper adds self-supervised and RL heads to existing models and proposes SQN and SAC. Experiments on two real-world e-commerce datasets show improved hit ratios, especially for purchase prediction.

  • Problem

    Offline RL for sequential recommendation is challenging because logged implicit feedback creates a pure off-policy setting and lacks negative rewards.

  • Method

    The approach augments existing sequential recommendation models with a cross-entropy self-supervised head and an RL head, yielding SQN and SAC frameworks.

  • Results

    The proposed SQN and SAC methods improve hit ratios on two real-world e-commerce datasets, especially when predicting purchase interactions.

  • Takeaways & Limitations

    Reward-driven RL regularization can steer existing recommendation models toward specified objectives while retaining self-supervised ranking signals.

  • Takeaways & Limitations

    Model-based alternatives depend heavily on constructed simulators, and the proposed framework’s future work includes online tests and additional use cases.

Abstract

from arXiv · show

In session-based or sequential recommendation, it is important to consider a number of factors like long-term user engagement, multiple types of user-item interactions such as clicks, purchases etc. The current state-of-the-art supervised approaches fail to model them appropriately. Casting sequential recommendation task as a reinforcement learning (RL) problem is a promising direction. A major component of RL approaches is to train the agent through interactions with the environment. However, it is often problematic to train a recommender in an on-line fashion due to the requirement to expose users to irrelevant recommendations. As a result, learning the policy from logged implicit feedback is of vital importance, which is challenging due to the pure off-policy setting and lack of negative rewards (feedback). In this paper, we propose self-supervised reinforcement learning for sequential recommendation tasks. Our approach augments standard recommendation models with two output layers: one for self-supervised learning and the other for RL. The RL part acts as a regularizer to drive the supervised layer focusing on specific rewards(e.g., recommending items which may lead to purchases rather than clicks) while the self-supervised layer with cross-entropy loss provides strong gradient signals for parameter updates. Based on such an approach, we propose two frameworks namely Self-Supervised Q-learning(SQN) and Self-Supervised Actor-Critic(SAC). We integrate the proposed frameworks with four state-of-the-art recommendation models. Experimental results on two real-world datasets demonstrate the effectiveness of our approach.

1 INTRODUCTION

Sequential recommendation must balance long-term objectives and multiple interaction types, but offline RL faces pure off-policy data and missing negative feedback. The paper addresses these challenges by augmenting recommendation models with self-supervised and RL heads, proposing SQN and SAC and evaluating them on real-world datasets.

  • Motivation: Sequential recommendation commonly predicts the next item from session interactions such as views, clicks, or purchases, but self-supervised prediction can produce sub-optimal recommendations.The task is used in e-commerce, video, and music recommendation.
  • Motivation: RL can optimize long-term cumulative rewards and support objectives such as diversity, but online training risks exposing users to irrelevant recommendations.Commercial systems therefore commonly learn off-policy from logged implicit feedback.
  • Challenges: Offline recommender RL faces policy–data mismatch, limited data, and absent negative feedback because logged interactions reveal what users selected but not what they disliked.These conditions make direct RL learning difficult, especially with many candidate items.
  • Challenges: Model-based alternatives depend heavily on user simulators, whose response modeling is substantially more complex than generating images.This creates a limitation for simulator-based recommendation training.
  • Proposed approach: The proposed approach adds an RL head to existing recommendation models, using the self-supervised head for ranking and the RL head to introduce reward-driven properties.The paper proposes SQN and SAC, integrates them with four state-of-the-art models, and evaluates them on two e-commerce datasets containing clicks and purchases.

2 PRELIMINARIES

Sequential recommendation predicts the next item from interaction histories and can be formulated as reinforcement learning to optimize long-term, potentially multi-objective rewards. However, logged implicit-feedback data creates off-policy, sparsity, and missing-negative-feedback challenges for standard RL methods.

  • 2.1 Next Item Recommendation: Sequential recommendation predicts the next item from previous user-item interactions, including clicks, purchases, and other behavior types.
  • 2.1 Next Item Recommendation: Self-supervised recommenders encode an interaction sequence into a hidden state and rank candidate items using classification logits.
  • 2.2 Reinforcement Learning: RL models recommendation as an MDP in which actions are recommended items and rewards can represent long-term objectives such as purchases, novelty, diversity, or watching time.
  • 2.2 Reinforcement Learning: Online RL is typically avoided because under-trained policies may expose users to irrelevant recommendations, so models learn from fixed logged feedback instead.
  • 2.2 Reinforcement Learning: Logged recommendation data creates distribution discrepancy between the target and behavior policies, while propensity-score correction can suffer from high variance.
  • 2.2 Reinforcement Learning: Implicit feedback omits disliked items, leaving limited state-action pairs and causing purely learned Q-values to produce poor rankings under biased state distributions.

3 METHOD

The method co-trains a conventional self-supervised recommendation head with an RL head, using the latter to align learning with specified rewards while retaining supervised ranking signals. SQN uses Q-learning as a regularizer, whereas SAC uses Q-values to weight the self-supervised actor loss.

  • 3 METHOD: The proposed frameworks add an RL output layer to an existing sequential recommender and co-train it with the self-supervised head.
  • 3.1 Self-Supervised Q-learning: SQN combines cross-entropy recommendation learning with an additional Q-learning loss trained on a replay buffer from logged implicit feedback.
  • 3.1 Self-Supervised Q-learning: SQN returns the supervised head’s top-k items, while the RL head regularizes the base model toward the chosen reward scheme rather than directly generating recommendations.
  • 3.2 Self-Supervised Actor-Critic: SAC treats the supervised head as an actor and the Q-learning head as a critic, using Q-values as weights for the self-supervised loss.
  • 3.2 Self-Supervised Actor-Critic: SAC stops gradient flow through critic values and delays their use until a threshold of update steps to improve stability in complex sequential models.
  • 3 METHOD: The frameworks can integrate with existing sequential recommenders and support reward designs tailored to recommendation goals, while the paper uses cross-entropy as the self-supervised loss.

4 EXPERIMENTS

The experiments evaluate SQN and SAC on two real-world sequential recommendation datasets in an e-commerce setting. They examine integration with existing models, RL reward and discount-factor effects, and recommendation using Q-learning alone.

  • 4 EXPERIMENTS: The evaluation uses two real-world sequential recommendation datasets in an e-commerce scenario.
  • 4 EXPERIMENTS: RQ1 evaluates how the proposed methods perform when integrated with existing recommendation models.
  • 4 EXPERIMENTS: RQ2 studies how the RL component affects performance under different reward settings and discount factors.
  • 4 EXPERIMENTS: RQ3 evaluates recommendation performance when only Q-learning is used.
  • 4 EXPERIMENTS: The paper reports dataset statistics in Table 1 and states that the experiments address the listed research questions.

4.1 Experimental Settings

The experiments use two real-world e-commerce datasets with click and purchase interactions, evaluate four sequential recommendation backbones, and compare top-k ranking performance under cross-validation.

  • Datasets: The study evaluates RC15 and RetailRocket, both containing sequential user-item interactions treated as clicks and purchases.RC15 contains clicks and purchases, while RetailRocket’s views and add-to-cart events are treated as clicks and purchases, respectively.
  • Evaluation Protocol: Performance is measured with HR and NDCG using whole-item-set ranking on validation and test sequences.HR checks whether the next-event item appears in the top-k list, while NDCG gives higher scores to higher-ranked items.
  • Baselines: The proposed SQN and SAC frameworks are integrated with four state-of-the-art sequential recommendation models, including GRU, Caser, NItNet, and SASRec.The models represent recurrent, convolutional, dilated-convolutional, and self-attention-based recommendation approaches.
  • Implementation Settings: The input consists of the last 10 items before the target timestamp, with shorter sequences padded and models trained using Adam.The embedding size is 64 for all models; learning rates are 0.01 for RC15 and 0.005 for RetailRocket.

4.2 Performance Comparison (RQ1)

Across RC15 and RetailRocket, SQN and SAC consistently improve recommendation performance over their corresponding baselines for click and purchase prediction. SAC is especially strong for ranking purchase-related items on RetailRocket and purchase prediction on RC15.

  • RC15: SQN consistently outperforms its corresponding baseline on RC15 for both click and purchase prediction.The Q-learning head models long-term cumulative reward and supplies an additional learning signal for selecting actions optimized for future interactions.
  • RC15: SAC achieves the best purchase-prediction performance on RC15, where Q-values weight the supervised loss toward purchases.The paper attributes this to better utilization of the strong but sparse purchase signal.
  • RetailRocket: On RetailRocket, both SQN and SAC consistently outperform their corresponding baselines for click and purchase prediction.SAC achieves the best purchase NDCG even when SQN sometimes obtains the highest purchase HR.
  • Overall Findings: SQN and SAC show consistent improvement over the selected baselines across the reported experiments.The paper presents this pattern as evidence of effectiveness and generalization ability.

4.3 RL Investigation(RQ2)

Reward design and discounting materially affect SQN and SAC: emphasizing purchases helps initially, while excessive reward differences or discount factors reduce performance. Both frameworks still outperform the basic GRU at equal click and purchase rewards.

  • Reward settings: Increasing the purchase-to-click reward ratio initially improves SQN and SAC for purchase prediction, but performance later declines.The decline may reflect instability in temporal-difference updates caused by large reward differences.
  • Reward settings: SQN click-recommendation performance remains initially stable or slightly improves before decreasing as the purchase-to-click reward ratio rises.Excessive reward discrepancy may reduce the relative importance of clicks, which dominate the data.
  • Reward settings: At rp/rc = 1, SAC reaches about 0.54 HR@10 for purchase prediction on RC15, versus 0.5183 for basic GRU.This indicates an advantage even without distinguishing clicks from purchases in the reward setting.
  • Discount factor: Figures 5 and 6 evaluate SQN and SAC with different discount factors on RC15 using GRU as the base model.RetailRocket exhibits similar trends, but those results are omitted.
  • Discount factor: Increasing γ from 0 improves SQN and SAC HR@10 and NDCG@10, but performance decreases when γ becomes too large.Because average interaction sequences contain only six items, smaller discount factors are more appropriate than common game-control values such as 0.95 or 0.99.

4.4 Q-learning for Recommendation (RQ3)

Using Q-learning alone to generate recommendations performs worse than the basic GRU, supporting the use of the RL component as a regularizer or critic rather than as the ranking mechanism.

  • Q-learning for Recommendation: Q-learning-only recommendation performs worse than the basic GRU on HR@10 and NDCG@10 on RC15.The experiment uses uniformly sampled unseen items to provide negative rewards and GRU as the base model.
  • Q-learning for Recommendation: SQN and SAC avoid this weakness by using Q-learning as a regularizer or critic instead of directly generating the recommendation ranking.The passage states that Q-value estimation remains unbiased in this role.

5 CONCLUSION AND FUTURE WORK

The paper introduces self-supervised reinforcement learning with jointly trained supervised and RL heads to address off-policy learning and missing negative rewards in sequential recommendation. SQN and SAC improve hit ratio, particularly for purchase prediction, while future work targets online testing, additional use cases, and slate recommendation.

  • Conclusion: The proposed approach augments existing recommendation models with an RL head that injects specified reward preferences into recommendation.The method is motivated by using an unbiased RL estimator to fine-tune recommendations according to chosen rewards.
  • Conclusion: SQN and SAC jointly train supervised and RL heads within four state-of-the-art recommendation models.Experiments are conducted on two real-world e-commerce datasets.
  • Conclusion: Experimental results show that SQN and SAC improve hit ratio, especially when predicting real purchase interactions.The conclusion reports effectiveness across the evaluated datasets and integrated recommendation models.
  • Future work: Future work includes online tests, experiments on recommendation diversity and video watching time, and extension to slate-based recommendation.The slate-based setting would recommend a set of items rather than a single item.
Loading 2006.05779v2…