Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning

Xiangyu Zhao; Liang Zhang; Zhuoye Ding; Long Xia; Jiliang Tang; Dawei Yin

Source-linked AI summary

Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning

Xiangyu Zhao, Liang Zhang, Zhuoye Ding, Long Xia, Jiliang Tang, Dawei Yin

arXiv:1802.06501v3Feb 19, 2018cs.IRcs.LGstat.ML

TL;DR

Traditional recommender systems use static strategies and must incorporate abundant negative feedback without burying positive feedback. The paper proposes DEERS, a deep reinforcement learning framework that models recommendations as an MDP and learns from both feedback types. Experiments on real-world e-commerce data report effective performance, with negative feedback improving recommendations.

Problem
Existing recommender systems commonly use fixed strategies and focus on immediate rewards, while negative feedback is abundant and difficult to combine with positive feedback.
Method
DEERS models recommendation sessions as an MDP and uses Deep Q-Network reinforcement learning to update strategies from positive and negative user feedback.
Results
The proposed framework outperforms representative baselines, and experiments show that negative feedback can improve recommendation performance.
Takeaways & Limitations
Both positive and negative user feedback contribute to recommendation performance within the proposed framework.
Takeaways & Limitations
Skipped items may reflect non-dislike behaviors, so weak negative feedback may fail to improve or may reduce performance; dwell time could provide stronger signals.

Abstract

from arXiv · show

Recommender systems play a crucial role in mitigating the problem of information overload by suggesting users' personalized items or services. The vast majority of traditional recommender systems consider the recommendation procedure as a static process and make recommendations following a fixed strategy. In this paper, we propose a novel recommender system with the capability of continuously improving its strategies during the interactions with users. We model the sequential interactions between users and a recommender system as a Markov Decision Process (MDP) and leverage Reinforcement Learning (RL) to automatically learn the optimal strategies via recommending trial-and-error items and receiving reinforcements of these items from users' feedback. Users' feedback can be positive and negative and both types of feedback have great potentials to boost recommendations. However, the number of negative feedback is much larger than that of positive one; thus incorporating them simultaneously is challenging since positive feedback could be buried by negative one. In this paper, we develop a novel approach to incorporate them into the proposed deep recommender system (DEERS) framework. The experimental results based on real-world e-commerce data demonstrate the effectiveness of the proposed framework. Further experiments have been conducted to understand the importance of both positive and negative feedback in recommendations.

1 INTRODUCTION

Traditional recommender systems use static, short-term strategies that may miss changing user preferences and future rewards. DEERS applies deep reinforcement learning to update recommendation strategies using both positive and negative feedback.

1 INTRODUCTION: Traditional recommender systems follow fixed strategies and often optimize immediate purchases rather than future rewards.These approaches may fail when user preferences change dynamically.
1 INTRODUCTION: Reinforcement learning models recommendations as sequential user-agent interactions and continually updates trial-and-error strategies.The framework seeks strategies that fit users’ dynamic preferences and maximize long-term cumulative rewards.
1 INTRODUCTION: DEERS uses Deep Q-Networks to estimate action values when the recommendation item space makes table-based reinforcement learning inflexible.The approach uses a nonlinear approximator for the action-value function.
1 INTRODUCTION: Positive feedback includes clicks or orders, while skipped items provide negative feedback that can improve understanding of user preferences.Negative feedback is typically much more frequent, making simultaneous modeling challenging because positive feedback may be buried.
1 INTRODUCTION: The paper identifies negative feedback as important, proposes DEERS to incorporate both feedback types, and evaluates it on real-world e-commerce data.The experiments also assess the importance of negative feedback for recommendation accuracy.

2 PROBLEM STATEMENT

The paper formulates recommendation as a Markov Decision Process in which an agent selects items and receives feedback-dependent rewards. The goal is to learn a policy maximizing cumulative reward from user interactions.

2 PROBLEM STATEMENT: The recommendation task is modeled as an MDP with states, actions, rewards, transition probabilities, and a discount factor.The recommender agent sequentially chooses items to maximize cumulative reward.
2 PROBLEM STATEMENT: A state records the user’s browsing history before time t, with items sorted chronologically.The state is represented by the preceding N recently clicked or ordered items.
2 PROBLEM STATEMENT: The agent’s action is to recommend an item, after which the user may skip, click, or order it and generate an immediate reward.The reward depends on the user’s feedback to the recommendation.
2 PROBLEM STATEMENT: The transition probability specifies how the user state changes after an action while satisfying the MDP’s Markov assumption.The next state depends on the current state and action rather than the full prior history.
2 PROBLEM STATEMENT: The recommendation policy maps states to actions and is optimized to maximize cumulative reward.The discount factor controls how future rewards contribute to present value.

3 THE PROPOSED FRAMEWORK BASED ON DEEP REINFORCEMENT LEARNING WITH NEGATIVE FEEDBACKS

The framework models recommendation as sequential user–agent interactions and extends deep Q-learning to incorporate both clicked/ordered and skipped items. DEERS separates positive and negative signals, uses recurrent representations, pairwise regularization, and offline training with a simulated online test.

Deep reinforcement learning: The framework uses DQN to approximate action values because large state and action spaces make Bellman-based estimation for every state–action pair infeasible.Training uses a Q-network with fixed previous-iteration parameters, stochastic optimization, experience replay, separate evaluation and target networks, and prioritized sampling.
Negative-feedback state: Negative feedback is incorporated because a positive-only system neither changes state nor updates its strategy when users skip recommended items.The extended state records both recent positive and negative feedback so skipped recommendations affect subsequent decisions.
Negative-feedback state: DEERS represents each state as positive history s+ and negative history s−, each containing the user’s previous N clicked/ordered or skipped items.Transitions update the corresponding history when the user accepts or skips an item, allowing both feedback types to enter the next state.
DEERS architecture: GRUs encode users’ sequential positive and negative preferences from item embeddings, using the final hidden states as representations for s+ and s−.The paper uses GRUs rather than LSTMs because it reports that GRUs outperform LSTMs for capturing sequential recommendation behavior.
DEERS architecture: The architecture processes positive and negative signals separately, using distinct early hidden layers to favor similarity to clicked items and dissimilarity to skipped items.Positive and negative inputs are formed by combining their respective states with the recommended item.
Online test: A simulated online environment trained on separate user logs achieved overall 90% precision for immediate-feedback prediction, enabling model testing on simulated online rewards.The simulator predicts immediate feedback from the current state and recommended item.

4 EXPERIMENTS

The experiments evaluate DEERS against representative baselines on real e-commerce data in offline and simulated online settings. Results indicate that reinforcement learning, negative feedback, sequential preference modeling, and pairwise regularization each contribute to recommendation performance.

Experimental Settings: The experiments use 1,000,000 JD.com recommendation sessions, split temporally into 70% training and 30% test data.States include 10 previously clicked or ordered items and 10 previously skipped items.
Offline Test: Offline evaluation compares DEERS with CF, FM, GRU, DQN, and DEERS-p using MAP and NDCG@40.DEERS-p is trained only on positive items, while DEERS incorporates positive and negative feedback.
Offline Test: DEERS outperforms representative baselines, while DEERS-p outperforms GRU, linking reinforcement learning with improved short-term and long-term reward optimization.CF and FM perform worse than sequence-aware and reinforcement-learning approaches because they ignore temporal browsing sequences.
Offline Test: DEERS performs better than DEERS-p, indicating that integrating skipped items with clicked and ordered items improves recommendation accuracy.The comparison attributes this difference to the inclusion of negative feedback in DEERS.
Online Test: DEERS outperforms DEERS-f, DEERS-t, and DEERS-r, supporting redesigned DQN architecture, GRU-based sequential preference modeling, and pairwise regularization.DEERS-p exceeds GRU significantly in long sessions, where long-term reward optimization matters more than immediate reward maximization.
Parameter Sensitivity: Offline performance peaks at α = 0.1, while increasing N improves performance and produces greater sensitivity to positive items than skipped items.The analysis states that clicked or ordered items mainly determine decisions, while skipped items also influence them.

5 RELATED WORK

The related work spans traditional recommendation techniques and reinforcement-learning approaches. The paper positions its method within sequential user-agent recommendation modeling while addressing scalability limitations of earlier approaches.

Traditional Recommendation Techniques: Traditional recommendation techniques include collaborative filtering and content-based methods for supplying users with potentially interesting items.Collaborative filtering relies on similarities among users’ preferences.
Reinforcement Learning for Recommendations: Reinforcement-learning recommendation models treat recommendation as sequential interactions between users and a recommender agent.The paper uses reinforcement learning to learn recommendation strategies rather than relying only on fixed recommendation procedures.
Reinforcement Learning for Recommendations: Earlier reinforcement-learning methods based on transition probabilities and Q-value tables may become inflexible as the number of recommendable items grows.The paper therefore uses a Deep Q-Network as a nonlinear action-value approximator.

6 CONCLUSION

The conclusion presents DEERS as a Markov Decision Process and deep reinforcement-learning framework that learns recommendation strategies from positive and negative feedback. Experiments on real e-commerce data report improved performance and a useful role for skipped items, while also identifying limits of interpreting weak negative feedback.

Conclusion: DEERS models recommendation sessions as a Markov Decision Process and uses reinforcement learning to learn optimal recommendation strategies.The framework is designed to update strategies during interactions and optimize long-term cumulative reward.
Conclusion: The framework combines a Deep Q-Network with a redesigned architecture that captures clicked, ordered, and skipped items simultaneously.This design distinguishes the approach from previous reinforcement-learning recommendation work.
Conclusion: Experiments on real e-commerce data show improved recommendation performance and indicate that negative feedback can assist item recommendation.The conclusion reports both framework effectiveness and the contribution of skipped items.
Future Scope and Limitations: Skipped items may reflect lack of attention or weaker preference rather than dislike, so weak negative feedback may fail to improve or may reduce performance.The paper suggests using additional behavioral information such as dwell time to capture stronger negative feedback.