Source-linked AI summary
Deep Reinforcement Learning for Page-wise Recommendations
Xiangyu Zhao, Long Xia, Liang Zhang, Zhuoye Ding, Dawei Yin, Jiliang Tang
TL;DR
Page-wise recommendation must adapt to real-time user feedback while generating complementary item sets with appropriate 2-D layouts. DeepPage addresses both requirements through deep reinforcement learning that jointly optimizes pages and display strategies. Experiments on a real-world e-commerce dataset demonstrate the framework’s effectiveness.
Problem
Page-wise recommenders must update strategies from real-time feedback and jointly generate complementary items with proper 2-D display, challenges traditional systems do not adequately address.
Method
DeepPage models recommendation as sequential reinforcement-learning interactions and jointly generates and displays a page of items using current user preferences.
Results
DeepPage outperforms its variants and improves long-session performance over GRU, while approaching convergence after 500,000 offline sessions.
Takeaways & Limitations
Deep reinforcement learning can optimize page-wise recommendations using real-time feedback while accounting for complementary items and display strategy.
Takeaways & Limitations
The item representations are pre-trained from company-collected browsing histories, and the offline pages contain only 5 rows and 2 columns to fit mobile screens.
Abstract
from arXiv · showhide
Recommender systems can mitigate the information overload problem by suggesting users' personalized items. In real-world recommendations such as e-commerce, a typical interaction between the system and its users is -- users are recommended a page of items and provide feedback; and then the system recommends a new page of items. To effectively capture such interaction for recommendations, we need to solve two key problems -- (1) how to update recommending strategy according to user's \textit{real-time feedback}, and 2) how to generate a page of items with proper display, which pose tremendous challenges to traditional recommender systems. In this paper, we study the problem of page-wise recommendations aiming to address aforementioned two challenges simultaneously. In particular, we propose a principled approach to jointly generate a set of complementary items and the corresponding strategy to display them in a 2-D page; and propose a novel page-wise recommendation framework based on deep reinforcement learning, DeepPage, which can optimize a page of items with proper display based on real-time feedback from users. The experimental results based on a real-world e-commerce dataset demonstrate the effectiveness of the proposed framework.
1 INTRODUCTION
Page-wise recommendation must adapt to users’ real-time feedback while jointly selecting complementary items and arranging them appropriately on a 2-D page. DeepPage addresses these challenges with deep reinforcement learning that optimizes recommendations over sequential interactions.
- Motivation: E-commerce recommendation interactions repeatedly present item pages, collect user feedback, and use that feedback to recommend subsequent pages.This interaction pattern motivates updating recommendation strategies during use.
- Motivation: Static greedy recommendation strategies may fail to capture dynamic user preferences or update efficiently from real-time feedback.The paper therefore models recommendation as sequential interaction between users and a recommender agent.
- Motivation: Traditional recommender systems often optimize immediate reward, whereas reinforcement learning maximizes expected long-term cumulative reward.This allows recommendations to account for items with small immediate rewards when they contribute to longer-term outcomes.
- Page-wise Recommendations: Page-wise recommendation jointly generates diverse, complementary items and arranges them in a 2-D display rather than treating recommendation only as 1-D ranking.Complementary bundles may receive higher rewards than pages containing only similar items.
- Contributions: DeepPage uses deep reinforcement learning to jointly optimize a recommendation page while incorporating users’ real-time feedback.The framework and its components are evaluated on a real-world e-commerce dataset.
2 THE PROPOSED FRAMEWORK
DeepPage models page-wise recommendation as a Markov decision process and uses an Actor-Critic framework to generate pages, update preferences from feedback, and estimate action values. Its Actor encodes initial and real-time preferences, then decodes the current state into a displayed recommendation page.
- Framework Overview: The framework models recommendation as an MDP in which a recommender agent selects pages from states, receives feedback rewards, and updates states over time.The policy seeks to maximize cumulative reward rather than selecting recommendations independently.
- Framework Overview: Deep reinforcement learning uses neural approximators for action values, avoiding explicit transition models and Q-value tables in large item spaces.The paper identifies large, dynamic action spaces and the computational cost of selecting an optimal page as central challenges.
- Framework Overview: Actor-Critic architecture maps the current state to a deterministic page, while the Critic evaluates its state-action value instead of evaluating every possible state-action pair.The Actor is updated according to the Critic’s judgment of whether the generated page matches the current preference.
- Actor Architecture: The Actor initializes session preference with a GRU over chronologically ordered previously clicked or purchased items and uses its final hidden state as the initial state.Items are first transformed into low-dimensional dense embeddings; GRU gates process the sequential history.
- Actor Architecture: During a session, a GRU with attention encodes page item representations and user feedback to capture evolving real-time preference.Each interaction can add another GRU unit using the displayed items, categories, and feedback as input.
- Actor Architecture: The decoder applies a deconvolution network to the current preference state to generate a page, while convolutional processing learns spatial display patterns.The page representation combines item, category, and feedback embeddings before 2D-CNN processing.
3 TRAINING AND TEST PROCEDURE
DeepPage uses online and offline reinforcement-learning procedures to train and test page-wise recommendations. Its mapping step converts proto-actions into valid, non-repeating items, while Actor-Critic training aligns these actions and learns from user feedback.
- Training and test procedures: DeepPage supports online-policy and off-policy procedures for interaction-based training and historical-data training, respectively.Off-policy training enables offline pre-training and evaluation before online deployment.
- Action mapping: The Actor generates a proto-action page, then maps each proto-item embedding to the most similar valid item in the existing item-embedding space.The mapped item retains the proto-item's page position.
- Action mapping: Removing each selected item from the candidate space prevents repeated recommendations within one page.The mapping algorithm adds the selected item at the corresponding position before removing it from the item-embedding space.
- Actor-Critic training: Offline training minimizes the difference between proto-actions and valid-actions because fixed valid-actions would otherwise disconnect the Actor from the Critic.The Actor is updated toward proto-actions that resemble the executed valid-actions.
- Actor-Critic training: DDPG trains the Critic from replay-buffer transitions using valid executed actions, while the Actor uses a policy gradient evaluated at its proto-action output.The online procedure alternates transition generation—recommendation, reward observation, and storage—with parameter updates from sampled minibatches.
- Training and test procedures: DeepPage provides online and offline tests, using real-time user feedback online and historical browsing data offline.After training, the framework learns Actor and Critic parameters before testing.
4 EXPERIMENTS
Experiments on real e-commerce data compare DeepPage with representative baselines and component variants in offline and simulated online tests. DeepPage performs better than conventional reinforcement-learning baselines and its variants.
- Experimental Setup: The experiments use 1,000,000 recommendation sessions containing 9,136,976 items, with the first 70% for training/validation and later 30% for testing.The data come from a September 2017 real e-commerce company dataset.
- Offline Test: DeepPage approaches convergence after training on 500,000 offline sessions and outperforms conventional DDPG in offline testing.The offline comparison includes CF, FM, GRU, DQN, DDPG, and DeepPage.
- Offline Test: DQN and DDPG outperform GRU offline, while DeepPage performs better than DDPG by jointly optimizing a page and learning real-time preferences with GRU.The comparison attributes the reinforcement-learning advantage to balancing short-term and long-term rewards.
- Online Test: Online testing uses a simulated environment built from data different from the training set and compares short sessions of 10 pages with long sessions of 50 pages.The simulator enables controlled evaluation of recommendation-session length.
- Online Test: In short sessions, GRU, DQN, and DDPG perform comparably; in long sessions, DQN and DDPG significantly outperform GRU, while DeepPage performs better than both conventional baselines.DDPG also trains faster than DQN in the online test.
- Effectiveness of Components: Ablations show that embeddings, category and feedback information, initial-preference modeling, real-time preference capture, attention, and item display strategy each contribute to performance.DeepPage outperforms all evaluated variants, including variants removing GRU, attention, or CNN components.
5 RELATED WORK
Related work applies reinforcement learning to recommendation settings including conversational, web-page, and sequential decision problems. These studies motivate modeling recommendation as a sequential interaction process.
- Reinforcement Learning for Recommendations: Prior studies use reinforcement learning for conversational recommendation, web-page recommendation, and sequential decision-making problems.The reviewed approaches include partial-observable-MDP-based collaborative filtering and Q-learning from web usage data.
6 CONCLUSION
The paper proposes DeepPage, a deep-reinforcement-learning framework that learns recommendation strategies while jointly optimizing page contents. Extensive experiments on real e-commerce data validate its effectiveness.
- Conclusion: DeepPage automatically learns recommendation strategies and simultaneously optimizes a page of items using deep reinforcement learning.The framework is designed for page-wise recommendation.
- Conclusion: Extensive experiments based on data from a real e-commerce company validate the effectiveness of the proposed framework.
- Conclusion: Future work includes reducing proto-action-to-valid-action mapping complexity and jointly handling search, bidding, advertisement, and recommendation.