Generative Adversarial User Model for Reinforcement Learning Based Recommendation System

Xinshi Chen; Shuang Li; Hui Li; Shaohua Jiang; Yuan Qi; Le Song

Source-linked AI summary

Generative Adversarial User Model for Reinforcement Learning Based Recommendation System

Xinshi Chen, Shuang Li, Hui Li, Shaohua Jiang, Yuan Qi, Le Song

arXiv:1812.10613v3Dec 27, 2018cs.LGcs.IRstat.ML

TL;DR

RL recommendation is challenged by unknown user rewards and dynamics, while model-free learning can require impractical user interactions. The paper learns a GAN-based user model and reward function, uses it as a simulation environment for cascading DQN policy learning, and reports better behavioral fit, long-term reward, and adaptation. The authors note that online A/B testing could produce stronger results.

Problem
RL recommendation lacks clearly defined user rewards and environment dynamics, while model-free methods may require many interactions and manually designed rewards.
Method
A GAN jointly models user behavior dynamics and reward, then serves as a simulation environment for a cascading DQN combinatorial recommendation policy.
Results
The GAN user model better fits held-out likelihood and click prediction, while its RL policy yields better cumulative long-term user reward and quickly adapts to new users.
Takeaways & Limitations
The learned user model enables more principled model-based recommendation and reduces interactions needed for policy adaptation compared with model-free approaches.
Takeaways & Limitations
The reported benefits come from offline experiments and realistic simulation; stronger results require future online A/B testing.

Abstract

from arXiv · show

There are great interests as well as many challenges in applying reinforcement learning (RL) to recommendation systems. In this setting, an online user is the environment; neither the reward function nor the environment dynamics are clearly defined, making the application of RL challenging. In this paper, we propose a novel model-based reinforcement learning framework for recommendation systems, where we develop a generative adversarial network to imitate user behavior dynamics and learn her reward function. Using this user model as the simulation environment, we develop a novel Cascading DQN algorithm to obtain a combinatorial recommendation policy which can handle a large number of candidate items efficiently. In our experiments with real data, we show this generative adversarial user model can better explain user behavior than alternatives, and the RL policy based on this model can lead to a better long-term reward for the user and higher click rate for the system.

1. Introduction

Conventional recommendation models optimize immediate responses, while RL can target evolving long-term user interest. Applying RL is difficult because user rewards are unknown and model-free learning requires impractical interaction volumes.

Immediate-response objectives do not explicitly account for users’ evolving long-term interests, which recommender actions may influence.
User preferences, represented by the reward function driving behavior, are typically unknown and manually designed click rewards may misrepresent item preferences.
Model-free RL requires many interactions, but users may quickly abandon services when recommendations appear random or irrelevant.
The paper proposes model-based RL that jointly learns a user behavior model and reward function in a unified mini-max framework before learning recommendation policies.

1. We develop a generative adversarial learning (GAN) for-

The paper develops a generative adversarial user model that jointly represents behavior dynamics and reward, then uses it to support efficient recommendation policy learning. Its cascading action-value design handles large candidate sets, while experiments report improved behavioral fit, long-term reward, and adaptation.

The GAN formulation jointly models user behavior dynamics and recovers the associated reward function through mini-max optimization.
The learned reward supports principled RL, while the user model provides a simulation environment for policy learning and adaptation to new users.
The cascading DQN finds recommendation subsets from large candidate pools with time complexity linear in the number of candidates.
The generative adversarial model better fits user behavior on held-out likelihood and click prediction, and its policy improves cumulative long-term user reward.

2. Related Work

Prior recommendation methods commonly simplify user choice, while model-free RL faces interaction and reward-design challenges. Model-based RL offers sample-efficiency benefits, but existing approaches are not tailored to complex discrete user behavior sequences.

Wide&Deep, XGBOOST, and DFM assume users choose items independently, while collaborative competitive filtering assumes choices within each page view are independent.
Session-based RNN methods model sequential behavior, but prior model-free RL recommendation approaches may require many interactions and manually designed rewards.
Model-based RL has reduced sample complexity in robotics, but user behavior consists of complex sequences of discrete choices under session context.
Figure 1 depicts recommender information flow in green and user information flow in orange.

3. Setting and RL Formulation

The formulation treats the user as the RL environment and models recommendation as sequential subset selection with feedback-driven state transitions. Rewards represent user utility, and unknown dynamics and rewards are learned before estimating an optimal policy.

The setting displays k items per page, receives a click on one item or no click, and then recommends another page of k items.
The framework can be extended to settings with more complex page views and user interactions.
A state is an ordered sequence of historical clicks, and an action is a subset of k items selected from the currently available candidates.
The user behavior model defines state transitions by returning the probability of the next state given the current state and displayed items.
The reward is user utility or satisfaction after choosing an item, and this work assumes the system’s reward equals the user’s utility.
Because reward and transition dynamics are unknown, they are learned from data before Q-learning or related methods estimate the optimal policy.

4. Generative Adversarial User Model

The paper models sequential user choices as reward-maximizing behavior with exploration, using history-aware rewards and a jointly trained adversarial user model. The model’s behavior generator imitates observed actions while its reward discriminator distinguishes real and generated trajectories.

Generative Adversarial Training: The user model jointly learns behavior and reward through a mini-max optimization over observed user trajectories.The behavior model acts as a generator, while the reward function acts as a discriminator between real and model-generated actions.
User Behavior As Reward Maximization: The reward depends on both the selected item and the user’s state, which captures historical experience.Non-click is treated as a special action, allowing the model to represent satisfaction, disinterest, and evolving preferences.
User Behavior As Reward Maximization: Users choose displayed items to maximize expected reward, while regularization encourages exploration.With entropy regularization, Gumbel noise permits less-rewarding choices, and η controls the exploration–exploitation trade-off.
Model Parameterization: User state is embedded from previously clicked items using position weighting or an LSTM.Position weighting uses a shallow network that is more efficient for forward computation and gradient backpropagation than an LSTM.
Model Parameterization: The reward and behavior functions are parameterized by neural networks over user-state and item features.The reward parameters are denoted θ, while the user-model parameters are denoted α.
Generative Adversarial Training: Non-convex mini-max training may be unstable, so entropy regularization provides a closed-form initialization for learning the reward function.The initialized reward can then support training with other regularization functions.

5. Cascading RL Policy for Recommendation

The recommendation policy operates over combinatorial item subsets, making direct Q-learning expensive. Cascading Q-networks decompose action selection into related Q-functions, reducing computation while estimating long-term reward from the learned user simulator.

Model-Based Recommendation: The learned user model and reward function serve as a simulation environment for estimating the recommendation policy.Q-learning repeatedly queries this model to learn long-term action values.
Cascading Q-Networks: The policy selects a subset of k items from K candidates, creating a large combinatorial action space.The action space is computationally challenging even for moderate candidate and display-set sizes.
Cascading Q-Networks: Cascading Q-networks use k related Q-functions to search for an optimal combinatorial recommendation efficiently.The cascade addresses both the large action space and long-term reward estimation.
Cascading Q-Networks: The cascading procedure obtains an optimal action in O(k|I|) computations.The Q-functions are applied sequentially to construct the recommendation action.
Q-Network Estimation: Each Q_j function is represented by a neural network and trained against a shared target with soft consistency constraints.At the optimum, each Q_j should equal the overall optimal action-value function Q∗.
Q-Network Estimation: Experience replay and ε-exploration are applied during cascading Q-learning.These techniques are incorporated into the overall policy-learning procedure.

6. Experiments

Experiments on real-world datasets evaluate the GAN user model, its recommendation policies, regularization choices, and adaptation to new users. The results support stronger predictive performance, higher user-level reward, and faster adaptation with GAN-assisted policies.

Experimental setup: Experiments use six real-world datasets, including MovieLens, Last.fm, Yelp, and Ant Financial News, with simulated display sets and user interactions.MovieLens uses 1,000 active users, while Ant Financial News includes click records from 50,000 users.
Predictive performance: GAN performs significantly better than the baselines on top-k precision, while GAN-PW nearly matches GAN-LSTM with greater training efficiency.The baselines include wide-and-deep, collaborative filtering, item-to-item, session-based recurrent, cascading, and gradient-boosted models.
Regularization: On Yelp, L2 regularization produces a better user model than Shannon entropy regularization when initialized with the entropy-based scheme.The broader regularization comparison finds Shannon entropy generally performs well and offers a closed-form solution.
Predictive performance: GAN captures user-interest evolution over time better than W&D-CCF, whose predictions concentrate on several movie categories.Figure 3 compares actual user-choice trajectories with simulated GAN and W&D-CCF trajectories across 80 movie categories.
Recommendation performance: GAN-CDQN yields higher averaged cumulative reward for most users than alternative recommendation policies.The evaluation uses a held-out test set of 1,000 users, with cumulative reward and CTR as performance metrics.
User-model-assisted adaptation: Pretraining CDQN with the GAN user model enables rapid achievement of high CTR on new users, whereas model-free CDQN and LinUCB require around 1,000 iterations, or 100,000 interactions.The GAN-assisted policy adapts using a simulated environment before online application.

7. Conclusion and Future Work

The paper concludes that its GAN-based user model supports cascading Q-network policies for combinatorial recommendation, while noting that online A/B testing remains future work.

7. Conclusion and Future Work: The framework models user behavior dynamics and reward with a GAN, then uses the model as a simulation environment for a cascading Q-network policy.The cascading policy is designed for combinatorial recommendation.
7. Conclusion and Future Work: Figure 5 evaluates Qj∗ against Q5∗ for the same recommended items, with diagonal alignment representing the ideal case.
7. Conclusion and Future Work: Figure 6 compares averaged click rates across recommendation policies as the number of user interactions increases.
7. Conclusion and Future Work: The cascading Q-network is intended to handle a large number of candidate items efficiently.
7. Conclusion and Future Work: The experiments show benefits in offline and realistic simulation settings, while stronger results could come from future online A/B testing.

A.1. Proof of lemma 1

The proof derives the optimal action distribution under entropy regularization and establishes its equivalence to a discrete-choice model with Gumbel noise.

A.1. Proof of lemma 1: For a fixed state and displayed set, φt can be any probability vector in the simplex over the k candidate items.
A.1. Proof of lemma 1: The optimal vector φt∗ is obtained by maximizing the regularized objective, yielding the closed-form solution referenced in the proof.
A.1. Proof of lemma 1: The resulting optimal policy is equivalent to a discrete-choice model in which the perturbation εt follows a Gumbel distribution.
A.1. Proof of lemma 1: The choice inequalities compare ηr(st, ai) + εi across items, combining reward-based preference with stochastic exploration.

A.2. Proof of lemma 2

The second proof reformulates the entropy-regularized objective as a maximum-likelihood problem under an assumption that state–action pairs do not repeat.

A.2. Proof of lemma 2: The proof recalls the optimization problem in Eq. (5) and the entropy term involving Pk i=1 φi log φi.
A.2. Proof of lemma 2: The derivation assumes that no state–action pair (st, at) is repeated.
A.2. Proof of lemma 2: Under this assumption, the mapping φ can assign probabilities to each state–action pair, with at represented by its feature vector.
A.2. Proof of lemma 2: The transformed objective becomes the negative log-likelihood function, establishing equivalence to Lemma 2.
A.2. Proof of lemma 2: The cascading deep Q-learning algorithm uses cascading Q-functions, experience replay, and ε-exploration to search actions and update the networks.

C. Dataset description

The experiments use several behavioral datasets, with detailed descriptions for MovieLens and Ant Financial news data, and compare user-model and policy behavior through simulated trajectories and click rates.

C. Dataset description: MovieLens provides ratings from 1,000 sampled active users, and the study treats rated movies as watched items while simulating missing choice context.Displayed movie sets are capped at 40 items, with additional movie information collected from IMDb.
C. Dataset description: The Ant Financial dataset contains one month of timestamped clicks and impressions from 50,000 users, with display sets averaging five articles.The experiments use 20-dimensional dense article embeddings derived from wide-and-deep features.
C. Dataset description: Last.fm, Yelp, RecSys15, and Taobao provide listening, review, click-stream, and clicking or buying behavior data, respectively.
C. Dataset description: GAN user models capture the evolution of users’ interests better than CCF in the reported trajectory comparisons.
C. Dataset description: Figure 7 compares true user-choice trajectories with trajectories simulated by GAN and CCF across 80 movie categories.
C. Dataset description: Figure 8 compares click rates for GAN-DQN and other policies across 1,000 users, with repeated experiments and standard-deviation shading.

D.3. Figures for section 6.3

The figure compares cumulative reward as policies interact with and adapt to 1,000 users over time across three display-set sizes. The CDQN policy pretrained on a GAN user model adapts faster than model-free policies and reduces early user-loss risk.

The results cover three display-set sizes and track cumulative reward averaged over 1,000 users during policy interaction and adaptation.Each recommender interacts with 1,000 users at each time step.
The CDQN policy pretrained on a GAN user model adapts to online users much faster than other model-free policies and reduces early user-loss risk.The policies are evaluated on a separate set of 1,000 test users, with no overlap between training and test users.
The x-axis counts recommender interactions with online users, while the y-axis reports click rate.Each interaction represents 100 online data points because 1,000 users are involved per interaction.