Source-linked AI summary
RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising
David Rohde, Stephen Bonner, Travis Dunlop, Flavian Vasile, Alexandros Karatzoglou
TL;DR
Recommendation research faces weak correspondence between offline metrics and online performance, especially when organic and advertising interactions are treated separately. RecoGym introduces an RL environment that simulates both user-traffic patterns and responses to publisher-site recommendations, with configurable behavioral factors. The simulator provides a setting for evaluating recommendation policies and supporting collaboration between recommender-systems and RL researchers.
Problem
Offline supervised-learning metrics can correlate poorly with online recommendation performance, while existing approaches and resources do not adequately combine organic and bandit information.
Method
RecoGym is an RL simulator that models organic shopping and bandit publisher sessions with configurable correlation, hidden user-item dimensions, and ad-exposure effects.
Results
RecoGym’s first release provides an environment with user shopping and publisher sequences and supports sanity checks for models trained on simulator data.
Takeaways & Limitations
The simulator is intended to support broader research connecting recommender systems and reinforcement learning.
Takeaways & Limitations
IPS-based offline evaluation can suffer from very large weights and biased estimates when the evaluated policy is far from the logging policy.
Abstract
from arXiv · showhide
Recommender Systems are becoming ubiquitous in many settings and take many forms, from product recommendation in e-commerce stores, to query suggestions in search engines, to friend recommendation in social networks. Current research directions which are largely based upon supervised learning from historical data appear to be showing diminishing returns with a lot of practitioners report a discrepancy between improvements in offline metrics for supervised learning and the online performance of the newly proposed models. One possible reason is that we are using the wrong paradigm: when looking at the long-term cycle of collecting historical performance data, creating a new version of the recommendation model, A/B testing it and then rolling it out. We see that there a lot of commonalities with the reinforcement learning (RL) setup, where the agent observes the environment and acts upon it in order to change its state towards better states (states with higher rewards). To this end we introduce RecoGym, an RL environment for recommendation, which is defined by a model of user traffic patterns on e-commerce and the users response to recommendations on the publisher websites. We believe that this is an important step forward for the field of recommendation systems research, that could open up an avenue of collaboration between the recommender systems and reinforcement learning communities and lead to better alignment between offline and online performance metrics.
1 INTRODUCTION
RecoGym addresses weak alignment between offline recommendation metrics and online performance by providing an RL simulator for online advertising. Its first release models both organic and bandit interactions with configurable behavioral factors and supports evaluation of arbitrary recommendation policies.
- Offline recommendation metrics often correlate poorly with measured online performance, motivating alternatives to classical supervised learning.
- Online advertising recommendation selects an item to show based on the user’s state and ad-display context, spanning organic and bandit interactions.
- Inverse Propensity Scoring enables counterfactual policy evaluation but can produce large weights and biased estimates when the evaluated policy differs substantially from the logging policy.
- RecoGym simulates users’ reactions to arbitrary recommendation policies without the exploding variance associated with IPS-weighted datasets.
- As an OpenAI Gym environment, RecoGym is intended to support broader interaction between recommender-systems and reinforcement-learning research.
- The first environment release models organic and bandit interactions, their correlation, hidden user-item dimensions, and the effect of repeated ad exposure on click-through rate.
2 RELATED WORK
Prior work largely separates organic user behavior modeling from bandit behavior modeling, while available evaluation resources do not jointly represent both. This leaves a gap for benchmarks that combine the two information sources.
- The literature is divided between recommendation methods modeling organic behavior and computational-advertising methods modeling bandit behavior.
- Organic recommendation commonly uses matrix completion or sequential models to predict user-item activity and next items.
- Some organic-behavior studies use behavioral rankings or uniformly evaluated actions to improve rankings for recommendations, supported by public organic datasets.
- Bandit behavior is studied with contextual bandits, counterfactual risk minimization, and Gaussian-process bandit optimization, but benchmark datasets remain limited.
- Existing bandit datasets requiring IPS produce noisy algorithmic-performance measures, and no public dataset is known to incorporate both organic and bandit information.
3 FORMALIZING RECOMMENDATION AS A GAME
RecoGym formalizes recommendation as user activity alternating between organic shopping sessions and bandit publisher sessions. Ads are shown during publisher sessions, and successful recommendations can return users to e-commerce activity.
- 3 FORMALIZING RECOMMENDATION AS A GAME: The idealized user model alternates between organic e-commerce sessions and bandit publisher sessions before stopping online activity.
- 3 FORMALIZING RECOMMENDATION AS A GAME: Advertising recommendations are available only during publisher sessions, while conversion probability is assumed constant after users return to product pages.
- 3 FORMALIZING RECOMMENDATION AS A GAME: The agent’s objective is to show personalized ads that increase transitions back to e-commerce, represented by user-ad click-through rates.
- 3.1 Notation: Table 1 defines the notation used to describe the environment and Table 2 provides an example of its data representation.
- 3.1 Notation: An example timeline follows a user from organic product visits to publisher ads, where a clicked recommendation restarts the retailer session.
- 3.1 Notation: The example contains six time steps for user 10, beginning with organic visits to products 104, 52, and 71 before advertising begins.
- 3.1 Notation: The notation and example-data tables provide complementary views of the environment’s variables and event sequence.
- 3.1 Notation: Table 2 is identified as an example-data table, while the surrounding notation section supplies the accompanying data context.
4 SIMULATOR
The simulator models recommendation as an RL environment linking organic shopping behavior with bandit ad interactions. It supports parameterized user–item structure, behavioral correlation, ad-exposure effects, and baseline agents for testing how these information sources affect recommendations.
- Environment interface: RecoGym provides a gym-based RL environment in which an agent recommends products and receives observations, click rewards, completion signals, and logging information.Reset generates a synthetic user, while Step processes each recommendation and returns the environment response.
- Agent strategies: Three model families use organic data, bandit data, or both to estimate and rank recommendations.The combined approach is intended to use organic ordering and bandit feedback together, while pure bandit methods require sufficient bandit data.
- Behavior model: The simulator represents organic product views with a Bernoulli model whose log odds connect organic behavior to bandit click-through rates.An increasing calibration function maps categorical organic behavior to Bernoulli bandit behavior, while time dependence can represent ad fatigue.
- Sanity checks: Expected performance varies with bandit-data availability and organic–bandit noise: pure organic methods depend on noise, pure bandit methods need abundant bandit events, and combined methods can outperform pure bandit methods at moderate data levels.As noise breaks the organic–bandit connection, pure organic performance declines, whereas combined and pure bandit methods avoid that specific issue.
- Baselines: RecoGym includes baseline agents such as Random, Logistic, and Supervised-Prod2Vec for interaction with the simulator.The Logistic agent uses exposure and click responses, representing users by their last organically viewed product.
5 CONCLUSIONS
RecoGym is introduced as an RL environment for online-advertising recommendation, with configurable organic and publisher sequences and user–item behavior. The paper presents sanity checks and guidelines intended to support research spanning reinforcement learning and recommender systems.
- Contribution: RecoGym is presented as the first RL environment for recommendation in online advertising.The simulator supports both user shopping and publisher sequences.
- Simulator scope: Experimenters can vary organic–publisher correlation, intrinsic user–item cluster dimensions, and the effect of repeated ad exposure on click-through rate.These controls define different simulated recommendation settings.
- Evaluation and outlook: The paper reports basic sanity checks for models trained on simulator data and proposes guidelines for using the environment.The stated broader goal is to encourage interdisciplinary work and better alignment between offline and online performance.