Source-linked AI summary
Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learning
Jing-Cheng Shi, Yang Yu, Qing Da, Shi-Yong Chen, An-Xiang Zeng
TL;DR
Physical-world reinforcement learning requires many costly trials, motivating a simulator-based approach for improving commodity search in Taobao. The paper builds Virtual Taobao from historical customer behavior with GAN-SD and MAIL, trains policies in it with ANC, and reports faithful recovery of real-environment properties alongside better online performance than supervised approaches.
Problem
Applying reinforcement learning to Taobao commodity search is difficult because physical-world training requires costly trials and customer interactions create a dynamic sequential environment.
Method
The paper builds Virtual Taobao from historical customer behavior using GAN-SD for customer distributions, MAIL for interactions, and ANC to reduce policy over-fitting.
Results
Virtual Taobao faithfully reconstructs important real-environment properties, and policies trained in it achieve better real-environment performance than traditional supervised approaches.
Takeaways & Limitations
The results support using a learned Virtual Taobao environment to train improved commodity-search policies without directly bearing the physical cost of reinforcement-learning trials.
Takeaways & Limitations
Customer behavior is learned from data collected under a fixed Taobao strategy, while changing the platform policy changes the customers’ environment.
Abstract
from arXiv · showhide
Applying reinforcement learning in physical-world tasks is extremely challenging. It is commonly infeasible to sample a large number of trials, as required by current reinforcement learning methods, in a physical environment. This paper reports our project on using reinforcement learning for better commodity search in Taobao, one of the largest online retail platforms and meanwhile a physical environment with a high sampling cost. Instead of training reinforcement learning in Taobao directly, we present our approach: first we build Virtual Taobao, a simulator learned from historical customer behavior data through the proposed GAN-SD (GAN for Simulating Distributions) and MAIL (multi-agent adversarial imitation learning), and then we train policies in Virtual Taobao with no physical costs in which ANC (Action Norm Constraint) strategy is proposed to reduce over-fitting. In experiments, Virtual Taobao is trained from hundreds of millions of customers' records, and its properties are compared with the real environment. The results disclose that Virtual Taobao faithfully recovers important properties of the real environment. We also show that the policies trained in Virtual Taobao can have significantly superior online performance to the traditional supervised approaches. We hope our work could shed some light on reinforcement learning applications in complex physical environments.
1 Introduction
Applying reinforcement learning to large online systems is difficult because customer-facing environments have high sampling costs and sequential, delayed feedback. The paper addresses this by building Virtual Taobao from customer data and training policies in the simulator, with ANC used to reduce over-fitting.
- Physical-world reinforcement learning is difficult to study because large online systems interact with customers and can affect user experience and social wealth.
- Taobao commodity search is a sequential decision problem in which ranked pages elicit customer feedback that guides later engine decisions.
- Historical customer data is collected under a fixed platform strategy, so imitation methods may fail when training changes the customer environment.
- Virtual Taobao generates customers with GAN-SD and interactions with MAIL, which jointly learns customer and platform policies.
- The simulator is trained from hundreds of millions of customer records, recovers properties close to the real environment, and supports policies with superior online performance to traditional supervised approaches.
2 Background
Reinforcement learning optimizes sequential decisions through interaction, while imitation learning infers behavior from expert trajectories. The background contrasts behavior cloning, inverse reinforcement learning, and adversarial imitation learning as approaches to learning from demonstrations.
- 2.1 Reinforcement Learning: Reinforcement learning models sequential decision making as an MDP with states, actions, transitions, rewards, and a discount factor.
- 2.1 Reinforcement Learning: The RL objective is to learn a policy that maximizes expected discounted return accumulated from immediate rewards.
- 2.2 Imitation Learning: Imitation learning infers an expert policy from trajectory samples instead of learning entirely from scratch or relying only on a manually designed reward.
- 2.2 Imitation Learning: Behavior cloning learns supervised state-action mappings, whereas inverse reinforcement learning infers a reward function and trains a policy under that reward.
- 2.2 Imitation Learning: Behavior cloning can suffer from compounding error and covariate shift, while GAIL uses environment interaction and a discriminator to compare generated and expert trajectories.
- 2.3 Generative Adversarial Networks: GANs train a generator and discriminator adversarially, with the discriminator distinguishing training examples from generated samples.
3 Virtual Taobao
Virtual Taobao models commodity search as a coupled sequential interaction between a search engine and customers. It generates customer distributions with GAN-SD, learns interacting customer and engine policies with MAIL, and uses ANC to limit over-fitting.
- 3.1 Problem Description: Commodity search is modeled as a multi-step decision process because customer feedback depends on sequences of displayed pages.The engine and customers are treated as each other’s environments, with customer behavior also viewed as sequential policy optimization.
- 3.2 GAN-SD: Generating Customers: Customers are represented with requests and simulated from the real request distribution using GAN-based generation.GAN-SD is introduced because ordinary GANs tend to generate the most frequent customer types.
- 3.2 GAN-SD: Generating Customers: GAN-SD combines entropy and KL-divergence constraints to generate a wider customer distribution guided by the training distribution.Entropy promotes a wider distribution, while KL-divergence guides generated customer types toward the training data.
- 3.3 MAIL: Generating Interactions: MAIL generates interactions by jointly learning customer and search-engine policies rather than imitating customers in a static environment.A discriminator distinguishes simulated from real interactions, and its signal is used as a reward for policy learning.
- 3.3 MAIL: Generating Interactions: MAIL uses customer distributions, historical engine trajectories, and reinforcement learning to update the joint policy iteratively.The customer distribution is learned by GAN-SD before MAIL, which returns a learned customer policy for virtual interactions.
- 3.4 ANC: Reduce Over-fit to Virtual Taobao: ANC penalizes actions whose norms exceed the norms of most historical actions to trade off virtual-environment accuracy against policy improvement.The modified reward is r′(s, a) = r(s, a) 1 + ρ max{||a|| −µ, 0}.
4.1 Experiment Setting
The experiments evaluate whether Virtual Taobao reproduces customer and purchase behavior using turnover, volume, and purchase-page metrics. Offline evaluation uses R2P, while online comparisons use trajectories collected from a random-engine Taobao bucket.
- Measurements: Total Turnover (TT) measures the value of commodities sold, while Total Volume (TV) measures the amount sold.Both metrics are used as indicators in online experiments.
- Measurements: Rate of Purchase Page (R2P) measures the ratio of page views where a purchase takes place.R2P is used in both online and offline experiments.
- Evaluation Design: Offline experiments use only R2P because the work does not predict customer number or commodity price.TT and TV therefore appear only in online evaluation.
- Evaluation Design: The comparison data come from an online A/B-test bucket using a random engine policy and about 400 million historical records.The trajectories provide real-environment data for comparing the virtual and real environments.
- Distribution Evaluation: Virtual Taobao customer proportions are compared with Taobao across query category, purchase power, and high-level indicator features.The experiment generates 1,000,000 customers and reports similar distributions between virtual and real environments.
4.2 On Virtual Taobao Properties
Virtual Taobao reproduces customer-feature distributions and their effects on R2P, while also reflecting R2P’s temporal trend relative to the real environment.
- Proportion & R2P over Features: Virtual Taobao’s customer proportions across query category, purchase power, and high-level indicator are similar to Taobao’s ground truth.The comparison uses 1,000,000 generated customers across the three listed features.
- Proportion & R2P over Features: The influence of customer features on R2P in Virtual Taobao is quite similar to the real environment.
- R2P over Time: Virtual Taobao reflects the trend of R2P over time observed in Taobao.The evaluation divides one day of historical data into 12 time-ordered parts and compares virtual and real R2Ps.
4.3 Reinforcement Learning in Virtual Taobao
Policies trained in Virtual Taobao generalize better when ANC reduces over-fitting, while MAIL-based environments retain performance better over time than BC-based environments. Online, RL+VTaobao consistently outperforms the supervised-learning baselines.
- Generalization Capability of ANC: ANC policies consistently outperform unconstrained TRPO policies in real Taobao, indicating reduced over-fitting to Virtual Taobao.The comparison trains TRPO and TRPO-ANC in Virtual Taobao and evaluates both in real Taobao.
- Generalization Capability of MAIL: R2P declines faster in the BC environment, whose policy becomes worse than random after one month.The MAIL and BC environments are evaluated by deploying policies across environments built from data collected at later times.
- Online Experiments: RL+VTaobao is always better than SL+Data in online Taobao experiments.Real-environment R2P is 0.096 for SL1, 0.098 for SL2, and 0.101 for RL.
- Online Experiments: The online comparison reports TT and TV improvements of RL+VTaobao over both SL1+data and SL2+data.The supervised baselines use historical records divided into purchase and non-purchase subsets, with SL1 and SL2 representing different supervised objectives.
5 Conclusion
The paper builds Virtual Taobao from historical data to avoid the high physical cost of reinforcement-learning training in Taobao. It uses ANC to train policies with better real-environment performance than traditional supervised approaches.
- Conclusion: Virtual Taobao addresses the high physical cost of training reinforcement learning for commodity search in Taobao.The simulator is trained from historical data using GAN-SD and MAIL.
- Conclusion: Virtual Taobao faithfully reflects important properties of the real environment according to the empirical results.The paper presents this as evidence supporting the simulator's use for training policies.
- Conclusion: Policies trained with ANC in Virtual Taobao achieve better real-environment performance than traditional supervised-learning approaches.The conclusion frames this as the demonstrated outcome of the policy-training experiments.