LADDER: A Human-Level Bidding Agent for Large-Scale Real-Time Online Auctions

Background

• Because little auction state is observable, the authors model the game as a POMDP with a huge but finite state space that terminates and restarts in cycles.Source: Background · p. 2

Related Work

• A3C and n-step Q-learning allowed multiple actions between training steps, but their semi-decoupling could not meet the auction system’s need for full decoupling (Mnih et al. 2016).Source: Related Work · p. 3
• Concurrent RTB work used dynamic programming and CTR prediction, extending beyond the traditional ECPM policy (Cai et al. 2017).Source: Related Work · p. 3

The Architecture of the DSP System in JD

• JD’s DSP recalls hundreds of candidate ads from millions, ranks them, and sends the top ad and bid to the ADX.Source: The Architecture of the DSP System in JD · p. 3
• Under GSP, a winning DSP knows only the next DSP’s bid, while losers know no rivals’ bids and DSPs do not know the rival count.Source: The Architecture of the DSP System in JD · p. 3
• DSPs buy impressions at CPM and sell to advertisers by CPC or CPA; JD uses ECPM = Q ∗ bid_click for ranking, leaving revenue and expenditure to control.Source: The Architecture of the DSP System in JD · p. 3
• JD’s ECPM policy calibrated Q using a massive ranking model that could not update fast enough, so human experts adjusted traffic-level bid coefficients tens of times daily.Source: The Architecture of the DSP System in JD · pp. 3–4
• In early 2016, JD began applying reinforcement learning to RTB auctions and devised LADDER, its learning ad exchange bidder.Source: The Architecture of the DSP System in JD · p. 4

Modeling

• Each day is an episode, and each auction’s reward is net profit, r_t = i_t - e_t; income i_t is zero unless the user clicks the ad.All related variables use CNY.Source: Modeling · p. 4
• Because click-linked income is usually 10^3–10^5 times larger than expense and rewards are mostly negative with a few high positives, the model needs high expressivity to capture both accurately.The authors identify CNNs as a suitable high-expressivity model.Source: Modeling · p. 4
• Bids are discrete actions from 0 to ceiling C in 0.01-CNY increments, creating an action space of thousands that is expected to be sparse in training data.Source: Modeling · p. 4
• User, SKU, and ad information forms partially observable states rendered as text; CNNs exploit similar plain-text entity IDs without elaborate feature engineering, and the model is under 1 Mb.The text is one-hot encoded before being fed to the CNN.Source: Modeling · p. 4
• The production model encodes input as a 600 × 71 matrix and uses shorthand text plus a traditional architecture rather than Inception or ResNets to save response time.All layers except the last use ReLU; the output layer size is omitted for commercial privacy.Source: Modeling · p. 4

Deep Asynchronous Stochastic Q-learning

• LADDER runs learning, bidding, and observation simultaneously, avoiding serial execution that cannot meet an online DSP’s millisecond response requirements when ad-click rewards arrive tens of minutes later.Each process can run on threads across multiple machines.Source: Deep Asynchronous Stochastic Q-learning · pp. 4–5
• Training samples a stochastic next state within a configurable time window and from the same publisher, treating publisher auctions as distinct games.The sampling window addresses uncertain user activity; same-publisher sampling responds to differences in publishers’ CTR, CVR, and ROI.Source: Deep Asynchronous Stochastic Q-learning · p. 5
• Using GSP auction properties, LADDER infers that bids above the observed winning bid would win and bids below the observed losing bid would lose, then redefines rewards across actions.The method combines the redefined rewards with the DQN loss; the original DQN loss remains applicable when actions are less correlated, and Double DQN variants can also be incorporated.Source: Deep Asynchronous Stochastic Q-learning · p. 5
• To maximize revenue while keeping breakeven, LADDER weights positive-reward sampling using hyperparameter π and uses experience memory; the authors call the full algorithm deep asynchronous stochastic Q-learning.The full procedure is presented as Algorithm 1.Source: Deep Asynchronous Stochastic Q-learning · p. 5

Experimental Results

• Experiments ran on four important publishers that account for a significant part of JD.com’s DSP revenue, aiming to improve both publisher revenue and profits.Source: Experimental Results · p. 5

• LADDER training ran on 4 Tesla K80 GPUs, while serving used 24 PC servers with a high-performance C++ sparse convolver in the asynchronous deployment shown in Figure 3.Source: Experiment Setup · p. 5, Figure 3 · p. 6
• Training used RMSProp at learning rate 5^-9 and restricted ε-greedy exploration to ε of 10^-3 to minimize business impact.Source: Experiment Setup · p. 5
• Imitation populated experience memory with data generated by the hybrid system’s ECPM policy, limiting environment interaction and keeping losses under control during cold start.Source: Experiment Setup · p. 6
• After several hours of imitation, with the duration depending on N, LADDER began learning from data generated by its own policy.Source: Experiment Setup · p. 6

Evaluation

• The May 2017 online A/B test used ECPM as baseline, began with LADDER on 10% of daily auctions, then launched it on 90% from day 8 while retaining a 10% baseline holdback for months.The experiment system proportionally normalized all experiments for comparison.Source: Evaluation · p. 6
• LADDER incurred huge losses on its first imitation-phase day, caught up with the baseline the next day, and outperformed it from day 5; its Q curve tracked rewards.The day-8 retreat coincided with launching LADDER at 90% of auctions, which mixed experimental data.Source: Evaluation · p. 6
• LADDER increased revenue by more than 50% since the first day and raised CTR by about 35%.The paper attributes the CTR increase as reasonable because the experimented publishers were on a CPC basis.Source: Evaluation · p. 6
• During JD.com’s 2017 June 18th sale, revenue for four publishers rose 54%, advertisers’ ROI rose 17%, JD’s DSP total revenue grew 17%, and sale-wide total ROI rose 7%.The paper presents these results as evidence of permanent improvement according to the holdback and of responsiveness in a volatile, competitive environment.Source: Figure 5 · p. 6, Evaluation · p. 6

Exploration and Exploitation

• At π=0.6, the deployed setting was chosen to maximize revenue; lowering π to 0.55 decreased revenue while increasing rewards and CTR.The authors interpret the change as the agent exploring less aggressively.Source: Exploration and Exploitation · p. 7

Visualization

• t-SNE visualization showed that LADDER mapped all four publishers into separate clusters.The analysis used publisher type as the label and visualized 1000,000 random samples.Source: Visualization · p. 7
• Within publisher 1, LADDER learned richer auction conditions, including that JDLN delivery may be more attractive for cold-start users.The visualization indicates that LADDER recognizes this condition from the plain-text inputs.Source: Visualization · p. 7

Future Work

• Future directions include quantitative trading, recommendation with a domain-specific loss function, and applying LADDER to the ranking phase of online advertising.Source: Future Work · p. 7

Conclusions

• In A/B tests, LADDER easily outperformed the existing industrial state-of-the-art solution.The paper attributes this performance to using high-level auction semantics without sophisticated feature engineering and reacting immediately to changing auction conditions.Source: Conclusions · p. 7
• DASQN decouples learning, observing, and acting in Q-learning, improving runtime performance and enabling large-scale real-world applications.Source: Conclusions · p. 7