Source-linked AI summary
Reinforcement Learning for Controller Parameters in the Auto-Bidding System at Pinterest
Yichu Zhou, Qinyi Chen, Fei Feng, Jacob Loh, Mehdi Ben Ayed, Lin Yang, Andrei Curelea, Jiajing Xu
TL;DR
Auto-bidding must balance budget delivery and cost efficiency across heterogeneous campaigns and changing conditions, but fixed PID tuning cannot adapt to them. RLBP uses RL to select campaign-specific PID parameters and adapts objective trade-offs at inference. Its online A/B test reports a statistically significant eCPC reduction with no revenue degradation.
Problem
Fixed PID settings cannot simultaneously suit heterogeneous campaigns and shifting market conditions, while direct-bid RL faces production safety, explainability, and offline-learning obstacles.
Source: §1 · p. 1Method
RLBP uses a multi-head Q-network to select daily campaign-specific PID parameter multipliers, with inference-time value fusion and rewards based on metric lift over the production baseline.
Source: §1 · p. 2, §4.4 · p. 5Results
A statistically significant 0.25% eCPC reduction (p < 0.05) was observed while revenue remained approximately unchanged.
Source: §6.2.1 · p. 8Takeaways & Limitations
RLBP demonstrates that an RL meta-controller can augment a production PID auto-bidder and deliver measurable value on live advertising traffic at scale.
Source: §8 · p. 9RL actions are limited to a small set of multipliers around production defaults, narrowing exploration and reducing adjustment precision and range.
Source: §7 · p. 8
Abstract
from arXiv · showhide
Auto-bidding systems in online advertising must balance competing objectives-maximizing budget delivery while minimizing cost per action-across heterogeneous campaigns under non-stationary market conditions. This trade-off evolves over campaign lifespans: early on, cost optimization dominates when budgets roll over; near deadlines, budget delivery becomes urgent. While PID controllers are widely used in production for their stability and interpretability, fixed tunings cannot adapt to diverse campaigns and shifting traffic patterns. We present RLBP (Reinforcement Learning for Bidding Parameters), a hybrid architecture deployed at Pinterest that uses RL as a meta-controller to select campaign-specific PID parameters daily while retaining PID for high-frequency bid adjustments. RLBP trains a multi-head Q-network using a relative-lift reward formulation normalized against the production baseline, and performs inference-time fusion that adapts the cost-delivery trade-off to campaign states without retraining. Through online A/B testing at Pinterest, RLBP achieves a statistically significant reduction in effective cost-per-click (eCPC) with no revenue degradation, and two months of post-rollout measurements against a holdout also show consistent cost reductions. These results demonstrate that meta-controller RL can successfully augment production control systems at scale.
1 Introduction
Auto-bidding must balance budget delivery with cost efficiency across campaigns and changing market conditions, while fixed PID settings cannot suit every campaign. RLBP uses RL to tune PID controllers rather than generate bids, and its live Pinterest evaluation reports lower eCPC without revenue degradation.
1 Introduction
- Fixed PID settings cannot suit hundreds of thousands of heterogeneous campaigns under shifting market conditions, while direct-bid RL raises safety, explainability, and offline-learning concerns.Bidding decisions occur billions of times daily, so fractional policy improvements can translate into substantial advertiser value and platform revenue.Source: §1 · pp. 1–2
- RLBP selects campaign-specific PID parameter multipliers daily, then leaves high-frequency bid adjustments to PID using a lookup table fixed for the next 24 hours.The bounded multipliers keep decisions near a validated production configuration, and serving loads a flat lookup table rather than a model artifact.Source: §1 · p. 2
- The online A/B test shows a statistically significant 0.25% eCPC reduction with no revenue degradation; post-rollout holdout measurements show consistent cost reduction and click-through rate improvement.RLBP was deployed and empirically validated at Pinterest with negligible infrastructure overhead.Source: §1 · p. 2
- A multi-head Q-network learns separate budget-utilization and cost-efficiency goals, whose values are fused at inference to adapt the delivery–cost trade-off to campaign states.Rewards use metric lift against the production baseline rather than absolute revenue or cost values.Source: §1 · p. 2
2 Related Work
Prior work addresses auto-bidding through feedback control, online bid optimization, and RL policies that often produce bids directly. RLBP retains PID pacing while learning campaign-specific controller settings and adapting the delivery–cost trade-off over campaign lifecycles.
2.1 Auto-Bidding and Budget Pacing
- Auto-bidding work includes feedback pacing controllers and dual-based bid-shading optimization; RLBP complements these mechanisms by learning campaign-specific controller settings across campaign lifecycles.Other approaches include allocating delivery rates across traffic segments.Source: §2.1 · p. 2
2.2 Reinforcement Learning for Bidding
- Many deep-RL bidding methods determine bids directly, creating deployment friction in safety-sensitive production; offline RL and safety-constrained exploration address related training and deployment concerns.The cited line includes direct-bid methods and Meta's offline-trained agent for adjusting an existing base policy.Source: §2.2 · p. 2
2.3 Hybrid RL and Classical Control
- Hybrid RL-controller work tunes PID gains rather than replacing the controller; RLBP focuses on daily campaign-specific discrete gain multipliers, multi-head goals, and adaptive value fusion.It retains PID feedback control and validates the design with live Pinterest A/B tests.Source: §2.3 · pp. 2–3
3 Problem Formulation
The formulation uses PID for within-day bid updates and an RL meta-controller for daily campaign-level parameter selection. Campaign state, bounded parameter adjustments, stochastic outcomes, and lifecycle-dependent objectives define the controller-tuning problem.
3.1 PID Controller for Auto Bidding
- PID tracks the error between performance and a target, then adjusts bids; its stability and interpretability make it practical, but fixed parameters cannot fit diverse campaigns or changing markets.The controller's proportional, integral, and derivative error terms are weighted by coefficients that influence bid adjustment.Source: §3.1 · p. 3
3.2 MDP for Controller-based Bidding
- The MDP objective is to maximize expected cumulative discounted reward, represented by action values over the state-action space.The formulation defines state and action spaces, transitions, rewards, and a discount factor.Source: §3.2 · p. 3
- At each day's start, the RL agent selects PID parameters from campaign state, and the PID controller uses them for bid updates throughout that day.State includes campaign identity, remaining lifetime, daily budget, controller targets, and previous-day CTR, CPC, and revenue.Source: §3.2 · p. 3
- Actions multiply the default PID parameters; implementation tunes proportional and integral multipliers over three discrete values each while fixing the derivative multiplier at 1.The nine bounded actions limit exploration around the validated production configuration; derivative error is excluded because it is fairly noisy.Source: §3.2 · p. 3
- Daily transitions depend on aggregate spend, clicks, revenue, and remaining budget, with stochastic outcomes under changing user behavior, auctions, and ad delivery.The non-stationary environment motivates adaptive policies for sustained performance.Source: §3.2 · p. 3
- The formulation balances budget utilization and CPC because higher bids can increase delivery at higher CPC, while lower bids may improve cost efficiency but leave budget unspent.Budget utilization means spending allocated budget to maximize revenue delivery; CPC is the cost-efficiency objective.Source: §3.2 · pp. 3–4
- Deployment sets γ=0 for a one-step contextual-bandit objective based on daily outcomes, avoiding offline bootstrapping errors while leaving cross-day trade-offs to lifecycle-dependent fusion.The passage describes this as a deployment choice; γ∈(0, 1) remains possible for longer-term optimization.Source: §3.2 · p. 4
4 Proposed Methods
RLBP uses a multi-head Q-network to learn campaign control values for separate objectives, then fuses them at inference to select PID parameter actions. Relative-lift rewards compare actions with the production baseline, while the fusion weight adapts the cost–delivery trade-off to campaign state.
4.1 Multi-head 𝑄-Networks
- A multi-head Q-network predicts separate budget-utilization and CPC values from a shared state representation, with outputs fused only at inference.Different fusion choices produce different policies from one trained model.Source: §4.1 · p. 4
4.1.1 Loss Functions.
- Each Q-head has a temporal-difference loss; with γ=0, the heads fit their corresponding daily rewards, and the CPC head uses a min Bellman backup.The CPC backup reflects that lower CPC lift indicates better cost performance.Source: §4.1.1 · p. 4
4.1.2 State Representation.
- The state representation combines embedded categorical features with batch-normalized numerical features, after normalizing the target spend curve and log-transforming budget features.Vector and scalar numerical features are concatenated before batch normalization.Source: §4.1.2 · p. 4
4.2 Inference-time Adaptive 𝑄-Value Fusion
- The cost-penalty weight decreases as the campaign deadline approaches, shifting policy emphasis toward budget utilization.The fused value subtracts the weighted CPC value from the budget-utilization value.Source: §4.2 · pp. 4–5
4.3 Policy Induction
- A top-K stochastic policy retains the actions with highest fused scores and applies a temperature-scaled softmax over that reduced set.The final bidding-control action is sampled from the resulting distribution.Source: §4.3 · p. 5
4.4 Reward Shaping
- Metric-lift rewards measure action outcomes relative to matched production-baseline estimates, aligning target scales across campaigns and simplifying inference-time objective balancing.CPC is total spend divided by clicks; budget utilization is total spend divided by allocated budget. Figure 2 compares relative action gaps under raw CPC and CPC-lift rewards.Source: §4.4 · p. 5, Figure 2 · p. 5
5 Production Integration
RLBP’s production integration has three stages: online exploration and exploitation, offline training, and daily serving. Daily decisions are materialized as campaign-level lookup tables for the existing bidder.
Online exploration-and-exploitation
- Online exploration reserves part of each campaign’s budget for uniform random actions while the remainder serves the RL policy, using separate user buckets and budgets.Assignments refresh daily, allowing exploratory actions to be evaluated within their allocated traffic and budget.Source: §5 · pp. 5–6
Offline training
- A periodic offline batch job aggregates historical transition samples and retrains the Q-network’s separate objective-specific functions.The resulting checkpoint generates serving-time decisions for active campaigns.Source: §5 · p. 6
Daily serving
- Daily serving selects campaign actions and stores their controller multipliers in lookup tables; the existing bidder fetches them for second-scale bid adjustments without request-time model inference.Figure 3 depicts the end-to-end pipeline’s online, offline-training, and daily-serving stages.Source: §5 · pp. 5–6, Figure 3 · p. 6
6 Experiments
Offline ablations compare reward shaping and adaptive fusion, followed by a live A/B test and two months of post-rollout holdout measurements. In the A/B test, RLBP reduced eCPC significantly without materially changing revenue; post-rollout results were directional because no statistical test was conducted.
6.1.1 IPS-based offline lift.
- The offline study used 21 days of training transitions and seven held-out days, estimating policy value with IPS from uniformly randomized exploration logs and reporting lifts separately for budget utilization and cost.The ablations varied relative-lift versus absolute rewards and the adaptive fusion function while holding other components fixed and averaging five training runs.Source: §6.1.1 · p. 6, §6.1.2 · p. 6
6.1.2 Ablation study.
- Relative-lift rewards yielded 2.77% revenue lift on the final campaign day and 3.02% CPC reduction on earlier days, versus 0.08% and 2.17% with absolute rewards.Revenue is evaluated at d=0, while CPC is evaluated at d>0.Source: Table 1 · p. 6, §6.1.2 · p. 6
- The linear adaptive-fusion schedule was the only tested variant with both nonnegative revenue lift (+0.05%) and reduced CPC (−0.34%) across campaigns’ last seven days, so it was selected for production.The sigmoid variant had +1.41% revenue alongside +0.49% CPC.Source: Table 2 · p. 6, §6.1.2 · p. 7
- Greater cost emphasis produced more CPC-reducing actions, gradually under the linear schedule and abruptly under the step schedule.The ablation compared step, linear, and sigmoid fusion-weight functions.Source: §6.1.2 · pp. 6–7, Figure 5 · p. 7
6.2 Online A/B Experiments
- The online evaluation used a seven-day A/B test of relative-lift rewards, linear adaptive fusion, and top-K softmax inference against the default controller without RL-driven adjustment.Results were reported for all treated campaign-days and for campaigns ending during the test window.Source: §6.2 · p. 7
6.2.2 Segmented analysis.
- eCPC point estimates were negative across all reported segments, with the largest gains in high-budget (−0.90% eCPC, +0.81% clicks) and high-CPC (−0.64% eCPC) segments.The segmented analysis groups campaigns by budget level and historical CPC tier.Source: §6.2.2 · p. 8, Table 4 · p. 8
6.3 Post-Rollout Performance
- Over two post-rollout months, RLBP showed nearly 1% lower eCPC where cost had greatest emphasis and higher delivery with lower eCPC where weighting shifted toward budget utilization.These holdout comparisons are directional; the paper reports no statistical test because traffic was imbalanced.Source: §6.3 · p. 8, Table 5 · p. 8
7 Discussion
The discussion outlines extensions to RLBP: multi-step optimization, a broader action space, and more frequent inference, each with practical caveats.
Multi-step learning
- RLBP uses γ=0 as an initial deployment step, while multi-step fusion remains a heuristic without formal guarantees of optimality.In the multi-step setting, each head bootstraps through its own greedy policy, so the fused Q-value is generally not optimal for the combined reward.Source: §7 · p. 8
Expanded action space
- Safety considerations limit actions to a small set of multipliers around production defaults, narrowing exploration and adjustment precision and range.Finer or continuous actions would require broader exploration coverage, additional safeguards, and more robust offline evaluation.Source: §7 · p. 8
More frequent inference
- Inference currently runs once daily; moving toward near-real-time updates would make offline lookup tables unusable and increase infrastructure costs.The paper is exploring shorter cadences, including hourly or minute-level updates, to track intraday fluctuations and market dynamics more precisely.Source: §7 · pp. 8–9
8 Conclusion
RLBP augments a production PID auto-bidder with an RL meta-controller and adaptive fusion. Pinterest’s online A/B experiments show statistically significant eCPC reductions with approximately unchanged revenue.
8 Conclusion
- RLBP augments rather than replaces a production PID auto-bidder, using a multi-head DQN, adaptive inference-time fusion, and relative-lift rewards.The system was deployed at Pinterest.Source: §8 · p. 9
- Online A/B experiments at Pinterest found statistically significant eCPC reductions with approximately unchanged revenue.Source: §8 · p. 9
- Future directions include multi-step optimization, continuous action spaces, and generalization to additional campaign types and controller parameters.Source: §8 · p. 9