Fine-Tuning Autobidders with Group Relative Policy Optimization

Anton Safin; Alexandra Khirianova; Andrey Pudovikov; Aleksandr Katrutsa; Egor Samosvat

Source-linked AI summary

Fine-Tuning Autobidders with Group Relative Policy Optimization

Anton Safin, Alexandra Khirianova, Andrey Pudovikov, Aleksandr Katrutsa, Egor Samosvat

arXiv:2608.28199v1Aug 28, 2026cs.GT

TL;DR

Autobidding requires sequential bid decisions that maximize campaign value under budget and efficiency constraints, yet actor–critic RL can be unstable in noisy, long-horizon auctions. This paper adapts critic-free GRPO to fine-tune a strong autobidding baseline, finding consistent click gains and best-or-second-best conversion performance across three benchmarks.

Problem
Actor–critic autobidding is difficult to train reliably because learned value estimates become unstable under long campaigns and noisy auction outcomes.
Method
AB-GRPO fine-tunes the ALM controller by sampling multiple trajectories for a shared campaign context and updating a single policy from their relative returns without a separate critic.
Results
Across BAT, iPinYou, and AuctionNet, AB-GRPO consistently outperforms baselines in clicks and is the best or second-best method in conversion volume.
Takeaways & Limitations
AB-GRPO provides a critic-free alternative for constrained autobidding and improves a strong heuristic baseline through policy fine-tuning.

Abstract

from arXiv · show

Automated bidding (autobidding) is a core component of modern online advertising systems. Within this component, advertisers delegate sequential bid decisions to algorithms that must maximize campaign value while adhering to constraints such as a limited budget and a target cost-per-click (CPC). One of the approaches to resolve the autobidding problem is to formulate it as a Markov decision process and use reinforcement learning (RL) to train a bid generation function. The standard RL framework is actor-critic, which consists of an actor network that generates actions and a critic network that estimates the value of those actions. In our setting, the action is typically a bid or related pacing multiplier, and the value is the expected return from the auction given the bid. However, the alternating training of actor-critic RL models leads to instability and reduced robustness to noise. To address these issues, we adapt the Group Relative Policy Optimization (GRPO) framework to the autobidding setting. This framework is a \emph{critic-free} policy-gradient method originally developed for large language model post-training, where the ground-truth target is unknown. The autobidding setting shares this property, since the optimal bid is unknown in advance. Moreover, GRPO in the LLM domain is used to fine-tune the pre-trained model, and we use the same technique to enhance the performance of the strong heuristic baseline. We empirically compare Autobidding GRPO with actor-critic models, simple heuristics, and controller-based methods on the BAT, iPinYou, and AuctionNet benchmarks. Extensive experiments show that Autobidding GRPO consistently outperforms baselines in clicks and is the best or second-best method in conversion volume.

1 Introduction

Autobidding is a sequential constrained decision problem, but actor–critic reinforcement learning can be unstable under long-horizon budget coupling and noisy auctions. The paper adapts critic-free GRPO to fine-tune a strong base autobidder and evaluates it against established alternatives.

Motivation: Autobidders adjust bids continuously to maximize campaign objectives while respecting budget and efficiency constraints such as target CPC or ROAS.These decisions affect advertiser return on investment and marketplace outcomes.
Motivation: Sequential bidding can be formulated as an MDP or constrained MDP because early decisions change remaining budget and later feasible actions.The bidder observes campaign and market state, selects a bid or pacing multiplier, and receives stochastic auction rewards and costs.
Challenge: Actor–critic methods can become unstable because their learned value estimates are difficult to fit reliably under long campaigns and noisy auctions.The instability affects policy updates when the critic is trained alongside the bidding policy.
Approach: AB-GRPO adapts critic-free GRPO by sampling multiple bidding trajectories for a fixed campaign context and updating the policy toward relatively better outcomes.The approach avoids training a separate critic and uses GRPO-style fine-tuning for autobidding.
Evaluation: AB-GRPO is evaluated on three benchmark datasets against heuristics, controller-based autobidders, and actor–critic reinforcement-learning methods.The reported comparison includes BAT, iPinYou, and AuctionNet.

2 Related works

Prior work applies RL and GRPO to bidding, language-model post-training, and advertising allocation, but these systems address different settings. The paper positions its method as a direct GRPO-based budget-pacing approach evaluated head-to-head with actor–critic autobidders.

RL autobidding: On-policy RL methods such as PPO face sample inefficiency, sparse and noisy rewards, non-stationary market dynamics, and the need for a separate critic network in RTB.These challenges motivate alternatives to standard actor–critic training.
GRPO: GRPO replaces an explicit value function with relative advantages computed by normalizing rewards across candidate actions sampled from the same state.The group baseline eliminates the separate critic network.
GRPO: Studies of GRPO report that larger groups can improve training stability and accuracy, while critic necessity may depend on the RL task and horizon.The cited comparisons include PPO, GRPO, and DAPO in LLM reasoning and classical single-task RL environments.
Related systems: Generative Bid Shading applies GRPO to per-request bid shading, a different objective from campaign-level budget pacing.Its goal is to reduce individual impression bids while maintaining win rate.
Related systems: DARA uses GRPO-Adaptive for few-shot budget allocation across multiple advertising channels and dynamically updates its reference policy.This differs from the autobidding setting compared in the paper.
Positioning: The paper claims the first GRPO-based corrective enhancement for budget pacing in second-price auctions and a direct comparison with actor–critic RL methods.Table 1 summarizes differences among closely related GRPO-based autobidding systems.

3 Problem statement: autobidding as RL

The paper models campaign bidding as a constrained sequential decision problem whose stochastic returns depend on auction outcomes, traffic, and user responses. GRPO replaces the difficult learned critic with relative comparisons among trajectories from a shared campaign context.

Campaign objective: A campaign spans T hours with total budget B and optional target average CPC C, while the autobidder submits nonnegative bids for auctions in each hour.Winning auctions incur clearing prices and can produce clicks or conversions.
Campaign objective: The campaign objective maximizes expected value subject to budget and average CPC constraints, with conversion maximization obtained by substituting conversions for clicks.Expectation covers auction competition, user responses, and traffic.
RL formulation: Sequential bidding is represented as an MDP or CMDP with states describing campaign progress and market context and actions consisting of bids or pacing multipliers.State features include remaining budget, remaining time, cumulative value and spend, predicted CTR, and local market signals.
RL formulation: Transitions arise from auction clearing, stochastic clicks and conversions, budget spending, and exogenous traffic dynamics.Rewards measure acquired value and may include penalties for constraint violations or pacing irregularity.
RL formulation: The policy πθ(at | st) maximizes expected return while satisfying budget and CPC constraints over an episode representing one campaign realization.Episodes use a fixed campaign context and auction-environment seed or statistics.
GRPO formulation: GRPO samples G trajectories from a shared context, assigns each a scalar return, and computes relative advantages using the group mean and standard deviation.This replaces the learned value function used by actor–critic methods.
GRPO formulation: The resulting policy update is clipped and importance-weighted like PPO but omits a value network, reducing memory and compute overhead.The design matches evaluation of alternative bidding behaviors under identical campaign conditions.

4 AB-GRPO framework

AB-GRPO adapts critic-free GRPO to autobidding by fine-tuning a stable ALM controller with bounded policy corrections. It uses group-relative rewards and policy optimization to train under campaign-level sequential bidding dynamics.

Framework adaptation: AB-GRPO adapts the original GRPO framework to autobidding and aligns its policy representation, state and action spaces, objective, and training loop with a campaign-level MDP.The policy is implemented with a lightweight MLP for continuous bid-control actions.
Adaptive linear model baseline: ALM is a traffic-aware pacing controller that allocates budget spending proportionally to campaign traffic and forecasts the remaining budget at campaign end.Its state uses normalized campaign features including budget, traffic, prior bids, and predicted click-through rate.
Policy and action representation: AB-GRPO trains a bounded correction δt to a fixed ALM gain, so the learned policy modifies the baseline rather than learning bids from scratch.The correction is constrained to δt ∈ [−∆max, ∆max] with ∆max = 1.5, while the resulting gain directly affects bid estimation.
Objective function: The objective combines a clipped policy loss with a KL penalty and entropy bonus, using β = 0.01 and α = 0.001, while refreshing the reference policy after each full epoch.The clipped surrogate uses ϵ = 0.2.
Training procedure: For each campaign context, AB-GRPO samples G independent full-horizon rollouts, computes rewards from total clicks or penalized and baseline-relative variants, and assigns group-standardized advantages to transitions.Collected on-policy data are then used for one or several Adam gradient steps and discarded.
Algorithm features: Eliminating the learned critic replaces it with a group-based empirical baseline, which the paper identifies as desirable for sparse, long-horizon budget-pacing settings.The design also targets actor–critic instability, noise, and additional memory and inference costs.

5 Numerical experiments

The experiments compare AB-GRPO with heuristics, controller-based bidders, and actor–critic RL across BAT, iPinYou, and AuctionNet, using clicks as the tuning objective. AB-GRPO generally leads in clicks, performs competitively in conversions, and has moderate pacing and complexity trade-offs.

Experimental setup: The evaluation covers BAT, iPinYou, and AuctionNet, comparing AB-GRPO with linear, controller-based, and actor–critic autobidders.The benchmarks span different auction logs and settings, including second-price environments and varying campaign structures.
Experimental setup: Total clicks on the test set are the primary metric, averaged over three random seeds, while conversions, pacing RMSE, and CPC are secondary trade-offs.Hyperparameters are selected using validation clicks before test reporting.
BAT: AB-GRPO achieves the highest clicks and conversions on BAT, while its RMSE and CPC remain moderate and below the base ALM method.PID has the smallest RMSE, but not the best click or conversion performance, indicating that pacing RMSE does not alone capture bidder quality.
iPinYou: AB-GRPO provides the most clicks on iPinYou and the second-highest conversion volume, matching ALM on conversions while improving the primary objective.Linear and FAB achieve the highest conversion count, but FAB produces substantially fewer clicks.
AuctionNet: AB-GRPO outperforms ALM and actor–critic RL methods in both clicks and conversions on AuctionNet, with RMSE and CPC in the middle of the baseline range.Its relative position is consistent with the BAT and iPinYou experiments.
Complexity: AB-GRPO is faster at inference and requires fewer training resources than actor–critic alternatives, but remains slower than ALM and Linear.The latter baselines lack an additional policy model, which accounts for their lower complexity.

6 Conclusion and future work

AB-GRPO is presented as a critic-free GRPO-based autobidding algorithm that improves a pre-tuned controller and performs strongly across three public datasets. The authors identify extensions to other controllers, auction settings, and real-world online evaluation as future directions.

Conclusion: AB-GRPO is a critic-free alternative to actor-critic autobidding algorithms, using the ALM controller as a pre-tuned model for correcting its factors.The design avoids a separate critic model and is intended to make policy training straightforward.
Conclusion: AB-GRPO is the best or second-best method in clicks and conversions across BAT, iPinYou, and AuctionNet.It consistently outperforms the actor-critic RL baselines USCB and FAB.
Conclusion: AB-GRPO is faster at inference than actor-critic RL methods and comparable in training and tuning to controller-based bidders.This complexity comparison supports practical efficiency relative to the evaluated alternatives.
Future work: The work takes a first step toward transferring GRPO through budget pacing into autobidding.The current approach uses ALM as the base model and trains a policy to correct its gain factor.
Future work: Future directions include extending AB-GRPO to other controllers and heuristics, other auction settings, and online evaluation in real-world auction environments.The authors describe these extensions as straightforward or promising directions for further work.