AdCoEvo: A Co-Evolving Multi-Agent Advertising Delivery System

1 Introduction

• Existing workflows rely on operator judgment, are difficult to scale, and make accumulated experience hard to reuse as operational capability.Prior automation addresses individual tasks such as bidding and budget allocation, while overall operations still require human coordination.Source: §1 · p. 2
• Role-specific training must account for differing scopes of influence and feedback timing, including cumulative returns and diagnosis that guides the next period.Planning and ad-unit adjustments affect subsequent delivery; diagnosis takes effect through later planning and adjustment.Source: §1 · p. 2
• Changing auction conditions can make historical strategies mismatched to current markets, motivating timely strategy updates.The paper frames continuous business feedback and collaboration as a way to study adaptation to market changes.Source: §1 · p. 2
• AdCoEvo improves advertiser value and reduces cost per acquisition in both environments, with co-evolution yielding greater returns than fixed parameters.The system updates one role per period using settled feedback while other roles’ parameters remain fixed.Source: §1 · p. 2
• AdCoEvo assigns account budget planning, ad-unit strategy adjustment, and post-delivery diagnosis to specialized roles across timescales and operational granularities.Diagnostic results guide planning and adjustment in the next period, while settled feedback supports rotating role-policy updates.Source: §1 · p. 2
• The system evaluates delivery across market sequences and compares role compositions, training stages, and role-update schemes.These comparisons examine role collaboration, operational capability development, and co-evolution.Source: §1 · p. 3

2.1 Advertising Delivery

• Advertising-delivery research has automated bidding and joint budget allocation, while AdPilot supports end-to-end delivery through diagnosis, decisions, and post-training.AdCoEvo instead connects specialized role policies with next-period diagnosis and updates based on settled feedback.Source: §2.1 · p. 3

2.2 Multi-Agent Collaboration

• Multi-agent research organizes collaboration through workflows, task allocation, supervision, and training signals from joint execution or interaction.AdCoEvo focuses on collaboration and continual evolution in advertising delivery tasks.Source: §2.2 · p. 3

2.3 Agentic RL

• Agentic reinforcement learning uses interaction feedback and trajectories to improve decision policies; AdCoEvo alternates role updates for mutual adaptation during delivery.Its training signals reflect each role’s scope of influence and feedback timing, with updates shaping other roles’ later feedback and learning.Source: §2.3 · p. 3

3 Problem Formulation

• The formulation treats continuous advertising delivery as sequential decision-making with value-generation and conversion-cost objectives under budget and adjustment-count constraints.The constraints apply across periods, accounts, and ad units.Source: §3 · p. 3

3.1 Modeling Delivery Decisions

• During delivery, the system plans ad-unit budgets and adjusts control tCPA; a fixed bidder uses these settings for traffic opportunities.Settled period outcomes generate diagnostic guidance for planning and strategy adjustment in the next period.Source: §3.1 · p. 4
• Decisions are indexed chronologically under policy π and use observable information available at the time of each decision.That information includes decision type, delivery state, execution records, settled feedback, and historical diagnostic guidance.Source: §3.1 · p. 4
• Decision outputs include account ad-unit budget allocations, control tCPA, and diagnostic guidance.Source: §3.1 · p. 4

3.2 Business Objectives and Constraints

• The system measures business outcomes with ADV for value generation and CPA for conversion costs across controlled accounts.ADV uses initial tCPA and summed predicted conversion probabilities for valid impressions; control-tCPA changes affect it through delivery outcomes.Source: §3.2 · p. 4
• The objectives use average ADV per account and aggregate CPA, defined as total spend divided by total conversions.The formulation optimizes both objectives subject to the stated constraints.Source: §3.2 · p. 4
• Ad-unit budget limits are nonnegative, reallocation preserves each account’s total budget, and valid tCPA adjustments per ad unit are capped at K per period.The policy set uses only information observable at decision time, and CPA requires positive expected conversions.Source: §3.2 · p. 4

4 Method

• Planner, Optimizer, and Inspector collaborate across periods, while two-stage training develops their capabilities offline and updates role parameters using settled feedback during delivery.Figure 2 contrasts delivery execution in its upper panel with role-policy updates in its lower panel.Source: §4 · p. 4, Figure 2 · p. 5

4.1 Role Specialization and Collaborative Execution

• Planner handles account budget planning, Optimizer adjusts ad-unit strategies, and Inspector aggregates period outcomes into guidance for the next period.Planner and Optimizer act for one account and its ad units at a time; Inspector aggregates across controlled accounts.Source: §4.1 · p. 5

4.2 Role Reward Construction

• Planner and Optimizer rewards reflect value generation and cost control, while Inspector’s reward measures changes in overall feedback between adjacent settled periods.Unsettled diagnoses are excluded from training.Source: §4.2 · pp. 5–6

4.3 Offline Development of Operational Capabilities

• Supervised fine-tuning on role-specific demonstrations precedes offline reinforcement learning, which compares candidate responses while holding other roles’ policies fixed.Planner and Optimizer use normalized leave-one-out advantages; Inspector’s advantage is the return difference from a baseline without diagnostic guidance.Source: §4.3 · p. 6

4.4 Co-Evolution during Continuous Delivery

• Settled delivery trajectories support one-at-a-time role updates; subsequent joint execution changes the states and feedback to which other roles adapt in later rotations.Updated policies take effect in the next period, while the other role parameters remain fixed during each update.Source: §4.4 · pp. 6–7

4.5 Inference Time Complexity Analysis

• Per-period model calls scale as O(|A|n), and fixed-size Transformer self-attention costs O(L^2) per call with KV caching.Here, |A| is the number of controlled accounts, n is the number of decision points per period, and L is the combined input and output length per call.Source: §4.5 · p. 7

5.1 Experimental Setup

• Experiments use AuctionNet Sparse and General, controlling 12 of 48 ad units across three accounts with Abid as the fixed underlying bidder.Each experimental group uses five matched market sequences of 50 consecutive periods.Source: §5.1 · p. 8

5.2 Overall Delivery Performance

• 14.02% higher ADV and 7.07% lower CPA than Rule in Sparse; AdCoEvo also improves both metrics in General and ranks highest on ADV and lowest on CPA across both environments.Table 1 aggregates five 50-period sequences.Source: §5.2 · pp. 8–9, Table 1 · p. 9

5.3 Comparison with Foundation Models

• AdCoEvo achieves the highest ADV, lowest CPA, and highest composite score; versus GPT-5.6-Terra, it raises ADV 3.73% and reduces CPA 3.23%, with average API cost CNY 0.235 per period, 65.59% below Gemini-3.1-Flash-Lite.Qwen3.5-397B-A17B, a larger model from the same Qwen family, does not achieve better delivery results, indicating operational capability cannot be assessed by model size alone.Source: §5.3 · p. 9

5.4.1 Role Composition Ablation

• Adding Inspector to the Planner–Optimizer combination increases ADV by 4.95% and reduces CPA by 9.57%; the full system beats No Operation on both metrics.Inspector turns execution outcomes into guidance for Planner and Optimizer’s subsequent decisions.Source: §5.4.1 · p. 10

5.4.2 Policy Learning Stage Ablation

• SFT increases ADV and reduces CPA, and subsequent GSPO further improves both; role collaboration alone does not guarantee high-quality decisions.Supervised demonstrations provide initial capabilities, while reinforcement learning optimizes delivery strategies using execution outcomes.Source: §5.4.2 · p. 10

5.4.3 Co-Evolution Ablation

• Against AdCoEvo (Fixed), co-evolution raises ADV 3.96% and reduces CPA 1.24% in Sparse; full role updates further raise ADV 2.88% and Score 2.78%, with CPA up approximately 0.10%.Table 5 compares role-update settings retaining the complete role composition; after market switches, AdCoEvo retains an ADV advantage and gradually recovers its CPA improvement after returning to General.Source: §5.4.2 · p. 10, §5.4.3 · pp. 10–11, Table 5 · p. 11

5.4.4 Diagnostic Guidance Ablation

• Inspector-generated diagnoses outperform fixed statistical summaries: ADV rises 1.61% and CPA falls 1.55% in Sparse, with corresponding gains of 1.64% and 1.34% in General.The results support using diagnostic guidance for subsequent account planning and strategy adjustment rather than statistics alone.Source: §5.4.4 · p. 11

5.5 Budget Constraints and Parameter Sensitivity

• At 60% of the original budget, AdCoEvo raises ADV 7.17% and reduces CPA 0.78% versus AdCoEvo (Fixed); W = 12 and K = 8 performs best among tested configurations.Extending W from 6 to 12 improves both metrics, while extending it to 18 reduces performance; reducing K to 4 lowers ADV 1.28% and raises CPA 0.31%, and increasing K to 12 also worsens both.Source: §5.5 · pp. 11–12, Figure 4 · p. 12, Table 6 · p. 12

6 Discussion and Conclusion

• Operational-level decisions, role collaboration, and post-training improve delivery performance, while co-evolution adds gains beyond offline reinforcement learning.The system organizes delivery through role specialization across timescales and operational granularities.Source: §6 · p. 12
• Co-evolution gains vary across environments, so its effectiveness still requires evaluation in specific delivery scenarios.Further study is also warranted on how operational roles’ decision frequencies and scopes of influence affect co-evolution, and on improving advantage estimation and update mechanisms.Source: §6 · p. 12

A.1 Rewards for Planner and Optimizer

• Planner and Optimizer rewards combine returns for the current account and all controlled accounts, using different aggregation scopes and feedback timing.Planner rewards settle at period end; Optimizer rewards use current-step increments, and the target cost allowance weights units’ initial tCPA by conversions.Source: §A.1 · p. 16

A.2 Period-Level Feedback for Inspector

• Inspector’s period-level feedback aggregates ADV, spend, and conversions across controlled accounts alongside cost-control and adjustment-execution signals.ADV and conversions are standardized using frozen training-set statistics and clipped to [−3, 3].Source: §A.2 · p. 16
• The CPA constraint component is positive below the target cost allowance and negative above it.The allowance is based on units’ initial tCPA weighted by their period conversions.Source: §A.2 · p. 16
• The forced-HOLD rate tracks SET requests converted to HOLD after the adjustment limit is reached, and enters the feedback vector negatively.Once period feedback settles, the previous diagnosis’s reward is computed from the weighted change between adjacent periods.Source: §A.2 · p. 16

B.1 Candidate Advantages and Sequence-Level Optimization

• Candidate advantages use leave-one-out comparisons for Planner and Optimizer, while Inspector compares diagnosis and no-diagnosis branches; sequence-level optimization uses length-normalized importance ratios and a KL penalty.Advantages remain fixed throughout offline optimization.Source: §B.1 · p. 17

B.2 Offline Reinforcement Learning Algorithm

• Offline reinforcement learning gathers role-specific decision snapshots, samples candidate responses, computes and fixes advantages, and stores valid candidates in training batches.For Inspector, the algorithm also evaluates a branch without a diagnosis.Source: §B.2 · p. 17

C Co-Evolution Update Details and Algorithm

• Continuous co-evolution uses settled execution trajectories, cycles updates through Planner, Optimizer, and Inspector, and applies updated policies in the next period.Newly settled records enter role buffers; used records are removed after forming a scheduled training set.Source: C Co-Evolution Update Details and… · p. 17

C.1 Value Baselines and Advantage Estimation

• Independent value models estimate scalar returns; Planner and Inspector use return-based advantages, while Optimizer applies GAE within each account-period sequence.Recursion does not cross accounts or periods, even though rewards include all-account feedback.Source: §C.1 · p. 18

C.2 Value Regression and Policy Updates

• Value models regress against fixed, detached return targets, while later scheduled rounds update the role policy using previously fixed action advantages.Actor updates stop and roll back policy and optimizer state if KL exceeds 0.01 or retained-token fraction falls below 0.4; completed value-model updates remain.Source: §C.2 · p. 18

C.3 Continuous Co-Evolution Algorithm

• The continuous algorithm executes delivery, settles feedback, selects valid records for the scheduled role, updates its value model, and then updates its policy when applicable.Updated role state is carried into subsequent periods.Source: §C.3 · p. 19

D.1 Environment Configuration

• Delivery periods use 48 time steps and 500,000 traffic opportunities, following AuctionNet’s official defaults.General uses a pValue generation baseline of 0.005 and full 48-dimensional budget and CPA-constraint configurations; other auction mechanisms remain unchanged.Source: §D.1 · p. 19
• Within each environment configuration, traffic volume, initial budgets and CPA constraints, competitor composition, and pValue baseline stay fixed across periods.Episodes resample predicted conversion probabilities, their variability, within-period traffic curves, and native competitors’ random states; Sparse and General split episodes 71–320 into five sequences of 50 periods.Source: §D.1 · p. 19

D.2 Training Configuration

• SFT trains Planner, Optimizer, and Inspector on 3,290 DeepSeek V4 Flash demonstration trajectories each, for 4, 3, and 3 epochs, respectively.Training uses a learning rate of 10^-4, microbatch size 8, gradient accumulation of 2, and a maximum sequence length of 2,560 tokens.Source: §D.2 · p. 19
• Offline GSPO uses a learning rate of 10^-5 for 5 epochs, 4 candidates per decision, clipping to [0.8, 1.2], and a KL penalty coefficient of 0.1.Both training stages use LoRA with rank 16 and alpha 32.Source: §D.2 · p. 19
• Co-evolution uses AdamW with actor and value-model learning rates of 10^-6 and 5 × 10^-4, respectively.Rounds with valid nodes update the value model once and the actor at most four times.Source: §D.2 · p. 19

• During delivery, responses use temperature 0.7 with thinking disabled, limits of 8,192 context and 512 response tokens, and HOLD or a listed tCPA multiplier as actions.Top-p is not explicitly specified; available multipliers are 0.5, 0.6, 0.7, 0.8, 1.0, 1.2, and 1.3.Source: §D.3 · p. 19
• If Planner parsing or format validation fails, the remaining budget is allocated in proportion to ad units’ initial budgets.Source: §D.3 · p. 20
• For Optimizer, parsing failure or an incorrect decision count produces four HOLDs, while each invalid multiplier is replaced by HOLD.Invalid responses are counted once during evaluation regardless of the number of errors, and format errors do not trigger resampling.Source: §D.3 · p. 20

• AdPilot* combines reflection and action generation in one model call, with SFT demonstrations from DeepSeek V4 Flash and SFT and GRPO training on AdCoEvo’s training markets.Source: §D.4 · p. 20
• Rule multiplies an ad unit’s budget by 1.1 when CPA does not exceed target and by 0.9 otherwise, then normalizes account budgets to preserve total budget.Source: §D.4 · p. 20
• Rule sets tCPA to 0.7 above 1.10 times target, 1.3 below 0.90 times target, and HOLD otherwise; no conversions or an exhausted adjustment limit also yield HOLD.The rules do not use within-period timing or spending progress to determine when to adjust.Source: §D.4 · p. 20
• Without Optimizer, delivery uses HOLD throughout and retains initial control tCPA; without Planner, each ad unit retains its initial budget limit.Without Inspector, Planner and Optimizer receive current-period delivery state but no cross-period diagnostic guidance or fixed statistical summaries.Source: §D.4 · p. 20