Source-linked AI summary
AdsWorldEngine: A Self-Evolving Conversational Advertising Agent through Orchestrator and Tool Coevolution
Simiao Zuo, Chenhui Xu, Yimeng Jia, Qiang Lou, Jian Jiao, Denis Charles
TL;DR
Conversational advertising must infer commercial intent across dialogue and decide when ads are suitable, extending beyond standalone-query retrieval and ranking. AdsWorldEngine jointly optimizes an ad-serving agent and its tools, improving offline diversity by 62.87% and relevance by 82.26% over the Production System.
Problem
Conversational ad serving must interpret multi-turn context and decide whether the current turn is suitable for ads.
Method
AdsWorldEngine combines an Opportunity Gate, Orchestrator, offline Evaluator, iterative actor-tool optimization, and label grounded judgment modeling.
Results
After three rounds, the final system improves diversity by 62.87% and relevance by 82.26% over the Production System.
Takeaways & Limitations
AdsWorldEngine provides a production-oriented approach for jointly learning ad decisions and downstream tools under conversational advertising constraints.
Takeaways & Limitations
For binary tasks, group-standard-deviation reward scaling can remove raw reward-gap information encoding the cost of different mistakes.
Abstract
from arXiv · showhide
Conversational advertising aims to deliver useful ads within multi-turn assistant interactions. Unlike conventional query-based advertising, where the user's intent is often expressed in a short standalone query, conversational ads must infer latent commercial intent from the current user query, the assistant response, and dialogue history while also deciding whether an ad would be helpful rather than intrusive. We propose AdsWorldEngine, an agentic framework for conversational advertising. AdsWorldEngine uses an Opportunity Gate to determine whether ads should be shown, an Orchestrator to generate commercial intents, call advertising tools, and construct a top-3 ad slate, and an Evaluator to score delivered ads for offline optimization. The central contribution is an iterative actor-tool training procedure: we first train the Orchestrator with supervised fine-tuning and agentic reinforcement learning, then use high- and low-reward rollouts to construct preference data to train tools. This creates a self-improving loop in which the system learns not only how to use advertising tools, but also how to improve them from rewarded behavior. To support subjective production decisions, we introduce label grounded judgment modeling, which trains judgment models from human labels collected under explicit guidelines. It enriches labels with thinking traces, filters inconsistent rationales through reflection, and further optimizes binary judgments with a cost sensitive GRPO variant that preserves asymmetric reward gaps. Offline, AdsWorldEngine improves diversity by 60% and relevance by 80% over the current production ad delivery system. In an online A/B test, it increases RPM by 22% and ads coverage by 74%.
1 Introduction
Conversational advertising requires inferring latent commercial intent across dialogue and deciding whether ads are appropriate, making ad serving a broader problem than retrieval and ranking. AdsWorldEngine addresses this with a gated agentic framework and iterative optimization of its Orchestrator, advertising tools, and judgment models.
- Problem: Conversational advertising must interpret accumulated or implied intent from prior turns and assistant responses, rather than relying on a standalone ad query.Commercial needs may be refined across turns through requests such as “cheaper ones” or “anything near me.”
- Problem: The system must decide whether ads suit the current turn, infer commercial intents, call retrieval and ranking tools, and construct a useful slate.These requirements make conversational advertising a full ad serving problem rather than only retrieval and ranking.
- Framework: AdsWorldEngine uses an Opportunity Gate for ad eligibility and an Orchestrator that resolves context, proposes intents, calls tools, and selects the best ads.The framework is designed for production settings and optimizes the advertising experience across the entire interactive process.
- Optimization: Iterative actor-tool optimization trains the Orchestrator and retrieval, relevance, and ranking tools from rewarded behavior, improving both tool use and tool quality.The technical challenge is that supervised training does not fully optimize interactions between intermediate decisions and downstream tools.
- Judgment modeling: Label grounded judgment modeling uses human labels, reflected thinking traces, consistency filtering, and cost-sensitive rewards to model production judgments with asymmetric error costs.The approach addresses judgments involving usefulness, relevance, and user experience, where inappropriate exposures may harm trust more than missed opportunities.
2 AdsWorldEngine Overview
AdsWorldEngine frames conversational advertising as a gated agentic process that separates whether to show ads from which ads to show. Its pipeline combines an Opportunity Gate, Orchestrator, Tool Set, and Evaluator with judgment modeling and iterative actor-tool optimization.
- Pipeline Overview: The online pipeline observes the user query, dialogue history, policy-allowed user context, and assistant response before deciding whether ads should be shown and which ads to show.Assistant recommendations and context carried across turns can clarify latent intent.
- System Components: The framework has four components: Opportunity Gate, Orchestrator, Tool Set, and Evaluator.The Opportunity Gate combines production rules with a learned triggering model to abstain or pass a turn.
- System Components: The Orchestrator resolves conversation state, generates commercial intents, calls advertising tools, and selects a slate with three ads.This combines dialogue reasoning with retrieval, ranking, relevance, pricing, and other advertising signals.
- Judgment Modeling: Label grounded judgment modeling uses guideline-conditioned thinking traces, reflection-based filtering, and cost sensitive GRPO for asymmetric binary judgments.The method allows false positive triggers to receive a larger penalty than false negatives.
- Actor-Tool Optimization: Iterative actor-tool optimization trains the Orchestrator with SFT and agentic reinforcement learning, then uses high- and low-reward rollouts to create tool preference data.Alternating updates let rewarded Orchestrator behavior improve retrieval, relevance, and ranking tools in later rollouts.
3 AdsWorldEngine Core Components
AdsWorldEngine coordinates ad serving through an Opportunity Gate, an Orchestrator, and an offline Evaluator. The gate decides whether advertising is appropriate, the Orchestrator generates intents and selects three ads, and the Evaluator supplies relevance and diversity signals for optimization.
- Opportunity Gate: The Opportunity Gate combines hard production rules with a learned model to show ads only when they help the current task, abstaining in sensitive, informational, or disruptive contexts.It rejects turns that violate policy or privacy constraints and treats false positives as especially costly.
- Orchestrator: After acceptance, the Orchestrator resolves dialogue state, preserves earlier constraints, generates one to three ad intents, calls tools, inspects candidates, and selects exactly three ads.Its tools can provide retrieval, relevance, ranking, pricing, and signal capabilities.
- Orchestrator: The Orchestrator is trained first with supervised trajectories for valid tool use and constraint preservation, then with agentic reinforcement learning that rewards complete trajectories producing better final slates.GRPO aligns intent decomposition, tool calls, and reflection with slate-level objectives such as relevance and diversity.
- Evaluator: The Evaluator operates offline, scoring ad quality to provide rewards for Orchestrator training and preference data for improving downstream tools rather than directly affecting online serving.Its scope includes user, advertiser, and publisher perspectives, with the current implementation focused on user signals.
- Evaluator: User-side evaluation measures relevance per ad and slate-level diversity through pairwise cosine similarity, penalizing near-duplicate ads among the three selected ads.The learned relevance model judges whether each candidate helps the user’s task given the conversation context.
4 Label Grounded Judgment Modeling
AdsWorldEngine grounds judgment models in human labels, task-specific guidelines, and filtered reasoning traces, then optimizes them with task rewards. For asymmetric binary decisions, its cost-sensitive GRPO variant preserves production-relevant reward gaps rather than normalizing them away.
- Data pipeline: Human labels are assigned under task-specific guidelines, expanded into guideline-conditioned thinking traces, and checked by reflection before supervised fine-tuning.Only examples that pass reflection are retained; inconsistent traces may contradict labels, add unsupported assumptions, or violate the required format.
- Judgment model training: The two-stage training procedure combines supervised fine-tuning with GRPO rewards for label correctness, format validity, and guideline consistency.Supervised fine-tuning teaches the task format and guideline-grounded reasoning before reward optimization.
- Cost-sensitive GRPO: Cost-sensitive rewards penalize false-positive ad triggering more strongly than false negatives, reflecting asymmetric production costs.In the stated binary reward matrix, predicting Yes for a No label receives −2, while predicting No for a Yes label receives −1.
- Cost-sensitive GRPO: Standard GRPO’s standard-deviation scaling can erase raw reward-gap information in binary tasks, so the method uses group centering without that scaling.The variant preserves within-group comparisons while retaining the magnitude of the cost-sensitive reward gap.
- Applications: The label-grounded framework supports binary Opportunity Gate judgments and conversation-to-ad relevance modeling, with task-specific labels, guidelines, traces, and rewards.The framework is applied separately to each task rather than using a shared judgment specification.
5 Orchestrator and Tool Iterative Training
AdsWorldEngine jointly improves its Orchestrator and advertising tools through alternating reward optimization and preference learning from high- and low-reward rollouts. This feedback loop strengthens tool use, expands the actor’s effective action space, and produces better supervision for subsequent updates.
- Iterative actor-tool training: The Orchestrator resolves conversations, generates intents, calls retrieval, relevance, pricing, signal, and ranking tools, and selects the final ad slate.The tools define the Orchestrator’s action space and are trained rather than treated as fixed infrastructure.
- Iterative actor-tool training: Training begins with supervised fine-tuning, then alternates GRPO optimization of the Orchestrator with preference-data construction for tools from improved rollouts.Evaluator rewards guide Orchestrator updates for intent decomposition, tool-use strategy, and reflection behavior.
- Rollout-derived supervision: High-reward slates identify preferred retrievable ads, while low-reward slates expose unsuitable retrieved candidates and provide realistic preference pairs for relevance and ranking.Final slate-level rewards are converted into supervision for intermediate tools.
- Feedback mechanism: The coupled loop improves Orchestrator tool use and tool quality, expanding the actor’s effective action space and generating more informative rollouts for future updates.Better tools support stronger actor decisions, while the stronger actor supplies improved training data for tools.
- Relevance preference training: For relevance models, each conversational intent forms a preference triple with a high-reward positive ad and a low-reward hard negative, scored against a frozen reference model.The method applies a DPO-like preference principle to relevance scores by increasing the positive-over-negative preference margin.
6 Experiments: Label Grounded Judgment
Label grounded judgment modeling aligns Opportunity Gate and conversation-to-ads relevance decisions with production preferences that prioritize user experience and fewer false positives. Cost-sensitive GRPO improves the false-positive/true-positive tradeoff beyond prompting or SFT alone.
- Opportunity Gate: Cost-sensitive GRPO adjusts the decision boundary to reduce false positives while preserving valid commercial opportunities, unlike prompting or SFT alone.The production operating point emphasizes fewer false positives because bad triggers are visible and can harm the conversation.
- Opportunity Gate: 30,000 human-labeled Opportunity Gate conversations were filtered to 27,000 guideline-consistent reasoning examples for training.GPT-5 generated reasoning traces conditioned on human labels and guidelines, and reflection removed contradictory traces.
- Opportunity Gate: 39.07% FPR improvement with SFT and cost-sensitive GRPO preserves baseline TPR and yields a 2.51% balanced accuracy gain.SFT alone reduced FPR too aggressively, decreasing TPR by 17.19% and balanced accuracy by 4.09%.
- Opportunity Gate: Product language can trigger false positives even when users seek information rather than a retailer or purchase path, such as choosing a concealer shade.Production-aligned reinforcement learning teaches that mentioning or selecting a product is insufficient evidence that an ad improves the experience.
- Conversation-to-ads relevance: 12.71% FPR reduction with GRPO incurs only a 0.78% TPR decrease and produces a 7.51% balanced accuracy improvement for conversation-to-ads relevance.The model was trained from 32,000 labeled examples, with 30,000 retained after reflection and split into 27,000 SFT and 3,000 GRPO examples.
7 Experiments: Orchestrator and Tools
AdsWorldEngine alternates Orchestrator and intent-to-ads tool training, using fixed Evaluator rewards and preference data to co-adapt intent generation and retrieval. After three rounds, it improves diversity by 62.87% and relevance by 82.26% over the Production System.
- Training procedure: Three training rounds alternate GRPO Orchestrator updates with fixed tools and preference-based updates to the intent-to-ads relevance model.The process begins with supervised warmup; the improved Orchestrator generates preference data for the next tool update.
- Training procedure: The fixed Evaluator scores conversation-to-ads relevance and slate diversity, rewarding diverse ad pairs whose embedding similarity is below 0.3.These judgments provide reward signals during Orchestrator optimization and final evaluation.
- Training procedure: Tool preference training makes ranking more intent-specific, reducing overlap among candidate sets and indirectly increasing diversity despite optimizing relevance.It promotes ads satisfying generated intents and demotes hard negatives.
- Results: 62.87% diversity and 82.26% relevance are achieved after three rounds over the Production System, with the largest gains occurring in the first round and diminishing thereafter.The progression suggests gradual co-adaptation between the Orchestrator and relevance tool.
- Case study: After GRPO, the Orchestrator generates grounded, product-specific intents that tools match more easily, yielding less-overlapping candidate sets than SFT-only comparison intents.The case study contrasts two overlapping comparison intents from SFT-only training with one intent per named vehicle after GRPO.
8 Real World Online Experiments
AdsWorldEngine was deployed worldwide across Microsoft Copilot surfaces, using an Opportunity Gate and Orchestrator to deliver ads on eligible conversational turns. In a 20-day online experiment, it increased revenue per mille (RPM) by 22% and ad coverage by 74%.
- 8 Real World Online Experiments: AdsWorldEngine was deployed across Microsoft Copilot’s consumer-facing surfaces worldwide.The deployment covered Copilot surfaces globally.
- 8 Real World Online Experiments: The Opportunity Gate identifies commercial opportunities where ads are unlikely to harm user experience.For eligible turns, the Orchestrator retrieves, ranks, and selects the final ad slate.
- 8 Real World Online Experiments: 22% higher revenue per mille (RPM) and 74% higher ad coverage were observed during a 20-day online experiment.These gains were measured after deploying the trained AdsWorldEngine in Microsoft Copilot.
9 Related Work
Related work studies conversational ad generation and assessment, ad insertion, and advertising economics, while AdsWorldEngine frames conversational advertising as an end-to-end serving problem. Its Orchestrator builds on agentic tool-use research, and its training and evaluation approach relates to rationale supervision, model-based evaluation, and preference learning.
- Recent advertising research uses dialogue context for ad generation and assessment, emphasizing contextual fit, linguistic quality, and personalization.
- Sponsored question answering and response insertion study where ads enter assistant outputs and how generated responses incorporate them.
- AdsWorldEngine treats conversational advertising as an end-to-end serving problem joining opportunity gating, intent formation, retrieval, ranking, and slate selection.
- The Orchestrator builds on agents that combine language reasoning with web interaction, external actions, and learned tool selection, alongside methods for API selection, planning, and error recovery.These prior methods mainly optimize the agent around a given environment.
- Related training and evaluation work uses rationale supervision, iterative feedback, model-based evaluators, detailed critiques, task-specific scores, and preference learning.These approaches provide richer training signals, estimate generation quality, and align behavior with preferred outputs.
10 Conclusion
AdsWorldEngine is a production-oriented conversational advertising framework that jointly learns ad-showing decisions, advertising-tool use, and tool improvement from rewarded outcomes. Its iterative Orchestrator–tool optimization and label grounded judgment modeling improve offline ad quality and online serving performance.
- Conclusion: AdsWorldEngine jointly learns when to show ads, how to use advertising tools, and how to improve those tools from rewarded outcomes.The framework is designed for production-oriented conversational advertising.
- Conclusion: Iterative Orchestrator-tool optimization combined with label grounded judgment modeling improves both offline ad quality and online serving performance.The two components are presented as complementary parts of the framework’s optimization approach.
- Conclusion: The results demonstrate the value of optimizing agent decisions and downstream tools as a coupled system.This conclusion frames Orchestrator and tool optimization as mutually reinforcing rather than independent processes.
A Appendix · A.1 Original Capacity Restore with On-Policy Distillation
Appendix A.1 applies general-domain on-policy distillation to restore broad behavior in specialized Triggering and Relevance Models while preserving their task-specific decision capabilities. Using a shared 10,000-example corpus and a teacher-student objective, OPD improves general-domain benchmark performance and maintains or improves held-out task metrics.
- A.1 Original Capacity Restore with On-Policy Distillation: OPD restores broad behavior lost when task-specific SFT and GRPO specialize the Triggering and Relevance Models away from heterogeneous real-user interactions.The target distribution includes open-domain, multilingual, code-switched, implicit, long-tail, and reasoning-heavy conversations.
- A.1 Original Capacity Restore with On-Policy Distillation: OPD runs separately for Trigger and Relevance, using each aligned checkpoint as the student and the original Qwen3-30B-A3B Thinking model as the teacher.The teacher provides pre-specialization broad-domain behavior, while the student generates on-policy trajectories from its current distribution.
- A.1 Original Capacity Restore with On-Policy Distillation: The shared general-domain mixture contains exactly 10,000 examples spanning mathematical reasoning, science question answering, comprehension, deduction, instruction following, Chinese knowledge, and safety.Its source quotas include 2,600 Orca Math, 900 SciQ, 1,100 SQuAD, 800 RuleTaker, 1,400 Dolly, 1,000 WizardLM, 1,200 BELLE, and 1,000 PKU-SafeRLHF examples.
- A.1 Original Capacity Restore with On-Policy Distillation: The restoration objective combines on-policy trajectory distillation with distillation on original responses and standard next-token supervision.The on-policy term evaluates teacher and student distributions under prefixes generated by the specialized student, while off-policy and SFT terms anchor training to corpus responses.
- A.1 Original Capacity Restore with On-Policy Distillation: 3.45 points: the Triggering Model’s macro average improves over the reported benchmark subset after OPD.The gains-only rule includes benchmarks improving by at least one absolute percentage point; largest gains occur on Winogrande, TruthfulQA, and MuSR.
- A.1 Original Capacity Restore with On-Policy Distillation: 2.46 percentage points: the Relevance Model’s aligned macro average improves across all seven retained benchmarks after general OPD.Largest improvements occur on HaluEval, HellaSwag, and DROP, with gains of 4.40, 4.25, and 4.06 points, respectively.
- A.1 Original Capacity Restore with On-Policy Distillation: 0.10 points: Triggering accuracy improves while FPR decreases by 0.11 percentage points and TPR is preserved relative to the Triggering Model baseline.The held-out triggering set uses deterministic decoding, indicating broad-domain gains without sacrificing existing triggering capability.
- A.1 Original Capacity Restore with On-Policy Distillation: 1.28 percentage points: Relevance TPR improves and accuracy rises by 0.93 points, while TNR decreases by 0.40 points relative to the Relevance Model baseline.The result preserves relevance decision capability while improving positive-example recall and aggregate accuracy, with a minor true-negative recall trade-off.