Source-linked AI summary
AgenticGen: Reward-Guided Agentic Video Generation for Advertising
Xingyuan Bu, Chengru Song, Hao Zhou, Tao Zhou, Dong Li, Wei Li, Shilong Li, Hao Shi, Yongxin Guo, Donghao Zhou, Qiangpeng Yang, Shilei Wen
TL;DR
Advertising video generation requires product-conditioned reasoning tied to online business performance, not only realistic synthesis. AgenticGen decomposes the task into strategy selection and draft generation, learns performance and rubric rewards, and optimizes both stages with DPO and GRPO. In TikTok online A/B experiments, it improves CTR, CVR, and Advv over SFT.
Problem
Advertising video generation must transform product materials into effective platform-native advertisements while connecting generation decisions to online business metrics.
Method
AgenticGen uses two trainable reasoning stages, performance-based and rubric-based rewards, and DPO followed by GRPO with process and outcome rewards.
Results
2.72% CTR, 2.63% CVR, and 9.61% Advv improvements over the SFT baseline were achieved after DPO and GRPO in TikTok online A/B experiments.
Takeaways & Limitations
The results support reward-guided agentic optimization as an approach for improving advertising video generation at scale.
Takeaways & Limitations
Online feedback collection uses a dedicated impression-balanced delivery pipeline rather than the standard advertising delivery pipeline to reduce confounding from cold-start and uneven delivery effects.
Abstract
from arXiv · showhide
Advertising video generation is not only a video synthesis task, but also a product-conditioned reasoning problem whose success is measured by online business metrics. Recent video foundation models can generate realistic clips from multimodal conditions, yet they do not optimize how a product should be transformed into an effective advertisement or how future generation should be improved from online business feedback. To close this loop, we propose AgenticGen, a reward-guided agentic framework that decomposes advertising video generation into two trainable reasoning stages, strategy selection and draft generation, thereby exposing optimization targets that online business feedback can supervise. AgenticGen learns a performance-based reward from accumulated online feedback and a complementary rubric-based reward aligned with human quality standards, then uses them to supervise policy optimization. DPO first moves the agentic policies toward online preferences, and GRPO further refines both stages with process and outcome rewards. Offline experiments validate the reward models and successive policy optimization. Online A/B experiments in the TikTok advertising system show that AgenticGen after DPO and GRPO improves CTR by 2.72%, CVR by 2.63%, and Advv by 9.61% over the SFT baseline.
1 Introduction
Advertising video generation must connect product-conditioned creative reasoning with measurable online business performance. AgenticGen closes this loop by optimizing strategy selection and draft generation with complementary rewards, achieving substantial gains over SFT.
- Motivation: Advertising video generation requires transforming heterogeneous product materials into platform-native videos that communicate value and perform effectively online.The task is framed as product-conditioned reasoning rather than standalone realistic video synthesis.
- Framework: AgenticGen decomposes advertising video generation into strategy selection and executable draft generation, exposing optimization targets for online feedback.A reasoning VLM selects feasible strategies and turns them into drafts that invoke video models and rendering tools.
- Results: 2.72% CTR, 2.63% CVR, and 9.61% Advv improvements over the SFT baseline were observed after DPO and GRPO in TikTok online A/B experiments.The evaluation combines offline experiments with online A/B testing in the TikTok advertising system.
- Reward design: The framework learns complementary performance-based and rubric-based rewards to capture online business value and human quality standards.Performance feedback supplies business preferences, while domain-specific standards address issues such as clickbait and severe video defects.
- Policy optimization: DPO learns business performance preferences from impression-balanced delivery data, while GRPO performs on-policy optimization using process and outcome rewards.Process rewards supervise strategy selection and outcome rewards supervise draft generation.
2 Related Work
Prior work advances multimodal video synthesis and agentic planning, but existing video generators remain reactive and reward-guided systems largely rely on fixed human or rubric feedback. Online business feedback has been used for other advertising modalities, motivating AgenticGen’s video-focused extension.
- Agentic video generation: Video foundation models have progressed to high-fidelity multimodal generation, but they do not autonomously decompose creative goals, select tools, or revise outputs over long horizons.Agentic video systems address this gap through planning, execution, evaluation, self-refinement, and reusable skills.
- Reward-guided generation: Reward-guided policy optimization includes off-policy methods such as DPO and on-policy methods such as PPO, REINFORCE, and GRPO.These methods optimize policies using fixed preferences or samples from the current policy, respectively.
- Reward-guided generation: Existing reward supervision commonly comes from human feedback, evaluation rubrics, Bradley-Terry models, reasoning-based evaluators, or rubric-based judges.Such rewards have improved open-ended textual and visual generation.
- Online advertising feedback: Industrial systems use online business feedback such as historical or click-derived CTR to supervise advertising text, images, and creative combinations.AgenticGen extends this line of work to advertising video generation and its agentic reasoning process.
3 The AgenticGen Framework
AgenticGen is a two-stage pipeline that selects feasible strategies from product materials and converts them into executable drafts for video generation and rendering. An impression-balanced delivery loop collects comparable feedback and trajectories for subsequent reward modeling and policy optimization.
- Pipeline: The framework first selects a feasible strategy group and then generates an executable draft before invoking video generation models and rendering tools.The decomposition exposes trainable targets while keeping final video execution separate from agentic reasoning.
- Strategy Selection: Strategy selection maps product context and material packages to strategies from a predefined catalog under validity constraints.The catalog includes asset editing, reference-guided generation, and cross-asset remixing families.
- Strategy Selection: The reasoning VLM balances product information, multimodal material features, historical performance, source quality, creative diversity, and expected advertising value when choosing strategies.It can favor lightweight editing for strong existing materials or reference-guided generation when source quality is insufficient.
- Draft Generation: Draft generation converts each selected strategy into structured executable fields such as editing choices, selling points, calls to action, storylines, and ordered source clips.The structured draft serves as the interface between agentic reasoning and the final video artifact.
- Initialization and Feedback: SFT initializes both reasoning stages and supplies the deployed policy for online data collection, while impression-balanced delivery reduces confounding from recall, ranking, and cold-start effects.The smaller SFT model reduces serving costs compared with direct deployment of the larger teacher model.
- Online Feedback Collection: Impression-balanced delivery attaches 12 generated videos per product to a dedicated traffic pool and serves them through random choices to balance impressions.The resulting trajectories contain contexts, strategies, drafts, rendered videos, and online feedback for reward modeling and reinforcement learning.
4 Reward Modeling and Policy Optimization
AgenticGen learns complementary business-performance and rubric-based rewards, then applies DPO warm-up followed by GRPO to optimize strategy selection and draft generation. The framework uses matched-context preferences and separate process and outcome supervision to guide its two-stage agentic pipeline.
- Reward Modeling: AgenticGen trains performance-based and rubric-based reward models to capture commercial value and human quality standards.The performance model learns from impression-balanced online feedback, while the rubric model evaluates domain-specific quality dimensions.
- Reward Modeling: Matched-context pairwise comparisons reduce confounding from traffic composition and advertising-system bias in performance reward learning.Videos are compared within shared product and delivery settings after receiving balanced impression opportunities.
- Reward Modeling: The rubric reward model evaluates script, content, decoration, and audio across platform-native style, attention, coherence, product consistency, and aesthetic harmony.Only video pairs with unanimous labels from three annotators are retained for a more stable learning signal.
- DPO Optimization: DPO moves the SFT policies toward business preferences using separate preference data for strategy selection and draft generation.For each stage, the other stage is fixed while online delivery identifies preferred and dispreferred outputs; an NLL term helps preserve valid generation format.
- GRPO Optimization: GRPO further optimizes both stages on-policy, using process rewards for strategy selection and outcome rewards for draft generation.This separates supervision for intermediate reasoning from final video quality after the DPO warm-up.
5 Experiments
The experiments use teacher-generated SFT trajectories, impression-balanced online feedback, human rubric annotations, and staged DPO/GRPO training data. Reward models and policies are trained with multimodal video, audio, narrative, and advertising-specific features.
- Data Construction: The SFT bootstrap uses 50K online product requests and teacher-generated strategy-selection trajectories filtered for output validity.Qwen3-VL-235B-A22B-Thinking generates the reasoning trajectories for the base policy.
- Data Construction: The performance reward dataset contains 1M video pairs collected over one month from impression-balanced online logs.Sampling across time windows reduces delivery-period bias, while balanced impressions provide cleaner pairwise labels.
- Data Construction: Rubric reward data retains video pairs with unanimous labels from three annotators and splits them 9:1 into training and validation sets.Reject sampling with Qwen3-Omni-30B-A3B-Thinking further constructs rubric-reward SFT data.
- Policy Training: DPO uses 50K impression-balanced strategy-selection pairs and 50K draft-generation pairs, while GRPO reuses their prompts for fresh on-policy rollouts.Both stages use separate training and validation data, and GRPO samples new trajectories from the current policy.
- Policy Training: DPO and GRPO are trained sequentially from SFT to DPO to GRPO with separately specified learning rates, rollout size, clipping, and KL settings.GRPO uses group size G = 8, while strategy selection balances global and local strategy priors with λsel = 0.5.
5.2 Offline Evaluation
Offline experiments validate the performance-based and rubric-based reward models, then show that DPO and GRPO improve agentic policy optimization. Pairwise reward modeling, complementary reward fusion, and preference training provide measurable gains across the evaluated stages.
- Performance-Based Reward Model: 60.85% pairwise reward-model accuracy exceeds pointwise prediction by 7.93% on the impression-balanced validation set.Pairwise comparisons under matched product contexts provide more stable relative preferences, although accuracy remains moderate because production CTR features are unavailable.
- Performance-Based Reward Model: 60.85% accuracy with all feature groups improves over 56.55% with video features alone.Audio contributes the largest gain at 1.88%, while storyline features add 0.93% and ad-specific features add 1.49%.
- Rubric-Based Reward Model: 69.50% HL accuracy and 53.30% IB accuracy follow rubric-reward SFT, up from 55.40% and 49.20%.The larger HL gain indicates stronger alignment with human quality judgments than with online business preferences.
- DPO Optimization: 57.41% average preference accuracy follows DPO, up from 50.18%, with gains in strategy selection and draft generation.Strategy selection rises from 49.72% to 56.48%, while draft generation rises from 50.64% to 58.34%.
- GRPO Reward Ablation: 58.19% average win rate makes weighted reward fusion the strongest GRPO variant.Fusion reaches 60.52% performance win rate and 55.86% rubric win rate, balancing business alignment and human-quality standards.
5.3 Online Evaluation
Online A/B tests compare AgenticGen against both the pre-agent pipeline and its SFT policy in an impression-balanced TikTok delivery pipeline. The results show gains from the agentic workflow and further gains from DPO and GRPO.
- Evaluation design: The RL comparison evaluates AgenticGen after DPO and GRPO against the SFT policy using relative lifts in CTR, CVR, and Advv.The online tests use the impression-balanced delivery pipeline.
- Metrics: Advv measures total advertiser-side value from attributed conversions, with each conversion weighted by its bid.
- SFT versus Pre-Agent: AgenticGen SFT improves CTR by 3.48%, CVR by 2.30%, and Advv by 9.83% over Pre-Agent.Pre-Agent uses mostly manual configuration and separately orchestrated components.
- RL versus SFT: Compared with SFT, AgenticGen RL increases CTR by 2.72%, CVR by 2.63%, and Advv by 9.61%.This comparison isolates the gain from DPO and GRPO optimization after the AgenticGen framework is established.
6 Conclusion
AgenticGen treats advertising video generation as a product-conditioned reasoning problem rather than a single synthesis step. It uses two trainable reasoning stages and reward-guided optimization, with online experiments showing improvements over SFT.
- Framework: AgenticGen decomposes advertising video generation into trainable strategy selection and draft generation stages.The framework targets online business performance rather than video synthesis alone.
- Reward-guided optimization: The framework learns performance-based and rubric-based reward models, then applies DPO and GRPO to improve both reasoning stages.The performance reward uses impression-balanced delivery data, while the rubric reward follows domain-specific human quality standards.
- Online result: 2.72% CTR, 2.63% CVR, and 9.61% Advv improvements over SFT are reported after DPO and GRPO in TikTok online A/B experiments.
A Strategy Definitions
The strategy catalog contains commercially sensitive rules that cannot be fully enumerated, so the paper gives abstract definitions and representative examples. Its public families cover asset editing, reference-guided generation, and cross-asset remixing.
- Catalog scope: The complete strategy catalog cannot be fully enumerated because it contains internal commercially sensitive rules.The paper therefore provides abstract definitions and representative examples.
- Asset editing: Asset editing applies lightweight transformations to existing or generated product videos while preserving usable content and the overall storyline.Examples include replacing music, adding hooks or overlays, highlighting selling points, and strengthening calls to action.
- Reference-guided generation: Reference-guided generation adapts effective narrative patterns from high-performing reference videos to the current product.Outputs can include product-specific storyboards, scripts, and generation instructions.
- Cross-asset remixing: Cross-asset remixing builds revised storylines by selecting, ordering, and combining eligible product-material clips.It can replace or extend clips with weak product presentation, low information density, or insufficient visual appeal.
B Strategy Selection Reward Details
The strategy-selection reward combines global strategy performance with local similarity to the highest-feedback historical strategy group. Global scores average feedback across products and campaigns, while local rewards use Jaccard overlap.
- Global reward: The global reward expands each selected strategy group into single-strategy assignments paired with scalar online feedback, namely CTR.This produces one assignment for each strategy in the selected group.
- Global reward: Each strategy receives an average feedback score across products and campaigns, and candidate groups are scored by the mean of their strategy scores.
- Local reward: The local reward selects the highest-feedback historical strategy group under the same product request as the local winner.A candidate rollout sample is then scored by its overlap with that winner.
- Local reward: The local reward is the Jaccard overlap between a candidate strategy group and the highest-feedback historical group.Algorithm 2 defines this as |S ∩ S+|/|S ∪ S+|.
C Preference Accuracy
Preference accuracy evaluates whether SFT and DPO policy scores recover online preference orderings for matched responses. DPO uses an implicit reward, while SFT compares sequence likelihoods directly.
- DPO admits an implicit reward through the optimized policy πψ and SFT reference policy πref.
- Because preferred and non-preferred responses share the same input, the input-dependent partition term cancels when their rewards are compared.A positive DPO preference margin indicates that the implicit reward ranks the online-preferred response higher.
- The SFT baseline has no implicit reward, so its preference margin is computed directly from the sequence-likelihood difference between Y + and Y −.The margin is log πSFT(Y + | X) − log πSFT(Y − | X).
- Stage-specific preference accuracy tests whether policy scores correctly recover the online preference ordering using an indicator function.The evaluation uses held-out impression-balanced triples containing higher- and lower-feedback responses for the same stage input.