VideoGen-Agent: Reinforcing Video Generation Agents

1 Introduction

• Generators can produce high-quality, coherent video yet miss prompt content [35], misrepresent named techniques [46], violate physics [12], or omit or reorder actions [37].These challenges affect uses including product demonstrations, educational content, sports analysis, and film pre-visualization [25].Source: §1 · p. 2
• VideoGen-Agent coordinates augmentation, generation, and verification tools across six tasks, using prompts and intermediate observations to guide its decisions.Augmentation retrieves knowledge or references and simulates motion; verification uses object detection and depth estimation to assess subjects and spatial structure. Task-dependent support includes visual references for identity and simulation for physical motion.Source: §1 · p. 2, Figure 1 · p. 3
• The shared policy is trained first on teacher-distilled trajectories through supervised fine-tuning, then through multitask reinforcement learning with a hybrid reward for tool use and video quality.Task advantage normalization balances learning signals across categories.Source: §1 · p. 2
• VABench is a held-out benchmark of 600 prompts spanning six capability categories, with category-specific VLM rubrics validated against human preferences.The categories include procedural knowledge, single- and multi-entity identity, physics simulation, compositional scenes, and multi-shot generation.Source: §1 · p. 2
• 19.1 points, from 56.5 to 75.6, is VideoGen-Agent’s improvement over its base T2V generator on VABench; upgraded tools raise the score to 86.1.Human raters prefer the upgraded configuration over the strongest standalone baseline in 84.3% of comparisons. Ablations show contributions from SFT and RL with generation tools held fixed.Source: §1 · p. 2

2.1 Multitask Agentic Workflow

• The rollout stops after requested segments are generated and verification thresholds are met, retains tool outputs or errors in history, then assembles segments in phase order.Tool observations inform later decisions in the reasoning–action–observation loop.Source: §2.1 · p. 3, §2.1.1 · p. 4

2.1.1 Tasks and Tools

• The shared policy retrieves procedural steps, visual references, or simulated motion, verifies compositions, and generates multi-shot phases sequentially.The task workflows emphasize procedural accuracy, identity, physical dynamics, spatial composition, or temporal order, depending on the prompt.Source: §2.1.1 · p. 4

2.3 Stage 1: Supervised Fine-Tuning

• Supervised fine-tuning applies next-token prediction to teacher trajectories, masking failed action spans while retaining them and their error observations as context.Valid later steps, including recovery actions, remain available for supervision.Source: §2.3 · p. 5

2.4 Stage 2: Agentic RL

• Starting from the SFT checkpoint, GRPO optimizes new rollouts; trajectories that ultimately fail because of external tool errors are excluded.Tool-response tokens provide context but do not contribute to the policy objective.Source: §2.4 · pp. 5–6

2.2 Data Construction

• The dataset contains 24K prompts: 16K teacher trajectories for supervised fine-tuning and 8K held out for reinforcement-learning rollouts.Two teacher agents generate trajectories that include reasoning, tool calls, observations, and final videos.Source: §2.2 · p. 5, Table 1 · p. 6

2.4.1 Multitask Hybrid Reward

• The hybrid reward combines format validity, category-specific VLM video-quality and prompt-consistency assessment, and task-appropriate tool use.The respective weights are 0.1, 0.5, and 0.4 for format, VLM, and tool-use rewards.Source: §2.4.1 · p. 6

2.4.2 Advantage Normalization

• After group-relative advantages are assigned to agent-emitted tokens, category-level normalization places token advantages from different tasks on comparable scales.The category statistics are computed over agent-token advantages in the global training batch.Source: §2.4.2 · p. 6

2.4.3 Token-Level GRPO Objective

• The token-level GRPO objective updates agent-emitted token positions and regularizes the policy toward the fixed SFT reference.Tool responses remain context; the KL term is weighted by β_KL.Source: §2.4.3 · pp. 6–7

3.1 Implementation Details

• The evaluation uses 600 held-out prompts, while Toolset 2 upgrades generation tools without changing augmentation or verification tools or retraining the agent.The experiments also compare standalone generators and training-stage variants; the benchmark protocol uses category-specific scoring and an overall unweighted mean across six categories.Source: §3.1 · p. 7, Table 2 · p. 8

3.2 Main Results

• 86.1 overall with Toolset 2, up from 75.6 with Toolset 1; the fixed agent policy scores highest in all six categories after generation-tool upgrades.The upgrade raises Multi-Entity Identity from 65.3 to 86.7, the largest category increase.Source: Table 2 · p. 8, §3.2 · p. 8
• With Toolset 1, VideoGen-Agent scores 75.6 overall, 19.1 points above its base Seedance 1.0 generator and 2.4 above Seedance 2.0 overall.It remains below Seedance 2.0 on Multi-Entity Identity and Compositional Scene.Source: §3.2 · p. 8
• Against Seedance 2.0, Toolset 2 gains are largest on Multi-Shot (25.7 points), Multi-Entity Identity (17.7), and Procedural Knowledge (14.1).The Compositional Scene gain is 2.9 points, where the strongest standalone baseline scores 87.8.Source: §3.2 · p. 8
• Human raters prefer VideoGen-Agent with Toolset 2 over Seedance 2.0 in 84.3% of ratings across 100 randomly sampled video pairs.The largest preference margins occur on Procedural Knowledge and Multi-Shot.Source: §3.2 · p. 9
• Under Toolset 1, SFT raises the overall score from 56.5 to 69.2, and RL adds 6.4 points to reach 75.6.Both training stages improve every category; prompt rewriting reaches 57.3 and zero-shot tool use 59.5.Source: §3.2 · p. 9, Table 3 · p. 9
• Removing the VLM reward lowers the overall score to 73.3 and removing the tool reward to 71.2; single-task RL scores 76.3, 0.7 above multitask RL.Multitask RL scores higher on the other categories and supports all six tasks with one agent at a similar overall score.Source: §3.2 · p. 9, Table 3 · p. 9
• For unseen combinations of procedural knowledge and identity constraints, the agent combines text and image retrieval to guide generation.In an additional robotic-manipulation pipeline, the agent uses π0.5 [30] for action prediction and Ctrl-World [13] for video generation; the videos more closely match ground truth than direct I2V generation.Source: §3.2 · p. 9

Video generation and tool augmentation

• Video-generation approaches use scene decomposition, editing, simulation, and force-based controls to address requirements difficult to satisfy through prompting alone.VideoGen-Agent builds on these capabilities by training a shared policy to use different tools across generation tasks.Source: §4 · p. 10

Agentic RL with tool use

• Agentic RL work studies tool-interleaved reasoning for retrieval, information seeking, code execution, visual understanding, GUI interaction, and embodied control.Video generation additionally requires evaluating generated visual content as well as tool-use decisions.Source: §4 · p. 10

Agentic visual generation

• Unlike image-generation agents focused on search augmentation or video refinement systems using test-time critique, VideoGen-Agent trains across six video tasks.Its multitask hybrid reward evaluates tool-call validity, task-appropriate use, and category-specific video quality.Source: §4 · p. 10

5 Conclusion

• VideoGen-Agent uses SFT followed by multitask RL to train a shared policy coordinating augmentation, generation, and verification tools.The work also constructs a category-balanced training dataset and a held-out benchmark covering six generation capabilities.Source: §5 · p. 10
• Experiments show gains over standalone video generators, and ablations support contributions from both training stages.Upgrading generation tools further improves performance without additional agent training.Source: §5 · p. 10
• The conclusion frames learned tool use and advances in generation models as complementary sources of performance improvement.Source: §5 · p. 10

6 Limitations and Future Work

• VideoGen-Agent is limited by generation-tool quality and latency, verification-feedback coverage, and its six-task training setting.These boundaries constrain the current system's tools, feedback, and task coverage.Source: §6 · p. 10
• Future work could explore on-policy distillation, stronger video-understanding feedback, and faster, higher-quality, easier-to-deploy generation tools.Source: §6 · p. 10
• The paper proposes jointly improving training, video feedback, and generation tools to support more complex workflows and a broader range of creative tasks.Source: §6 · p. 10

• The appendix is organized into sections A and B.Source: Overview · p. 14
• Appendix A covers VideoGen-Agent implementation, including reward computation, tool configurations, the system prompt, the physics simulator, and category-specific evaluation rubrics.Source: Overview · p. 14
• Appendix B presents qualitative comparisons across all six task categories and evaluates agreement between the VLM judge and human preferences.Source: Overview · p. 14

A.1 Reward and RL Implementation Details

• The section documents the RL implementation, including the hybrid reward, task advantage normalization, token-level policy objective, and tool-failure handling.Source: §A.1 · p. 14

A.1.1 Format Reward

• Format reward checks whether each action follows the required protocol, including valid tool-call structure, parseable JSON, available tools, valid arguments, and final-answer termination.Malformed actions receive zero format score for that turn.Source: §A.1.1 · p. 14

A.1.2 Tool-Use Reward

• Tool-use reward scores whether the agent selects suitable tools and functionally uses their outputs, including retrieval evidence, conditioning artifacts, and verification results.Category-specific checks score one when satisfied and zero otherwise; no applicable check yields a tool-use reward of zero.Source: §A.1.2 · p. 14, Table 4 · p. 15

A.1.3 Category-Specific VLM Reward

• The category-specific VLM reward scores the final video against the prompt and, when applicable, category-dependent reference information across weighted evaluation dimensions.Dimensions are normalized to [0, 1]; Physics Simulation motion correctness is computed by code and weighted 0.20.Source: §A.1.3 · p. 15

A.1.4 Hybrid Reward

• The hybrid trajectory reward combines format validity, category-specific video quality, and task-appropriate tool use with weights 0.1, 0.5, and 0.4, respectively.All components and the resulting reward lie in [0, 1].Source: §A.1.4 · p. 15

A.1.5 Token-Level Group-Relative Advantage

• Group-relative advantage is computed for sampled trajectories and initially assigned to every agent-emitted token in each trajectory.Source: §A.1.5 · pp. 15–16

A.1.6 Task-Level Advantage Normalization

• Task-level normalization equalizes advantage distributions across categories within a training batch after collecting advantages for agent-emitted tokens in each category.Group-relative normalization compares trajectories sampled for the same prompt, while task-level normalization balances categories.Source: §A.1.6 · p. 16

A.1.7 Token-Level Policy Objective

• The token-level policy objective uses agent-emitted token positions, with the behavior policy generating rollouts and the fixed reference policy set to the initial SFT checkpoint.The behavior policy is updated during RL optimization.Source: §A.1.7 · p. 16

A.2 Concrete Tool Implementations

• Toolset 2 upgrades generation tools while retaining augmentation and verification tools, and the trained policy is evaluated with it without additional training.Both toolsets include augmentation, generation, and verification; verification uses Grounding DINO 1.5 and Depth Anything V2 (ViT-L).Source: §A.2 · p. 17, Table 5 · p. 17

A.3 System Prompts

• The system prompt requires tool-mediated video generation, one tool-call block per turn, and a final answer only after generation and any applicable verification.Turns begin with a <think> block; the final turn evaluates prompt-video alignment before the <answer> signal.Source: §A.3 · p. 17
• For physical interactions, the agent runs a simulation with prompt-matched parameters and guides video generation with the returned optical-flow video.The prompt specifies independent rigid-body parameters and shared restitution for collisions, and directs the agent not to use grounding_dino_video on simulation outputs.Source: §A.3 · pp. 18–19
• For co-occurring generic subjects, the prompt requires detection checks and revised generations when any subject is missing, allowing up to two regeneration attempts.Grounding checks sample the video; the agent submits only when all queried subjects are detected at least once or the retry limit is reached.Source: §A.3 · pp. 19–20
• For a real-world process, mechanism, or scientific phenomenon, search factual details then use video_gen_text; for one named entity whose appearance matters, image_search candidates and use one selected reference path.Source: §A.3 · p. 19
• For ordered events, the prompt uses separate segments linked by the preceding segment’s last frame and checks each phase independently; sequences of three or more phases use chained first-frame generation.For two phases, missing subjects trigger phase-specific regeneration, with the second phase conditioned on the first phase’s latest frame; the supplied instructions also require one segment per phase for N ≥ 3.Source: §A.3 · p. 20
• Multi-phase generation chains each segment’s last frame into the next segment as its first frame, maintaining subject, environment, and style continuity.For phases beyond the third, the same last-frame chaining rule continues.Source: §A.3 · p. 20

A.4 Custom Function: Rigid-Body Collision Simulator

• The collision simulator turns policy-produced JSON specifications into deterministic motion videos that guide Wan 2.1-VACE generation.It renders at 832×480, 5 s, and 24 fps; its planar rigid-body routines support spheres, cubes, cuboids, and triangular prisms, with restitution derived from a target post-collision velocity when supplied.Source: §A.4 · pp. 20, 23

A.5 Reward Prompt

• The reward judge uses Gemini 3.1 Pro and category-specific rubrics, with discrete_3 for qualitative criteria and int_1_10 for factual correctness and identity preservation.It returns per-dimension JSON scores that are normalized and combined using category-specific weights.Source: §A.5 · p. 23, Table 6 · p. 24
• Physics Simulation combines VLM judgments of physical plausibility with code-based quantitative checks weighted at 0.20.The VLM evaluates contact, rebound, motion direction, and induced rotation, while tracking and numerical analysis check measurable motion.Source: §A.5 · p. 23, Table 6 · p. 24
• For multi-shot prompts, unsupported requested durations are adjusted to supported segment lengths while preserving event order, and evaluation uses the adjusted timing requirements.The adjustment also updates timestamps in the generation prompt, avoiding penalties solely for unsupported output durations.Source: §A.5 · p. 23

Reward Prompt — Identity and Multi-Entity Identity

• The rubric assigns identity_match weight 0.70, comparing the specifically named subject in the video with 3-5 distinctive visual features distilled from reference images; action_correctness weighs 0.20, and aesthetics weighs 0.10.Source: §A.5 · p. 25

Reward Prompt — Physics Simulation

• Physics evaluation weights physical_realism at 0.70, assessing whether motion, collisions, trajectories, and prompt-specified dynamics are plausible, while aesthetics weighs 0.10.A visually smooth video with incorrect physics should still score poorly.Source: §A.5 · p. 26

Reward Prompt — Compositional Scene

• Scene-composition scoring weights subjects_correctness at 0.50 and interaction_logic at 0.40, evaluating subject presence and attributes as well as prompt-specified interactions and spatial relations.Aesthetics is separately scored for visual qualities such as lighting, composition, color, and sharpness.Source: §A.5 · p. 27

Reward Prompt — Multi-Shot

• Multi-shot scoring weights phase_transition at 0.50 and intraphase_correctness at 0.40, checking chronological phases, crisp transitions, continuity, timing, and segment-specific content.Aesthetics is also scored; missing or misordered phases or a scene reset fail the phase-transition requirement.Source: §A.5 · p. 27

• Figures 3–8 compare VideoGen-Agent with standalone video generators across all six VABench task categories.Source: §B.1 · p. 28
• Each figure shows uniformly sampled frames from videos generated for the same prompt, highlighting differences in how methods satisfy its requirements.Source: §B.1 · p. 28
• Together, the examples illustrate how retrieval, simulation, verification, and sequential generation support procedural accuracy, identity preservation, physical consistency, scene composition, and temporal ordering.Source: §B.1 · p. 28

Agreement with Human Preferences

• The main benchmark results use Gemini 3.1 Pro with the same category-specific rubrics and scoring weights as the training reward.Gemini 3.1 Pro provides the VLM reward Rvlm used during reinforcement learning.Source: §B.2 · p. 28

Setup

• Raters assessed category-specific requirements, using overall prompt faithfulness as the tie-breaker when comparing each video pair.The study sampled 100 pairs per category across the strongest standalone baseline, VideoGen-Agent-SFT, and VideoGen-Agent-RL.Source: §B.2 · p. 28

Agreement with Human Preferences

• 92.0% of 600 video pairs: Gemini 3.1 Pro agrees with the human majority preference.Three raters independently compared each pair without knowing which models generated the videos.Source: §B.2 · p. 28
• Agreement is 72% for Procedural Knowledge, 98% for Single-Entity Identity, 100% for Multi-Entity Identity, 94% for Physics Simulation, 88% for Compositional Scene, and 100% for Multi-Shot.Procedural Knowledge has the lowest agreement; the other five categories range from 88% to 100%.Source: §B.2 · p. 28
• Gemini 3.1 Pro achieves agreement comparable to GPT-4o, while Qwen3-VL-235B-A22B-Instruct performs particularly well on Single-Entity Identity and Multi-Entity Identity, but shows lower agreement on other categories.Source: §B.2 · p. 28