Source-linked AI summary
MVAgent: Multi-Agent Video Generation via Consistent Condition Construction and Shot-Level Policy Optimization
Xiangyu Kong, Wenjie Zhou, Fengping Tian, Lihua Fang, Haoqin Sun, Chenyang Lyu, Longyue Wang, Weihua Luo
TL;DR
Multi-shot video generation with a frozen black-box generator must maintain character appearance, spatial layout, and character state across shots, which repeated text alone does not determine. MVAgent coordinates typed conditioning inputs and trains its Orchestrator with shot-level candidate comparisons. It reports leading cross-shot and narrative-quality results among compared methods and is preferred over agentic baselines in human evaluation.
Problem
Separate requests to a frozen video generator share no internal state, and repeated textual descriptions do not determine consistent appearance, spatial arrangement, or character state across shots.
Source: §1 · p. 1Method
MVAgent uses typed conditioning inputs for spatial references and observed continuity, then trains the Orchestrator with Trunk-GDPO by comparing candidate renders at each shot.
Source: §1 · p. 1, §2 · p. 1, §2.4 · p. 3Results
MVAgent achieves the highest score in eight of nine columns, including overall GC (0.5689 versus 0.5571 for ViMax), the highest novel-to-video mean (4.256 versus 4.107 for the strongest agentic baseline), and human preference rates above 50% on every criterion.
Source: §3 · pp. 3–4Takeaways & Limitations
The paper formulates multi-shot narrative video generation as condition construction for a frozen generator, with shot-level feedback training the Orchestrator that composes requests.
Source: §4 · p. 4The approach is subject to the generator’s capabilities and the judges’ reliability.
Source: §4 · p. 4
Abstract
from arXiv · showhide
Multi-shot agentic video generation requires consistent character appearance, stable spatial layout across camera angles, and continuous character state between shots. When every shot is a separate request to a frozen generator, repeated text does not determine appearance, layout or state. We therefore recast the problem as condition construction and present MVAgent, a multi-agent pipeline whose agents collaborate through typed conditioning inputs. Because an environment image shows one viewpoint, a Spatial Grounding agent samples views from generated camera-traversal clips and anchors each shot to the view matching its framing. As generated shots drift from the plan, an Observer records how each shot ends in a continuity memory, from which a Transition agent builds character action and spatial references for the next shot. An Orchestrator composes these inputs into each request. Since a request reveals its effect only after rendering, we train it by agentic reinforcement learning with Trunk-GDPO, which compares rendered candidates at every shot rather than once per video and continues the best as the trunk. With generator and judges frozen, MVAgent attains the highest cross-shot consistency and narrative-planning quality among the compared methods on ViMax-Bench and is preferred over the strongest agentic baseline in human evaluation.
1. INTRODUCTION
MVAgent frames multi-shot consistency as a condition-construction problem for frozen video generators, using typed agent inputs and shot-level policy training. The introduction motivates this approach by showing why repeated text and existing methods do not adequately address black-box generation.
1. INTRODUCTION
- A black-box generator’s separate shot requests share no internal state, so repeated descriptions do not ensure consistent character appearance, stable layout across viewpoints, or compatible character states.Existing approaches such as identity adapters, reference attention, camera control, explicit scene representations, and joint multi-shot generation require generator parameters or internal features.Source: §1 · p. 1
- For ViMax [9], NEWTON [10], VISTA, and UniVA [12], roles exchange only text plans, prompts, and keyframes, underdetermining rendered output, while fixed role prompts never learn from produced shots.Source: §1 · p. 1
- MVAgent constructs typed conditioning inputs for a frozen generator, combining spatial anchoring and observation-grounded continuity with shot-level training of the Orchestrator.The Spatial Grounding agent samples views from camera-traversal clips, while the Observer and Transition agent use observed shot endings to build references for the next shot.Source: §1 · p. 1
- Trunk-GDPO trains the Orchestrator by comparing rendered candidates at each shot, using group-relative advantages and continuing the best candidate as the trunk.This shot-level approach is presented as an alternative to crediting every step with a single video-level evaluation or relying on states that recur across rollouts.Source: §1 · p. 1
2. METHOD
MVAgent constructs typed conditioning inputs for a frozen video generator, combining persistent assets with shot-specific spatial and continuity references. Trunk-GDPO trains the Orchestrator using rendered shot-level candidate comparisons.
2. METHOD
- MVAgent combines persistent assets, per-shot environment views, continuity memory, and transition references into requests to a frozen generator.The Screenwriter plans shots; the Orchestrator composes the conditioning inputs.Source: §2 · pp. 1–2, Figure 1 · p. 2
2.1. Problem statement
- Because a frozen generator receives separate requests and text constrains semantics rather than appearance, MVAgent constructs requests carrying persistent assets and observations of generated state.The shot plan specifies cast, environment, camera, action, dialogue, and intended opening and end states.Source: §2.1 · p. 2
2.2. Generative spatial anchoring
- The Spatial Grounding agent samples camera-traversal frames, admits views distinct from earlier selections, describes their fixed elements, and retrieves a view matching each shot’s camera specification.The selected view and its description provide visual and textual layout references.Source: §2.2 · p. 2
2.3. Continuity memory
- After each shot, the Observer records visible characters’ ending positions, orientations, actions, and camera configuration; the Transition agent renders a two-shot reference for the next shot.Spatial Grounding checks the reference’s fixed-element arrangement and requests one regeneration with a new seed if verification fails.Source: §2.3 · p. 2
2.4. Trunk-GDPO
- For each shot, Trunk-GDPO renders and judges K candidates from the same state, normalizes available reward components, and continues the highest-advantage candidate as the trunk.This produces shot-level comparison groups rather than relying only on a terminal video reward.Source: §2.4 · p. 3
3. EXPERIMENTS
Experiments compare MVAgent’s cross-shot consistency, narrative quality, and human preference with prior methods, then test its conditioning and training components. Results favor MVAgent across these evaluations, with specific advantages varying by metric and comparison.
Setup and benchmarks
- Evaluation covers 35 ViMax-Bench stories and 50 NarrativeQA novels, measuring consistency, six narrative dimensions, and three human-preference criteria.At inference, each final shot uses one policy output without best-of-K selection or post-generation repair.Source: §3 · p. 3
Cross-shot consistency
- With Gemini 3 Pro and Veo 3.1, MVAgent leads eight of nine consistency columns, including GC 0.5689 versus 0.5571 for ViMax.With Gemini 3.1 Flash and Veo 3.1, it exceeds ViMax in every column.Source: §3 · p. 3, Table 1 · p. 4
Qualitative comparison
- Across four shots, MVAgent preserves character appearance and the store’s arrangement under different framings, unlike the text-only conditioning strategies.The comparisons hold the generator, story, and camera specifications fixed.Source: Figure 2 · p. 4, §3 · p. 4
Narrative quality and LLM backbone
- MVAgent scores 4.256 versus 4.107 for the strongest agentic baseline in novel-to-video narrative quality.Its advantage does not extend to faithfulness, consistent with the absence of novel retrieval; GPT-5.4 scores highest on all six backbone dimensions, averaging 4.19.Source: Table 2 · p. 4, §3 · p. 4
Subjective evaluation
- MVAgent exceeds 50% preference on every criterion against each baseline, with average rates from 60.06% against ViMax to 80.12% against StoryDiffusion.Against ViMax, rates are 61.39% for cross-scene consistency, 63.02% for semantic following, and 55.78% for aesthetic quality.Source: Table 3 · p. 4, §3 · p. 4
Ablation
- The full model achieves the highest CC, IC, and GC and is preferred over every conditioning variant, with preference margins shrinking as components are added.It is preferred over the untrained policy in 56.3% and over episode-level RL in 53.0% of comparisons, chiefly on cross-scene consistency (63.6% and 55.0%).Source: Table 4 · p. 4, §3 · p. 4
4. CONCLUSION
MVAgent frames multi-shot narrative video generation as condition construction for a frozen generator, using agents that exchange typed conditioning inputs. Its approach combines spatial references, continuity memory, and shot-level policy training, subject to generator capabilities and judge reliability.
- MVAgent casts multi-shot narrative video generation as condition construction for a frozen generator, with agents exchanging typed conditioning inputs.Source: §4 · p. 4
- Spatial Grounding supplies reusable spatial references, while Observer and Transition agents convert VLM observations into continuity memory and action and position references.Source: §4 · p. 4
- Trunk-GDPO trains the Orchestrator to compose these conditions into generator requests using shot-level feedback.Source: §4 · p. 4
- The approach is subject to the frozen generator’s capabilities and the judges’ reliability.Source: §4 · p. 4