PaperlayerView exact revision on arXiv

Source-linked AI summary

FRAMEWORKERS: A Dynamic Multi-Agent Framework for AI-Generated Video Production

Zhendong Li, Lei Sun, Letian Shi, Deheng Zhang, Ruibo Ming, Mengshun Hu, Dannong Xu, Jian Wang, Danda Paudel, Luc Van Gool, Jinjin Gu

arXiv:2608.29814v1cs.AI

TL;DR

Complete video production requires coordinating interdependent steps, while fixed workflows struggle to accommodate diverse inputs. FRAMEWORKERS uses dynamic task-stack orchestration and workspace-grounded sub-agent execution, improving dependency-aware routing and replanning after disruptions.

  • Problem

    Fixed pipelines are difficult to adapt to diverse video-production inputs and workflows involving interdependent creative steps.

    Source: §1 · p. 1

  • Method

    FRAMEWORKERS uses a Director to dynamically select and revise tasks and an Assistant to route them to descriptor-defined sub-agents using shared workspace assets and context.

    Source: §1 · p. 2, §3.1 · p. 3, §3.2 · p. 3

  • Results

    FRAMEWORKERS improves dependency-aware routing, integrates new sub-agents without retraining, and recovers from disruptions; its Director beats Gemini 3 Pro by +10.7 Chain points on DCE and +34.0 on DCH.

    Source: §4.2 · p. 8, §5 · pp. 12–13

  • Takeaways & Limitations

    Descriptor-based sub-agent integration and dynamic task-stack editing form an effective basis for flexible video-production systems.

    Source: §5 · p. 13

    Real-generation evaluation remains limited in scale, so robustness across open-world production requests, sub-agent combinations, and generation backends is not established.

    Source: §5 · p. 13

Abstract

from arXiv · show

Modern video generators excel at synthesizing individual clips, but complete video production requires coordinating a long sequence of interdependent creative steps, including scripting, storyboarding, generation, and editing. It further demands persistent asset management and dynamic task orchestration as intermediate outputs, dependencies, and execution states evolve over time. Existing automated systems typically rely on rigid pipelines that are difficult to adapt to diverse inputs and changing workflows, while general-purpose large language models (LLMs) remain unreliable for long-horizon orchestration and multimodal asset routing. We introduce FRAMEWORKERS, a task-centric and workspace-grounded multi-agent framework for open-ended video production. A central Director formulates video creation as dynamic task management, continuously editing a Task Stack to determine which subtask to execute next and which sub-agent to invoke. An Assistant serves as the execution layer, grounding each selected task in a shared Workspace, retrieving the required assets and context, invoking the assigned sub-agent, and persisting the resulting artifacts. Execution capabilities are exposed through modular sub-agents with registered descriptors, allowing new sub-agents to be integrated without redesigning the orchestration workflow. To improve orchestration reliability, we fine-tune the Director via supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) for descriptor-conditioned task routing. Experiments show that FRAMEWORKERS outperforms strong LLM planners in routing accuracy, recovers reliably from runtime failures, generalizes to unseen sub-agents without retraining, and achieves higher end-to-end video quality and broader task coverage than fixed pipelines, single-agent systems, and prior multi-agent approaches.

1. Introduction

FRAMEWORKERS targets complete, open-ended AI-generated video production rather than isolated clip synthesis. It replaces fixed workflows with dynamic task management, shared asset handling, and modular agents to accommodate diverse inputs and changing execution states.

Introduction

  • Diverse inputs require different workflows, making fixed pipelines poorly suited to tasks such as preserving supplied character identities or incorporating product-specific information into promotional videos.Inputs may range from short ideas and detailed scripts to character portraits, brand constraints, and visual references.Source: §1 · p. 1
  • Production involves interdependent steps, extensive asset coordination, and possible failures, while pretrained language models can misprioritize tasks, omit steps, or invoke sub-agents inappropriately.Manually designing workflows for each task type risks a combinatorial explosion; unreliable asset communication can also cause lost context, inconsistent results, or redundant operations.Source: §1 · pp. 1–2
  • FRAMEWORKERS models video creation as dynamic task management, with a Director selecting the next subtask and sub-agent based on the request, task state, assets, and execution results.The Director maintains a Dynamic Task Stack rather than following a fixed pipeline.Source: §1 · p. 2
  • An Assistant manages sub-agent interfaces, inputs and outputs, and a unified asset space while preserving cross-step context through Global Memory.The workspace can hold scripts, character designs, storyboards, reference images, generated clips, and other intermediate results.Source: §1 · p. 2
  • New creative functions can be added by registering sub-agents without redesigning the workflow, while Director training combines verified routing data with GRPO on self-generated plans.The routing data is template-generated and manually verified; training uses supervised learning followed by Group Relative Policy Optimization.Source: §1 · p. 2

2. Related Work

Prior work advances clip generation, long-form video production, visual-agent tooling, and agent training, but leaves open-ended production orchestration—especially dynamic scheduling, persistent media assets, and failure recovery—underexplored.

Video Generation Models and Agents

  • Clip-generation models [5] [6] [7] [8] improve video quality but remain clipcentric, while long-form methods leave dynamic scheduling, heterogeneous assets, and failure recovery underexplored.Long-form systems add narrative structure and multi-scene coherence, but typically optimize specific formulations or cover only part of asset management and scheduling.Source: §2 · p. 2
  • Long-form production systems [22] [24] [25] [26] add scripts, memory, cinematic direction, and synchronization, yet typically address specific formulations or only parts of asset management and scheduling.Earlier multi-scene systems [9] [10] [11] and agentic production planners [14] [15] [16] [19] [20] improve coherence and automation but generally assume fixed or task-specific stage orders.Source: §2 · p. 2

Agentic Visual Media

  • Visual-agent frameworks [28] [29] [32] establish modular tool use, while media agents [36] add structure but target narrower tasks than open-ended video production.The broader visual-agent frameworks lack a persistent media-asset layer spanning scripts, storyboards, references, clips, and editable files.Source: §2 · p. 2

Agent Training

  • Agent-training work [45] motivates a trained director through interaction trajectories, planner-executor separation, and learning critical planning steps.Scheduling errors can propagate across story, storyboard, generation, and editing, motivating fine-tuning rather than relying on prompting alone.Source: §2 · p. 2

3. Method

FRAMEWORKERS coordinates open-ended video production through a Director that dynamically schedules tasks and an Assistant that executes them against a persistent shared Workspace. Descriptor-based sub-agents and a revisable Task Stack support changing workflows, while the Director is trained for reliable routing.

3.1 Overview

  • The Director dynamically plans and revises work while the Assistant retrieves Workspace artifacts, runs modular sub-agents, checks outputs, and records results.Source: §3.1 · p. 3

3.2 Multi-Agent Scheme

  • Sub-agent descriptors specify roles, inputs, outputs, and trigger conditions, enabling the Director to route calls and new or revised agents to be added without changing the framework.Source: §3.2 · p. 3

3.3 Director and Dynamic Task Stack

  • The editable Task Stack records task prerequisites, assignments, statuses, and history; ADD and DELETE revise unexecuted work while preserving execution records.The Director dispatches the earliest pending task whose prerequisites are satisfied, and can revise the remaining tasks after failures or new instructions.Source: §3.3 · p. 3

3.4 Assistant and Workspace

  • The Assistant resolves task inputs from a persistent Workspace, validates and stores outputs, and adapts heterogeneous sub-agents through semantic retrieval and structured output checks.Workspace components include file records, generated assets, global memory, and execution logs.Source: §3.4 · p. 4

3.5 Director Training

  • Director training uses SFT for descriptor-conditioned routing, then GRPO to optimize output validity, sub-agent coverage, and dependency-consistent ordering.Source: §3.5 · pp. 6–7
  • GRPO optimization after SFT improves Chain accuracy and Edit distance on longer chains compared with SFT alone.Source: §3.5 · p. 7

4. Experiments

Across its experiments, FRAMEWORKERS demonstrates strong end-to-end video quality, accurate task-specific routing, and robust recovery from runtime failures. Its descriptor-based design also supports execution with unseen sub-agents and broader production workflows.

4.1 Data Construction and Task Setup

  • The held-out DCE and DCH sets contain 722 cases across 68 chain structures, testing routing accuracy and structural generalization.Training uses separate SFT and GRPO splits; the SFT split has 1,404 instruction-plan pairs across eleven routing categories, and GRPO has 1,383 prompts.Source: §4.1 · p. 7

4.2 Director Planning Capability

  • 91.0% on DCE and 81.5% on DCH Chain accuracy make the SFT+GRPO Director the only evaluated system above 80% on both datasets.It beats Gemini 3 Pro by +10.7 Chain points on DCE and +34.0 points on DCH. The ablations show that structured-CoT after GRPO reaches 88.4% Chain and 93.4% Step accuracy, while GRPO adds +33.5 Chain points on DCH for the 8B structured-CoT variant.Source: Table 1 · p. 8, §4.2 · p. 8, Table 2 · p. 8

4.3 Failure Recovery and Replanning

  • Across 991 runtime-perturbed cases, the system recovered 96.7% overall, including 494 of 495 missing-producer cases (99.8%) and 464 of 496 quality-gate failures (93.5%), yielding valid executable continuations.Recovery does not delete or invalidate completed tasks; the Director receives no recovery-specific training.Source: §4.3 · p. 9, Table 3 · p. 10

4.4 Assistant and Sub-agent Execution Capability

  • 100% Pass Total and 97.55% First-shot for Gemini 2.5 Flash show reliable execution across the 204-case suite, with the remaining 2.45% recovered through rework.Weaker models mainly fail at pre-generation semantic grounding: GPT-4o and GPT-4o-mini have Resolver failure rates of 32.35% and 11.76%, while Gemini 2.5 Flash eliminates observed Resolver, Missing, and Rework Fail cases.Source: Table 4 · p. 10, §4.4 · p. 10, Table 5 · p. 10
  • 93.0% and 99.2% correct selection of ComedyAgent and ScreenwriterAgent, respectively, show that the frozen Director can route to unseen sub-agents without retraining.The resulting workflows achieve 99.5% and 88.8% pass rates, respectively, using only each agent’s descriptor and execution wrapper.Source: §4.4 · p. 10, Table 6 · p. 11

4.5 End-to-End Video Evaluation

  • Call = 4.32 gives FRAMEWORKERS the best overall average on the main video benchmark, with the highest CLIP and stronger C1–C4 scores than the baselines.UniVA is slightly better on style consistency and overall video-text consistency, but FRAMEWORKERS leads on the overall average.Source: Table 7 · p. 11, §4.5 · p. 11
  • On the story-level benchmark, FRAMEWORKERS achieves the best automatic evaluation performance against MovieAgent and Anim-Director and the best human evaluation across all three dimensions.On the 20-case main benchmark, it also outperforms S-Agent and UniVA across instruction following, temporal consistency, and realism in the user study.Source: §4.5 · p. 11
  • FRAMEWORKERS supports broader input and output coverage than prior systems through plug-and-play sub-agents dynamically composed around user requests, available assets, and execution state.The Director selects and organizes agents while the Assistant resolves and transfers heterogeneous artifacts through a shared Workspace.Source: §4.5 · p. 11, Table 8 · p. 12

5. Limitations and Discussion

FRAMEWORKERS contributes a modular coordination mechanism that dynamically constructs video-production workflows rather than a new generation backbone. Its evaluation supports the orchestration design but does not establish robustness across open-world production requests, sub-agent combinations, or generation backends.

5. Limitations and Discussion

  • FRAMEWORKERS dynamically constructs workflows through modular coordination; its fine-tuned Director routes dependencies, incorporates registered sub-agents without retraining, and replans after unseen disruptions.The authors identify descriptor-based sub-agent integration and dynamic task-stack editing as the basis for flexible video-production systems.Source: §5 · pp. 12–13
  • On the main benchmark, FRAMEWORKERS scores highest on every human-evaluation dimension and 8 of 9 sub-questions; UniVA is slightly higher on (1b) and (1c).Human evaluation compares FRAMEWORKERS against S-Agent and UniVA.Source: Figure 6 · p. 13
  • Evaluation uses a controlled configuration and predefined sub-agent inventory; costly complete pipelines limit real-generation evaluation, so open-world robustness remains unestablished.The routing, recovery, execution, and extensibility experiments are at substantially larger scale, but the results do not establish robustness across open-world requests, sub-agent combinations, or generation backends.Source: §5 · p. 13
  • FRAMEWORKERS inherits the quality, safety, and reliability limitations of its underlying generative models.Source: §5 · p. 13

6. Conclusion · Appendix · A. More Details About FRAMEWORKERS Pipeline

FRAMEWORKERS combines Director-led dynamic orchestration, Assistant-supported execution, and unified asset management to support end-to-end AI-generated video production. Its pipeline repeatedly plans, executes, evaluates, and replans toward the user’s goal.

6. Conclusion

  • FRAMEWORKERS combines Director-based dynamic orchestration, Assistant-supported sub-agent execution, and unified asset management to address fixed-pipeline limitations in end-to-end AIGC video production.The paper presents it as a scalable foundation for automated video creation, with improved stability, flexibility, and extensibility across diverse user inputs.Source: §6 · p. 13

Appendix

  • The appendix adds a pipeline walkthrough, backend robustness analysis, reproducibility details, and the user-study methodology and rating rubrics.It includes the S-Agent baseline design and the complete prompt template for MLLM-as-a-judge evaluation.Source: Appendix · p. 16

A. More Details About FRAMEWORKERS Pipeline

  • The Director considers the user’s instructions, Task Stack, execution state, and available sub-agents to plan operations and dynamically update the stack.The workflow begins with user instructions forwarded through the system interface and cycles through planning, execution, evaluation, and replanning.Source: A. More Details About FRAMEWORKERS… · p. 16
  • The Assistant dispatches the next executable task, retrieves required Workspace content and artifacts, formats inputs for the assigned sub-agent, then stores its outputs and execution traces.The Assistant acts as the execution runtime between the shared Workspace and the sub-agent.Source: A. More Details About FRAMEWORKERS… · p. 16
  • After execution, the Assistant checks artifacts for schema and structural validity, content quality, and asset existence, then stores the evaluation results in the Workspace.It gathers the context needed to apply the evaluator’s checks from the shared Workspace.Source: A. More Details About FRAMEWORKERS… · p. 16
  • The Assistant returns a task and evaluation summary so the Director can revise the plan, recover from failure, or continue, while the interface relays updates until the user’s goal is achieved.The Director updates the Task Stack and prepares status messages; the interface periodically polls for updates and new user information.Source: A. More Details About FRAMEWORKERS… · p. 16

B. Robustness and Failure Modes

Gemini 2.5 Flash passes all 204 cases and completes every planned step after recovering the five cases requiring rework, while Gemini 2.0 Flash has failures across multiple categories and agents.

B. Robustness and Failure Modes

  • Gemini 2.0 Flash has category pass rates of audio 66.7%, style 85.7%, storytelling 86.1%, bilingual 88.9%, cr 91.7%, and complex 92.9%.The authors say this pattern suggests long-horizon multimodal coordination remains challenging under deeper generation chains.Source: B. Robustness and Failure Modes · p. 16, Table 11 · p. 16
  • Gemini 2.0 Flash step-level completion rates are BriefEnricherAgent 88.1%, AudioMixAgent 93.8%, and MusicAgent 96.9%.The authors interpret the distributed agent failures as indicating instability in long-context planning and audio-scene synchronization.Source: B. Robustness and Failure Modes · p. 16, Table 11 · p. 16
  • 204 cases pass and every planned step completes with Gemini 2.5 Flash; all five cases requiring rework recover within the rework loop.No category or agent falls below 100%.Source: Table 11 · p. 16, B. Robustness and Failure Modes · p. 16

C. More Implementation Details

The section specifies S-Agent, a sequential single-agent baseline without multi-agent collaboration, for comparison with FRAMEWORKERS.

C. More Implementation Details

  • S-Agent uses one MLLM call to plan ordered shots and keyframe prompts, then generates T2I keyframes and I2V video segments.The I2V process guarantees visual quality and spatio-temporal consistency only within each individual shot.Source: C. More Implementation Details · pp. 16–17

C.1 Sub-agent Descriptors

Sub-agent descriptors expose each agent’s inputs, output, and invocation purpose, while the Director infers dependencies from matching artifact labels. The same descriptors form the Director’s capability catalog during training and inference.

C.1 Sub-agent Descriptors

  • Each descriptor specifies consumed artifact labels and their cardinality, requirement status, and resolution guidance, the produced artifact, and when the sub-agent should be invoked.The Assistant uses the input information to retrieve and select concrete Workspace assets.Source: §C.1 · p. 17, Figure 7 · p. 17
  • Dependencies are implicit in matching input and output artifact labels, so the Director must infer which sub-agents depend on others during planning.Descriptors describe only their own sub-agent and do not explicitly reference other sub-agents.Source: §C.1 · p. 17
  • At planning time, all registered descriptors are rendered verbatim as a capability catalog using the same representation in training and inference.In Section 4, the catalog is about 30K characters, contains no example plans, and is dominated by descriptors rather than handcrafted examples.Source: §C.1 · p. 17

C.2 Training and Reproducibility Details · C.3 Prompt for the MLLM Judge

The appendix specifies the Director’s SFT/GRPO training and production backends, then defines an MLLM judge that scores videos against visible evidence across content, spatial relations, action, storytelling, style, and text consistency.

C.2 Training and Reproducibility Details

  • The Director is trained from Qwen3-8B using LoRA-based SFT followed by on-policy GRPO.LoRA uses rank 32, α = 64, and dropout 0.05. SFT uses a 2×10−5 learning rate, effective batch size 8, four epochs, and a 12,288-token maximum sequence; GRPO uses group size 4, reward weights λval = 0.1, λcov = 0.5, λord = 0.4, β = 0.04, and a 1×10−6 learning rate for four epochs.Source: §C.2 · p. 17
  • The Assistant and sub-agents use Gemini 2.5 Flash; generation uses gemini-3-pro-image-preview at 2K and self-hosted HunyuanVideo-I2V at 25 denoising steps.CLIP scores use ViT-B/32, and the MLLM judge is Gemini 2.5 Pro.Source: §C.2 · pp. 17–18

C.3 Prompt for the MLLM Judge

  • The judge evaluates one video using only provided frames, timestamps, and text/control information, producing structured scores with traceable evidence and no claims about unseen content.Source: §C.3 · p. 18
  • Semantic content scoring checks whether specified objects and scene types appear and remain consistent with the prompt and visual context.Source: §C.3 · p. 18
  • Action scoring assesses recognizable, temporally continuous behavior and motion, with action set to null when none is specified.Source: §C.3 · p. 18
  • Multi-object scoring checks object counts, perspective-consistent spatial relations, plausible occlusions, and physically plausible interactions.Source: §C.3 · p. 18
  • Story-richness scoring rewards diverse scenes, meaningful narrative progression, and coherent, engaging transitions rather than repetitive or static content.Source: §C.3 · p. 18
  • Style and overall-consistency scoring checks whether appearance and camera movement match and remain stable, and whether the video holistically satisfies its text prompt.Source: §C.3 · p. 19

D. User Study · D.1 Rating Instrument · D.2 Operational Notes for Raters

The user study rated each case–system tuple with a multilingual instrument covering instruction following, temporal consistency, and realism. Operational reminders guided raters’ attention, and only explicitly submitted sessions were analyzed.

D.1 Rating Instrument

  • Each case–system tuple received 1–5 Likert ratings across three dimensions, each with three independently answered sub-questions; definitions appeared in the rater’s selected language.Raters saw a short hint before each dimension’s questions; the English rubric was shown in the paper.Source: §D.1 · p. 19
  • Instruction following assessed requested control signals, prompt-content match, and irrelevant frames, with control phrases highlighted and text prompts prioritized over reference images.The rubric covered BGM, subtitles, ambient sound, and style tags; higher scores reflected fewer unrequested frames throughout the video.Source: §D.1 · p. 20
  • Temporal consistency measured whether characters, objects, and environments remained stable and coherent across frames.The rubric specifically considered faces, identities, body proportions, props, clothing, object appearance and position, backgrounds, lighting, and flicker.Source: §D.1 · p. 20
  • Realism assessed natural appearance and movement, penalizing unnatural moments and judging specified styles against their intended style rather than photorealism.Static appearance included uncanny or broken artifacts; the rubric also treated implausible motion as a defect.Source: §D.1 · p. 20
  • The realism rubric required plausible body and object motion and intentional camera movement, assigning low scores to videos with little or no motion or camera movement.It identified warping, teleportation, impossible movement, and locked-off cameras as problems raters should not reward.Source: §D.1 · p. 20

D.2 Operational Notes for Raters

  • In-app reminders directed raters to attend to highlighted prompt phrases, track changes across frames, and listen to audio when the prompt mentioned sound.The help popup opened automatically once per device and could be reopened using the “?” button.Source: §D.2 · p. 20
  • Raters could save progress at any time, but only explicitly submitted sessions were included in the analysis.Source: §D.2 · p. 20

Related papers

Loading 2608.29814v1…