PaperlayerView exact revision on arXiv

Source-linked AI summary

FRAMEWORKERS: A Dynamic Multi-Agent Framework for AI-Generated Video Production

Zhendong Li, Lei Sun, Letian Shi, Deheng Zhang, Ruibo Ming, Mengshun Hu, Dannong Xu, Jian Wang, Danda Paudel, Luc Van Gool, Jinjin Gu

arXiv:2608.29814v1cs.AI

TL;DR

Complete AI-generated video production remains difficult because interdependent steps, persistent assets, and diverse inputs challenge fixed pipelines and long-horizon orchestration. FRAMEWORKERS uses a Director-managed dynamic task stack, an Assistant grounded in shared workspace state, and descriptor-based modular sub-agents, with SFT followed by GRPO for routing. Experiments report stronger routing, failure recovery, unseen-sub-agent integration without retraining, and broader production capability, while evaluation remains limited in scale and configuration.

  • Problem

    Complete video production requires coordinating interdependent creative steps and persistent assets across diverse inputs that fixed workflows struggle to accommodate.

  • Method

    FRAMEWORKERS combines Director-based dynamic task-stack orchestration, Assistant-supported workspace execution, descriptor-based sub-agents, and SFT followed by GRPO for routing.

  • Results

    Experiments report improved routing, recovery from disruptions, integration of newly registered sub-agents without retraining, and higher stability, flexibility, and extensibility.

  • Takeaways & Limitations

    Dynamic task-stack editing and descriptor-based sub-agent integration form an effective basis for flexible automated video-production systems.

  • Takeaways & Limitations

    Evaluation uses a controlled configuration and predefined sub-agent inventory, with real-generation experiments limited in scale and open-world robustness not yet established.

Abstract

from arXiv · show

Modern video generators excel at synthesizing individual clips, but complete video production requires coordinating a long sequence of interdependent creative steps, including scripting, storyboarding, generation, and editing. It further demands persistent asset management and dynamic task orchestration as intermediate outputs, dependencies, and execution states evolve over time. Existing automated systems typically rely on rigid pipelines that are difficult to adapt to diverse inputs and changing workflows, while general-purpose large language models (LLMs) remain unreliable for long-horizon orchestration and multimodal asset routing. We introduce FRAMEWORKERS, a task-centric and workspace-grounded multi-agent framework for open-ended video production. A central Director formulates video creation as dynamic task management, continuously editing a Task Stack to determine which subtask to execute next and which sub-agent to invoke. An Assistant serves as the execution layer, grounding each selected task in a shared Workspace, retrieving the required assets and context, invoking the assigned sub-agent, and persisting the resulting artifacts. Execution capabilities are exposed through modular sub-agents with registered descriptors, allowing new sub-agents to be integrated without redesigning the orchestration workflow. To improve orchestration reliability, we fine-tune the Director via supervised fine-tuning (SFT) followed by Group Relative Policy Optimization (GRPO) for descriptor-conditioned task routing. Experiments show that FRAMEWORKERS outperforms strong LLM planners in routing accuracy, recovers reliably from runtime failures, generalizes to unseen sub-agents without retraining, and achieves higher end-to-end video quality and broader task coverage than fixed pipelines, single-agent systems, and prior multi-agent approaches.

1. Introduction

Complete AI-generated video production requires coordinating interdependent creative steps, managing persistent assets, and adapting workflows to diverse user inputs. FRAMEWORKERS addresses this with dynamic task management and modular multi-agent execution.

  • AI video systems can synthesize visually rich clips, but complete production also requires scripting, planning, asset organization, generation, and editing.
  • Diverse inputs require different workflows, such as preserving provided scripts, maintaining character identity, or incorporating product-specific information.
  • Interdependent production steps and possible failures make manually designed workflows difficult to extend across heterogeneous creation tasks.
  • FRAMEWORKERS models video creation as dynamic task management through a Director that selects subtasks and sub-agents using current state and available assets.
  • An Assistant executes selected tasks through a unified asset space and Global Memory, while registered sub-agents enable new creative functions without workflow redesign.

2. Related Work

Prior work improves video generation, multi-scene coherence, agentic production planning, and agent training. These approaches motivate combining structured video-production agents with a trained director for long-horizon scheduling.

  • Recent video-generation models improve motion coherence, fidelity, editing, personalization, and audio support, but primarily generate clips.
  • Long-form video systems introduce narrative structure through scripting, role decomposition, reference conditioning, retrieval-guided motion, and modular story representations.
  • Agentic video systems automate production planning by expanding prompts into scene plans and coordinating specialized production functions.
  • Agent-training research shows benefits from interaction trajectories, planner-executor separation, and learning critical planning steps.
  • Because scheduling errors propagate across story, storyboard, generation, and editing, prior work suggests training a director rather than relying only on prompting.

3. Method

FRAMEWORKERS replaces fixed video-production workflows with closed-loop orchestration that dynamically schedules tasks, routes them to modular sub-agents, and grounds execution in a persistent Workspace. The Director revises the Dynamic Task Stack from execution feedback, while the Assistant resolves inputs, invokes sub-agents, validates outputs, and records artifacts for continued planning.

  • Overview: FRAMEWORKERS closes the loop among planning, execution, evaluation, and replanning, allowing the system to incorporate intermediate results, failures, and changing user intent.The Director performs strategic scheduling, while the Assistant handles concrete multimodal execution against persistent shared state.
  • Multi-Agent Scheme: Independent sub-agents expose descriptors containing roles, inputs, outputs, and trigger conditions, enabling the Director to select and revise calls without a hand-crafted workflow.Because routing uses semantic descriptions rather than fixed identifiers, new or revised sub-agents can be integrated without modifying the framework itself.
  • Director and Dynamic Task Stack: The Director edits an ordered Dynamic Task Stack to schedule remaining work, preserve execution history, and adapt plans during execution.Tasks record assigned sub-agents, prerequisites, expected outputs, status, and runtime metadata; ADD and DELETE operations support refinement, recovery, and new instructions.
  • Director and Dynamic Task Stack: The earliest pending task with satisfied prerequisites is dispatched to the Assistant, which returns failures or execution summaries to the Director for replanning.Tasks can be enabled by completed tasks or available Workspace artifacts, and failed-task summaries are written into Global Memory.
  • Assistant and Workspace: The Assistant grounds each selected task in a shared Workspace by retrieving schema-compatible inputs, invoking the assigned sub-agent, validating outputs, and persisting artifacts.The Workspace maintains files, generated multimedia assets, and summarized global memory; compact projections prevent low-level execution details from overloading the Director.
  • Director Training: Compared with SFT alone, GRPO improves Chain accuracy and Edit distance on longer chains by optimizing executable plan properties after supervised routing training.The training procedure uses a frozen SFT reference and group-normalized policy optimization; groups with identical rewards provide no update signal.

4. Experiments

FRAMEWORKERS is evaluated as a descriptor-conditioned routing system, a runtime recovery mechanism, an extensible sub-agent framework, and an end-to-end video-production system. Across these evaluations, it improves routing, recovers from perturbations, supports unseen capabilities, and produces strong video-quality results.

  • Evaluation setup: The evaluation covers routing across 68 chain structures, using Chain accuracy, Step accuracy, and set-aware Edit distance on easy and hard held-out datasets.DCE contains 522 simpler cases, while DCH contains 200 longer and more structurally demanding cases.
  • Director planning capability: 91.0% DCE Chain accuracy and 81.5% DCH Chain accuracy make the SFT+GRPO Director the only system above 80% on both datasets.It exceeds Gemini 3 Pro by +10.7 Chain points on DCE and +34.0 points on DCH, with Edit scores of 0.15 and 0.50, respectively.
  • Director planning capability: GRPO improves every SFT checkpoint, with the 8B structured-CoT variant gaining +33.5 DCH Chain points from 48.0 to 81.5.After GRPO, structured-CoT reaches 88.4% Chain and 93.4% Step accuracy, while scaling mainly benefits longer, harder chains.
  • Failure recovery and replanning: Runtime replanning recovers 99.8% of missing-producer cases and 93.5% of quality-gate failures, for a 96.7% overall recovery rate across 991 cases.Recovery preserves completed execution history and revises only the unexecuted Task Stack portion.
  • Assistant and sub-agent execution capability: With the Director frozen, unseen ComedyAgent and ScreenwriterAgent are selected correctly in 93.0% and 99.2% of cases, respectively.New capabilities are introduced through functional descriptors and execution wrappers rather than retraining or workflow redesign.
  • End-to-end video evaluation: FRAMEWORKERS achieves the strongest overall automatic and MLLM-based video evaluation, while also attaining the best human scores across instruction following, temporal consistency, and realism.It records the best overall average score, Call = 4.32, although UniVA is slightly better on style and overall video-text consistency.

5. Limitations and Discussion

FRAMEWORKERS’ contribution is a modular coordination mechanism that dynamically constructs video-production workflows rather than introducing a new generation backbone. Its evidence supports flexible routing, sub-agent integration, and recovery, while evaluation remains bounded by controlled configurations and limited real-generation scale.

  • Discussion: FRAMEWORKERS transforms fixed video-production pipelines into dynamically constructed workflows through descriptor-based sub-agent integration and Dynamic Task Stack editing.The framework’s main contribution is coordination rather than a new generation backbone.
  • Discussion: The fine-tuned Director improves dependency-aware routing and incorporates newly registered sub-agents without retraining.This supports modular extensibility within the evaluated framework.
  • Discussion: Human evaluation finds the method highest on every dimension and 8 of 9 sub-questions, with UniVA slightly higher on sub-questions (1b) and (1c).Figure 6 reports mean human-evaluation scores against S-Agent and UniVA.
  • Discussion: Task-stack replanning recovers from unseen disruptions, supporting flexible video-production workflows.The conclusion links recovery with descriptor-based integration and dynamic task-stack editing.
  • Limitations: The evaluation uses a controlled configuration and predefined sub-agent inventory, while real-generation testing remains limited in scale.The paper therefore does not establish robustness across the full diversity of open-world requests, sub-agent combinations, or generation backends.

6. Conclusion

The paper presents FRAMEWORKERS as an autonomous multi-agent framework for end-to-end AIGC video production. Its closed loop coordinates planning, execution, evaluation, and replanning through the Director, Assistant, and Workspace until the user’s goal is achieved.

  • Conclusion: FRAMEWORKERS combines Director-based dynamic orchestration, Assistant-supported execution, and unified asset management for end-to-end AIGC video production.The conclusion presents this combination as the framework’s central design.
  • Conclusion: The paper concludes that FRAMEWORKERS handles diverse user inputs with improved stability, flexibility, and extensibility.It characterizes the framework as a scalable foundation for automated AI video creation.
  • Conclusion: The workflow continuously cycles through planning, execution, evaluation, and replanning in response to user instructions and execution state.The Director updates the Task Stack, while the Assistant executes selected tasks using available context.
  • Conclusion: The Assistant retrieves Workspace artifacts, invokes sub-agents, stores outputs and evaluation results, and displays updates as the loop continues.This execution cycle persists generated changes until the overarching user goal is achieved.

B. Robustness and Failure Modes

Robustness varies substantially across evaluated backends and failure categories. Gemini 2.5 Flash completes all 204 cases, whereas Gemini 2.0 Flash exhibits distributed failures associated with deeper multimodal coordination.

  • Failure Modes: Gemini 2.0 Flash shows distributed failures across audio, style, storytelling, bilingual, cr, and complex segments.The reported pass rates include 66.7% for audio and 85.7% for style.
  • Failure Modes: The Gemini 2.0 Flash pattern suggests that long-horizon multimodal coordination remains challenging under deeper generation chains.Step-level failures are also distributed across multiple agents.
  • Robustness: Gemini 2.5 Flash passes all 204 cases and completes every planned step, recovering all five rework cases within the rework loop.No category or agent falls below 100% in this breakdown.
  • Evaluation: Table 11 reports case-level pass rate and step-level completion rate, with N denoting the number of cases or planned steps.Segments and agents with perfect accuracy across all backends are grouped as Other.

C. More Implementation Details

The S-Agent baseline uses a single MLLM query to create a sequential shot plan and keyframe prompts, followed by sequential visual generation. It therefore represents a straightforward single-agent pipeline rather than collaborative orchestration.

  • S-Agent Baseline: S-Agent is a single-agent baseline that operates through a straightforward sequential pipeline without multi-agent collaboration.The system uses one MLLM for the planning phase.
  • S-Agent Baseline: Given a user instruction, S-Agent decomposes the request into sequential shots and generates detailed keyframe prompts for each shot.This planning step precedes the visual-generation process.
  • S-Agent Baseline: S-Agent then applies a two-step visual-generation process beginning with a Text-to-Image model.The supplied description identifies the first stage but does not provide the complete second-stage details.

C.1 Sub-agent Descriptors

Sub-agent descriptors expose standardized capability and artifact requirements to the Director and Assistant. Dependencies are inferred through shared artifact labels rather than explicit cross-agent references.

  • Descriptor Structure: Each descriptor defines Inputs, Output, and Purpose / Trigger fields for exposing a sub-agent’s capability.Inputs describe consumed artifact labels and provide cardinality, requirement status, and artifact-selection guidance.
  • Descriptor Structure: The Assistant uses descriptor input specifications to retrieve and select concrete artifacts from the shared Workspace.
  • Dependency Representation: Dependencies between sub-agents are implicit when one agent’s output artifact label satisfies another agent’s required input.The Director must infer these dependencies during planning.
  • Capability Catalog: Registered descriptors are rendered verbatim into the Director prompt as a capability catalog during planning, training, and inference.The catalog is preceded by routing rules and allowed identifiers.

C.2 Training and Reproducibility Details

The appendix specifies implementation details for Director training and a multimodal judge rubric. The judge evaluates video quality through structured criteria covering semantic accuracy, spatial relations, actions, and scene-level properties.

  • Director Training: The Director is initialized from Qwen3-8B and fine-tuned with LoRA adapters using supervised training settings detailed in the appendix.The reported configuration includes rank 32, α = 64, dropout 0.05, a learning rate of 2×10^-5, effective batch size 8, 4 epochs, and maximum sequence length 12,288 tokens.
  • MLLM Judge: The multimodal judge is Gemini 2.5 Pro and evaluates individual videos from provided frames, timestamps, and text or control information.The judge is instructed not to hallucinate unseen content and to provide structured scoring with traceable evidence.
  • Judge Criteria: The rubric scores semantic content accuracy by checking specified objects, scene type, and consistency with the text prompt and visual context.The semantic scale ranges from major omissions or mismatches to fully correct and stable content.
  • Judge Criteria: Action and behavior evaluation checks whether specified actions or poses are recognizable, continuous, and maintained over time.If no action is specified, the corresponding score is set to null.
  • Judge Criteria: Multi-object evaluation checks object counts, spatial relations, perspective consistency, occlusions, and physically plausible interactions.Examples include above/below, inside/outside, left/right, and front/back relations.
  • Judge Criteria: Story richness, style consistency, and overall video-text consistency assess scene diversity, narrative progression, visual coherence, and alignment with the intended instruction.The story-richness scale distinguishes repetitive or static videos from diverse, coherent, and engaging visual storytelling.

D.1 Rating Instrument

The rating instrument uses 1–5 Likert scores across realism, instruction following, and temporal consistency. Each dimension contains three independently answered sub-questions, with procedural reminders and submission controls standardizing evaluation.

  • Instrument Structure: Each case–system tuple receives 1–5 Likert ratings across three dimensions, each decomposed into three independently answered sub-questions.Definitions are displayed in the rater’s selected language, with the English version reported.
  • Instruction Following: Instruction following measures controllability, prompt adherence, and absence of irrelevant frames.Raters consider requested controls such as BGM, subtitles, ambient sound, and style tags, and prioritize text over a reference image when both are present.
  • Temporal Consistency: Temporal consistency measures subject, object, and environment stability as characters, props, and scenes evolve across frames.The criteria cover identity and proportions, object appearance and position, and background, lighting, and layout coherence.
  • Realism: Realism measures natural-looking and natural-moving video through static appearance, motion plausibility, and camera motion.Raters penalize uncanny artifacts, physically impossible motion, and little or no motion.
  • Rating Procedure: Procedural reminders direct raters to inspect highlighted prompt controls, temporal changes, and audio when relevant to the questions.Only sessions explicitly submitted by raters are included in the analysis.
Loading 2608.29814v1…