PaperlayerView exact revision on arXiv

Source-linked AI summary

Automated Movie Generation via Multi-Agent CoT Planning

Weijia Wu, Zeyu Zhu, Mike Zheng Shou

arXiv:2503.07314v1cs.CV

TL;DR

Existing long-form video systems require manual planning for narratives, scenes, cinematography, and character interactions. MovieAgent uses hierarchical multi-agent Chain-of-Thought planning to automate filmmaking, and experiments report state-of-the-art results in story and movie-generation quality.

  • Problem

    Existing video-generation frameworks do not automate the planning required for long-form narrative structure, cinematography, and scene composition.

  • Method

    MovieAgent uses specialized agents and hierarchical Chain-of-Thought reasoning to plan scripts, scenes, shots, camera settings, and cinematography before customized video and audio generation.

  • Results

    MovieAgent achieves state-of-the-art automated storytelling and movie-generation performance, with strong character consistency and narrative coherence.

  • Takeaways & Limitations

    The framework defines an automated movie-generation paradigm that reduces human intervention while supporting coherent stories, preserved characters, and audiovisual synchronization.

  • Takeaways & Limitations

    Current technology cannot fully support simultaneous audio-video generation in a single model, so movie-generation settings are categorized by whether audio is required.

Abstract

from arXiv · show

Existing long-form video generation frameworks lack automated planning, requiring manual input for storylines, scenes, cinematography, and character interactions, resulting in high costs and inefficiencies. To address these challenges, we present MovieAgent, an automated movie generation via multi-agent Chain of Thought (CoT) planning. MovieAgent offers two key advantages: 1) We firstly explore and define the paradigm of automated movie/long-video generation. Given a script and character bank, our MovieAgent can generates multi-scene, multi-shot long-form videos with a coherent narrative, while ensuring character consistency, synchronized subtitles, and stable audio throughout the film. 2) MovieAgent introduces a hierarchical CoT-based reasoning process to automatically structure scenes, camera settings, and cinematography, significantly reducing human effort. By employing multiple LLM agents to simulate the roles of a director, screenwriter, storyboard artist, and location manager, MovieAgent streamlines the production pipeline. Experiments demonstrate that MovieAgent achieves new state-of-the-art results in script faithfulness, character consistency, and narrative coherence. Our hierarchical framework takes a step forward and provides new insights into fully automated movie generation. The code and project website are available at: https://github.com/showlab/MovieAgent and https://weijiawu.github.io/MovieAgent.

1. Introduction

MovieAgent frames automated movie generation as a hierarchical, multi-agent alternative to manual filmmaking, producing coherent long-form videos while automating planning and shot design.

  • Motivation: Existing video-generation research primarily targets short videos, leaving narrative planning, cinematography, and scene composition dependent on manual input.Examples include diffusion and spatiotemporal transformer systems focused on short clips.
  • Motivation: Movie-level generation must coordinate narrative themes, scene transitions, character consistency, and low-level cinematographic parameters across specialized filmmaking roles.The paper contrasts this hierarchical, collaborative process with short-video generation and single-model approaches.
  • Approach: MovieAgent simulates filmmaking professionals with multi-agent Chain-of-Thought reasoning to automate script breakdown, scene planning, shot design, and cinematography.Its hierarchical process separates narrative structure, scene and shot composition, camera settings, emotional tone, and subtitles.
  • Capabilities: Given a script and character bank, MovieAgent generates multi-scene, multi-shot videos with coherent narratives, character consistency, and synchronized subtitles.The figure comparison presents these automated planning capabilities as an alternative to traditional manual production.
  • Results: MovieAgent is reported to achieve state-of-the-art performance in automated storytelling and movie generation, particularly for character consistency and narrative coherence.The contribution passage also identifies reduced human effort as an outcome of automated planning.

2. Related Works

Related work advances video quality, story visualization, and LLM-based cinematic planning, but existing approaches still leave fully automated long-form production unresolved.

  • Video Generation: Diffusion and transformer frameworks have improved video synthesis quality, temporal modeling, efficiency, controllability, and text-video understanding.Representative systems include VDM, SVD, CogVideo, and VideoPoet.
  • Remaining Gap: Despite these advances, existing frameworks still rely on manual input for narrative planning, cinematography, and scene composition.The paper positions MovieAgent’s multi-agent design as addressing this remaining limitation.
  • Story Visualization: Story visualization methods improve narrative consistency, temporal coherence, motion dynamics, personalization, and character-consistent generation.These approaches span GANs, diffusion models, and adapter-based techniques.
  • LLM-driven Planning: LLM-driven frameworks support narrative structuring, interactive storytelling, scene composition, long-form coherence, and adaptive shot planning.Examples include VideoDirectorGPT, VideoStudio, Mora, DreamFactory, and StoryAgent.

3. Method

MovieAgent transforms a script synopsis and character bank into a hierarchically planned, multi-scene and multi-shot movie. Specialized agents decompose narratives into sub-scripts, scenes, and shots before customized models generate video and, when required, audio.

  • Task formulation: MovieAgent maps a script synopsis and character bank to a sequence of shots forming a multi-scene, multi-shot long-form video.The objective includes narrative coherence, character consistency, and audiovisual synchronization.
  • Generation pipeline: A hierarchical multi-agent Chain of Thought workflow coordinates director, scene-plan, and shot-plan agents before customized shot and audio generation.The generation stage invokes image, video, talking-human, and audio models; joint video-audio generation uses a two-stage strategy because current models do not generate both simultaneously.
  • Director Agent: The Director Agent segments the synopsis into self-contained sub-scripts while preserving temporal, thematic, and character consistency.Its reasoning identifies acts, plot points, turning points, and logical story progression.
  • Scene Plan Agent: The Scene Plan Agent refines sub-scripts into detailed scenes containing characters, plot, emotional tone, visual style, cinematography notes, and scene boundaries.It identifies narrative transitions and natural breaks such as location shifts, time jumps, and emotional climaxes.
  • Shot Plan Agent: The Shot Plan Agent decomposes each scene into shots annotated with characters, plot, camera movements, shot types, and subtitles.It also specifies framing, cinematographic techniques, lighting, visual effects, and continuity across shots.

4.1. Experiment Setting

MovieAgent is evaluated on a newly constructed automated movie-generation dataset using automated metrics and human voting, with baselines spanning script processing, image generation, and video generation.

  • Evaluation combines automated metrics such as VBench with human voting because automated metrics may not fully match preferences and ground-truth videos are unavailable.FID cannot be computed because generated movies vary in shot count and lack ground-truth videos.
  • Baselines cover LLM-based script processing, image generation, and video generation components, because existing approaches do not address automated movie generation end to end.
  • Table 2 compares script-to-keyframe and script-to-video performance using VBench metrics, excluding models without character consistency.
  • MoviePrompts contains 10 script prompts derived partly from well-known movies and maps script and character inputs to shot videos.

4.2. Performance Comparisons and Analysis

MovieAgent outperforms nearly all baselines on automatic video metrics and receives strong human ratings across narrative, visual, script, character, and physical-law criteria.

  • Automatic Metric on MoviePrompts: MovieAgent achieves the highest reported motion smoothness 97.84, dynamic degree 76.27, and aesthetic quality 58.63 in movie generation.It also records the highest CLIP score 22.12 and Inception score 7.23 for keyframe generation.
  • Human Rating: MovieAgent outperforms the best baseline by up to 2 points on a five-point human-rating scale.Ratings were limited to key baselines because human assessment is costly.
  • Human Rating: MovieAgent receives ratings of 3.49 for Narrative Coherence, 4.01 for Visual Appeal, 3.89 for Script Faithfulness, 4.04 for Character Consistency, and 3.42 for Physical Law.

4.3. Ablation Study

Ablations examine internal CoT, LLM choice, and multi-agent collaboration using human assessment, showing gains from reasoning and collaboration alongside model-specific trade-offs.

  • Ablation Setup: Human assessment is used because script faithfulness and narrative coherence lack automated metrics, and prior work indicates human evaluation is more reliable for generation tasks.
  • Ablation Setup: The ablation study evaluates three movies because human evaluation can require up to 100 shots per movie and is costly.
  • Effect of Internal Chain of Thought: Internal CoT raises the average score from 3.55 to 3.61 and Narrative Coherence from 3.09 to 3.29 for GPT-4o.
  • Effect of Large Language Model: GPT-4o with multi-agent collaboration achieves the highest average score of 3.82, exceeding Deepseek-V3 at 3.54 and Deepseek-R1 at 3.78.GPT-4o nevertheless trails Deepseek-R1 in Narrative Coherence, 3.55 versus 3.79.
  • Effect of Multi-Agent: Multi-agent collaboration increases GPT-4o’s average score from 3.55 to 3.72, with script faithfulness and narrative coherence gains of 0.33 and 0.22.The authors attribute this to hierarchical collaboration that translates a synopsis into detailed scene- and shot-level movie scripts.

5. Conclusion

MovieAgent defines automated movie generation as a multi-agent CoT-based framework that reduces human intervention while targeting coherent, consistent, and synchronized long-form films.

  • 5. Conclusion: MovieAgent automates story structuring, scene planning, and shot composition through hierarchical reasoning and specialized AI agents.The framework is presented as a paradigm for automated movie and long-video generation.
  • 5. Conclusion: Experiments show improvements in story consistency, character preservation, and audiovisual synchronization.These outcomes address key challenges identified for AI-driven filmmaking.

7. Metrics of Human Evaluation

Human evaluation rates generated long videos on five 0-5 criteria covering visual quality, script adherence, narrative flow, character identity, and physical realism.

  • Evaluation framework: Generated long videos are evaluated on a 0-5 scale across five aspects.The criteria are visual appeal, script faithfulness, narrative coherence, character consistency, and physical-law adherence.
  • Visual Appeal: Visual appeal assesses realism, aesthetic consistency, artifacts, motion smoothness, and rendering quality.Detailed criteria and examples correspond to each score.
  • Script Faithfulness: Script faithfulness measures whether plots, character actions, settings, and narrative structure follow the described script.The scoring system is intended to assess narrative integrity consistently.
  • Narrative Coherence: Narrative coherence measures logical storyline progression and smooth transitions between scenes.Low scores reflect random scenes, abrupt cuts, illogical progressions, or weak continuity.
  • Character Consistency: Character consistency measures stable appearance, behavior, role, identity, facial features, and clothing throughout the video.The highest criterion requires stable identity, facial features, and clothing across the entire video.
  • Physical Law: Physical-law adherence evaluates realistic motion, object interactions, environmental consistency, and basic physics.Examples of violations include floating objects, wall phasing, and movements that defy gravity.

8. More Visualization and Analysis

MovieAgent’s hierarchical generation organizes stories from synopsis to sub-scripts, scenes, and shots, while its failure analysis identifies persistent visual, identity, interaction, transition, and physics problems.

  • 8.1. Advantages of Hierarchical Generation: Hierarchical generation progresses from Script Synopsis to Sub-Script to Scene to Shot, organizing themes, characters, emotions, and cinematography.This structure provides control over story details and supports logical progression across generation levels.
  • 8.2. Discussion on Failure Cases and Improvements: Failure cases include severe visual distortions and glitches, especially in facial structures and animations.The paper suggests higher-quality data or reward models as potential improvements.
  • 8.2. Discussion on Failure Cases and Improvements: Character identities can vary across frames, particularly for small targets, causing continuity errors.Suggested directions include larger datasets, temporal consistency mechanisms, embedding tracking, and feature matching.
  • 8.2. Discussion on Failure Cases and Improvements: Complex interactions such as walking and talking remain difficult to synchronize accurately, limiting script faithfulness.The paper identifies higher-quality data and improved prompt-following strategies as potential remedies.
  • 8.2. Discussion on Failure Cases and Improvements: Abrupt scene transitions can omit intermediate actions, as illustrated by the jump from charging to defending in a battle sequence.More precise internal CoT and hierarchical storyboarding are proposed to improve transitions.
  • 8.2. Discussion on Failure Cases and Improvements: Generated frames may contain distorted hands, limbs, body movements, and other anatomically unrealistic proportions.Potential improvements include physics-based rendering and biomechanical constraints.
Loading 2503.07314v1…