PaperlayerView exact revision on arXiv

Source-linked AI summary

CamPilot: A Multi-Agent Cinematic Assistant for Camera-Controlled Movie Generation

Yang Wu, Stefano Petrangeli, Ishita Dasgupta, Yu Shen

arXiv:2609.10943v1cs.CV

TL;DR

Current text-to-video systems remain limited in professional cinematographic language and multi-shot continuity. CamPilot separates story planning from camera-controlled execution, learning camera grammar from professional movies with GRPO and coordinating multiple agents. Experiments report improved camera-work and generation quality, while the paper acknowledges limits from finite cinematographic coverage, discrete labels, and incomplete automatic evaluation.

  • Problem

    Existing text-to-video methods remain less refined than professional filmmaking in cinematographic language and have difficulty maintaining multi-shot continuity.

  • Method

    CamPilot is a multi-agent framework that plans scenes and shots, learns camera grammar from 14K professional movies with GRPO, and generates video under explicit camera-work control.

  • Results

    CamPilot consistently improves camera-work classification and downstream generation quality across multiple planner backbones compared with baselines.

  • Takeaways & Limitations

    Separating story planning from cinematographic execution supports structured camera work, multi-shot continuity, and overall cinematic quality.

  • Takeaways & Limitations

    CamPilot may miss rare or highly stylized cinematographic patterns because it learns from a finite corpus and fixed discrete camera-work labels.

Abstract

from arXiv · show

The integration of large language models (LLMs) into video generation has enabled rapid text-to-video creation and improved visual quality. However, it still falls short of professional filmmaking, where cinematographic language is less refined than human-crafted camera work and multi-shot continuity remains challenging. To address these limitations, we introduce CamPilot, a multi-agent framework that integrates cinematographic planning and camera-work control to produce more coherent, logically structured, and human-aesthetic movies. CamPilot adopts a GRPO-based learning paradigm to learn camera work planning from 14K real-world professional movies, internalizing motion patterns and composition principles that support reasoning over shooting techniques (e.g., camera angle, motion, and focal behavior) and cross-shot relationships for controllable camera-viewpoint generation. Multiple agents further collaborate and evolve to improve overall output quality. To support this work and further studies in this domain, we establish CamEval, a benchmark for evaluating camera work quality and cinematic engagement. Empirical results show that CamPilot outperforms state-of-the-art text-to-movie generation methods on cinematographic control and quality, highlighting the impact of professional camera design on movie generation.

1 Introduction

Existing text-to-video systems enable rapid generation but remain less refined than professional filmmaking in cinematographic language and multi-shot continuity. CamPilot addresses this gap through planning-first, camera-controlled generation, while CamEval evaluates camera work and cinematic engagement.

  • 1 Introduction: Existing text-to-video methods still struggle with refined cinematographic language and coherent multi-shot continuity compared with professional filmmaking.The paper attributes this limitation to a mismatch between current generation and professional camera-work design.
  • 1 Introduction: CamPilot first plans a professional shooting script and then generates camera-controlled movies to better express intended scenes and emotions.The plan specifies shot-level camera angle, motion, and focal behavior.
  • 1 Introduction: CamPilot integrates cinematographic planning with explicit camera-work control and learns camera grammar from 14K professional movies using GRPO-based training.The framework supports shot-level technique reasoning and cross-shot relationship modeling.
  • 1 Introduction: CamEval is introduced as a benchmark for evaluating camera-work quality and cinematic engagement against state-of-the-art text-to-movie methods.
  • 1 Introduction: CamPilot improves professional camera-work quality, video generation quality, visual fidelity, and multi-shot coherence in empirical experiments.

2 Related Work

Related work advances text-to-video generation, agentic planning, and multi-scene consistency through structured scripts and iterative refinement. CamPilot extends this direction by learning camera language from real movies for controllable cross-shot cinematographic planning.

  • 2 Related Work: Diffusion-based text-to-video models improve realism and temporal coherence by extending image diffusion into the spatiotemporal setting.Latent-space formulations further improve efficiency and scalability.
  • 2 Related Work: LLM-based agentic systems decompose tasks, invoke tools, and iteratively refine outputs through feedback or self-critique.
  • 2 Related Work: Recent systems expand prompts into scripts, storyboards, or scene plans to improve long-form and multi-scene consistency.VideoDirectorGPT uses explicit layouts, while VideoStudio produces multi-scene scripts for downstream generation.
  • 2 Related Work: CamPilot focuses on learning camera language from real movies for controllable, cross-shot cinematographic planning within a multi-agent text-to-movie pipeline.

3 Methodology

The methodology formulates long-form movie generation as producing a coherent, cinematic sequence of scene-organized shots from a textual prompt. A hierarchical function expands the prompt into structured plans and generates each shot under explicit camera-work control.

  • 3.1 Problem Definition: The task is to generate a multi-scene, multi-shot movie that remains coherent in narrative and cinematic style from a textual prompt.
  • 3.1 Problem Definition: The output movie is organized as shot videos indexed by scene and shot, with N scenes and Mi shots in scene i.
  • 3.1 Problem Definition: Function F expands the prompt into structured scene and shot plans, generates each shot with explicit camera-work control, and concatenates the shots into the final movie.

3.2 CamPilot Overview

CamPilot separates story planning from cinematographic execution, using structured shot context to plan camera work before video synthesis. Multi-agent evaluation, revision, character references, and frame conditioning support quality refinement and cross-shot continuity.

  • 3.2 CamPilot Overview: CamPilot expands a user prompt into scene and shot plans, then generates each shot under explicit camera-work control.The pipeline separates narrative planning from cinematographic execution and uses an off-the-shelf text-to-video backbone for synthesis.
  • 3.2.1 Scene Planner: The Scene Planner specifies narrative intent, location, time, and mood, while the Shot Planner converts each scene into an ordered, structured shot list.Shot descriptions include characters, action, environment, mood, tone, and intention for downstream camera reasoning.
  • 3.2.3 Camera Work Planner: The Camera Work Planner reasons over scene context, preceding shots, and the current shot to output structured camera angle, shot size, and motion.Motion is represented through type, focal length, speed, rotation intensity, and translation intensity.
  • 3.2.4 Training the Camera Work Planner with GRPO: GRPO training maps planning contexts to camera-work outputs using task-specific rewards, reference-policy regularization, and attribute-level supervision.The reward factorizes over camera angle, shot size, motion, and five motion sub-attributes.
  • 3.2.5 Evolution Loop: An evaluator–reviser loop repeatedly edits shot descriptions, camera work, or generator conditions when quality checks fail, up to a fixed iteration limit.Accepted shots are appended to the sequence, and each ending frame conditions the next shot for improved continuity.
  • 3.2.6 Character Bank and Video Generation: A Character Bank supplies relevant character references, while previous-shot ending frames provide start-frame conditions for smoother long-form transitions.The backbone generator can be any existing text-to-video model.

4 Dataset Construction

CamEval links professional movie footage with structured shooting scripts and camera-work labels to provide scalable supervision for camera-language learning. Its construction segments movies into clips, samples frames, and uses a vision–language model to produce structured annotations.

  • 4 Dataset Construction: CamEval is distilled from 14K professional movies and pairs clip-level visual observations with structured shooting scripts and camera-work labels.The dataset is designed to link real-world footage with structured camera-work attributes.
  • 4 Dataset Construction: The construction pipeline segments each source movie into short clips before annotation.Clip segmentation provides the units for subsequent frame sampling and structured labeling.
  • 4 Dataset Construction: For each clip, uniformly sampled frames are summarized by Qwen/Qwen3-VL-32B-Instruct into scene, shot, and camera-work descriptions.The labels cover camera angle, shot size, and camera motion, with scripts describing location, characters, action, mood, and intention.

5 Experiments

CamPilot is evaluated against multiple planning baselines across camera-work classification, keyframe quality, and movie-generation metrics, including qualitative multi-shot comparisons. Results consistently favor CamPilot, while discussion attributes its gains to structured, context-aware planning and iterative refinement.

  • 5 Experiments: Experiments compare Standard, Vanilla-SFT, DreamFactory, and MovieAgent across three open-source planner backbones while holding the downstream Firefly generator fixed.The evaluation covers camera-work classification, keyframe generation, and long-form movie generation.
  • 5.4 Experimental Results: CamPilot consistently achieves the strongest camera-work classification and downstream generation quality within each planner-backbone family.It also improves cross-shot subject stability and aesthetics, with Sub_Cons around 95 and Aesthetic around 59 across backbones.
  • 5.4 Experimental Results: 63.7 Macro-Acc for CamPilot exceeds Claude-Sonnet-4.5 at 60.1 and GPT-5 at 53.2 under the reported comparison.The comparison indicates that CamPilot remains competitive with substantially larger closed-source commercial models.
  • 5.5 Qualitative Results: CamPilot produces more dynamic camera work, clearer motion intent, and more coherent shot progression than baseline methods in qualitative comparisons.Its richer camera movements and deliberate composition changes make the generated videos appear more cinematic and engaging.
  • 5.6 Ablation Study: 62.1 Macro-Acc and 26.5 Macro-F1 improve over single-shot Standard results of 44.3 and 9.1, respectively.CamPilot also reports CLIP/Inception of 22.0/9.9 and Sub_Cons/Aesthetic of 95.0/59.0, exceeding Standard’s 87.6/51.2.
  • 5.7 Discussion: The Camera Work Planner conditions decisions on scene intent, preceding-shot context, and the current shot description to coordinate multi-shot camera relationships.An evaluator–reviser loop further identifies and refines shots that fail to satisfy intended camera-work or continuity requirements.

6 Conclusion and Future Work

CamPilot separates story planning from cinematographic execution through scene and shot scripts, a GRPO-trained Camera Work Planner, and refinement. Across multiple planner backbones, it improves camera-work classification and downstream generation quality, while future work targets stronger backbones and broader evaluation.

  • 6 Conclusion and Future Work: CamPilot separates story planning from cinematographic execution by constructing scene and shot scripts before generating structured camera work.The Camera Work Planner is trained with GRPO on CamEval to learn camera grammar from real-world professional movies.
  • 6 Conclusion and Future Work: CamPilot consistently improves camera-work classification and downstream generation quality across multiple planner backbones compared with baselines.The conclusion connects structured camera control with stronger multi-shot continuity and overall cinematic quality.
  • 6 Conclusion and Future Work: Future work will extend CamPilot to stronger video backbones, richer evaluation protocols, adaptive planning and revision, and more user-controllable cinematic styles.The stated targets include long-form coherence and character consistency.

Limitations

CamPilot’s limitations concern incomplete coverage of cinematographic language, possible planning errors that propagate across shots, and imperfect alignment between automatic metrics and human cinematic judgments.

  • Coverage of Cinematographic Language: CamPilot may miss rare or highly stylized cinematographic patterns because it learns from a finite movie corpus and fixed label schema.Examples include unconventional lens behavior, complex blocking, and genre-specific shot conventions.
  • Coverage of Cinematographic Language: Discrete camera-work labels can underrepresent continuous creative variations in camera movement and composition.The authors propose discrete-continuous representations as one possible future direction.
  • Planning Faithfulness and Error Propagation: Planner mistakes can make generated videos deviate from the intended narrative or exhibit inconsistent motion across shots.Because the pipeline is planning-first, errors in prompt interpretation or implausible plans may propagate downstream.
  • Evaluation: Automatic metrics may not fully reflect human judgments of cinematic quality, especially for subtle transitions or creative shot choices.Future CamEval extensions could add richer human evaluation, preference modeling, and task-specific rubrics.

Ethics Statement

The authors report compliance with relevant ethical guidelines and state that the study uses publicly available data without private or personally identifying information.

  • Ethics Statement: The work complies with the relevant ACL ethical guidelines after review of the ACL Ethics Policy.
  • Ethics Statement: All study data come from publicly available sources and exclude private or personally identifying information.

A Performance Comparison on Subtasks

CamPilot achieves the strongest reported performance across camera-work subtasks, with particularly clear gains on camera angle, shot size, and motion-related classification.

  • Subtask Performance: 92.7 Camera Angle Macro-ACC and 83.7 Shot Size Macro-ACC are CamPilot’s reported results, exceeding the cited baselines.Camera Angle compares with MovieAgent at 84.0 and DreamFactory at 80.4; Shot Size compares with Standard at 78.3.
  • Subtask Performance: 61.2 Camera Motion (Translation) Macro-ACC exceeds MovieAgent’s 51.8, alongside CamPilot scores of 31.0 Macro-Prec, 30.4 Macro-Rec, and 28.3 Macro-F1.

B Prompt

The prompt operationalizes camera-work annotation as structured, reproducible JSON grounded in observable frames, fixed cinematography labels, conservative judgments, and shot continuity rules.

  • Prompt Design: The prompt assigns the model a professional cinematographer and camera-operator role for camera-work planning in text-to-movie generation.
  • Prompt Design: Given time-ordered frames, the model must describe observable shot content, infer camera attributes, provide a brief intent rationale, and output valid JSON.
  • Annotation Scope: The annotation focuses on shot size, shot angle, camera motion, focal behavior, motion speed, and motion intensity.
  • Annotation Scope: Multiple frames are treated as consecutive moments from one shot unless a cut is evident, in which case the dominant shot is annotated.
  • Reliability Guidelines: The prompt enforces conservative, concrete, internally consistent judgments, including compatibility between static shots and translation speed.
  • Label Schema: The schema requires one option per field for camera angle, shot size, motion type, focal behavior, speed, rotation intensity, and translation intensity.
Loading 2609.10943v1…