PaperlayerView exact revision on arXiv

Source-linked AI summary

FilmFlow: Agentic System for 3D Virtual Film Cinematography

Yiming Zhang, Jiarui Guo, Haojia Wei, Yifei Liu, Yifan Yang, Hongjie Zhang, Xue Yang, Zhihang Zhong

arXiv:2610.30753v1cs.CV

TL;DR

Translating cinematographic intent into controllable, executable camera trajectories remains challenging because spatial and temporal decisions are often implicit in language or reference footage. FilmFlow uses an agentic Director and concurrent camera operations to generate and refine trajectories, and it is reported to achieve the strongest benchmark performance and highest user preference among evaluated baselines.

  • Problem

    Camera instructions in language or reference footage leave spatial and temporal decisions implicit, making it challenging to translate cinematographic intent into controllable, executable trajectories.

    Source: §1 · p. 2

  • Method

    FilmFlow's Director composes concurrent atomic camera operations into executable plans, supported by agents for intent interpretation, initial viewpoint selection, and visual critique and refinement.

    Source: §1 · p. 2

  • Results

    FilmFlow achieves the best performance across reported automatic metrics and the highest user-study preference for both text-directed and video reference-directed generation among evaluated baselines.

    Source: §1 · p. 2

  • Takeaways & Limitations

    CinemaBench evaluates text- and video-directed camera generation, and FilmFlow's executable trajectories coordinate camera motion, subject tracking, and framing to realize cinematographic intent.

    Source: §1 · p. 2, §5 · p. 10

Abstract

from arXiv · show

Complex camera motion is central to visual storytelling, yet translating cinematographic intent into controllable, executable camera trajectories remains challenging. We introduce FilmFlow, an agentic framework that converts natural-language instructions or reference videos into explicit camera trajectories for dynamic 3D environments. To generate these trajectories, a Director agent constructs executable camera plans by composing high-level motions from concurrent atomic operations that jointly control translation, rotation, and framing. The Director is supported by three specialized agents: an Interpreter that extracts scene-grounded cinematographic intent, a Scout that selects initial viewpoints, and a Critic that evaluates rendered results and provides visual feedback for iterative plan refinement. We also introduce CinemaBench, comprising 150 Blender scenes from self-created and licensed sources, with five camera instructions per scene for 750 text-directed tasks and 60 reference videos associated with a subset of scenes. CinemaBench supports evaluation of both text-directed camera generation and video reference-directed camera generation. Across a diverse set of evaluated baselines, FilmFlow achieves the strongest benchmark performance and the highest user preference, demonstrating effective spatial reasoning, temporal coordination, and visual composition for executable 3D cinematography.

1 INTRODUCTION

Camera motion must coordinate framing, movement, and subject motion, but translating implicit intent from language or reference footage into executable trajectories remains challenging. FilmFlow combines concurrent camera operations with specialized agents and introduces CinemaBench to evaluate text- and video-directed generation.

1 INTRODUCTION

  • FilmFlow addresses the challenge of translating implicit spatial and temporal decisions from natural-language instructions or reference footage into executable camera trajectories for dynamic 3D scenes.Camera movement, framing, and subject motion must be coordinated to realize narrative effects.Source: §1 · p. 2

Agentic cinematography in dynamic scenes

  • The Interpreter extracts scene-grounded intent, the Scout selects initial viewpoints, and the Critic evaluates and refines executed plans and rendered results.The Director constructs the executable camera plans.Source: §1 · p. 2

Concurrent motion representation and optimization

  • The Director composes high-level motions from concurrent atomic operations, with explicit parameters and timing to coordinate translation, rotation, and framing.Critic-guided refinement uses execution and visual feedback, while in-context demonstrations guide planning.Source: §1 · p. 2

CinemaBench

  • FilmFlow achieves the best performance across reported automatic metrics and the highest user-study preference for both text-directed and video reference-directed generation among evaluated baselines.CinemaBench includes 150 Blender scenes, 750 text-directed tasks, and 60 reference videos associated with a subset of scenes.Source: §1 · p. 2

2 RELATED WORK

FilmFlow builds on computational cinematography, compositional camera control, and language-agent workflows, combining executable camera planning with refinement in dynamic 3D scenes.

Computational Cinematography and Camera Control

  • FilmFlow unifies text-directed and video reference-directed generation through executable camera planning and refinement in dynamic 3D scenes.Prior approaches include rule- and optimization-based control, text-conditioned trajectory generation, and reference-directed optimization.Source: §2 · p. 3

Compositional Camera Motion and Trajectory Optimization

  • Unlike sequentially concatenated motion primitives, FilmFlow uses independently parameterized atomic operations with explicitly overlapping intervals, refined using execution checks and rendered feedback.This supports coordinated actions such as orbiting, rising, and tracking a moving subject.Source: §2 · p. 3

Language Agents for Filmmaking and Visual Feedback

  • FilmFlow assigns intent interpretation, opening-view selection, and camera planning to specialized agents, while a shared Critic evaluates views and motion and guides revisions.The executable camera plan is the shared representation for planning, evaluation, and refinement.Source: §2 · p. 3

3 METHODS

FilmFlow turns a natural-language instruction or reference video and a target 3D scene into an executable camera trajectory through four specialized agents. Its plan composes concurrent camera operations, then uses Critic feedback for evaluation and refinement.

3 METHODS

  • FilmFlow uses an Interpreter to ground input in the scene, a Scout to select an opening viewpoint, a Director to plan motion, and a Critic to guide evaluation and refinement.For video input, the Interpreter also analyzes reference motion and framing and supplies sampled frames and estimated relative camera poses as auxiliary evidence; their metric scale is unverified.Source: §3 · pp. 3–4, Figure 2 · p. 4

3.1 CONCURRENT ATOMIC CAMERA OPERATIONS

  • The Director represents a shot as timed atomic operations whose overlapping intervals let compatible camera changes jointly control position, orientation, and focal length.The executor combines operation-specific relative changes with a reference or tracking-derived camera state, then samples camera states into animation keyframes.Source: §3.1 · pp. 4–5, Table 1 · p. 5

3.2 CRITIC-GUIDED EVALUATION AND REFINEMENT

  • A shared Critic evaluates rendered evidence against the script to guide both opening-view search and operation-plan revision.Scouting assesses instruction alignment, subject readability, and composition; directing adds spatial storytelling, motion coherence, and cinematic quality, after hard checks for collisions and execution validity.Source: §3.2 · p. 5

3.3 IN-CONTEXT CINEMATOGRAPHIC PLANNING

  • FilmFlow stores only validated, Critic-accepted demonstrations that meet overall and per-dimension quality thresholds, using development scenes disjoint from test scenes.Retrieval uses up to three Interpreter-assigned cinematography keywords and Jaccard similarity; the Director adapts a selected example to the current scene and opening, or proceeds without one if none qualifies.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments compare FilmFlow with baselines on CinemaBench, test component ablations, and assess human judgments for both input modes.

4 EXPERIMENTS

  • CinemaBench has 150 Blender scenes, including 73 dynamic and 77 static scenes, and evaluation reports means over 500 T2V and 40 V2V test tasks.The scene-based split reserves 50 development scenes for the ICL example bank and uses 100 test scenes.Source: §4 · p. 6

4.1 COMPARISON WITH BASELINES

  • Baseline comparisons use shared task inputs and report four metrics separately for T2V and V2V; qualitative examples show FilmFlow revealing instructed subjects and maintaining visibility as subjects move.The V2V comparison concerns camera behavior rather than scene appearance.Source: §4.1 · p. 6, Table 2 · p. 6, Figure 3 · p. 7

4.2 ABLATION STUDY

  • In text-directed ablations, removing Scout leaves a distant opening view, while removing Critic or ICL allows foreground characters to obstruct the intended subject.The video reference-directed ablation also shows less consistent target framing, obstructed views, or target loss in ablated variants.Source: Figure 5 · p. 8, Figure 6 · p. 9
  • Full FilmFlow leads the ablation metrics except V2V Probe, where w/o ICL scores 0.766 vs. 0.750; Probe is an exploratory soft score, not a pass rate.The external evaluation protocol remains fixed across variants.Source: §4.2 · pp. 7–8, Table 3 · p. 8

4.3 USER STUDY

  • The blinded study used 42 participants, each assigned 10 T2V and 10 V2V trials, with 420 T2V and 397 V2V completed rankings analyzed.Participants rated alignment, visual quality, and motion smoothness, judged pass or fail, and ranked outputs by preference.Source: §4.3 · pp. 8–9
  • 61.0% of T2V judgments and 63.2% of V2V judgments ranked FilmFlow first; it also had the highest pass rates and mean ratings in both tasks.Human ratings covered alignment, visual quality, and motion smoothness.Source: Table 4 · p. 9, Table 5 · p. 9

5 CONCLUSION

The conclusion presents FilmFlow and CinemaBench, reports strong benchmark performance and user preference, and identifies future extensions.

5 CONCLUSION

  • Experiments show FilmFlow achieves the strongest benchmark performance and highest user preference among evaluated baselines.The framework generates executable camera trajectories for dynamic 3D scenes from natural-language instructions or reference videos.Source: §5 · p. 10
  • Future work will test whether smaller models can match performance through task-specific fine-tuning and extend the system to automate scene construction alongside camera planning.Source: §5 · p. 10
  • The authors report using generative AI to improve manuscript clarity, debug code, generate synthetic datasets, and create the teaser figure, with author review and responsibility for final content.Source: AI USE STATEMENT · p. 11
  • The release will preserve asset attribution and licensing restrictions, exclude materials lacking redistribution permission, and provide acquisition instructions where possible.The user study received institutional ethics clearance; participants gave informed consent and responses were anonymized.Source: ETHICS STATEMENT · p. 11
  • The authors plan to publicly release the implementation and CinemaBench annotations with instructions for obtaining required scene assets.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 AGENT ROLES AND STRUCTURED INTERFACES

FilmFlow’s agents turn scene-grounded intent into an opening view and an executable camera plan, then use rendered feedback to refine it. The section also describes how reference-video evidence enters the shared planning pipeline.

A.1 AGENT ROLES AND STRUCTURED INTERFACES

  • The Interpreter grounds the instruction in scene observations, identifying relevant objects and separating opening composition and shot intent from the eventual operation timeline.The example distinguishes scene description G and cinematographic script B; B specifies the requested dolly zoom, duration, and final hold.Source: §A.1 · pp. 14–15
  • The Scout submits rendered candidate views for Critic scoring and selects the highest-scoring opening when its two-round search ends below the 0.70 threshold.The selected 0.68 view is provisional, and its pose and focal length remain fixed during motion-plan refinement.Source: §A.1 · pp. 15–16
  • The Director returns a complete timeline of concurrent, time-bounded operations, which the executor composes into camera states and renders for evaluation.The example’s initial plan combines a screen-position lock, zoom, orbit, and dolly-in.Source: §A.1 · p. 16
  • Re-evaluation selects the revised plan at q(1)_motion = 0.7367, although divider overlap and final-hold drift remain.The revision shortens the orbit from 120° over nine seconds to 18° over 2.5 seconds and extends the framing lock through the shot.Source: §A.1 · p. 17
  • For reference-video generation, the Interpreter grounds motion and framing cues in the target scene, while the Critic compares timestamped reference and rendered frames.Relative poses with unverified metric scale provide motion cues, not target-scene camera coordinates; the Critic does not receive LingBot poses.Source: §A.1 · p. 18

A.2 ATOMIC CAMERA OPERATIONS AND PARAMETERS

The camera codebook exposes atomic operations with configurable parameters and explicit time intervals. Compatible operations compose through channel-specific relative changes; the section also reports system budgets and average task cost.

A.2 ATOMIC CAMERA OPERATIONS AND PARAMETERS

  • The 22-operation codebook specifies configurable parameters, start and end times, and units, with common easing and transition defaults.Angles use degrees, focal lengths use millimeters, explicit distances use scene world units, and dolly scales are endpoint-to-initial distance ratios.Source: §A.2 · p. 18, Table 6 · p. 18
  • Relative changes combine by channel: translations and compatible angles add, positive scale ratios multiply, and orbit and Dolly precede Truck and Pedestal offsets.Orbit is represented by an angular offset and radius ratio, not a simple additive position offset.Source: §A.2 · p. 19

A.3 HYPERPARAMETERS AND COMPUTATIONAL COST

  • FilmFlow averages 2.23 minutes and 69.5k tokens per task across 6.23 model calls, including unsuccessful attempts but excluding final video export.The reported token average comprises 65.9k input and 3.7k output tokens.Source: §A.3 · p. 19

B.1 DATASET COMPOSITION

CinemaBench pairs Blender scenes with text-directed camera tasks and a smaller set of reference-video tasks. Its scene-disjoint development and test splits support evaluation on held-out environments.

B.1 DATASET COMPOSITION

  • CinemaBench contains 150 Blender scenes, including 73 dynamic and 77 static scenes, built from self-created and licensed assets.The environments vary in setting, subject count, spatial layout, occlusion, and subject motion.Source: §B.1 · p. 20, Table 7 · p. 20
  • Five instructions per scene yield 750 text-driven tasks, balanced per scene as one easy, two medium, and two hard tasks.Difficulty ranges from single-subject, single-movement instructions to coordinated operations, attention shifts, or precise temporal constraints.Source: §B.1 · p. 20
  • A subset of 60 scenes has one reference-video task each, and the scene-disjoint split assigns 50 scenes to development and 100 to test.The split contains 250 T2V and 20 V2V development tasks, and 500 T2V and 40 V2V test tasks.Source: §B.1 · p. 20

B.2 LANGUAGE OF CAMERA MOTION

CinemaBench instructions specify subjects, motion, framing, and temporal structure. They test whether a camera trajectory realizes both requested visual outcomes and the ordering or concurrency of actions.

B.2 LANGUAGE OF CAMERA MOTION

  • Camera instructions describe subjects, motion, framing, and temporal structure, including subject placement, scale, visibility, and attention shifts.Motion descriptions can specify action direction and extent.Source: §B.2 · p. 20
  • Temporal wording distinguishes sequential actions from concurrent ones, and valid trajectories must realize both the actions and their temporal organization.“Orbit, then move closer” is ordered; “orbit while moving closer” requires simultaneous rotation and approach.Source: §B.2 · p. 20
  • Visual intent does not prescribe a unique operation sequence, so text-driven fulfillment is judged from executed camera behavior and rendered composition rather than a ground-truth trajectory.Requests to reveal a subject, shift attention, or hold a close-up can be satisfied by multiple operation combinations.Source: §B.2 · p. 21

B.3 BASELINE IMPLEMENTATION

The baseline adaptations align each method’s camera trajectory with the target scene while preserving the methods’ native planning choices where applicable.

B.3 BASELINE IMPLEMENTATION

  • Baselines share FilmFlow’s inputs and are not initialized from its outputs; trajectory coordinates, scale, and focal lengths are adapted to the target scene, while CinemaTraj plans its own opening.For video-reference baselines, estimated poses are converted to Blender coordinates and applied without scene-aware placement, VLM-based motion planning, or closed-loop refinement; GenDoP and LAMP do not track moving subjects.Source: §B.3 · p. 21

C ADDITIONAL QUALITATIVE RESULTS

Additional comparisons illustrate camera behavior across text-directed and video reference-directed tasks, while case studies show how Scout and Critic feedback affect planning and refinement.

C ADDITIONAL QUALITATIVE RESULTS

  • The additional examples compare methods under the same scene and conditioning input, using sampled frames to show camera motion, subject framing, and composition.Supplementary videos provide temporal context beyond the sampled frames.Source: C ADDITIONAL QUALITATIVE RESULTS · p. 21

D CASE STUDY: SCOUT REFINEMENT

  • Scout selects a higher, downward-pitched opening with a shorter lens, revealing more of the subject and leaving room for the requested push-in.The Director’s fixed-focal-length opening crops the hair and shows little of the chest, so forward motion would tighten the framing further.Source: D CASE STUDY: SCOUT REFINEMENT · p. 21, Figure 9 · p. 23

E CASE STUDY: DIRECTOR REFINEMENT

  • With the opening fixed, Critic feedback links execution problems to changes in operation selection, parameters, or timing.The before-and-after plans pair rendered frames with operation timelines; supplementary video provides motion-continuity context.Source: E CASE STUDY: DIRECTOR REFINEMENT · p. 21, Figure 10 · p. 23
  • For a bullet-time orbit, FilmFlow keeps the figure centered as the rooftop slides behind it and ends on a stable frontal view.LAMP and GenDoP drift toward the wall or empty frames, while CinemaTraj approaches the figure but leaves it off-center under an oblique view.Source: Figure 7 · p. 22
  • For a group-to-subject reference, FilmFlow reproduces the progression and ends on a close-up matching the reference framing.LingBot-Map remains near a static wide view, VGGT-Ω crops the subject in intermediate frames, and the zero-shot VLM loses the subject in the final frame; the comparison concerns camera behavior, not scene appearance.Source: Figure 8 · p. 22

F CASE STUDY: GENERALIZATION THROUGH ICL

The case studies show in-context demonstrations guiding operation choices across different scenes, and grey-model renders serving as structural conditions for video generation.

F CASE STUDY: GENERALIZATION THROUGH ICL

  • A retrieved demonstration transfers operation choice and timing across scenes: pairing Dolly In with Frame Subject and Aim Follow keeps the bunny nearly constant in frame during a forward push.Without ICL, the Director uses a forward push alone and the bunny fills the view by 6.0 s; the demonstration transfers structure, while target references, distances, and focal lengths are resolved in the current scene.Source: F CASE STUDY: GENERALIZATION THROUGH ICL · pp. 23–24, Figure 11 · p. 24

G APPLICATION: GREY MODEL REFERENCE AS VIDEO-GENERATION CONDITIONS

Related papers

Loading 2610.30753v1…