Source-linked AI summary
ShotVerse: Advancing Cinematic Camera Control for Text-Driven Multi-Shot Video Creation
Songlin Yang, Zhe Wang, Xuyi Yang, Songchun Zhang, Xianghao Kong, Taiyi Wu, Xiaotong Zhao, Ran Zhang, Alan Zhao, Anyi Rao
TL;DR
Cinematic multi-shot video generation lacks precise, practical camera control because textual instructions are unreliable and explicit trajectories are laborious and difficult to execute. ShotVerse addresses this with aligned data, a VLM Planner, and a camera-conditioned Controller, achieving camera-accurate, cross-shot-consistent videos with superior aesthetics while remaining limited to single-scene multi-shot generation.
Problem
Cinematic multi-shot camera control remains difficult because textual conditions are unreliable, while manual explicit trajectory plotting is laborious and complex trajectories can fail during execution.
Method
ShotVerse uses aligned (Caption, Trajectory, Video) data to decouple VLM-based cinematic trajectory planning from camera-conditioned holistic video rendering.
Results
ShotVerse achieves the lowest FVD and highest Aesthetic Quality (5.465), while its Controller achieves the lowest translation and rotation errors and highest CAS.
Takeaways & Limitations
ShotVerse bridges unreliable textual control and labor-intensive manual plotting, producing multi-shot videos that are camera-accurate and cross-shot consistent.
Takeaways & Limitations
The current method is limited to single-scene, multi-shot video, constraining duration and flexibility; longer multi-scene generation remains future work.
Abstract
from arXiv · showhide
Text-driven video generation has democratized film creation, but camera control in cinematic multi-shot scenarios remains a significant block. Implicit textual prompts lack precision, while explicit trajectory conditioning imposes prohibitive manual overhead and often triggers execution failures in current models. To overcome this bottleneck, we propose a data-centric paradigm shift, positing that aligned (Caption, Trajectory, Video) triplets form an inherent joint distribution that can connect automated plotting and precise execution. Guided by this insight, we present ShotVerse, a ``Plan-then-Control'' framework that decouples generation into two collaborative agents: a VLM (Vision-Language Model)-based Planner that leverages spatial priors to obtain cinematic, globally aligned trajectories from text, and a Controller that renders these trajectories into multi-shot video content via a camera adapter. Central to our approach is the construction of a data foundation: we design an automated multi-shot camera calibration pipeline aligns disjoint single-shot trajectories into a unified global coordinate system. This facilitates the curation of ShotVerse-Bench, a high-fidelity cinematic dataset with a three-track evaluation protocol that serves as the bedrock for our framework. Extensive experiments demonstrate that ShotVerse effectively bridges the gap between unreliable textual control and labor-intensive manual plotting, achieving superior cinematic aesthetics and generating multi-shot videos that are both camera-accurate and cross-shot consistent.
1 Introduction
ShotVerse addresses the difficulty of precise, cinematic multi-shot camera control by separating trajectory planning from video rendering and grounding both in aligned data.
- Existing methods struggle with textual camera control, unified multi-shot coordinates, and trajectories that follow cinematic patterns.
- Manual cinematic trajectory plotting imposes a prohibitive design burden because poses must align with narrative flow in a global coordinate system.
- Complex cinematic trajectories can fall outside camera-controlled video models’ training distribution, causing generation failures.
- Aligned (Caption, Trajectory, Video) triplets support separate optimization of trajectory planning and video generation.This data-centric view models P(Trajectory | Caption) and P(Video | Caption, Trajectory), avoiding unstable joint training while preserving compatibility.
- ShotVerse uses a VLM Planner for explicit cinematic trajectories and a Controller with a camera adapter for high-fidelity multi-shot rendering.The two agents share the aligned data distribution and the Controller uses a holistic multi-shot backbone.
- ShotVerse-Bench aligns disjoint trajectories through automated camera calibration and evaluates planning, execution fidelity, and multi-shot consistency.
2 Related Work
Related work spans multi-shot generation and camera-control datasets, while ShotVerse emphasizes holistic generation and automated cinematic plotting from textual descriptions.
- Multi-Shot Video Creation: Cascaded multi-shot approaches struggle with consistency across temporal gaps, while memory-based methods face error accumulation.
- Camera Trajectory Datasets: ShotVerse-Bench is presented as a unified trajectory dataset supporting cinematic camera-control research.
- Multi-Shot Video Creation: Holistic generation models entire video sequences jointly instead of iteratively synthesizing separate shots.
- Camera Control for Text-Driven Video Generation: Earlier camera-control methods encode camera extrinsics or use 3D priors, but they lack multishot storytelling capability.
3 Methodology: ShotVerse
ShotVerse’s methodology combines hierarchical, VLM-based trajectory planning with camera-conditioned holistic video generation, using unified global coordinates and cross-shot temporal modeling.
- Overview: ShotVerse decouples multi-shot camera control into a Planner that generates trajectories and a Controller that renders them into video.
- Planner: Shot-Aware Cinematic Trajectory Plotting: The Planner represents each shot as a trajectory of poses in a unified global coordinate system.Camera poses are represented with translation and rotation parameters and converted through reversible tokenization and de-tokenization.
- Planner: Shot-Aware Cinematic Trajectory Plotting: Hierarchical prompts combine global and per-shot text with learnable query placeholders that provide slots for shot-specific camera plans.
- Planner: Shot-Aware Cinematic Trajectory Plotting: VLM query-token states aggregate global context, previous shots, and current-shot instructions to encode context-aware camera codes.
- Planner: Shot-Aware Cinematic Trajectory Plotting: A lightweight autoregressive Transformer decodes concatenated shot codes into variable-length trajectory-token sequences with temporal modeling across shots.
- Controller: Cinematic Camera Control: The Controller injects camera features into the video model and uses 4D Rotary Positional Embedding to encode shot-aware structure.
- Controller: Cinematic Camera Control: The controller objective conditions flow matching on textual embeddings and explicit camera-condition embeddings.
4 Dataset and Benchmark: ShotVerse-Bench
ShotVerse-Bench provides aligned, globally unified multi-shot camera trajectories with hierarchical captions for training and evaluating cinematic video generation. Its three-track protocol separately assesses planning, execution, and end-to-end quality.
- Dataset Curation: ShotVerse-Bench contains 20,500 high-production clips with hierarchical captions and unified multi-shot trajectories.The dataset spans 27+ fine-grained trajectory categories, multiple framing scales, explicit transitions, and four content domains.
- Dataset Curation: The dataset includes diverse camera motions, compositions, shot transitions, durations, and content domains.Examples include 1,639 zoom/scale samples, 1,720 explicit cuts or transitions, and clips averaging 5.50 seconds.
- Evaluation Benchmark: The three-track protocol evaluates text-to-trajectory planning, trajectory-to-video execution fidelity, and end-to-end text-to-video generation quality.Track A uses hierarchical prompts and globally aligned camera sequences, while Track B evaluates execution from trajectories.
- Dataset Curation: ShotVerse-Bench is presented as the first large-scale dataset combining multi-shot cinematic camera trajectories with rich, multi-level caption annotations.This alignment addresses the lack of semantic descriptions paired with globally unified camera trajectory annotations.
- Evaluation Benchmark: Track A compares representative trajectory planners, while Track C includes open-source and closed-source multi-shot video models.The benchmark adapts trajectory-generation baselines for multi-shot evaluation using the proposed calibration pipeline.
5 Experiments
Experiments evaluate ShotVerse across alignment, camera-control fidelity, cinematic quality, and component ablations. The method achieves strong benchmark performance while error analysis identifies limits in long-context persistence, dense scenes, and single-scene scope.
- Benchmark Results: ShotVerse’s three-track evaluation measures alignment, control, and cinematic quality using rendered outputs, VLM scoring, and human studies.The evaluation emphasizes rendered video because numerical trajectory metrics may not fully reflect cinematographic experience.
- Benchmark Results: The VLM-driven Planner achieves the best results on both evaluated benchmarks, indicating stronger cross-domain generalization than the compared planners.GenDoP performs competitively on its native benchmark but shows a domain gap after retraining on ShotVerse-Bench.
- Benchmark Results: The Controller achieves the lowest translation and rotation errors and the highest CAS among compared methods under ground-truth trajectory control.These results suggest stronger cross-shot consistency under the CAS proxy.
- Benchmark Results: ShotVerse achieves the lowest FVD and an Aesthetic Quality score of 5.465 in end-to-end text-to-video evaluation.Its 4D RoPE raises Shot Transition Accuracy to 0.933, compared with 0.927 for MultiShotMaster and 0.645 for HoloCine.
- Ablation Studies: Ablations show that removing the VLM encoder degrades F1-Score and CLaTr-CLIP, supporting the role of semantic-spatial priors in trajectory planning.The reported degradation compromises narrative alignment and trajectory-text consistency.
- Generalizability: Long shot-reverse-shot sequences and high-density crowds remain difficult, producing occasional off-target trajectories, background drift, or subject flickering.These failures arise in settings involving long-context persistence, many independently moving subjects, and complex occlusions.
- Generalizability: The method is limited to single-scene, multi-shot video, constraining duration and flexibility despite compatibility with autoregressive generation.Extending the approach to longer, multi-scene generation remains future work.
6 Conclusions
ShotVerse decouples cinematic multi-shot generation into VLM-driven trajectory planning and geometry-aware rendering, supported by the ShotVerse-Bench dataset. Its evaluation reports technical camera accuracy alongside cinematic pacing and visual salience.
- 6 Conclusions: ShotVerse uses a Plan-then-Control framework that separates VLM-driven cognitive plotting from geometry-aware video rendering.This decoupling targets the spatial logic required for multi-shot storytelling.
- 6 Conclusions: ShotVerse-Bench unifies disjoint shot trajectories into a global coordinate system through a novel calibration pipeline.The dataset is presented as a high-fidelity foundation for teaching cinematic structure.
- 6 Conclusions: A three-track evaluation protocol assesses ShotVerse's technical accuracy and its implicit understanding of cinematic pacing and visual salience.The conclusion frames these evaluations as confirming both geometric and cinematic capabilities.