PaperlayerView exact revision on arXiv

Source-linked AI summary

ShotVerse: Advancing Cinematic Camera Control for Text-Driven Multi-Shot Video Creation

Songlin Yang, Zhe Wang, Xuyi Yang, Songchun Zhang, Xianghao Kong, Taiyi Wu, Xiaotong Zhao, Ran Zhang, Alan Zhao, Anyi Rao

arXiv:2603.11421v2cs.CV

TL;DR

Precise camera control remains difficult in cinematic multi-shot text-to-video, where textual prompts are imprecise and manual trajectory plotting is burdensome. ShotVerse separates trajectory planning from video control, and reports strong control fidelity and cinematic quality in its evaluation.

  • Problem

    Cinematic multi-shot video generation lacks precise camera control, while manual trajectory plotting is burdensome and existing models struggle to execute complex trajectories.

    Source: §1 · p. 2

  • Method

    ShotVerse uses a VLM-based Planner to map text to explicit camera trajectories and a Controller to render them into multi-shot videos.

    Source: §1 · p. 2

  • Results

    Across the three-track evaluation, existing methods struggle with complex multi-shot planning, while ShotVerse’s Controller has the lowest translation and rotation errors and its end-to-end videos achieve the highest Aesthetic Quality (5.465).

    Source: §5.2 · pp. 6–7

  • Takeaways & Limitations

    ShotVerse-Bench provides a high-fidelity training foundation and benchmark for evaluating cinematic planning, execution fidelity, and multi-shot consistency.

    Source: §1 · p. 2

    ShotVerse is currently limited to single-scene, multi-shot video, constraining duration and flexibility; longer multi-scene generation remains future work.

    Source: §5.4 · p. 8

Abstract

from arXiv · show

Text-driven video generation has democratized film creation, but camera control in cinematic multi-shot scenarios remains a significant block. Implicit textual prompts lack precision, while explicit trajectory conditioning imposes prohibitive manual overhead and often triggers execution failures in current models. To overcome this bottleneck, we propose a data-centric paradigm shift, positing that aligned (Caption, Trajectory, Video) triplets form an inherent joint distribution that can connect automated plotting and precise execution. Guided by this insight, we present ShotVerse, a ``Plan-then-Control'' framework that decouples generation into two collaborative agents: a VLM (Vision-Language Model)-based Planner that leverages spatial priors to obtain cinematic, globally aligned trajectories from text, and a Controller that renders these trajectories into multi-shot video content via a camera adapter. Central to our approach is the construction of a data foundation: we design an automated multi-shot camera calibration pipeline aligns disjoint single-shot trajectories into a unified global coordinate system. This facilitates the curation of ShotVerse-Bench, a high-fidelity cinematic dataset with a three-track evaluation protocol that serves as the bedrock for our framework. Extensive experiments demonstrate that ShotVerse effectively bridges the gap between unreliable textual control and labor-intensive manual plotting, achieving superior cinematic aesthetics and generating multi-shot videos that are both camera-accurate and cross-shot consistent.

1 Introduction

ShotVerse reframes cinematic multi-shot camera control as a data-centric Plan-then-Control task, pairing text-to-trajectory planning with trajectory-conditioned video generation. ShotVerse-Bench provides a calibrated training foundation and three-track evaluation of planning, execution fidelity, and multi-shot consistency.

1 Introduction

  • Textual camera prompts lack precision, multi-shot cameras may lack unified coordinates or cinematic patterns, and explicit trajectories impose laborious plotting and can cause generation failures.Manual plotting requires synchronizing camera poses in a global coordinate system with narrative flow.Source: §1 · p. 2
  • Aligned (Caption, Trajectory, Video) triplets motivate modeling P(Trajectory | Caption) for plotting and P(Video | Caption, Trajectory) for generation, enabling decoupled optimization.The approach uses VLM spatial priors and avoids joint-training instability while keeping trajectories compatible with the generator’s execution domain.Source: §1 · p. 2
  • ShotVerse pairs a VLM Planner that maps textual descriptions to cinematic trajectories with a Controller that renders them through a lightweight camera adapter in a holistic multi-shot backbone.The holistic backbone lets the method focus on camera control while generating multi-shot content.Source: §1 · p. 2
  • The introduction reports that ShotVerse enables superior cinematic plotting and generates multi-shot videos that are camera-accurate and cross-shot consistent.The benchmark’s three tracks assess cinematic planning, execution fidelity, and multi-shot consistency.Source: §1 · p. 2
  • ShotVerse-Bench combines hierarchical captions with an automated calibration pipeline that aligns disjoint single-shot trajectories in a unified global coordinate system.The benchmark is designed as both a high-fidelity training foundation and an evaluation benchmark.Source: §1 · p. 2
  • Unlike full-stack filmmaking systems [Wu et al. 2025b; Xiao et al. 2025] and generic trajectory generation or control [Zhang et al. 2025c; Bai et al. 2025a; He et al. 2024], ShotVerse targets explicit cinematic control for holistic multi-shot models.Its focus is unified 3D trajectories and professional camera patterns that support cross-shot coherence.Source: §1 · p. 2

2 Related Work

Prior work addresses multi-shot generation, trajectory data and plotting, and camera control through approaches that leave gaps in cross-shot consistency, cinematic spatial coherence, or cut orchestration. ShotVerse addresses these gaps with calibrated multi-shot data and VLM-based plotting for holistic multi-shot generation.

Multi-Shot Video Creation

  • Cascaded generation struggles with temporal consistency, memory-based synthesis risks error accumulation, and holistic approaches instead model entire sequences jointly [Dehghanian et al. 2025; Wang et al. 2025a; Xiao et al. 2025; Zhang et al. 2025a; Zheng et al. 2025; Zhou et al. 2024; He et al. 2025a; Zhang et al. 2025b; Cai et al. 2025b; Guo et al. 2025; Jia et al. 2025; Kara et al. 2025; Luo et al. 2026; Wang et al. 2025b; Xiao et al. 2025].ShotVerse-Bench supplies multi-shot sequences calibrated to a unified coordinate system for learning cross-shot spatial logic and cinematic consistency.Source: §2 · pp. 2–3

Camera Trajectory Datasets

  • Existing trajectory datasets cover basic static-environment paths, human-centric tracking, or cinematic moves, but remain single-shot and lack global spatial coherence [Yu et al. 2023; Zhou et al. 2018; Ling et al. 2024; Jiang et al. 2024; Courant et al. 2024; Zhang et al. 2025c].ShotVerse-Bench provides multi-shot sequences calibrated in one global coordinate system.Source: §2 · p. 3

Camera Trajectory Generation

  • Early camera-trajectory approaches relied on optimization or heuristic constraints [Bonatti et al. 2020], while recent generative methods often depend on explicit 3D priors, including character motion, object-centric data, or RGBD information.Source: §2 · p. 3

Camera Control for Text-Driven Video Generation

  • Camera-control methods encode extrinsics or use 3D priors but lack multi-shot storytelling; synchronized multi-camera work overlooks cinematic cuts, while ShotDirector uses fixed-point or template-constrained trajectories [Guo et al. 2023; He et al. 2024; Wang et al. 2024; Bahmani et al. 2024a,b; Zheng et al. 2024; Bai et al. 2025a,b; He et al. 2025b; Kuang et al. 2024; Wu et al. 2025a].ShotVerse unifies planning and control for holistic multi-shot video generation.Source: §2 · p. 3

3 Methodology: ShotVerse

ShotVerse separates text-driven multi-shot video creation into trajectory planning and video control, treating aligned (Caption, Trajectory, Video) triplets as a joint distribution. Its VLM-based Planner generates globally unified camera trajectories, and its Controller renders them into video.

3 Methodology: ShotVerse

  • The Plan-then-Control framework models P(Trajectory | Caption) for planning and P(Video | Caption, Trajectory) for rendering with a holistic Diffusion Transformer [Meng et al. 2025; Peebles and Xie 2023].This decomposition is intended to bridge unreliable textual camera control and labor-intensive manual plotting.Source: §3 · pp. 3–4

3.1 Planner: Shot-Aware Cinematic Trajectory Plotting

  • The Planner interleaves global and per-shot text with query slots, using VLM spatial priors to produce shot-aware camera codes that incorporate global context, prior shots, and current-shot instructions.The design addresses shallow spatial reasoning in text encoders and the inefficiency of directly predicting variable-length trajectories with a VLM.Source: §3.1 · p. 4
  • The Planner’s autoregressive decoder expands fixed-count shot codes into variable-length pose-token sequences, modeling temporal dependencies within and across shots for cinematic continuity.Reversible tokenization maps 12D poses to discrete bins and back to explicit trajectories in a unified global coordinate system.Source: §3.1 · p. 4

3.2 Controller: Cinematic Camera Control for Text-Driven Multi-Shot Video Generation

  • The Controller fine-tunes a holistic multi-shot video model with LoRA and injects projected camera-pose embeddings before self-attention in each transformer block to condition generated motion on trajectories.Training on real-world ShotVerse-Bench data is used to align camera control with natural cinematic textures and mitigate texture-distribution drift.Source: §3.2 · pp. 4–5
  • The Controller’s 4D Rotary Positional Embedding represents shot, frame, height, and width positions, assigning frames in each shot a shared shot embedding to enforce intra-shot consistency.Its attention dimensions allocate greater capacity to spatial dimensions while reserving capacity for hierarchical temporal modeling.Source: §3.2 · p. 5
  • The Controller is trained with Flow Matching to predict the velocity from interpolated clean-video latents and Gaussian noise, conditioned on text and explicit camera embeddings.The interpolation is v_σ = (1 − σ)v_0 + σv_1, with σ in [0, 1].Source: §3.2 · p. 5

4 Dataset and Benchmark: ShotVerse-Bench

ShotVerse-Bench supplies cinematic clips with hierarchical captions and globally unified multi-shot trajectories, supported by a calibration pipeline and a three-track evaluation protocol. The section also specifies planner, controller, and video-generation baselines for comparison.

4.1 Dataset Curation

  • ShotVerse-Bench contains 20,500 high-production cinema clips with hierarchical captions and globally unified multi-shot trajectories.Table 1 describes it as the first large-scale dataset providing multi-shot cinematic trajectories with rich, multi-level caption annotations.Source: §4.1 · p. 5, Table 1 · p. 6
  • The dataset spans 27+ trajectory categories, varied camera motions and compositions, explicit transitions, and multiple content domains.It includes 1,639 zoom/scale, 463 translation, 158 rotation/arc, 1,295 complex/reveal, 4,874 close-up, 3,543 mediumrange, and 512 wide/long-range samples; 1,720 clips have explicit cuts or transitions. Domains are TV series (10,864), movies (3,962), animation (3,441), and documentaries (2,233); mean and maximum durations are 5.50 and 19.98 seconds.Source: §4.1 · p. 5
  • A four-step calibration pipeline aligns shot-local trajectories globally, and the resulting 1,100 multi-shot scenes are split into 1,000 training and 100 test scenes.It masks dynamic foregrounds with SAM, reconstructs shots independently and jointly across keyframes with PI3, then uses shared anchor frames and a similarity transform to align local and global poses. The scenes contain 2, 3, or 4 shots in a 6:3:1 ratio; train and test have no scene overlap.Source: §4.1 · p. 5

4.2 Evaluation Benchmark: A Three-Track Protocol

  • The three-track protocol evaluates text-to-trajectory planning, trajectory-to-video execution fidelity, and end-to-end text-to-video quality.Track A uses F1-Score and CLaTr-CLIP for motion-tag and semantic alignment. Track B measures Transition Error, Rotation Error, and CAS, which compares DINOv2 similarity for cross-shot frames with high field-of-view overlap. Track C measures global and shot-level ViCLIP consistency, LAION Aesthetic Quality, Shot Transition Accuracy, FVD, and VLM/user-study judgments on four cinematic-planning dimensions.Source: §4.2 · pp. 5–6

4.3 Baseline Selection

  • Baseline comparisons cover trajectory planners, shot-by-shot camera-control models, and open- and closed-source multi-shot video models.Track A includes CCD [Jiang et al. 2024], E.T., Director3D [Li et al. 2024a], and GenDoP [Zhang et al. 2025c], the strongest autoregressive planner baseline. Track B includes CameraCtrl [He et al. 2024], MotionCtrl [Wang et al. 2024], and ReCamMaster [Bai et al. 2025a], adapted shot-by-shot and aligned with the calibration pipeline. Track C includes HoloCine [Meng et al. 2025], MultiShotMaster [Wang et al. 2025b], Sora2, VEO3 [Google Deepmind 2025], Kling3.0, and Seedance2.0 [Gao et al. 2025]; closed-source models receive hierarchical prompts for zero-shot evaluation.Source: §4.3 · p. 6

5 Experiments

ShotVerse’s experiments evaluate trajectory alignment, camera control, cinematic quality, and system design. The results support its Planner and Controller while identifying limitations in long-context persistence and dense scenes.

5.1 Implementation Details

  • The Planner uses a Qwen3-VL-2B backbone with an OPT-based decoder, while the Controller adapts HoloCine with rank-128 LoRA and a camera encoder.Training optimizes the camera encoder only in the high-noise stage (0.875 ≤𝜎≤1), then refines details through LoRA in the low-noise stage.Source: §5.1 · p. 6

5.2 Benchmark Results

  • The VLM-driven Planner achieves the best results on both benchmarks, while GenDoP’s competitive performance on its native benchmark weakens after retraining on ShotVerse-Bench.CCD, E.T., and Director3D underperform, especially on complex multi-shot narrative prompts; the results indicate stronger cross-domain generalization from the VLM backbone.Source: §5.2 · p. 6, Table 2 · p. 7
  • ShotVerse achieves the lowest FVD and highest Aesthetic Quality (5.465), and leads across all four cinematic-quality dimensions in VLM and user-study evaluations.Commercial models achieve competitive aesthetics but significantly higher FVD; ShotVerse shows particularly strong gains in Subject Emphasis & Saliency and Cinematic Pacing.Source: §5.2 · p. 7, Table 4 · p. 7, Table 5 · p. 7
  • Given ground-truth trajectories, the Controller achieves the lowest translation and rotation errors and highest CAS, while single-shot baselines lack cross-shot coordination.ReCamMaster reduces rotation error but retains coordinate misalignment across shots.Source: §5.2 · p. 7, Table 3 · p. 7
  • ShotVerse’s 4D RoPE raises Shot Transition Accuracy to 0.933, compared with 0.927 for MultiShotMaster and 0.645 for HoloCine.The ablation reports that replacing 4D RoPE with 3D RoPE drops Shot Transition Accuracy from 0.933 to 0.429, indicating the shot axis matters for respecting shot boundaries.Source: §5.2 · p. 7, Table 4 · p. 7, §5.3 · p. 7

5.3 Ablation Study

  • Planner ablations support the VLM spatial priors, dedicated shot-aware decoder, and learnable query tokens; Controller ablations support camera encoding, high-noise-only injection, global calibration, and real cinematic triplets.Removing query tokens degrades F1-Score and CLaTr-CLIP; adding a low-noise camera encoder slightly trades off perceptual quality, while removing global calibration reduces inter-shot consistency and aesthetics.Source: §5.3 · p. 7, Table 6 · p. 7, Table 7 · p. 7

5.4 Discussion on Generalizability

  • ShotVerse generalizes to unseen multi-shot scenes and underrepresented atmospheric content, but long shot-reverse-shot sequences and dense crowds expose persistence, trajectory, background-drift, and flickering limitations.The method is currently limited to single-scene multi-shot video; autoregressive intra-scene and inter-scene extensions are described as compatible directions, while longer multi-scene generation remains future work.Source: §5.4 · p. 8

6 Conclusions

ShotVerse uses a “Plan-then-Control” framework to connect narrative intent with geometric precision in multi-shot video generation. Its ShotVerse-Bench dataset and three-track evaluation support the paper’s claims of state-of-the-art technical accuracy and cinematic understanding.

6 Conclusions

  • The three-track evaluation reports state-of-the-art technical accuracy and implicit understanding of cinematic pacing and visual salience.The passage characterizes the evaluation as extensive and rigorous.Source: §6 · p. 8
  • ShotVerse decouples multi-shot storytelling into VLM-driven cognitive plotting and geometry-aware rendering through a “Plan-then-Control” framework.The framework is presented as a way to handle the spatial logic required for multi-shot storytelling.Source: §6 · p. 8
  • ShotVerse-Bench is built on a calibration pipeline that unifies disjoint shot trajectories in a global coordinate system, forming a high-fidelity dataset for cinematic video generation.The conclusion describes it as the first dataset of its kind for teaching AI the “grammar of film”.Source: §6 · p. 8

Related papers

Loading 2603.11421v2…