PaperlayerView exact revision on arXiv

Source-linked AI summary

ShotPlan: Cinematic Video Generation with Learnable Planning Token

Su Guo, Guangce Liu, Haosen Yang, Jiepeng Wang, Cong Liu, Junqi Liu, Haibin Huang, Hongxun Yao, Chi Zhang, Xuelong Li

arXiv:2607.17675v1cs.CV

TL;DR

Cinematic video generation still lacks effective explicit shot planning for coherent multi-shot narratives. ShotPlan introduces learnable planning tokens with fractional temporal position encoding, and improves transition accuracy, inter-shot consistency, and narrative coherence over existing methods.

  • Problem

    Existing video generation models produce realistic single-shot videos but lack effective explicit shot planning for coherent multi-shot narrative production.

  • Method

    ShotPlan integrates learnable shot-transition planning tokens with video tokens and uses Fractional Temporal RoPE for frame-level transition control.

  • Results

    ShotPlan improves transition accuracy, inter-shot consistency, and narrative coherence, achieving a transition deviation of 0.64 versus 1.12 for MultiShotMaster and 2.71 for HoloCine.

  • Takeaways & Limitations

    ShotPlan offers more precise temporal control while preserving cross-shot information flow and scene structure within the pretrained video-generation architecture.

  • Takeaways & Limitations

    Existing keyframe-based and architectural shot-planning methods remain limited by independent shot generation, sparse keyframes, or restricted cross-shot interaction.

Abstract

from arXiv · show

Current video generation models achieve impressive results in single-shot generation, yet remain limited in cinematic video generation, where coherent narratives and effective multi-shot composition require explicit shot planning. To address this challenge, we propose ShotPlan, a framework for explicit multi-shot cinematic video generation built upon a video diffusion foundation model. Our method introduces learnable planning tokens that capture shot-level transition cues and can be seamlessly integrated with the original video generation tokens to control transition timestamps. Unlike standard video generation tokens, the proposed planning tokens are equipped with Fractional Temporal Rotary Position Embedding (FRoPE), enabling shot transitions to be modeled at the frame level. Experiments demonstrate that ShotPlan significantly outperforms existing cinematic video generation methods, offering more flexible shot management and stronger inter-shot consistency.

1. Introduction

ShotPlan addresses the limitations of single-shot video generation and prior multi-shot methods by explicitly planning shot transitions within one coherent generation process. It uses learnable planning tokens with FRoPE to provide controllable, frame-level transition timing while preserving the pretrained video generation architecture.

  • Motivation: Recent video diffusion models produce visually realistic single-shot videos but remain insufficient for practical video production settings requiring multi-shot cinematic composition.The introduction identifies strong single-shot quality alongside limitations for practical production.
  • Limitations: Keyframe-based approaches generate shots independently, preventing coherent inter-shot interaction during generation.These methods first generate consistent keyframes, then condition each shot on its corresponding keyframe.
  • Limitations: Direct cinematic video methods modify diffusion architectures with structured attention masks or RoPE-based positional discontinuities, but can restrict cross-shot information interaction.These mechanisms improve shot separation while introducing constraints on inter-shot communication.
  • ShotPlan: ShotPlan explicitly represents shot transitions with learnable planning tokens that act as temporal anchors and encode shot-level cues for cross-shot generation.Each planning token is associated with a transition event.
  • ShotPlan: Integrating planning tokens with original video tokens enables controllable transition timing while preserving the underlying video generation architecture.The method addresses the challenge of aligning transitions precisely in temporally compressed latent spaces.
  • Contributions: FRoPE enables frame-level shot-transition control, while ShotPlan formulates multi-shot generation as explicit planning within a single coherent process.The framework studies planning-token design, injection strategy, and FRoPE without replacing the pretrained video generation architecture.

2. Related Work

Related work spans text-to-video models, keyframe- and unified-process multi-shot generation, and temporal positional-encoding approaches. ShotPlan instead uses learnable planning tokens with Fractional Temporal RoPE to control transitions while preserving cross-shot interaction and the pretrained video-token positional prior.

  • Text-to-Video Generation: Text-to-video models have advanced through temporal modules, motion layers, spatio-temporal diffusion, and Transformer-based designs, but cinematic multi-shot structure remains challenging.Cinematic production requires explicit shot boundaries, varying viewpoints, and coherent cross-shot semantics.
  • Multi-Shot Video Generation: Keyframe- or visual-plan-based pipelines synthesize shots largely independently, relying on sparse keyframe quality and coverage that limit inter-shot interaction.These methods decompose generation into keyframe or visual-plan generation followed by image-to-video synthesis.
  • Multi-Shot Video Generation: Unified denoising methods improve shot separation using transition tokens, local attention masks, cinematic-transition masks, or structured attention for multi-shot generation.ShotAdapter controls shot number and duration with learnable transition tokens and local attention masks.
  • Multi-Shot Video Generation: Shot-level temporal positional encodings distinguish shots or add phase offsets, but changing video-token positions may disturb pretrained temporal priors, especially under VAE temporal compression.RoPE is widely used in Transformer-based video generation, with later work studying video-oriented RoPE designs.
  • Multi-Shot Video Generation: ShotPlan inserts learnable planning tokens into the video sequence, uses standard self-attention for cross-shot interaction, and applies Fractional Temporal RoPE while keeping video-token RoPE unchanged.This design enables frame-level transition control without disrupting the pretrained positional prior.

3. Method

ShotPlan enables one-pass multi-shot video generation by adding learnable planning tokens to a DiT-based video diffusion process. These tokens encode user-specified transition timestamps with FRoPE while preserving the original visual-token positional encoding, and training uses curated multi-shot clips with hierarchical annotations.

  • Video Diffusion Backbone: The video diffusion backbone encodes videos with a pretrained 3D VAE, patchifies the latent representation into visual tokens, and denoises them with a DiT.The model follows a rectified-flow formulation for video diffusion.
  • Learnable Planning Tokens: ShotPlan concatenates learnable planning tokens with patchified visual tokens and jointly processes them through all DiT blocks as temporally localized transition cues.The planning tokens interact with visual tokens through self-attention during denoising and are discarded afterward.
  • Learnable Planning Tokens: Hard cuts use replicated hard-cut tokens at user-specified cut points, while gradual transitions use learned start and end tokens.Each hard-cut token copy receives the temporal coordinate of its corresponding cut point.
  • Fractional Temporal RoPE: FRoPE assigns planning tokens continuous temporal coordinates for precise frame-level transition control without changing the visual tokens’ original temporal RoPE.User timestamps are mapped from original frame space to fractional latent coordinates, while planning tokens use fixed spatial coordinates.
  • Multi-Shot Data Curation: Training data is built from VideoEvent by detecting shot boundaries, extracting 5-second windows containing at least two shots, and filtering candidates for coherent multi-shot structure.Selected samples receive global descriptions and per-shot captions covering scene attributes, actions, viewpoints, and motion patterns.

4. Experiment

Experiments show that ShotPlan improves transition precision and inter-shot consistency on a multi-shot benchmark, while ablations support learnable tokens, in-context injection, and Fractional RoPE. The framework also extends to temporally localized camera movement control.

  • Evaluation: The evaluation uses 100 Gemini-generated multi-shot prompts covering cross-shot character consistency, scene consistency, and long-take sequences.The prompt set contains 60 character-consistency, 30 scene-consistency, and 10 long-take prompts, with specified shot pairs for targeted evaluation.
  • Quantitative Comparison: On the multi-shot benchmark, ShotPlan achieves 0.64 transition deviation versus 1.12 for MultiShotMaster and 2.71 for HoloCine.The result demonstrates more precise shot-boundary control.
  • Quantitative Comparison: ShotPlan improves character consistency from 0.39 to 0.46 and scene consistency from 0.32 to 0.37 over the strongest baseline.It performs best on the reported inter-shot consistency metrics.
  • Ablation Study: In-context token modulation better preserves pretrained DiT structure and enables more precise temporal control than alternative injection mechanisms.Latent Addition is limited to coarse latent-level positions, while Global AdaLN Modulation is also evaluated as an alternative.
  • Ablation Study: Fractional RoPE is essential for frame-level temporal alignment because removing fractional coordinates degrades transition control from physical-frame to coarser latent-level granularity.Planning-token attention concentrates around assigned cut timestamps and spreads smoothly to neighboring frames, supporting soft temporal modulation.
  • Temporally Localized Camera Movement: ShotPlan extends its token mechanism to temporally localized camera movement, where control tokens propagate motion effects from specified starting timestamps across subsequent frames.A user study compares ShotPlan with Wan 2.1, Kling 2.6, and SeedDance 1.5 Pro on motion type and timing accuracy.

5. Conclusion

ShotPlan is a lightweight framework for explicit multi-shot cinematic video generation built on a pretrained video diffusion model. Its learnable planning tokens with Fractional Temporal RoPE provide frame-level shot-transition control without modifying the backbone’s original attention structure or positional encoding.

  • Conclusion: ShotPlan enables explicit multi-shot cinematic video generation using a pretrained video diffusion model.The framework is described as lightweight.
  • Conclusion: Learnable planning tokens equipped with Fractional Temporal RoPE provide frame-level control of shot transitions.The planning tokens function as temporal anchors and in-context conditioning signals.
  • Conclusion: ShotPlan preserves the video backbone’s original attention structure and positional encoding while adding planning-based transition control.This enables the framework to control shot transitions without modifying those components.

Supplementary Material

The supplementary material provides additional ShotPlan details and results, covering a multi-shot video generation gallery, camera-motion dataset construction, and a planning-token noise ablation.

  • Generation Gallery: Section A presents a generation gallery of multi-shot cinematic videos.The gallery provides additional qualitative results for ShotPlan.
  • Camera-Motion Dataset: Section B details construction of the camera-motion dataset for temporally localized camera movement experiments.These experiments are referenced as Section 4.6 of the main paper.
  • Planning-Token Ablation: Section C provides an ablation testing whether planning tokens should be noised and included in the training loss.The ablation examines the training treatment of planning tokens.

A. Generation Gallery

ShotPlan generates diverse multi-shot cinematic videos with coherent shot transitions, maintaining subject identity and scene layout across shots.

  • A. Generation Gallery: ShotPlan’s generation gallery shows diverse scenes, characters, and shot transitions while preserving subject identity and scene layout across shots.Figures 6 and 7 present these examples.

B. Camera-Motion Data Construction

ShotPlan constructs training videos with camera motion beginning at a known intermediate timestamp while preserving the preceding segment, then filters concatenated results for temporal seams and scene inconsistency.

  • Data requirement: Training data are designed for temporally localized camera-motion control, with prescribed motion starting at a known intermediate timestamp after a motion-free preceding segment.Natural videos rarely provide these annotations, so the data are synthesized.
  • Synthesis procedure: A random truncation point separates the unchanged source prefix from an I2V-generated remainder conditioned on the truncation frame.The I2V model is Wan2.2 fine-tuned with camera-motion LoRA adapters.
  • Synthesis procedure: The generated remainder is assigned one of six camera movements: circle-left, circle-right, move-left, move-right, zoom-in, or zoom-out.The preceding segment remains unchanged while the remainder is regenerated to exhibit the selected movement.
  • Quality filtering: Naive concatenation can create a visible temporal seam at the boundary and degrade scene consistency within the generated segment.The seam occurs between the last original frame and first generated frame, while single-frame I2V conditioning may not preserve the original scene content.
  • Quality filtering: All concatenated videos are manually screened, and samples exhibiting either temporal discontinuity or scene inconsistency are discarded.Filtering addresses both failure modes introduced by naive concatenation.

C. Ablation on Noising the Planning Tokens

ShotPlan keeps planning tokens clean during training rather than perturbing them with diffusion noise or including them in the training loss. Compared with a noised variant, this design converges faster and is adopted for its simplicity.

  • Clean-token design: The final design keeps planning tokens clean throughout training: they are neither perturbed by diffusion noise nor included in the training loss.This creates a sequence with clean planning tokens alongside noised video tokens, while remaining compatible with the flow-matching objective because the training loss is applied only to t.
  • Noised variant: The noised variant perturbs planning tokens with video tokens and regresses toward detached values from the previous iteration.It uses a 1,000-step learning-rate warmup.
  • Convergence comparison: 11,000 training steps are required for the noised variant to reach the clean-token design’s performance within 3,500 steps.Under this configuration, the noised variant converges substantially more slowly.
  • Design choice: The clean-token design is adopted because it offers a simpler training configuration.The choice follows the slower convergence observed for the noised variant.
Loading 2607.17675v1…