Source-linked AI summary
HoloCine: Holistic Generation of Cinematic Multi-Shot Long Video Narratives
Yihao Meng, Hao Ouyang, Yue Yu, Qiuyu Wang, Wen Wang, Ka Leong Cheng, Hanlin Wang, Yixuan Li, Cheng Chen, Yanhong Zeng, Yujun Shen, Huamin Qu
TL;DR
Text-to-video models generate isolated clips well but struggle with coherent multi-shot narratives. HoloCine addresses this through holistic scene generation with localized shot prompting and sparse inter-shot attention, achieving reported state-of-the-art consistency and shot control while supporting minute-scale generation. The paper identifies causal reasoning as a remaining challenge.
Problem
Existing text-to-video systems excel at isolated clips but lack coherent multi-shot narrative generation, while holistic approaches face control and computational challenges.
Method
HoloCine jointly generates entire multi-shot scenes using Window Cross-Attention for per-shot control and Sparse Inter-Shot Self-Attention for efficient long-range communication.
Results
HoloCine establishes reported state-of-the-art performance in long-term consistency, narrative fidelity, and shot-transition control, with persistent character and scene memory and nuanced cinematic-language control.
Takeaways & Limitations
Minute-scale holistic generation shifts the focus from isolated clip synthesis toward directing entire cinematic scenes.
Takeaways & Limitations
HoloCine can fail at causal reasoning, such as rendering an empty glass after water is poured into it because it prioritizes visual consistency over the physical consequence.
Abstract
from arXiv · showhide
State-of-the-art text-to-video models excel at generating isolated clips but fall short of creating the coherent, multi-shot narratives, which are the essence of storytelling. We bridge this "narrative gap" with HoloCine, a model that generates entire scenes holistically to ensure global consistency from the first shot to the last. Our architecture achieves precise directorial control through a Window Cross-Attention mechanism that localizes text prompts to specific shots, while a Sparse Inter-Shot Self-Attention pattern (dense within shots but sparse between them) ensures the efficiency required for minute-scale generation. Beyond setting a new state-of-the-art in narrative coherence, HoloCine develops remarkable emergent abilities: a persistent memory for characters and scenes, and an intuitive grasp of cinematic techniques. Our work marks a pivotal shift from clip synthesis towards automated filmmaking, making end-to-end cinematic creation a tangible future. Our code is available at: https://holo-cine.github.io/.
1. Introduction
HoloCine addresses the narrative gap between high-quality single-shot video generation and coherent multi-shot storytelling. It combines localized shot control, efficient holistic attention, and joint scene generation to support minute-scale cinematic narratives.
- Existing multi-shot methods often generate chunks, keyframes, or shots through decoupled processes, limiting long-range coherence.
- Holistic generation improves global consistency but makes per-shot instruction control difficult and full self-attention prohibitively expensive for minute-scale videos.
- Window Cross-Attention aligns each per-shot text prompt with its corresponding video segment for precise directorial control and sharp narrative transitions.
- Sparse Inter-Shot Self-Attention uses dense within-shot attention and sparse between-shot communication, reducing complexity toward linear scaling with shot count.This makes minute-scale holistic generation feasible while preserving long-range consistency.
- HoloCine significantly outperforms strong baselines across pre-trained, keyframe-to-video, and holistic paradigms, establishing state-of-the-art long-term consistency, narrative fidelity, and shot-transition control.The authors also report persistent character and scene memory and nuanced control over cinematic language.
2. Related Work
Prior work progresses from single-shot diffusion models toward multi-shot pipelines, holistic scene generation, and efficient long-video Transformers. However, independent shot synthesis risks consistency drift, while long sequences remain constrained by attention cost.
- Single-shot T2V models produce high-fidelity clips but lack explicit mechanisms for coherent narratives across distinct shots.
- 2.2. Multi-Shot and Scene-Level Video Generation: Hierarchical and keyframe-based pipelines improve organization or keyframe consistency, but video shots are still generated independently.
- 2.2. Multi-Shot and Scene-Level Video Generation: Holistic generation jointly models all shots to enforce global consistency, offering a more promising alternative to decoupled pipelines.
- 2.3. Long Video Generation: Autoregressive long-video generation is computationally manageable but prone to consistency drift and error accumulation as sequences lengthen.
- 2.3. Long Video Generation: Efficient Transformer approaches reduce attention cost through localized windows, token compression, or structured sparse masks for longer videos.
3. Method
HoloCine jointly processes all shots in a holistic diffusion pipeline, using hierarchical data and localized cross-attention for control. Sparse inter-shot attention preserves within-shot motion modeling while making long multi-shot generation computationally feasible.
- HoloCine targets coherent multi-shot generation from hierarchical text prompts in one holistic pass, built on the Wan2.2 DiT-based video diffusion model.
- 3.1. Data Curation and Annotation: The dataset pipeline segments, filters, and groups contiguous cinematic shots into 400k multi-shot samples across 5-, 15-, and 60-second duration tiers.
- 3.2. Holistic Multi-Shot Generation: Jointly processing all shot latents helps preserve character identity, background, and style across shot boundaries.
- 3.2.1. Window Cross-Attention: Window Cross-Attention restricts each video segment to global text and its corresponding per-shot prompt, enabling temporally aligned shot transitions.
- 3.2.2. Sparse Inter-Shot Self-Attention: Sparse Inter-Shot Self-Attention applies dense bidirectional attention within shots and compact summary-token communication between shots.For N_s shots of length L_shot summarized by S tokens, complexity falls from O((N_sL_shot)^2) to approximately O(N_s × (L_shot^2 + L_shot · N_s · S)).
4. Experiments
HoloCine is evaluated against pre-trained, pipeline-based, and holistic baselines on multi-shot video generation, including prompt adherence, consistency, and shot-transition control. Qualitative comparisons, ablations, and emergent-capability tests examine narrative control, architectural components, memory, and limitations.
- Comparison: HoloCine parses hierarchical prompts into coherent distinct shots with accurate per-shot descriptions and consistent characters and style, outperforming tested baseline paradigms qualitatively.The evaluation includes a 100-prompt benchmark spanning genres and narrative structures, with metrics covering quality, semantic consistency, intra-shot consistency, inter-shot consistency, and transition control.
- Ablation Studies: Removing Window Cross-Attention severely degrades shot control, while dense full self-attention preserves quality but is computationally prohibitive for long sequences.The ablation reports lower Shot Cut Accuracy and per-shot semantic consistency without windowed attention, whereas sparse attention retains most generative quality with a more practical trade-off.
- Comparison: HoloCine executes complex shot transitions that Vidu and Kling 2.5 Turbo miss, while matching Sora 2 qualitatively on directed medium-shot-to-close-up sequences.The commercial-model comparison reports that Vidu and Kling 2.5 Turbo produce single continuous clips instead of following multi-shot instructions.
- Ablation Studies: Removing inter-shot summary-token communication causes catastrophic character inconsistency across shots, demonstrating its role in maintaining narrative continuity.The isolated-within-shot variant changes the old man’s identity and appearance drastically between shots.
- Emergent Memory Capability: HoloCine exhibits emergent memory by preserving character identity across viewpoints, recalling subjects after distractor shots, and retaining a non-salient magnet’s position.These capabilities are evaluated as character permanence, long-range consistency, and fine-grained detail persistence.
- Limitations: The model’s visual consistency has a causal-reasoning limitation: after water is poured into an empty glass, it can regenerate the glass as empty.The reported failure prioritizes consistency with the initial shot over the physical consequence of the intervening action.
5. Conclusion
HoloCine generates complete multi-shot scenes holistically, combining narrative coherence with precise shot control and minute-scale efficiency. It also exhibits persistent memory and cinematographic-language understanding, while causal reasoning remains a challenge.
- HoloCine combines holistic scene synthesis, precise shot control, and efficient minute-scale generation to support coherent visual narratives.Its Window Cross-Attention enables shot control, while Sparse Inter-Shot Self-Attention addresses computational cost.
- A causal-reasoning failure shows the model can prioritize visual consistency over an action’s logical consequence.After water is poured into an empty glass, a later shot incorrectly restores the glass to its empty state.
- The model develops persistent memory for characters and scenes and demonstrates nuanced understanding of cinematic language.
A. Details on Evaluation metrics
The evaluation measures generated videos across five dimensions covering visual quality, semantic and temporal consistency, and shot-transition control.
- The evaluation covers aesthetic quality, semantic consistency, intra-shot consistency, inter-shot consistency, and transition control.Intra-shot consistency measures subject and background stability.
A.1. Transition Control Evaluation Metrics
Shot Cut Accuracy evaluates whether generated videos reproduce instructed cuts with both the correct count and precise temporal placement. Its normalized discrepancy formulation penalizes timing errors and missed or extra cuts.
- Shot Cut Accuracy measures both the number of generated cuts and the temporal precision of their placement against ground-truth cuts.Generated cut locations are obtained with TransNet V2 before comparison.
- SCA uses matched frame-wise deviation and penalties for missed or extraneous cuts, producing scores in (0, 1] where 1 is a perfect match.The exponential formulation heavily penalizes large temporal deviations.
A.2. Aesthetic Quality
Aesthetic quality is assessed with the LAION aesthetic predictor, which estimates human-perceived visual and artistic qualities in generated frames.
- The LAION aesthetic predictor evaluates composition, color harmony, realism, naturalness, and overall artistic appeal.
A.3. Semantic Consistency.
The evaluation measures semantic consistency globally, at the shot level, within shots, and across shots grouped by character identity.
- Global consistency compares ViCLIP representations of the full prompt and video using cosine similarity.
- Shot-level consistency compares each generated shot with its corresponding shot-level prompt using ViCLIP cosine similarity.
- Intra-shot consistency evaluates subject and background stability across frames after detecting shot boundaries with TransNet V2.
- Cross-shot consistency groups shots by character identity before computing ViCLIP similarity, avoiding biased comparisons between different characters or scenes.