Source-linked AI summary
CausalCine: Real-Time Autoregressive Generation for Multi-Shot Video Narratives
Yihao Meng, Zichen Liu, Hao Ouyang, Qiuyu Wang, Ka Leong Cheng, Yue Yu, Hanlin Wang, Haobo Li, Jiapeng Zhu, Yanhong Zeng, Xing Zhu, Yujun Shen, Qifeng Chen, Huamin Qu
TL;DR
Existing autoregressive video models struggle to generate coherent multi-shot narratives with evolving events, viewpoint changes, and shot boundaries. CausalCine addresses this with causal multi-shot training, content-aware memory routing, and few-step distillation, outperforming autoregressive baselines while approaching bidirectional models’ quality and coherence.
Problem
Existing autoregressive video models are mainly designed for short-horizon single-scene continuation, limiting coherent long-form generation across evolving events, viewpoint changes, and shot boundaries.
Method
CausalCine trains causal multi-shot transitions, routes bounded KV memory by content relevance, and distills the model for interactive streaming generation with prompt updates.
Results
CausalCine improves prompt following, shot-cut control, and identity preservation over autoregressive baselines while achieving visual quality and cross-shot coherence competitive with bidirectional models.
Takeaways & Limitations
CausalCine supports interactive multi-shot video generation with shot-level prompt updates, long-range recall, and continued streaming without regenerating earlier video.
Takeaways & Limitations
Real-time deployment requires a 14B-parameter backbone and 8 NVIDIA H200 GPUs, placing it beyond consumer-grade GPU capability.
Abstract
from arXiv · showhide
Autoregressive video generation aims at real-time, open-ended synthesis. Yet, cinematic storytelling is not merely the endless extension of a single scene; it requires progressing through evolving events, viewpoint shifts, and discrete shot boundaries. Existing autoregressive models often struggle in this setting. Trained primarily for short-horizon continuation, they treat long sequences as extended single shots, inevitably suffering from motion stagnation and semantic drift during long rollouts. To bridge this gap, we introduce CausalCine, an interactive autoregressive framework that transforms multi-shot video generation into an online directing process. CausalCine generates causally across shot changes, accepts dynamic prompts on the fly, and reuses context without regenerating previous shots. To achieve this, we first train a causal base model on native multi-shot sequences to learn complex shot transitions prior to acceleration. We then propose Content-Aware Memory Routing (CAMR), which dynamically retrieves historical KV entries according to attention-based relevance scores rather than temporal proximity, preserving cross-shot coherence under bounded active memory. Finally, we distill the causal base model into a few-step generator for real-time interactive generation. Extensive experiments demonstrate that CausalCine significantly outperforms autoregressive baselines and approaches the capability of bidirectional models while unlocking the streaming interactivity of causal generation. Demo available at https://yihao-meng.github.io/CausalCine/
1 Introduction
CausalCine addresses the limitations of short-horizon autoregressive video models by enabling causal multi-shot generation with prompt updates and long-range context reuse. It first learns multi-shot causal behavior, routes relevant KV memory across shots, and distills the model for real-time interactive synthesis.
- Short-horizon causal video models often stagnate, loop, or drift semantically when rolled out beyond a single local motion pattern.Bidirectional diffusion models offer visual fidelity but make long, interactive generation expensive.
- Interactive multi-shot generation requires introducing new content at shot changes, following appended prompts, and selecting relevant long-range information without regenerating previous shots.CausalCine generates across shot changes, accepts new prompts during rollout, and reuses historical context.
- CausalCine first trains a full-step causal multi-shot base model on native long-form sequences with teacher forcing before accelerating generation.Training exposes shot boundaries, changing prompts, and long-range entity reappearance under the same causal dependency structure used with KV caching.
- Content-aware KV memory is needed because useful context may come from distant scenes rather than temporal neighbors when viewpoints, environments, or new content change.The model may recall a distant character, ignore the immediately preceding scene, or combine semantic information.
- The causal multi-shot base model is distilled with Distribution Matching Distillation into a few-step generator for real-time interactive synthesis.The resulting model generates chunk by chunk with KV caching, supports prompt updates, and preserves cross-shot consistency.
- CausalCine enables real-time online directing on a 14B-parameter video generator, allowing users to add events or viewpoints and continue without recomputing previous shots.The system streams video causally rather than rendering a complete video offline.
2 Related Works
Prior autoregressive video models enable long-horizon, low-latency streaming but primarily target single-scene continuation. Multi-shot methods provide explicit story control, yet cross-shot coherence and long-range context remain challenging.
- Autoregressive Video Generation: Autoregressive video generation factorizes videos into sequential frames or chunks, enabling long-horizon rollout, KV-cache reuse, and interactive continuation.Recent approaches often causalize pretrained diffusion models for chunk-by-chunk generation, including CausVid’s few-step causal distillation for low-latency streaming.
- Autoregressive Video Generation: Existing autoregressive methods primarily extend single-scene motion patterns rather than causally introducing new shots, prompts, and events while preserving long-range story context.This limitation motivates studying autoregressive generation in the multi-shot setting.
- Multi-Shot Video Generation: Multi-shot video methods decompose stories into scripts, shots, or keyframes and generate segments with short-video models, but cross-shot consistency requires separate linking or refinement.These approaches provide explicit control over story planning while treating coherence recovery as an additional stage.
- Memory Mechanisms: Video-generation memory mechanisms retain recent frames, fixed anchors, compressed history, or multi-scale memories, with newer methods retrieving history using scene or content cues.Examples include retrieval based on camera pose, field-of-view overlap, 3D scene structure, or content.
3 Method
CausalCine first learns causal multi-shot structure with a full-step generator, then adds content-aware memory routing and distills the result into a four-step interactive model. Its method uses chunk-wise autoregression, shot-indexed conditioning, long multi-shot supervision, and bounded semantic memory.
- Method overview: The framework converts a pretrained bidirectional diffusion model into a full-step causal multi-shot generator before distilling it into a four-step generator.Content-aware memory routing is shared by training and inference.
- Causal multi-shot generation: Long videos are factorized into contiguous autoregressive chunks, using L=3 latent frames, approximately 12 video frames, per chunk.Frame-wise autoregression is the special case L=1.
- Causal multi-shot generation: Shot-indexed prompts change at latent-frame boundaries, requiring each new chunk to reflect its new shot rather than extrapolate the previous shot.Both clean and noisy segments receive the same shot prompt through segment-level cross-attention, with cross-segment attention forbidden.
- Causal multi-shot generation: Parallel teacher forcing packs clean and noisy copies of all chunks into one 2N-segment input with a block-sparse causal attention mask.The mask reproduces inference-time KV-cache visibility while replacing sequential rollout with one parallel forward pass.
- Content-Aware Memory Routing: Content-Aware Memory Routing retrieves out-of-window history frames by query-key relevance, complementing a local window with bounded semantic memory.The implementation uses W=3 chunks and k=5 frames, with chunk-shared frame-level routing.
- Few-step distillation: The resulting causal model is distilled into a four-step autoregressive generator with DMD and adversarial regularization to reduce sequence-level drift.A lightweight GAN head is attached to intermediate features during the adversarial objective.
4 Experiments
Experiments evaluate CausalCine on a 100-prompt multi-shot benchmark spanning shot changes, viewpoint shifts, subject reappearance, and long temporal gaps. CausalCine outperforms autoregressive baselines, matches bidirectional models in quality and coherence while running faster, and benefits from causal multi-shot tuning and content-aware memory.
- Evaluation Protocol: The 100-prompt benchmark tests visual quality, prompt following, temporal consistency, long-range consistency, and shot structure across diverse multi-shot scenarios.Prompts include global story descriptions, five shot-level captions, and target shot-cut locations.
- Comparisons: CausalCine better follows shot progression than autoregressive baselines while preserving subjects across shots and maintaining coherent intra-shot motion.Compared methods include Self-Forcing, Infinity-RoPE, LongLive, MemFlow, and ShotStream.
- Comparisons: CausalCine achieves comparable visual quality and cross-shot coherence to bidirectional models while being substantially faster at inference.The comparison uses the bidirectional baselines’ 15-second setting and also highlights interactive continuation.
- Ablation on Long Multi-Shot Causal Tuning: Skipping long multi-shot causal tuning causes unstable intra-shot content and inconsistent cross-shot identity, whereas the full pipeline remains more stable.The full framework learns long-context causal multi-shot generation before ODE initialization and DMD distillation.
- Ablation on Memory Design: Content-aware memory routing better preserves character identity across long temporal gaps than no-memory and first-frame sink variants.The dedicated memory evaluation set contains 100 prompts emphasizing subject disappearance and reappearance across shots.
5 Conclusion
CausalCine is a causal framework for interactive multi-shot video generation that learns long-form shot transitions before distillation and routes KV memory by content. It enables shot-level prompt updates, bounded-attention rollout, and cross-shot recall without regenerating earlier video, while improving key capabilities over autoregressive baselines and remaining competitive in visual quality with a bidirectional multi-shot model.
- Conclusion: CausalCine learns long-form shot transitions before distillation and routes KV memory by content for interactive multi-shot video generation.This supports shot-level prompt updates, bounded-attention rollout, and cross-shot recall without regenerating earlier video.
- Conclusion: CausalCine improves prompt following, shot-cut control, and identity preservation over autoregressive baselines.Its visual quality is competitive with a bidirectional multi-shot model.
A More Results
The supplementary results provide diverse multi-shot video examples and a recorded real-time interactive demo. An HTML gallery organizes these materials for convenient browsing.
- Additional video results: Supplementary videos cover character reappearance, viewpoint changes, scene transitions, and long-range cross-shot consistency.These results extend beyond the still frames shown in the main paper.
- Interactive generation demo: The recorded demo shows chunk-by-chunk generation, newly appended shot-level prompts during rollout, and continuation without regenerating previous shots.It demonstrates CausalCine’s real-time interactive generation process.
- Supplementary browsing: An HTML gallery organizes all video results and the interactive demo in one place for direct viewing.Readers can browse the results without opening each video file individually.
B Causal Base Model vs. Four-Step Student
The four-step DMD student retains the causal base model’s coherent multi-shot structure, subject consistency, prompt following, and visual quality while substantially reducing denoising cost. It follows the same shot progression as the 50-step base despite using only four denoising steps.
- Causal base model: The causal base model produces coherent multi-shot rollouts with clear shot transitions, strong prompt following, and stable recurring subjects.It is trained with long multi-shot teacher forcing and serves as the high-quality autoregressive teacher for distillation.
- Four-step student: The four-step student remains close to the causal base on visual quality, text alignment, consistency, and shot-cut accuracy while reducing inference cost.DMD compresses the denoising trajectory while maintaining the multi-shot structure.
- Four-step student: The four-step student preserves the 50-step causal base’s shot-level structure, subject identity, visual composition, and shot progression.Figure S1 shows that it maintains the main subject across the sequence despite using only four denoising steps.
C Effect of Adversarial Regularization
Adversarial regularization during DMD distillation stabilizes long causal rollouts at the sequence level. Removing the GAN head preserves prompt following but causes drift in camera motion and subject framing.
- Adversarial regularization ablation: The GAN head stabilizes subject framing and camera motion in four-step student rollouts, whereas removing it causes sequence-level drift and irregular camera motion.Without adversarial regularization, recurring subjects may move away from the frame center despite continued adherence to the multi-shot prompt.
D Limitations and Failure Case
CausalCine incurs higher inference cost by using a 14B backbone, despite reaching real-time generation in distributed deployment. Its main failure mode is fine-grained physical-state continuity across shot cuts, where plausible shots may not form one consistent physical process.
- Limitations: CausalCine uses Wan2.1-T2V-14B instead of smaller 1.3B-scale backbones, increasing inference cost.The larger backbone supports multi-shot fidelity but raises computational demands.
- Limitations: 16 FPS on 8 NVIDIA H200 GPUs enables real-time generation through distributed deployment.This real-time rate is achieved despite the model’s larger backbone.
- Failure case: CausalCine can produce individually plausible shots that fail to preserve fine-grained physical-state continuity across cuts.The model lacks explicit structured state for small objects, contact geometry, and ongoing physical interactions.
- Failure case: In coffee-making shots, the milk stream, pitcher pose, hand position, and latte-art pattern do not evolve as one physically consistent action.The shots remain visually plausible while becoming inconsistent across cuts.