Source-linked AI summary
ShotStream: Streaming Multi-Shot Video Generation for Interactive Storytelling
Yawen Luo, Xiaoyu Shi, Junhao Zhuang, Yutian Chen, Quande Liu, Xintao Wang, Pengfei Wan, Tianfan Xue
TL;DR
Long-form multi-shot video generation needs both narrative coherence and lower latency for interactive use. ShotStream reformulates generation as streaming next-shot prediction, distills a bidirectional teacher into a causal model, and adds dual-cache memory with two-stage self-forcing. It generates coherent videos at 16 FPS on one NVIDIA H200 GPU, with evaluations reporting state-of-the-art metrics and decisive user preference.
Problem
Existing multi-shot video generation faces limited interactivity and high latency despite the need for coherent long-form narratives.
Method
ShotStream reformulates synthesis as next-shot generation, distills a bidirectional teacher into a causal model, and uses dual-cache memory with two-stage progressive self-forcing.
Results
16 FPS on a single NVIDIA H200 GPU accompanies coherent long multi-shot videos, state-of-the-art consistency, prompt adherence, and shot-transition control, plus decisive user preference.
Takeaways & Limitations
Streaming prompts let users dynamically guide ongoing narratives while ShotStream supports lower-latency interactive multi-shot generation.
Takeaways & Limitations
Complex scenes and prompts can produce visual artifacts and inconsistencies, and the method still has room for acceleration.
Abstract
from arXiv · showhide
Multi-shot video generation is crucial for long narrative storytelling, yet current bidirectional architectures suffer from limited interactivity and high latency. We propose ShotStream, a novel causal multi-shot architecture that enables interactive storytelling and efficient on-the-fly frame generation. By reformulating the task as next-shot generation conditioned on historical context, ShotStream allows users to dynamically instruct ongoing narratives via streaming prompts. We achieve this by first fine-tuning a text-to-video model into a bidirectional next-shot generator, which is then distilled into a causal student via Distribution Matching Distillation. To overcome the challenges of inter-shot consistency and error accumulation inherent in autoregressive generation, we introduce two key innovations. First, a dual-cache memory mechanism preserves visual coherence: a global context cache retains conditional frames for inter-shot consistency, while a local context cache holds generated frames within the current shot for intra-shot consistency. And a RoPE discontinuity indicator is employed to explicitly distinguish the two caches to eliminate ambiguity. Second, to mitigate error accumulation, we propose a two-stage distillation strategy. This begins with intra-shot self-forcing conditioned on ground-truth historical shots and progressively extends to inter-shot self-forcing using self-generated histories, effectively bridging the train-test gap. Extensive experiments demonstrate that ShotStream generates coherent multi-shot videos with sub-second latency, achieving 16 FPS on a single GPU. It matches or exceeds the quality of slower bidirectional models, paving the way for real-time interactive storytelling. Training and inference code, as well as the models, are available on our
1. Introduction
ShotStream targets coherent, interactive long-form multi-shot video generation by reformulating synthesis as streaming next-shot generation. Its causal design combines dual-cache consistency mechanisms with progressive self-forcing and achieves efficient, user-preferred generation.
- Current multi-shot methods must preserve subject and scene consistency while advancing narratives through varied sequential shots.
- 25 minutes for 240 frames illustrates the prohibitive latency of existing multi-shot generation.HoloCine is cited as the example.
- ShotStream generates subsequent shots from previous shots and accepts streaming prompts that dynamically alter content, visual styles, or characters.
- A dual-cache mechanism preserves inter-shot consistency with global context and intra-shot consistency with locally generated frames.
- Two-stage self-forcing progresses from ground-truth historical shots to self-generated histories, bridging training and inference for long-horizon generation.
- 16 FPS on a single NVIDIA H200 GPU accompanies state-of-the-art consistency, prompt adherence, and shot-transition control, with decisive user preference across 54 participants.
2. Related Work
Related work spans keyframe-based and autoregressive approaches to long video generation. ShotStream builds on causal diffusion distillation and next-shot conditioning to address long-form, low-latency synthesis.
- Keyframe-based methods extend initial shot frames with image-to-video models but often struggle with global coherence.
- Autoregressive video models support gradual rollout for long video generation, while diffusion integration targets causal high-quality synthesis.
- CausVid distills multi-step diffusion into a 4-step causal generator for low-latency streaming.
- Self Forcing and Rolling Forcing address exposure bias and error accumulation using self-generated outputs or progressive noise levels.
3. Preliminary
Distribution Matching Distillation accelerates diffusion generation by matching teacher and student output distributions. Self forcing addresses autoregressive error accumulation caused by the training-inference discrepancy.
- Distribution Matching Distillation converts slow multi-step diffusion models into fast few-step student generators while maintaining high quality.
- DMD matches smoothed data and generator distributions by minimizing reverse KL divergence across random timesteps.
- Autoregressive error accumulation arises because training uses ground-truth data while inference relies on imperfect predictions.
- Self forcing bridges this gap by unrolling autoregressive training with previously generated outputs instead of ground-truth frames.
4. Method
ShotStream fine-tunes a bidirectional next-shot teacher and distills it into a causal generator using sparse historical context, dual caches, and progressive self-forcing. The method separates historical and current-shot information while training the causal rollout to resemble inference.
- Architecture: The method first fine-tunes a text-to-video model into a bidirectional next-shot teacher, then distills it into an efficient 4-step causal model.
- Historical context: Historical shots are represented by dynamically sampled sparse context frames under a maximum conditional budget, with remaining capacity allocated to the most recent shot.The experimental context budget is 6 frames.
- Historical context: Specific captions for conditional frames bind past visual information to textual descriptions through global and local cross-attention.
- Teacher model: Condition latents and noisy target latents are patchified and concatenated along the frame dimension before entering the DiT blocks.The condition tokens remain clean while noise is added only to target tokens.
- Causal architecture: The dual-cache mechanism stores sparse historical frames globally for inter-shot consistency and recent generated frames locally for intra-shot consistency.
- Distillation: Two-stage distillation first uses ground-truth histories for intra-shot causal rollout, then self-generated histories for inter-shot rollout.
5. Experiments
Experiments evaluate ShotStream through quantitative benchmarks, qualitative comparisons, user preferences, and ablations. The results support strong multi-shot consistency, prompt adherence, transitions, and the proposed design choices.
- Quantitative Results: ShotStream achieves the highest visual consistency while maintaining precise shot-transition control and strong prompt alignment across major evaluation metrics.The reported evaluation also finds higher overall aesthetic quality than competing methods.
- Qualitative Results: Qualitative comparisons show that ShotStream adheres to prompts, preserves visual coherence, and produces natural transitions between shots.The comparison presents the initial frames of each shot generated by all methods.
- User Study: 54 participants consistently preferred ShotStream in a user study comparing 24 multi-shot videos against competing baselines.Participants evaluated visual consistency, prompt following, and visual quality, with multiple selections allowed.
- Teacher Model Ablations: Dynamic context-frame sampling outperforms naive historical-frame sampling, while corresponding historical-shot prompts benefit in-context learning.The strategy allocates sparse context frames according to the number of historical shots and the conditional context budget.
- Causal Student Ablations: Inter-shot self-forcing significantly improves long-term visual style and color consistency compared with using only stage-one distillation.The ablation also reports that both distillation stages are indispensable, with stage one establishing next-shot capability and stage two simulating inference.
- Causal Student Ablations: Explicit cache distinction and the training-free RoPE offset outperform alternative cache-disambiguation designs in the causal student ablation.The comparison contrasts explicit distinction and a learnable embedding against the proposed RoPE offset.
6. Conclusion
ShotStream combines streaming next-shot generation, causal distillation, and dual-cache memory to support interactive long-form storytelling. It achieves 16 FPS on one GPU, while complex scenes can still produce artifacts and inconsistencies.
- Contributions: ShotStream reformulates multi-shot generation for streaming prompts and distills a bidirectional next-shot teacher into a causal architecture.Its dual-cache memory mechanism targets visual consistency across and within shots.
- Efficiency and Interactivity: 16 FPS on a single GPU enables interactive long narrative storytelling with substantially reduced latency relative to existing bidirectional multi-shot models.The paper describes runtime adaptation of upcoming shots based on previously generated content.
- Limitations: Highly complex scenes and text prompts can cause visual artifacts and inconsistencies because the current backbone has limited capacity.The authors expect scaling the base model to improve performance and stability in challenging scenarios.
- Limitations: The method remains open to further acceleration, with sparse attention and attention sink identified as possible future extensions.The stated goal is to provide better interactive experiences.
Supplementary Material
The supplementary file provides the text-to-video base model, the Distribution Matching Distillation objective, and training details.
- Supplementary materials include the text-to-video base model, the Distribution Matching Distillation objective function, and training details.
7. Text-to-Video Base Model
ShotStream builds on a transformer-based latent diffusion architecture with a 3D VAE, temporal transformer blocks, and text conditioning through cross-attention. Rectified Flow defines the noising process and trains the velocity network with an MSE objective.
- ShotStream builds on the transformer-based latent diffusion architecture Wan2.1-T2V-1.3B.
- A 3D VAE maps video features into latent representations, while transformer blocks model temporal dynamics.Each transformer block combines 3D spatial-temporal attention, cross-attention, and a feed-forward network.
- The umT5 text encoder injects text conditioning through cross-attention layers.
- Rectified Flow constructs x_t = (1 − t)x + tz from data x and Gaussian noise z ∈ N(0, I).
- The training objective is an MSE loss between the parameterized velocity v_θ and the target velocity z − x.
8. Distribution Matching Distillation Objective
Distribution Matching Distillation trains a student generator by matching its output distribution to the smoothed data distribution. Its gradient uses the difference between score functions for the data and student-generated distributions.
- DMD minimizes the reverse KL divergence between the smoothed data distribution and the student generator’s output distribution.
- The optimization gradient is approximated by subtracting the student-generated distribution’s score function from the data distribution’s score function.The score functions are trained using a denoising loss.
- The student generator G_ϕ is evaluated through the forward diffusion process Ψ with random Gaussian noise ϵ.
9. Training Details
ShotStream is trained on an internally curated dataset of 320K multi-shot videos with hierarchical narrative and shot-level annotations. Teacher training optimizes only the 3D self-attention layers in the DiT blocks.
- Dataset and Preprocessing: The training dataset contains 320K multi-shot videos, each comprising 2 to 5 shots and up to 250 frames.
- Dataset and Preprocessing: Each video has a global caption for the narrative arc, characters, and visual style, plus shot-level captions for segment actions and content.
- Bidirectional Teacher Training: Bidirectional teacher training optimizes only the 3D self-attention layers within the DiT blocks.