Source-linked AI summary
Long Context Tuning for Video Generation
Yuwei Guo, Ceyuan Yang, Ziyan Yang, Zhibei Ma, Zhijie Lin, Zhenheng Yang, Dahua Lin, Lu Jiang
TL;DR
Narrative video generation needs multi-shot scenes with visual and dynamic consistency, whereas existing systems largely generate single-shot clips and prior scene-level methods still struggle with complex coherence. LCT expands a pre-trained single-shot diffusion model’s context across an entire scene and learns cross-shot relations directly from data. Evaluations report consistent multi-shot scene generation, compositional generation, and shot extension, while current autoregressive conditioning retains potential redundancy.
Problem
Narrative videos require coherent multi-shot scenes, but current video generators primarily provide single-shot synthesis and existing scene-level methods struggle with complex coherence.
Method
LCT expands a pre-trained single-shot diffusion model’s context across all shots, using interleaved 3D position embeddings, asynchronous timesteps, and optional context-causal attention with KV-cache.
Results
LCT produces visually and semantically consistent multi-shot scenes and exhibits compositional generation and autoregressive shot extension beyond the pre-trained model’s capabilities.
Takeaways & Limitations
LCT supports scene synthesis without additional parameters while enabling compositional generation and interactive modification through shot extension.
Takeaways & Limitations
Current autoregressive generation includes all preceding tokens as history conditions, introducing potential redundancy.
Abstract
from arXiv · showhide
Recent advances in video generation can produce realistic, minute-long single-shot videos with scalable diffusion transformers. However, real-world narrative videos require multi-shot scenes with visual and dynamic consistency across shots. In this work, we introduce Long Context Tuning (LCT), a training paradigm that expands the context window of pre-trained single-shot video diffusion models to learn scene-level consistency directly from data. Our method expands full attention mechanisms from individual shots to encompass all shots within a scene, incorporating interleaved 3D position embedding and an asynchronous noise strategy, enabling both joint and auto-regressive shot generation without additional parameters. Models with bidirectional attention after LCT can further be fine-tuned with context-causal attention, facilitating auto-regressive generation with efficient KV-cache. Experiments demonstrate single-shot models after LCT can produce coherent multi-shot scenes and exhibit emerging capabilities, including compositional generation and interactive shot extension, paving the way for more practical visual content creation. See https://guoyww.github.io/projects/long-context-video/ for more details.
1. Introduction
LCT addresses the gap between realistic single-shot video synthesis and narrative videos composed of coherent multi-shot scenes. It expands pre-trained single-shot models to learn cross-shot consistency and supports scene generation alongside emerging interactive and compositional capabilities.
- 1. Introduction: Narrative videos require multiple shots with consistent visual appearance and temporal dynamics, beyond current single-shot generation.Visual coherence includes identity, background, lighting, and color tone; temporal coherence includes actions and camera movement.
- 1. Introduction: Existing approaches use appearance conditioning or keyframes followed by image-to-video animation, but they struggle with abstract cross-shot elements or ineffective conditioning.Appearance-conditioned methods rely on predefined conditions and curated datasets, limiting consistency for lighting and color tone.
- 1. Introduction: LCT expands a pre-trained single-shot model’s context window to learn scene-level cross-shot consistency directly from scene-level video data.The method adapts full attention across all shots in a scene.
- 1. Introduction: Interleaved 3D positional embeddings preserve shot positions and within-shot token relationships, while independent diffusion timesteps unify conditioning inputs and samples.The asynchronous strategy enables joint denoising of all shots or using some shots as low-noise conditions.
- 1. Introduction: LCT generates visually and semantically consistent scenes, including an example of around 20 shots lasting 3 minutes.The model also supports compositional generation from identity and environment images and autoregressive shot extension for interactive modification.
2. Related Works
Prior work on video generation spans single-shot synthesis, scene-level generation, and methods for extending video duration. Existing scene-level approaches primarily condition on appearance or use keyframes with image-to-video animation, while complex coherence remains difficult.
- 2. Related Works: Single-shot video generation progressed from GAN-based methods to diffusion-based models that extend image diffusion architectures with temporal layers.Representative systems include Align-Your-Latents, AnimateDiff, Stable Video Diffusion, Lumiere, and Emu Video.
- 2. Related Works: Scene-level generation synthesizes sequential videos depicting continuous events and primarily uses appearance conditioning or keyframe-based image-to-video methods.Appearance conditioning preserves character or environment appearance, while keyframe methods generate initial keyframes before independent animation.
- 2. Related Works: Existing methods still struggle to model complex scene-level coherence, and keyframe-based approaches can suffer from ineffective conditioning.This limitation motivates approaches that learn scene-level relationships more directly.
- 2. Related Works: Long-video research addresses duration limits through training-free methods, autoregressive frameworks, hierarchical strategies, and distributed inference.Examples include noise rescheduling with window-based temporal attention and autoregressive video extension.
3. Method
LCT adapts a pre-trained single-shot video diffusion model for scene-level generation by learning cross-shot consistency from structured scene data. It combines long-context attention, interleaved positional encoding, asynchronous diffusion timesteps, and causal attention fine-tuning for multiple generation modes.
- Data Preparation: LCT expands a pre-trained single-shot diffusion model’s context window to learn scene-level correlations directly from multi-shot video data.Scene data uses global prompts for shared characters, environments, and stories, plus per-shot prompts for individual events.
- Data Preparation: The training data combines approximately 500K scene samples averaging five shots with roughly 1M additional samples created by segmenting temporally varied single-shot videos.Videos are segmented into scenes and shots, then described with global and shot-level prompts.
- Architecture Design: Long-context MMDiT jointly attends to all text and video tokens in a scene, while remaining compatible with single-shot generation when only one video is present.The architecture avoids adding an inductive bias through vanilla full attention and can retain pre-trained single-shot behavior.
- Architecture Design: Interleaved 3D RoPE preserves each shot’s relative text-video positions while assigning distinct absolute positions to distinguish tokens across shots.Scene tokens are arranged in an interleaved [text]-[video]-[text]-... sequence along the space diagonal.
- Conditioning and Generation: LCT assigns independent diffusion timesteps to shots, allowing cleaner samples to condition noisier samples and strengthening cross-shot dependencies without auxiliary conditioning networks.The asynchronous timestep strategy unifies visual conditioning inputs with diffusion samples.
- Conditioning and Generation: After LCT, bidirectional models can be fine-tuned with context-causal attention for efficient auto-regressive generation using KV-cache.Attention remains bidirectional within each shot while restricting context flow across shots, matching cleaner-history conditioning.
- Conditioning and Generation: Human-selection generation retrieves relevant earlier character and environmental shots from a history pool instead of relying only on recent shots.This addresses non-sequential cross-shot dependencies and avoids the recency assumption of common auto-regressive generation.
4. Experiments
Experiments show that LCT produces coherent, diverse multi-shot scenes while supporting multiple inference modes and emergent interactive capabilities. Ablations indicate that LCT preserves single-shot ability, while causal attention improves history fidelity and enables efficient adaptation.
- Settings: The 3B model is trained with a maximum context of nine shots for 135K iterations, followed by 9K iterations of causal-attention fine-tuning.Training uses 128 NVIDIA H800s at 480 × 480 area resolution with untouched aspect ratio.
- Qualitative Results: LCT produces more diverse framing and better prompt alignment than baselines, while avoiding the keyframe-based reappearance problem.It includes wide-, mid-, and close-up shots and directly generates consistent character identities across shots.
- Quantitative Results: The method slightly trails visually conditioned baselines in visual quality but substantially outperforms them in semantic alignment and scene-level coherence.Evaluation combines VBench shot-level metrics with user studies emphasizing cross-shot consistency.
- Single-shot Generation after LCT: Post-LCT models improve consistently across all VBench metrics for single-shot generation, indicating that LCT preserves and may enhance the pretrained capability.LCT introduces no additional parameters.
- Effects of Inference Modes: Joint and autoregressive inference improve text alignment, while context-causal attention provides superior fidelity to preceding history conditions.The authors attribute the latter to sequential dependency that gives greater weight to preceding conditions.
- Emerging Capabilities: Autoregressive generation supports narrative continuation, compositional identity-environment integration, minute-long shot extension, and interactive shot-by-shot development.The model can autoregressively generate 10-second segments while maintaining visual consistency.
5. Conclusion
The paper concludes that LCT adapts single-shot video models to scene-level generation without additional parameters and supports efficient context-causal autoregression. It also identifies planner-based generation and token routing as future directions.
- 5. Conclusion: LCT expands scene context, uses interleaved 3D position embeddings and asynchronous training timesteps, and supports context-causal generation with KV-cache.The approach enables flexible scene synthesis without additional parameters.
- 5. Conclusion: The authors suggest MLLMs as future planners for scene-level video generation and token dynamic routing to reduce redundancy in autoregressive history conditioning.These directions are motivated by story generation, reasoning, and the inclusion of all preceding tokens as history.