Source-linked AI summary
Context Forcing: Consistent Autoregressive Video Generation with Long Context
Shuo Chen, Cong Wei, Sun Sun, Ping Nie, Kai Zhou, Ge Zhang, Ming-Hsuan Yang, Wenhu Chen
TL;DR
Long-video generation is limited by a mismatch between long-context students and short-context teachers, which cannot supervise global temporal dependencies. Context Forcing uses a long-context teacher, contextual distillation, error-recycling training, and Slow-Fast Memory to address this gap. The method achieves 20+ second effective context lengths and improves long-duration consistency over state-of-the-art baselines, while memory compression remains open for optimization.
Problem
Short-context teachers cannot guide long-context students on global temporal dependencies, limiting learnable context length and long-term consistency.
Method
Context Forcing distills a long-context teacher into a long-context student and uses Slow-Fast Memory plus error-recycling training for robust long-context generation.
Results
20+ seconds of effective context, a 2–10× improvement over current state-of-the-art baselines, accompanies improved long-duration consistency.
Takeaways & Limitations
The framework supports consistent long-form video generation by maintaining long-term history while reducing visual redundancy.
Takeaways & Limitations
The current memory compression strategy still leaves room for optimization, and the method assumes teacher reliability near student-generated contexts.
Abstract
from arXiv · showhide
Recent approaches to real-time long video generation typically employ streaming tuning strategies, attempting to train a long-context student using a short-context (memoryless) teacher. In these frameworks, the student performs long rollouts but receives supervision from a teacher limited to short 5-second windows. This structural discrepancy creates a critical \textbf{student-teacher mismatch}: the teacher's inability to access long-term history prevents it from guiding the student on global temporal dependencies, effectively capping the student's context length. To resolve this, we propose \textbf{Context Forcing}, a novel framework that trains a long-context student via a long-context teacher. By ensuring the teacher is aware of the full generation history, we eliminate the supervision mismatch, enabling the robust training of models capable of long-term consistency. To make this computationally feasible for extreme durations (e.g., 2 minutes), we introduce a context management system that transforms the linearly growing context into a \textbf{Slow-Fast Memory} architecture, significantly reducing visual redundancy. Extensive results demonstrate that our method enables effective context lengths exceeding 20 seconds -- 2 to 10 times longer than state-of-the-art methods like LongLive and Infinite-RoPE. By leveraging this extended context, Context Forcing preserves superior consistency across long durations, surpassing state-of-the-art baselines on various long video evaluation metrics.
1. Introduction
Causal video models offer streaming and long-horizon generation, but their effective context remains short because memoryless teachers cannot supervise long-term dependencies. Context Forcing addresses this mismatch with long-context distillation and Slow-Fast Memory, enabling more consistent long-form generation.
- Causal video architectures can theoretically generate infinite-length videos by predicting future frames from past context, but their computational cost motivates their use for long-form generation.
- Current causal video models often lose identity and temporal coherence beyond a few seconds because short-window teachers cannot guide global temporal dependencies.
- The Forgetting-Drifting Dilemma trades short memory, which limits error accumulation but forgets prior content, against long context, which preserves identity but exposes accumulated errors.
- Context Forcing distills a long-context teacher into a long-context student through Contextual Distribution Matching Distillation, transferring long-term dependencies and global consistency.
- 2–10× longer KV-cache history, exceeding 20 seconds versus 1.5–9.2 seconds for prior state of the art, is enabled by Slow-Fast Memory and supports consistent long-form generation.
2. Related Work
Related work extends video generation through autoregressive prediction, causal architectures, and explicit memory mechanisms, while distillation supports efficient real-time generation. Context Forcing builds on these directions.
- High computational costs of Diffusion Transformers have limited video generation to short clips, motivating autoregressive approaches for extending temporal horizons.
- Figure 2 contrasts self-forcing, LongLive’s memoryless 5-second teacher, and Context Forcing’s long-context teacher aware of full generation history.
- Recent methods improve efficiency through drifted-input training or inference-time recaching, sampling strategies, and feedback.
- Causal video generation imposes strict temporal ordering, enabling streaming inference and long-horizon synthesis, but early autoregressive models suffered from error accumulation and poor scalability.
- Explicit memory structures and recurrent latent states are used to accumulate scene information or model long-range dependencies efficiently.
3. Methodology
Context Forcing trains a causal video student against long-context teacher guidance while decomposing long-horizon learning into local and continuation objectives. A Slow-Fast KV-cache system and robust context training support efficient, consistent long rollouts.
- Context Forcing: The causal framework models long videos as conditional frame or chunk generation and seeks to match the student’s long-video distribution to real data.The global objective targets long-horizon KL divergence, but direct optimization is computationally intractable for large sequences.
- Context Forcing: The KL objective is decomposed into local dynamics and global continuation dynamics, motivating a two-stage curriculum.Stage 1 learns short-window dynamics; Stage 2 matches continuation predictions over long contexts.
- Stage 1: Local Distribution Matching: Stage 1 matches 1–5-second student windows to a teacher distribution, producing high-quality contexts for subsequent long-context training.The DMD gradient uses student and teacher scores with a weighting function.
- Stage 2: Contextual Distribution Matching: Stage 2 matches student-generated continuations against a pretrained Context Teacher, training on the student’s own rollouts to mitigate exposure bias.The teacher serves as a proxy for real-data continuation when student prefixes remain near the real-data manifold.
- Context Management System: A linearly growing KV cache is partitioned into attention sink, Slow Memory, and Fast Memory, retaining salient long-term information alongside recent local context.Slow Memory stores high-entropy keyframes, while Fast Memory is a rolling FIFO queue; salient history is compressed into lower indices as recent history slides through higher indices.
- Robust Context Teacher Training: Error-Recycling Fine-Tuning perturbs teacher contexts with accumulated model residuals so the teacher can correct degraded student histories.This training is intended to keep the teacher’s conditional distribution reliable when student contexts drift.
4. Experiments
Experiments evaluate Context Forcing across teacher continuation, short- and long-video generation, and ablations of its memory and training components. The method maintains consistency over extended sequences while preserving comparable short-video performance.
- Evaluation setup: The evaluation uses VBench, DINOv2, CLIP-F, and CLIP-T to assess short-video quality and long-term structural, semantic, and prompt consistency.Results use official VBench prompts, MovieGenBench prompts, window-based sampling, and five random seeds per prompt.
- Video continuation: The context teacher generates reliable continuations from student-generated contexts, supporting its use for contextual distillation.The teacher is evaluated on complete sequences combining the initial context and generated continuation.
- Text-to-short video generation: Short-video generation achieves performance comparable to the baselines on VBench.This evaluation concerns 5-second videos using official extended prompts.
- Text-to-long video generation: A context span exceeding 20s with a KV cache size of 21 improves consistency and mitigates content drifting during 60-second generation.Qualitative comparisons report superior consistency against the baselines.
- Ablation studies: Similarity-based slow-memory selection preserves long-term consistency by sampling historically dissimilar chunks instead of using fixed uniform intervals.The strategy prioritizes critical semantic information and is compared with uniform sampling at intervals of 1 and 2 chunks.
- Ablation studies: Removing Context DMD degrades semantic and temporal consistency, while removing Bounded Positional Encoding reduces background stability and subject consistency.The ablations identify both components as important for coherent long-horizon generation and temporal-drift mitigation.
- Ablation studies: Error-Recycling Fine-Tuning makes the context teacher more robust to accumulated errors and produces fewer artifacts in 30s rollouts.The resulting teacher distribution is used for further contextual distillation.
5. Conclusion
Context Forcing addresses the student-teacher mismatch in long-horizon causal video generation with long-context supervision and Slow-Fast Memory. It achieves 20+ second effective context lengths while leaving memory compression as an optimization target.
- 5. Conclusion: Context Forcing uses a long-context teacher to supervise a long-context student, addressing the supervision gap in streaming-tuning methods.The teacher maintains awareness of long-term history.
- 5. Conclusion: Slow-Fast Memory reduces visual redundancy to make extreme-duration generation computationally feasible.The conclusion identifies learnable compression and adaptive memory as future directions.
- 5. Conclusion: 20+ seconds is the reported effective context length, representing a 2–10× improvement over current state-of-the-art baselines.The method is also reported to mitigate drifting errors and enhance temporal coherence.
- 5. Conclusion: The current memory compression strategy remains limited in information density, leaving efficiency and semantic retention open for optimization.The paper proposes learnable context compression and adaptive memory mechanisms as future work.
Impact Statement
The paper frames improved temporal consistency in long-video generation as beneficial for creative and professional applications. It also notes that more consistent generation increases risks from sophisticated synthetic media and deepfakes.
- Impact Statement: Improved temporal consistency supports more coherent and realistic visual sequences for digital storytelling, filmmaking, world models, and professional video editing.The statement describes potential applications rather than measured deployment outcomes.
- Impact Statement: Highly consistent long-form video generation increases the risk of synthetic media and deepfakes being used for misinformation.The paper advocates digital watermarking, provenance standards, transparency, and detection mechanisms.
A. Preliminaries
Causal autoregressive video models generate frames or short chunks sequentially under strict temporal causality. Their formulation conditions each chunk on preceding chunks while permitting bidirectional attention within the current chunk.
- A. Preliminaries: Causal autoregressive models generate videos frame-by-frame or short-chunk by short-chunk while enforcing strict temporal causality.Information flow across chunks is restricted.
- A. Preliminaries: Block-wise causal attention permits bidirectional self-attention within each chunk X_t but restricts information flow across chunks.The conditional formulation is P(X_t | X_<t).
B. Visual artifacts in LongLive.
LongLive can produce abrupt scene resets and repetitive cyclic motion patterns during long video generation.
- LongLive frequently exhibits abrupt scene resets and repetitive, cyclic motion patterns.These artifacts are illustrated in Figure 8.
C. Algorithm of Context Forcing.
The Context Forcing algorithm trains with variable-length rollouts, cached context, and contextual DMD supervision. It alternates gradient-enabled and disabled denoising computations before computing loss on a context video and target noise.
- Contextual DMD requires a pretrained teacher, a stage-1 student, an autoregressive diffusion model, prompts, rollout settings, and a context window.
- The algorithm samples prompts, rollout lengths, and random exit steps, then initializes generated outputs and a KV cache for training.
- During each rollout, the procedure switches to a full denoising schedule near the teacher segment and otherwise uses the sampled exit step.
- The method enables gradients only at selected denoising computations and disables them for the remaining computations while generating rollout frames.
- For each rollout, it constructs context and target videos, then computes contextual DMD loss using student and teacher score functions.