PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression

Jung Yi, Wooseok Jang, Paul Hyunbin Cho, Jisu Nam, Heeji Yoon, Seungryong Kim

arXiv:2512.05081v1cs.CV

TL;DR

Long-horizon autoregressive video generation suffers from error accumulation and fidelity degradation beyond the training distribution. Deep Forcing addresses this with training-free Deep Sink and Participative Compression, achieving state-of-the-art evaluations and minute-long generation without fine-tuning. Its scope remains constrained by the frozen backbone and may include gradual drift in extremely long sequences.

  • Problem

    Long-horizon autoregressive generation accumulates errors from imperfect prior frames, degrading fidelity as models extrapolate far beyond their short-video training domain.

  • Method

    Deep Forcing combines Deep Sink, which preserves a large attention-sink region with adjusted RoPE, and Participative Compression, which retains KV-cache tokens important to recent queries.

  • Results

    Deep Forcing achieves state-of-the-art performance on VBench-Long, user studies, and VLM evaluation without fine-tuning, surpassing training-based methods.

  • Takeaways & Limitations

    The training-free KV-cache management paradigm enables minute-long autoregressive video generation while preserving visual fidelity and motion dynamics.

  • Takeaways & Limitations

    The frozen-backbone method is constrained by pretrained capacity and biases, lacks explicit long-term memory, and may gradually drift in extremely long sequences with repeated occlusions.

Abstract

from arXiv · show

Recent advances in autoregressive video diffusion have enabled real-time frame streaming, yet existing solutions still suffer from temporal repetition, drift, and motion deceleration. We find that naively applying StreamingLLM-style attention sinks to video diffusion leads to fidelity degradation and motion stagnation. To overcome this, we introduce Deep Forcing, which consists of two training-free mechanisms that address this without any fine-tuning. Specifically, 1) Deep Sink dedicates half of the sliding window to persistent sink tokens and re-aligns their temporal RoPE phase to the current timeline, stabilizing global context during long rollouts. 2) Participative Compression performs importance-aware KV cache pruning that preserves only tokens actively participating in recent attention while safely discarding redundant and degraded history, minimizing error accumulation under out-of-distribution length generation. Together, these components enable over 12x extrapolation (e.g. 5s-trained to 60s+ generation) with better imaging quality than LongLive, better aesthetic quality than RollingForcing, almost maintaining overall consistency, and substantial gains in dynamic degree, all while maintaining real-time generation. Our results demonstrate that training-free KV-cache management can match or exceed training-based approaches for autoregressively streaming long-video generation.

1. Introduction

Autoregressive video generation supports real-time streaming but accumulates errors over long horizons. Deep Forcing addresses this with training-free attention-sink preservation and selective KV-cache compression.

  • Autoregressive video generation streams frames sequentially for interactive applications, unlike offline generation of complete clips.
  • Long-horizon generation accumulates errors from imperfect prior frames, causing color drift, blurred textures, and disappearing fine details.
  • Attention sinks stabilize autoregressive attention by retaining a small set of initial global tokens, but video generation requires deeper context preservation.
  • Deep Forcing introduces Deep Sink and Participative Compression as training-free mechanisms for stabilizing context and removing redundant KV-cache tokens.
  • Deep Sink preserves intermediate sink tokens with dynamically adjusted RoPE, while Participative Compression retains tokens important to queries from recent frames.
  • Evaluations using VBench, user studies, and VLM assessments show improvements over Self Forcing without fine-tuning, including state-of-the-art results on several metrics.

2. Related Work

Related work combines autoregressive prediction, diffusion, attention sinks, and KV-cache compression to support streaming and longer-context generation. Deep Forcing builds on these ideas without relying on additional training.

  • Autoregressive Video Diffusion: Autoregressive video diffusion generates frames or chunks sequentially, with methods using progressive denoising, causal conversion, or self-generated histories.
  • Attention Sink: Attention sinks are initial tokens that receive disproportionate attention; retaining them in a sliding window enables generation beyond the training context length.
  • Attention Sink: Recent autoregressive video models preserve the first three frames as attention sinks through distillation or fine-tuning, whereas Deep Forcing leverages inherent sink behavior without training.
  • KV Cache Compression: KV-cache compression reduces memory growth by selecting important tokens, addressing attention dilution as increasingly many tokens compete for focus.

3. Preliminaries

Autoregressive video diffusion factorizes a video distribution into frame-wise conditional distributions. Each frame is generated by conditioning on previously generated frames and their cached keys and values.

  • Autoregressive video diffusion produces each frame or chunk conditioned on previously generated frames within a denoising diffusion process.
  • For a sequence of N frames, the autoregressive model factorizes the joint distribution into conditional terms p(x_i | x_<i).
  • The conditional diffusion model generates frame x_i using the previously generated frames x_<i as context.
  • Previously generated clean frames provide the KV cache, while the predicted clean frame is perturbed with Gaussian noise for the next denoising iteration.

4. Method

Deep Forcing addresses long-horizon degradation in autoregressive video generation through deep temporal attention sinks and importance-aware KV-cache compression. The method preserves intermediate context, realigns sink positions, and removes redundant history to stabilize quality beyond the training horizon.

  • Overview: Deep Forcing uses two training-free mechanisms: Deep Sink for persistent temporal anchoring and Participative Compression for selective KV-cache pruning.The framework is designed to mitigate error accumulation without fine-tuning.
  • Deep Sink: Substantial attention in pretrained Self Forcing extends from initial to intermediate tokens across the context window, motivating deeper sinks than StreamingLLM-style initial-token retention.The analysis examines how newly generated frames attend to earlier KV-cache entries across layers and heads.
  • Deep Sink: Increasing sink frame size improves Overall Consistency and decreases Aesthetic Quality Drift in 50-second videos.The ablation evaluates sink depth using VBench metrics on the first 21 MovieGen prompts.
  • Deep Sink: Deep Sink dedicates a large portion of the sliding window to sink tokens and applies a temporal RoPE offset while preserving spatial indices.The offset reduces the temporal gap between sink and tail tokens, aligning their relative temporal positions.
  • Participative Compression: Participative Compression targets degeneration beyond 12× the training length by pruning redundant history that can dilute attention and accumulate errors.The method retains contextually relevant tokens while discarding low-contribution history.
  • Participative Compression: Participative Compression replaces FIFO eviction with token-level selection based on aggregated attention from recent queries.It retains sink and recent tokens, ranks intermediate candidates by importance, and compresses the cache when it reaches maximum length.

5. Experiments

Deep Forcing is evaluated as a training-free extension of Self Forcing for long-video generation, using VBench-Long, qualitative comparisons, VLM assessment, user studies, and component ablations. Across these evaluations, it improves visual stability and dynamic motion while remaining competitive with or surpassing trained baselines, though subject consistency is lower on VBench-Long.

  • Evaluation setup: Deep Forcing is evaluated with VBench-Long, qualitative comparisons, VLM visual-stability assessment, user preference studies, and ablations against autoregressive baselines.The implementation uses Self Forcing as the base model and compares against CausVid, Self Forcing, Rolling Forcing, and LongLive.
  • Quantitative results: Deep Forcing achieves superior overall consistency and imaging quality compared to LongLive, and better aesthetic quality than Rolling Forcing.The comparison is reported despite Deep Forcing being training-free and built on a model not trained for long-video generation.
  • Quantitative results: Deep Forcing produces more dynamic camera and subject motion than existing approaches, yielding more visually expressive videos.The method also shows gains in dynamic degree relative to trained methods, although it was not explicitly optimized for this aspect.
  • Limitations: Subject consistency is lower in VBench-Long metrics, although qualitative results show better overall quality with limited degradation compared to training-based methods.The authors provide additional qualitative results in Appendix F.
  • Ablation studies: Adding Deep Sink and Participative Compression progressively improves dynamic degree, overall consistency, and image quality in the component ablation.The ablation uses sink size S = 10 frames, budget N = 16, and recent-token count R = 4.
  • Ablation studies: In a 30-second qualitative ablation, Self Forcing shows severe color drift, Deep Sink reduces degradation but leaves artifacts, and the complete method preserves consistent visual quality.The complete framework effectively eliminates noticeable degradation while preserving overall quality and fine-grained details.

6. Conclusion

Deep Forcing is a training-free approach that mitigates error accumulation in autoregressive long-video generation through Deep Sink and Participative Compression. It achieves strong evaluation results without fine-tuning, while remaining constrained by the frozen backbone and lacking explicit long-term memory.

  • Deep Forcing mitigates error accumulation through Deep Sink and Participative Compression.
  • The method achieves state-of-the-art performance on VBench-Long, user studies, and VLM evaluation without fine-tuning.
  • Deep Forcing enables minute-long video generation while preserving visual fidelity and motion dynamics.
  • The frozen backbone constrains performance to the pretrained model’s capacity and biases.
  • The lack of explicit long-term memory may cause gradual drift in extremely long sequences with repeated occlusions.

A. The Tale of Three Sinks

The comparison shows that Deep Sink outperforms LongLive-style and Rolling Forcing-style sinks in the training-free setting. Both deep sinking and RoPE adjustment are essential for stable long-video generation.

  • LongLive’s fixed three-frame sink without RoPE adjustment develops fidelity degradation, flickering, and rollback during generation.Failures appear at frames 800, 801, and 802, respectively.
  • Rolling Forcing’s Dynamic RoPE still exhibits severe fidelity degradation at frames 800–801.
  • Both deep sink and RoPE adjustment are essential for long-video generation.

B. Qualitative Results on Different Sink Size

Larger attention sinks reduce fidelity degradation in 60-second videos, but excessively large sinks cause repetitive generation. The reported optimal range is 10–15 frames, or 40–60% of the sliding window.

  • Without an attention sink, fidelity degradation appears by frame 230 and quality completely collapses by frame 690 in the monster scene.The SUV scene also exhibits significant fidelity degradation.
  • Increasing sink size to 4 and 9 frames progressively reduces degradation, although fine-detail degradation remains visible.
  • Once sink size exceeds 10 frames, fidelity degradation is substantially reduced, whereas an 18-frame sink produces repetitive generation.
  • The optimal sink range is 10–15 frames, corresponding to 40–60% of the sliding window.Deep Sink alone remains insufficient to maintain visual fidelity throughout minute-long generation across diverse scenes.

C. Participative Compression Details

Participative Compression preserves sink and recent context while pruning intermediate KV-cache tokens according to query-based importance. Selection at the initial diffusion timestep remains relevant throughout denoising and outperforms random selection qualitatively.

  • Participative Compression Details: Participative Compression compresses only intermediate tokens while preserving initial sink and recent context unchanged.
  • Participative Compression Details: Compression is applied independently per layer when the cache exceeds its maximum window length, only at t = 1000.The selected tokens remain fixed throughout subsequent denoising steps.
  • Participative Compression Details: Attention patterns remain consistent across denoising timesteps, validating Top-C selection at t = 1000.The evaluated timesteps are 1000, 750, 500, and 250.
  • Participative Compression Details: Combining clean past and current denoising queries achieves the highest scores across the evaluated metrics.Using only clean past tokens gives image quality 68.54 and overall consistency 20.47, while using only denoising tokens gives image quality 68.24 and motion smoothness 97.86.
  • Participative Compression Details: Denoising query-based selection preserves subject consistency and contextual coherence, unlike Gaussian random selection, which causes severe artifacts and context drift.
  • Participative Compression Details: Denoising-based selection concentrates on semantically important positional slots rather than distributing selections uniformly across candidates.The concentrated positions bridge established sink context and recent generation.
  • Participative Compression Details: The method’s effectiveness is hypothesized to stem from four-step distilled diffusion and per-layer KV caching.

E. FPS measurements

Deep Forcing maintains throughput comparable to Self Forcing despite compression overhead, balancing slower compression phases with faster generation phases. Additional visualization shows how Top-C token selection varies across persistent sink, compressible, and recent tokens.

  • Throughput: 15.75 FPS versus 15.78 FPS: Deep Forcing maintains throughput comparable to baseline Self Forcing on 60-second videos.The measurement uses a single NVIDIA H100 GPU.
  • Latency trade-off: Compression from 21 to 16 frames adds computation, while subsequent generation reduces attention costs compared with attending over 21 cached frames.Throughput alternates between slightly slower compression phases and slightly faster generation phases.
  • Token selection: The Top-C heatmap separates tokens into Deep Sink tokens, compression candidates, and recent tokens across a 1-minute generation.Color intensity indicates how often each token is selected as Top-C.

F. More Qualitative Results

Additional qualitative examples report Deep Forcing results on par with training-based methods, while the user study compares videos using consistency and motion criteria across multiple baselines.

  • More Qualitative Results: Deep Forcing produces qualitative results on par with training-based methods in additional examples.The cited passage refers to Figures 14 and 15.
  • User Study: The user study uses Two-Alternative Forced Choice comparisons between videos generated from the same prompt.Participants selected their preferred video according to color consistency, dynamic motion, subject consistency, and another stated criterion.
  • User Study: 384 total video comparisons were collected from 24 participants comparing Deep Forcing with four baselines.Each participant evaluated 16 video pairs, with four comparisons per baseline.

H. Additional Attention Visualization

Additional visualizations show substantial attention to intermediate tokens across layers and heads, while qualitative figures compare Deep Forcing with training-based and training-free baselines on extended videos.

  • Additional Attention Visualization: Substantial attention to intermediate tokens appears consistently across layers and heads, alongside attention to initial tokens.This pattern supports the hypothesis that deep sinks are fundamental to Self Forcing.
  • More Qualitative Results: On 30-second videos, Deep Forcing is reported to match baseline temporal consistency and visual quality while producing more dynamic content with greater subject consistency.The comparison includes CausVid, Self Forcing, LongLive, and Rolling Forcing.
  • More Qualitative Results: On 60-second videos, Deep Forcing is reported to match baseline temporal consistency and visual quality while producing more dynamic content with greater subject consistency.The comparison spans the same four baselines as the 30-second results.
Loading 2512.05081v1…