Source-linked AI summary
LoL: Longer than Longer, Scaling Video Generation to Hour
Justin Cui, Jie Wu, Ming Li, Tao Yang, Xiaojie Li, Rui Wang, Andrew Bai, Yuanhao Ban, Cho-Jui Hsieh
TL;DR
Long-form autoregressive video generation can suffer sink-collapse, in which outputs revert to attention-sink frames and lose temporal continuity. LoL analyzes this behavior as a multi-head RoPE synchronization problem and applies training-free frequency shifts across heads. The method mitigates sink-collapse while preserving generation quality and supports real-time infinite streaming, including demonstrations lasting up to 12 hours.
Problem
Autoregressive ultra-long video generators using attention sinks can abruptly fall back to initial sink frames, producing sink-collapse and repeated content.
Method
LoL shifts the base frequencies of different attention heads around the original RoPE base to reduce inter-head homogenization, together with streaming RoPE generation and noise sampling.
Results
The method mitigates sink-collapse while preserving generation quality, achieving real-time infinite streaming generation with little quality degradation using a 1.3B model and KV cache.
Takeaways & Limitations
LoL demonstrates that training-free multi-head RoPE frequency shifting can support indefinitely long real-time video streaming without collapse in the reported setting.
Takeaways & Limitations
Long-term memory remains challenging for multi-hour videos, and generation quality is constrained by the limited-capacity underlying model and its teacher.
Abstract
from arXiv · showhide
Recent research in long-form video generation has shifted from bidirectional to autoregressive models, yet these methods commonly suffer from error accumulation and a loss of long-term coherence. While attention sink frames have been introduced to mitigate this performance decay, they often induce a critical failure mode we term sink-collapse: the generated content repeatedly reverts to the sink frame, resulting in abrupt scene resets and cyclic motion patterns. Our analysis reveals that sink-collapse originates from an inherent conflict between the periodic structure of Rotary Position Embedding (RoPE) and the multi-head attention mechanisms prevalent in current generative models. To address it, we propose a lightweight, training-free approach that effectively suppresses this behavior by introducing multi-head RoPE jitter that breaks inter-head attention homogenization and mitigates long-horizon collapse. Extensive experiments show that our method successfully alleviates sink-collapse while preserving generation quality. To the best of our knowledge, this work achieves the first demonstration of real-time, streaming, and infinite-length video generation with little quality decay. As an illustration of this robustness, we generate continuous videos up to 12 hours in length, which, to our knowledge, is among the longest publicly demonstrated results in streaming video generation.
1 Introduction
LoL addresses sink-collapse in autoregressive ultra-long video generation, where outputs regress toward attention-sink frames and abruptly repeat content. It identifies a multi-head RoPE-related mechanism and introduces frequency shifting to support indefinite streaming generation while preserving quality.
- 1 Introduction: Autoregressive video generation supports longer temporal modeling but remains vulnerable to instability, overexposure, train–test mismatch, and other long-sequence failures.These approaches replace computationally expensive bidirectional processing with prediction from previously generated frames, often using KV caches for streaming.
- 1 Introduction: Attention sinks can improve alignment and stability, yet state-of-the-art models repeatedly regress toward sink frames, causing sink-collapse at recurring latent frame indices.LongLive and Self-Forcing++ both collapse at latent indices 132 and 201, with additional collapses across prompts and input noise.
- 1 Introduction: LoL finds that sink-collapse reflects synchronized high attention to sink frames across multiple heads and temporal dimensions, rather than a single periodic component.The paper links collapse points to local maxima in summed phase alignment and multi-head attention concentration.
- 1 Introduction: Shifting the base frequencies of different attention heads around the original base θ reduces inter-head homogenization and mitigates sink-collapse without retraining.The method is combined with streaming RoPE generation and noise sampling for real-time, indefinitely long generation.
- 1 Introduction: Extensive experiments show that the method mitigates sink-collapse while preserving generation quality compared with baseline methods.The reported system achieves real-time infinite streaming generation with little quality degradation using a 1.3B model and KV cache.
3 Method
This section identifies sink-collapse as a long-horizon failure caused by RoPE phase realignment and synchronized attention across heads, then introduces multi-head RoPE jitter to disrupt that synchronization. The approach preserves streaming generation while mitigating collapse and supporting effectively unlimited length.
- 3.1 Background: Sink frames stabilize autoregressive video generation but can cause generated frames to abruptly revert to the initial frames.This sink-collapse appears as repeated fallback to the sink frames at recurring positions.
- 3.1 Background: RoPE’s periodic rotations can realign distant positions, causing multiple frames to share nearly identical positional embeddings at long horizons.The resulting positional aliasing resets distinctions that are useful for autoregressive attention.
- 3.3 Mitigating Sink Collapse: Sink-relative phase concentration measures how strongly multiple RoPE frequency components align a generated frame with a sink frame.Higher concentration indicates phase synchronization associated with sink-collapse.
- 3.3 Mitigating Sink Collapse: Sink-collapse is a collective multi-head effect: many heads simultaneously assign high attention to generated and sink frames, producing abrupt scene transitions.Additional attention visualizations show this copying pattern across multiple layers and heads rather than a single localized component.
- 3.3 Mitigating Sink Collapse: Multi-head RoPE jitter shifts the base frequency separately across attention heads to disrupt global phase alignment and reduce simultaneous overlap with sink frames.The method is presented as a lightweight intervention that changes rotated queries and keys through head-wise frequency perturbations.
- 3.3 Mitigating Sink Collapse: With sink-collapse mitigated, dynamic sampling of initial noise and RoPE enables effectively infinite streaming generation with little additional overhead.The method also addresses length constraints associated with RoPE and VAE memory consumption.
4 Experiments
Experiments compare positional-embedding extensions and evaluate sink-collapse using maximum and average distance drops. Multi-head RoPE jitter mitigates collapse while preserving motion and generation quality across long-video models, with jitter magnitude and head count affecting the trade-off.
- Metrics: Sink-Collapse Max measures the worst single-prompt distance drop, while Sink-Collapse Avg measures the average distance drop across test prompts.Both scores are computed across the whole generation range.
- Comparisons: Naive position extrapolation produces severe sink-collapse, while position interpolation suppresses it but makes motion nearly stagnant.NTK preserves more motion but offers limited mitigation, whereas YARN strongly suppresses collapse while greatly hampering dynamics.
- Comparisons: LoL effectively mitigates sink-collapse while preserving video generation quality on LongLive and Self-Forcing++.The evaluations include 100-second videos and comparisons with other autoregressive methods on 75- and 100-second videos.
- Ablations: Changing a single RoPE dimension or the RoPE base does not eliminate sink-collapse; base changes instead shift when collapse occurs.Altering the dimension identified by RIFLEx and neighboring dimensions is ineffective, and bases from 6000 to 20,000 shift rather than remove collapse.
- Ablations: Increasing jitter intensity reduces sink-collapse, with σ=0.8 reported as a balance between generation quality and mitigation.Higher intensity can further mitigate the problem but may reduce motion or quality.
- Ablations: Increasing the number of jittered attention heads progressively alleviates sink-collapse, and jittering all heads gives the strongest mitigation while maintaining high quality.The result supports a multi-head rather than single-head origin for the phenomenon.
5 Limitations and Further work
The method remains bounded by the capabilities of its underlying models and does not fully solve long-term memory for multi-hour videos.
- Fine-tuning or retraining may further improve the training-free method’s overall performance.
- Generation quality is constrained by underlying models that rely on local attention and sink frames for alignment and stability.
- Maintaining long-term memory remains challenging, especially for multi-hour videos.
- Future work may explore alternative position embeddings, advanced training strategies, stronger control signals, and sparse or linear attention.
6 Limitations and Further work
The method’s suppression is strongest in settings where sink-collapse is rarely observed, and its quality and efficiency remain bounded by the base model and hardware.
- The method only significantly suppresses repetition where sink-collapse is rarely seen during hour-long video generation.
- Infinite-generation potential is demonstrated with Wan2.1-T2V-1.3B, whose limited capacity bounds performance.
- Because the model was never trained on real datasets, its performance is bounded by that of the teacher model.
- The current streaming system generates about 16 FPS on a single GPU.
More related work
Video generation research has been reshaped by Diffusion Transformers and subsequent systems that improve realism, temporal consistency, resolution, or efficiency through scaling and large-scale pretraining.
- Diffusion Transformers have reshaped video generation, with performance scaling remarkably with computational capacity.
- Sora delivers highly realistic and temporally consistent videos with diverse, natural motion.
- Wan 2.1 highlights large-scale pretraining for high-resolution video synthesis.
- CogVideoX enhances cross-model video generation capabilities.
Phase alignment of other θ values
Under alternative RoPE base values, sink-collapse remains closely aligned with phase concentration, with representation distances dropping at local phase-alignment maxima.
- Alternative RoPE base values reveal a sink-collapse pattern highly consistent with the θ = 10000 setting.
- Sink-collapse remains closely aligned with the emergence of phase concentration under a different θ value.
- The normalized L2 distance between consecutive representations drops when phase alignment reaches local maxima.
- At θ=20000, sink-collapse locations highly correlate with local maxima of phase concentration.
The impact of sink frame number
Changing the number of attention-sink frames does not eliminate sink-collapse. The phenomenon persists with both larger and minimal sink-frame configurations.
- Three sink tokens with a local attention window of nine form the default total window size of twelve.This configuration follows the default setting used in LongLive.
- Increasing the sink-frame count from three to five does not prevent repetitive patterns or sink-collapse.The results indicate that increasing the sink-frame budget alone does not resolve the instability.
- Reducing the sink-frame count to one likewise leaves sink-collapse artifacts and repetition behavior.The collapse persists even under the minimal possible sink-frame configuration.
Visualization of other model layers
Attention-map visualizations show copying behavior across multiple model layers and a wide range of attention heads, rather than in one localized component.
- Copying behavior appears consistently across multiple attention layers and a wide range of attention heads.This widespread pattern supports interpreting sink-collapse as a structural effect rather than a localized anomaly.
Visualization of single prompt
Single-prompt visualizations demonstrate stable, coherent streaming generation for videos lasting up to 12 hours, including sequences with faster and more complex camera motion.
- Up to 12-hour videos retain coherent structure, visual consistency, and motion stability with little quality degradation.The visualizations illustrate long-form generation under a single prompt.
- A 12-hour sequence with faster and more dynamic camera motion remains coherent and temporally consistent throughout.The example indicates stable performance under long-range, fast-motion conditions.
Visualization of prompt switching
The method supports streaming generation with prompt switching, including frequent semantic transitions, while maintaining coherent and stable sequences. The paper also identifies limits tied to the underlying architectures, memory, and base-model capacity.
- Visualization of prompt switching: The method supports both single-prompt generation and prompt-switching scenarios.The prompt-switching demonstrations include controlled transitions and substantially more frequent semantic changes.
- Visualization of prompt switching: The visualizations include one-hour sequences with evolving prompts and 12-hour sequences generated from a single prompt.These examples cover both prompt transitions and extended single-prompt generation.
- Visualization of prompt switching: A 10-minute visualization demonstrates large scene transitions across successive cinematic prompts.The prompts move through a nighttime chase, freeway explosion, and finale sequence.
- Discussion & Future Work: The method suppresses sink-collapse without retraining, while retraining may further improve generation quality.The discussion also identifies alternative position embeddings, larger models, and long-term memory as future directions.
- Limitations: Overall generation quality is bounded by the LongLive and Self-Forcing++ architectures underlying the training-free extension.The paper frames this as a limitation of building on those base architectures.
- Limitations: Without long-term memory, subject consistency may fail after objects leave the frame or remain occluded for extended periods.This is a stated limitation of the underlying generation framework.
- Limitations: The distilled 4-step Wan2.1-T2V-1.3B base model can show reduced visual diversity in extremely long single-prompt videos.The paper identifies larger-capacity base models as a future direction.