PaperlayerView exact revision on arXiv

Source-linked AI summary

Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusion

Haodong Li, Shaoteng Liu, Zhe Lin, Manmohan Chandraker

arXiv:2602.07775v6cs.CV

TL;DR

Autoregressive video diffusion models face a train-test gap because finite-duration training does not cover open-ended generation, causing degradation at longer horizons. The paper analyzes cache maintenance and introduces training-free Rolling Sink, which enables a 5-second-trained model to synthesize ultra-long videos with stable identities, colors, structures, and dynamics while outperforming state-of-the-art baselines. The authors present it as a simple baseline because residual mismatch can remain and further cache-maintenance methods may improve open-ended synthesis.

  • Problem

    Finite training horizons cannot cover open-ended test-time generation, so autoregressive video diffusion models can suffer long-horizon drift and rapid visual degradation.

  • Method

    Rolling Sink uses systematic AR cache-maintenance analysis to keep the cache consistent with within-duration behavior without additional training.

  • Results

    Rolling Sink scales a model trained on 5-second videos to 5-30-minute synthesis while maintaining stable identities, colors, structures, and dynamics, with superior fidelity and temporal consistency versus state-of-the-art baselines.

  • Takeaways & Limitations

    Training-free cache maintenance can mitigate long-horizon autoregressive video drift under a strictly bounded cache and extend synthesis far beyond the training duration.

  • Takeaways & Limitations

    Rolling Sink cannot fully close the limited-horizon/open-ended mismatch and is presented as a simple baseline whose residual mismatch may be improved by more advanced cache-maintenance methods.

Abstract

from arXiv · show

Recently, autoregressive (AR) video diffusion models have achieved remarkable performance. However, due to their limited training durations, a train-test gap emerges when testing at longer horizons, leading to rapid visual degradations. Following Self Forcing, which studies the train-test gap within the training duration, this work studies the train-test gap beyond the training duration, i.e., the gap between the limited horizons during training and open-ended horizons during testing. Since open-ended testing can extend beyond any finite training window, and long-video training is computationally expensive, we pursue a training-free solution to bridge this gap. To explore a training-free solution, we conduct a systematic analysis of AR cache maintenance. These insights lead to Rolling Sink. Built on Self Forcing (trained on only 5s clips), Rolling Sink effectively scales the AR video synthesis to ultra-long durations (e.g., 5-30 minutes at 16 FPS) at test time, with consistent subjects, stable colors, coherent structures, and smooth motions. As demonstrated by extensive experiments, Rolling Sink achieves superior long-horizon visual fidelity and temporal consistency compared to SOTA baselines. Project page: https://rolling-sink.github.io/

1 Introduction

The paper studies why autoregressive video diffusion models degrade when generating beyond their limited training horizons and proposes a training-free cache-maintenance approach for open-ended synthesis. Rolling Sink, built on a model trained on 5-second videos, supports ultra-long generation with improved visual and temporal consistency.

  • Motivation: Open-ended video generation requires models to continue producing arbitrary-length videos from sequences of prompts, including shots lasting seconds, minutes, or hours.Autoregressive models are suited to this setting because they predict subsequent frames conditioned on previous ones.
  • Motivation: 5s-at-16-FPS training covers only a narrow range of test durations, causing long-horizon generation to exhibit inconsistent subjects, saturated colors, vanished dynamics, and collapsed structures.These degradations become especially pronounced when testing beyond the training duration.
  • Motivation: Finite-length training cannot fully cover open-ended testing, while extending training to very long videos is computationally expensive, motivating a training-free solution.The paper frames this mismatch as exposure bias between limited-horizon training and open-ended test-time generation.
  • Approach: Rolling Sink analyzes AR cache maintenance and keeps the cache consistent with within-duration behavior during open-ended synthesis.The approach is built on Self Forcing, uses a strictly bounded cache, and requires no additional training.

3 Methodology

The method analyzes how autoregressive cache construction creates a train–test gap and develops cache-maintenance operations that preserve within-duration behavior during open-ended synthesis. Rolling Sink combines static cache structure with sliding indices and rolling semantics to mitigate long-horizon drift.

  • Preliminaries: Autoregressive video generation factorizes a video into conditional block distributions, with each block denoised while conditioning on preceding context and the user prompt.Each denoising step progressively converts a noisy block into a clean prediction before producing the next autoregressive block.
  • Preliminaries: Teacher forcing and diffusion forcing cache ground-truth context during training but self-generated context during testing, creating a train–test gap.Self Forcing instead uses model-generated frames during training, improving the cache-distribution match within the training duration, but long-horizon synthesis can still fail beyond that duration.
  • Cache Maintenance: Because the prompt and initial noise distribution remain fixed, cache corruption during longer rollouts can bias subsequent predictions and amplify autoregressive drift.The resulting degradations include inconsistent subjects, over-saturated colors, vanished dynamics, collapsed structures, and other long-horizon artifacts.
  • Systematic Analysis: The analysis first pins minimally drifted early latents as static sink blocks, then examines sliding their time indices and semantic content within a bounded cache.The cache uses a finite capacity, while the rolling operations approximate how sink indices and semantics should evolve over an indefinitely extending video.
  • Rolling Sink: Rolling Sink rolls a segment of minimally drifted within-duration blocks over the entire cache instead of fixing the sink to the first blocks, further mitigating drift and improving subject consistency.The method is built by applying the rolling operation to the within-duration block set x[0,K).

4 Experiments

Experiments compare Rolling Sink with strong autoregressive video synthesis baselines under matched evaluation settings. Qualitative and quantitative results show reduced long-horizon drift and stronger performance across 1-minute and 5-minute generation.

  • 4.1 Experimental Settings: Rolling Sink is implemented on top of Self Forcing, with cache-based conditioning formed by concatenating prior clean visual tokens with current-block tokens.The cache provides keys and values for self-attention in the DiTs.
  • 4.1 Experimental Settings: VBench-Long evaluates long-video generation through multiple diagnostic dimensions measured by pretrained expert models.The evaluation compares Self Forcing and LongLive under a shared 5-second training duration at 16 FPS, excluding LongLive’s additional LoRA weights in the main comparison.
  • 4.2 Qualitative Comparisons: Rolling Sink qualitatively preserves subject identity, scene geometry, and coherent motion beyond the training horizon, unlike baselines that accumulate over-saturated colors and collapsed structures.The comparisons cover Fig. 7 and supplementary Figures S9–S18.
  • 4.3 Quantitative Comparisons: On both 1-minute and 5-minute settings, Rolling Sink achieves the best average rank and top scores on most VBench-Long dimensions.These results reflect reduced drift, improved visual quality, and more stable autoregressive rollouts beyond the training window.
  • 4.3 Quantitative Comparisons: Fig. 8 summarizes quantitative comparisons on 1-minute and 5-minute synthesis, where Rolling Sink attains the highest scores on most VBench-Long dimensions without additional training.Rolling Sink is built on top of Self Forcing.

5 Summary

Rolling Sink is presented as a training-free approach for extending autoregressive video diffusion beyond limited training horizons. The paper reports stronger long-horizon quality while noting a scope boundary around single-shot generation under a fixed prompt.

  • 5 Summary: Rolling Sink targets 5–30-minute autoregressive video synthesis despite a limited 5-second training duration, without additional training.The method maintains stable identities, colors, structures, and smooth dynamics while preserving efficiency.
  • 5 Summary: The method primarily targets single-shot long-video synthesis under a fixed prompt rather than movie-like generation requiring new prompts and semantics over time.The paper notes that multiple shots are needed in more general long-video scenarios.
  • 5 Summary: Rolling Sink achieves lower average rank than LongLive with LoRA on both 1-minute and 5-minute synthesis despite its much shorter training duration.The comparisons are reported in supplementary Tables S3 and S4.
  • 5 Summary: The cache analysis evaluates attention sinks, sliding indices, and sliding semantics across multiple sink sizes on 1-minute and 5-minute synthesis.These settings are reported in supplementary Tables S5–S10.

A Text Prompts in Fig. 1

The figure prompts describe two dynamic action scenes: a snowboarder descending a snowy mountain slope and a surfer carving through a powerful wave.

  • Upper figure: The upper figure depicts a young snowboarder accelerating through powdery snow between pine trees with a misty mountain vista behind.The scene is framed as a high-energy action shot with sunlight highlighting the snowboarder’s movements.
  • Bottom figure: The bottom figure depicts a surfer accelerating across a powerful wave, balancing on the board amid splashing water and a sunlit ocean.Distant white-capped waves and seagulls appear in the background.

B Related Works

The related-work discussion situates video diffusion models within applications such as neural simulation and world modeling, emphasizing their development from image diffusion models.

  • Video Diffusion Models: Video generation supports neural simulators and world models, while photorealistic video synthesis has become a community standard following image diffusion’s success.The passage attributes this development to the scaling abilities of video diffusion models and internet-scale data.

C Quantitative Results during Our Analysis

Quantitative VBench-Long analysis across 1-minute and 5-minute synthesis shows performance improves as the sink ratio increases, with Rolling Sink performing best at 83%.

  • VBench-Long evaluations cover both 1-minute and 5-minute autoregressive video synthesis across multiple dimensions and sink sizes.The detailed scores are reported in Tables S5-S10.
  • Increasing the sink ratio generally yields higher average VBench-Long scores for both 1-minute and 5-minute synthesis.
  • 83% sink ratio gives Rolling Sink the best overall performance after applying Sliding Semantics.This setting corresponds to S = 5 and K = 6.

D Additional Qualitative Comparisons

Additional qualitative studies show that Rolling Sink preserves long-horizon video quality more reliably than baseline methods, including methods trained on longer videos.

  • Rolling Sink better preserves stable appearance and coherent structure than LongLive with LoRA despite LongLive using longer video training.
  • At approximately 33s and 50s, baseline methods exhibit abrupt changes and may collapse into repetitive frames, whereas Rolling Sink largely suppresses these failures.The study uses a sink ratio of 83%.
  • Intermittent frame flickers persist across training-based and training-free methods during 30-60s synthesis, showing that attention sinks alone are insufficient.
  • LongLive with LoRA still exhibits frame flickers during 30-60s synthesis, although at a lower frequency than without the longer-video training.
  • Without additional training, Rolling Sink scales a 5s-trained autoregressive video diffusion model to 30 minutes with coherent identity, structures, colors, and smooth dynamics.

E Rolling Sink vs. LongLive (w/ LoRA)

Against LongLive with its train-long-test-long LoRA, Rolling Sink remains more stable and temporally consistent despite being trained only on 5s clips.

  • LongLive’s train-long-test-long LoRA improves long-horizon quality over its version without LoRA but still exhibits noticeable autoregressive drift.
  • Rolling Sink produces more stable and temporally consistent long-horizon rollouts than LongLive with LoRA while using only 5s training clips.
  • Rolling Sink achieves the lowest, and therefore best, average rank in the quantitative comparisons on 1-minute and 5-minute synthesis.
  • The comparison indicates that stabilizing the autoregressive cache can be more effective than expensively extending the training horizon.

F Why Not Building on LongLive?

The paper builds on Self Forcing because it offers a standardized base for systematic cache analysis, while LongLive and other methods retain long-horizon instabilities.

  • LongLive revisits both recent and initially self-generated blocks, whereas Self Forcing revisits only recent blocks.
  • LongLive’s 1-minute train-long-test-long LoRA extends the feasible generation horizon, but LongLive still exhibits flickers and repetition collapse.
  • Self Forcing provides a standardized base that makes Rolling Sink’s step-by-step performance gains and effect more transparent.
  • Static sink prefixes stabilize colors but leave flickers, because long-horizon stability also requires sliding time indices and semantics.
  • Rolling Sink achieves superior performance to Self Forcing, LongLive, and LongLive with LoRA despite using Self Forcing as its base.

G 30-Minute AR Video Synthesis

Rolling Sink extends autoregressive video synthesis from 5-second training clips to 30-minute open-ended test rollouts while maintaining long-horizon visual stability across realistic and animated content.

  • 30 minutes: Rolling Sink maintains strong long-horizon stability across realistic and animated video content.The rollout preserves consistent subject identity, coherent structures and colors, and smooth dynamics.
  • Across the entire rollout, subject identity remains consistent in appearance and shape.
  • Coherent structures and colors are preserved throughout the 30-minute synthesis.
  • The generated videos retain smooth dynamics without long-horizon autoregressive drift such as over-saturation.

H Abbreviation Legend & Prompt Lists

The supplementary materials provide abbreviation references and prompt lists spanning quality, consistency, temporal behavior, appearance, scenes, colors, and human actions.

  • Quality and consistency: Prompt lists cover aesthetic quality, imaging quality, and overall consistency across varied visual scenarios.
  • Appearance and style: Appearance-style prompts vary rendering styles including animation, cyberpunk, watercolor, Ukiyo-e, surrealism, and Van Gogh-inspired imagery.
  • Scenes and attributes: Background-consistency and scene prompts span locations such as tracks, stations, pools, theaters, reefs, rivers, and supermarkets.
  • Scenes and attributes: Color prompts specify objects in colors including black, pink, red, orange, green, blue, and yellow, while human-action prompts cover varied activities.
  • Temporal behavior: Temporal flickering prompts use tranquil tableaux and static environments to assess temporal stability.
  • Temporal behavior: Temporal-style prompts specify camera movements and motion patterns such as panning, zooming, tilting, shaking, and slow motion.
Loading 2602.07775v6…