PaperlayerView exact revision on arXiv

Source-linked AI summary

Rolling Sink: Bridging Limited-Horizon Training and Open-Ended Testing in Autoregressive Video Diffusion

Haodong Li, Shaoteng Liu, Zhe Lin, Manmohan Chandraker

arXiv:2602.07775v6cs.CV

TL;DR

Autoregressive video diffusion models trained on finite clips face a horizon gap when asked to generate open-ended videos. Rolling Sink uses training-free cache maintenance to address this gap, and the authors report superior long-horizon visual fidelity and temporal consistency to state-of-the-art baselines.

  • Problem

    Finite training clips cannot cover open-ended test horizons, motivating a training-free approach to bridge the horizon mismatch for long-video generation.

    Source: §1 · pp. 2–3

  • Method

    Rolling Sink is a training-free approach derived from systematic AR cache analysis that aims to keep cache behavior consistent with within-training-duration behavior.

    Source: §1 · p. 3, Key Issue: Cache Maintenance · p. 6

  • Results

    Rolling Sink achieves superior long-horizon visual fidelity and temporal consistency to state-of-the-art baselines, scaling synthesis to 5–30 minutes despite 5-second training clips.

    Source: §5 · p. 14

  • Takeaways & Limitations

    The study identifies maintaining cache consistency with within-duration behavior as a key issue in bridging limited-horizon training and open-ended testing.

    Source: Key Issue: Cache Maintenance · p. 6

    Rolling Sink only approximates necessary cache properties, so residual mismatch may remain and additional cache maintenance or more advanced methods may improve open-ended synthesis.

    Source: Quantitative Results during Analysis · p. 10

Abstract

from arXiv · show

Recently, autoregressive (AR) video diffusion models have achieved remarkable performance. However, due to their limited training durations, a train-test gap emerges when testing at longer horizons, leading to rapid visual degradations. Following Self Forcing, which studies the train-test gap within the training duration, this work studies the train-test gap beyond the training duration, i.e., the gap between the limited horizons during training and open-ended horizons during testing. Since open-ended testing can extend beyond any finite training window, and long-video training is computationally expensive, we pursue a training-free solution to bridge this gap. To explore a training-free solution, we conduct a systematic analysis of AR cache maintenance. These insights lead to Rolling Sink. Built on Self Forcing (trained on only 5s clips), Rolling Sink effectively scales the AR video synthesis to ultra-long durations (e.g., 5-30 minutes at 16 FPS) at test time, with consistent subjects, stable colors, coherent structures, and smooth motions. As demonstrated by extensive experiments, Rolling Sink achieves superior long-horizon visual fidelity and temporal consistency compared to SOTA baselines. Project page: https://rolling-sink.github.io/

1 Introduction · 2 Related Works

The paper frames long-horizon video drift as a mismatch between limited training horizons and open-ended testing, and proposes a training-free cache-maintenance method to address it. It reports improved long-video fidelity and consistency with Rolling Sink.

1 Introduction

  • Open-ended generation targets videos of arbitrary duration, while autoregressive models enable it by predicting subsequent frames from prior ones, unlike bidirectional models that denoise all frames simultaneously.Long videos may consist of multi-shot prompts whose shots range from seconds to minutes or hours.Source: §1 · pp. 2–3
  • Training on fixed, limited durations such as Self Forcing’s 5s at 16 FPS [39] leaves a horizon mismatch that can cause inconsistent subjects, oversaturated colors, vanished dynamics, and collapsed structures.The paper interprets this long-horizon drift as exposure bias: predictions may gradually drift when testing beyond the duration on which the model was regularized.Source: §1 · p. 3
  • Longer-video training cannot cover every possible test horizon, and scaling training to very long durations is computationally expensive, motivating a training-free solution.The paper notes that AR video models after Self Forcing [39] have been trained on clips ranging from 5s to 100s.Source: §1 · p. 3
  • Because prompts and per-block noise remain fixed in their respective ways during synthesis, the paper identifies cache context as the major factor in long-horizon drift and derives Rolling Sink from systematic cache analysis.The proposed cache maintenance aims to preserve the cache behavior observed within the training duration.Source: §1 · p. 3
  • Built on Self Forcing [39] trained on 5s videos, Rolling Sink generates 5–30-minute videos with consistent identities and structures, stable colors, smooth dynamics, and streaming efficiency under the same bounded cache size.It uses the same few-step denoising per autoregressive generation step as Self Forcing.Source: §1 · pp. 3–4
  • Rolling Sink achieves the highest scores on most VBench-Long dimensions and the lowest averaged rank, while prior SOTA methods [39] [99] often show color, subject, and surroundings inconsistencies.The evaluations cover 1-minute and 5-minute AR video synthesis; qualitative comparisons also report superior long-horizon visual fidelity.Source: §1 · p. 4

2 Related Works

  • The Related Works section directs readers to Supplementary Section B.Source: §2 · p. 4

3 Methodology

Rolling Sink addresses long-horizon drift by maintaining a bounded autoregressive cache whose indices and semantic content evolve to approximate within-training behavior. Systematic analysis improves VBench-Long quality across 1- and 5-minute synthesis, but finite-duration training leaves a residual gap.

3.1 Preliminaries: Autoregressive Video Diffusion Models

  • AR video diffusion factorizes video into prompt-conditioned next-frame distributions, each modeled by iterative denoising, while self-generated training caches narrow—but do not eliminate—the long-horizon train–test gap.Self Forcing better matches training and testing cache distributions, yet still falls short beyond its training duration.Source: §3.1 · pp. 4–5

3.2 Systematic Analysis & Rolling Sink

  • Building on Self Forcing, Rolling Sink keeps cache capacity bounded while combining pinned early blocks, sliding time indices, and rolling sink semantics to support open-ended generation.The figure illustrates the design with K = 3 and S = 2; the method is designed for a strictly bounded cache.Source: §3.2 · p. 5, Figure 3 · p. 6, Key Issue: Cache Maintenance · p. 6

Key Issue: Cache Maintenance.

  • Within-duration cache behavior requires minimally drifted blocks whose indices slide on a global timeline and whose semantic content moves along an enduring video manifold; beyond 5 seconds, corrupted latents can amplify drift.A constant prompt embedding and identically distributed initial noise leave conditioning context—the cache—as the key factor to maintain consistently.Source: Key Issue: Cache Maintenance · p. 6

Quantitative Results during Analysis.

  • Across 1- and 5-minute synthesis, analysis steps progressively improve VBench-Long scores, with Rolling Sink best at S/K = 83%, though finite-clip training leaves a residual gap.The figure reports averaged scores across VBench-Long dimensions; no specific scores are provided in the passages.Source: Figure 4 · p. 7, Quantitative Results during Analysis · p. 10

Key Issue: Cache Maintenance.

  • Larger pinned prefixes stabilize color, but flicker and eventual repetition remain; an all-sink cache is excluded because recent blocks are needed for local smoothness.Prominent flickers in 1-minute rollouts typically occur around 33 and 50 seconds before generation tends to collapse into repetition.Source: Key Issue: Cache Maintenance · p. 7
  • Sliding sink indices further reduce drift, especially flicker, while rolling sink semantics from within-duration history—alternating forward and reversed content—further improves consistency, particularly subject consistency.The rolling segment is drawn from all K minimally drifted within-duration blocks rather than only the first S blocks.Source: Key Issue: Cache Maintenance · pp. 8–9, Figure 6 · p. 8

Quantitative Results during Analysis.

  • Rolling Sink is a simple baseline that mitigates long-horizon drift but cannot fully close the mismatch from finite-horizon training; additional cache requirements or more advanced methods may improve synthesis.The cache properties studied are not an exhaustive characterization of within-duration behavior.Source: Quantitative Results during Analysis · p. 10

4 Experiments

With training held to 5-second clips, Rolling Sink suppresses visual drift beyond the training horizon and achieves the best average rank in 1- and 5-minute VBench-Long evaluations.

4.3 Quantitative Comparisons

  • On both 1- and 5-minute VBench-Long tests, Rolling Sink achieves the best average rank and top scores on most dimensions.The reported advantage over prior methods is more pronounced at 5 minutes.Source: Table 1 · p. 12, Table 2 · p. 12, §4.3 · p. 13

4.1 Experimental Settings

  • Comparisons use VBench-Long and Self Forcing and LongLive baselines with matched 5-second, 16-FPS training; LongLive’s further-trained LoRA weights are excluded.Rolling Sink uses a strictly bounded cache of K = 6 and a 4-step video diffusion sampler; it is built on Self Forcing, which builds upon CausVid and Wan.Source: §4.1 · pp. 10, 13

4.2 Qualitative Comparisons

  • Beyond the training horizon, Rolling Sink suppresses AR drift, preserving subject identity, scene geometry, and coherent motion where baselines degrade.Baseline degradation includes over-saturated colors and collapsed structures.Source: Figure 7 · p. 11, §4.2 · p. 13

5 Summary

Rolling Sink addresses long-horizon drift in autoregressive video diffusion, which the paper attributes to exposure bias between limited-horizon training and open-ended testing. The training-free method supports ultra-long synthesis, while the paper identifies multi-shot generation as a future extension.

  • The paper attributes long-horizon drift to exposure bias and proposes training-free Rolling Sink, based on systematic AR cache-maintenance analysis, to keep cache behavior consistent with the training duration.Source: §5 · p. 14
  • Rolling Sink targets single-shot synthesis under a fixed prompt; future work would extend its drift-mitigation principle to multi-shot generation with coherent transitions and continued long-horizon stability and fidelity.The paper notes that multi-shot settings need new semantics over time rather than only preserving and extrapolating existing content.Source: §5 · p. 14
  • Despite much shorter training, Rolling Sink achieves lower (better) average rank than LongLive (w/ LoRA) on 1- and 5-minute synthesis, while supporting 5–30-minute generation after 5s training.The paper reports stable identities, colors, and structures, smooth dynamics, and no efficiency sacrifice during ultra-long synthesis.Source: §5 · p. 14, Table S3 · p. 15, Table S4 · p. 16

A Text Prompts in Fig. 1

Fig. 1’s prompts describe two dynamic action scenes: a snowboarder speeding through a snowy mountain forest and a surfer carving across a sunlit wave.

A Text Prompts in Fig. 1

  • Fig. 1 pairs a high-energy snowboarder weaving through snowy pines with a low-angle surfer carving a sunlit wave, both framed as dynamic action shots.The snowboarder scene emphasizes speed and agility against a mountain vista; the surfer scene highlights balance, splashing water, and ocean waves.Source: A Text Prompts in Fig. 1 · p. 30

B Related Works

Video diffusion models produce impressive but generally short clips, while autoregressive video diffusion combines denoising with prior-conditioned generation for longer synthesis. Progress toward efficient open-ended generation has reduced denoising and cache costs, but long videos still degrade beyond training durations.

Video Diffusion Models

Autoregressive Video Diffusion Models

  • Earlier AR video diffusion methods NOVA [17], SkyReels-V2 [13], and MAGI-1 [86] use inefficient multi-step denoising, while Pyramid Flow [45] and CausVid use few-step generation for temporal efficiency.As cached history grows, computational resource demands rise sharply, constraining generation length.Source: B Related Works · p. 30
  • Self Forcing [39] and LongLive [99] cache bounded context windows, improving spatial efficiency and architecturally enabling open-ended generation, yet still fall short on long videos beyond their training durations.These methods address the resource burden of growing cached history, but do not resolve long-horizon synthesis degradation.Source: B Related Works · p. 30

C Quantitative Results during Our Analysis

Quantitative VBench-Long evaluations across 1-minute and 5-minute synthesis show scores rising through the analysis, with Rolling Sink performing best overall at an 83% sink ratio after applying Sliding Semantics.

C Quantitative Results during Our Analysis

  • The analysis evaluates VBench-Long [113] on 1-minute and 5-minute AR video synthesis, following the experimental setting in Sec. 4.3.Dimension-level results are reported in Tables S5–S7 for 1-minute synthesis and Tables S8–S10 for 5-minute synthesis.Source: C Quantitative Results during Our… · p. 31
  • Rolling Sink with Sliding Semantics achieves the best overall performance at an 83% sink ratio (S = 5, K = 6), with scores trending upward through the analysis.Increasing the sink ratio generally yields higher average scores on both 1-minute and 5-minute synthesis.Source: C Quantitative Results during Our… · p. 31

D Additional Qualitative Comparisons

Across additional qualitative comparisons, Rolling Sink better preserves appearance, scene structure, and motion at long horizons than prior baselines, while suppressing severe flickers and repetition collapse. These results also show that attention sinks do not eliminate all intermittent frame flickers.

D Additional Qualitative Comparisons

  • Rolling Sink suppresses long-horizon degradation better than prior SOTA AR video diffusion baselines [39] [99], preserving subject identity, scene geometry, and coherent motion.The comparisons cover diverse prompts and visual styles; baselines often show over-saturated colors and distorted scene structures when extrapolating beyond training horizons.Source: D Additional Qualitative Comparisons · p. 31
  • Despite training on 1-minute clips, LongLive (w/ LoRA) shows identity inconsistency and structural instability, while Rolling Sink better preserves stable appearance and coherent structure.Rolling Sink consequently yields more consistent long-horizon generations.Source: Figure S19 · p. 32
  • At T1 ≈33s and T2 ≈50s, training and training-free methods show abrupt changes and may collapse into repetitive frames, while Rolling Sink largely suppresses flickers and maintains coherent motion.This 1-minute synthesis study uses S_K = 83%.Source: Figure S21 · p. 34
  • Intermittent frame flickers, usually every several seconds during 30–60s synthesis, persist in training and training-free methods, indicating attention sinks are only part of the solution to long-horizon AR drift.LongLive (w/ LoRA), trained on longer videos, still flickers but at a lower frequency.Source: Figure S26 · p. 39, Figure S27 · p. 40
  • Without additional training, Rolling Sink scales a 5s-trained AR video diffusion model to 30-minute synthesis with coherent identity, structures, colors, and smooth dynamics.The appendix shows this result in two qualitative examples.Source: Figure S28 · p. 41, Figure S29 · p. 42

E Rolling Sink vs. LongLive (w/ LoRA)

Against LongLive with its train-long-test-long LoRA, Rolling Sink trained on 5s clips produces more stable long-horizon rollouts and the lowest average rank. The results highlight AR cache stabilization as a key factor in bridging limited-horizon training and open-ended testing.

E Rolling Sink vs. LongLive (w/ LoRA)

  • Rolling Sink continuously achieves the lowest (best) average rank against LongLive (w/ LoRA) at 1- and 5-minute horizons.The comparison also includes Self Forcing and LongLive (w/o LoRA) for reference.Source: E Rolling Sink vs. LongLive (w/ LoRA) · p. 43
  • LongLive’s train-long-test-long LoRA improves long-horizon synthesis over its w/o LoRA variant, but still shows AR drift, especially inconsistent subject identities.The qualitative comparisons are reported in Fig. S19 and S20.Source: E Rolling Sink vs. LongLive (w/ LoRA) · p. 43
  • Rolling Sink, built on Self Forcing and trained on only 5s clips, produces more stable and temporally consistent long-horizon rollouts than LongLive (w/ LoRA).The results suggest that stabilizing the AR cache can be more effective than expensively extending the training horizon.Source: E Rolling Sink vs. LongLive (w/ LoRA) · p. 43

F Why Not Building on LongLive?

Rolling Sink, built on Self Forcing, outperforms Self Forcing and LongLive, including LongLive with a 1-minute-video LoRA. The section argues that LongLive’s attention sinks and longer training do not eliminate long-horizon drift, motivating systematic study of cache maintenance and sliding time indices and semantics.

Why Not Building on LongLive?

  • Self Forcing [39] is a standardized base for systematic cache-maintenance analysis and transparent, stepwise gains; LongLive [99] extends it with sink-block revisiting and 1-minute-video LoRA training.Self Forcing revisits recent blocks, whereas LongLive revisits recent and initially self-generated blocks.Source: F Why Not Building on LongLive? · p. 43
  • LongLive [99] stabilizes colors with attention sinks but still flickers at long horizons and can collapse into repetition; 1-minute-video LoRA only partially reduces flicker.For 1-minute synthesis, prominent flickers are reported around 33s and 50s, followed by repetition collapse; with LoRA, a prominent flicker remains around 50s.Source: F Why Not Building on LongLive? · p. 44
  • LongLive’s static sink time indices and semantics create a gap that grows beyond its 5s training window, while RoPE instability and attention to earliest blocks may also contribute to drift.The passage discusses RoPE’s high- and low-frequency positional limitations and findings that current latent blocks attend predominantly to earliest self-generated blocks.Source: F Why Not Building on LongLive? · p. 44
  • Training and training-free methods show similar temporal instability, indicating that minimally drifting AR caches are insufficient without sliding time indices and semantics.The passage frames sliding in time indices and semantics as characteristics needed for long-horizon stability.Source: F Why Not Building on LongLive? · p. 44

SOTA performance based on Self Forcing

  • Rolling Sink outperforms Self Forcing [39] and LongLive [99], including LongLive with LoRA trained on 1-minute videos.The reported comparisons are in Fig. 7, Fig. S9–S20, and Tables 1, 2, S3, and S4.Source: F Why Not Building on LongLive? · p. 44

G 30-Minute AR Video Synthesis

At 30-minute test-time horizons, Rolling Sink maintains stability across realistic and animated video, preserving subjects, coherent structures and colors, and smooth dynamics without long-horizon AR drift.

G 30-Minute AR Video Synthesis

  • Across the entire 30-minute rollout, Rolling Sink maintains strong long-horizon stability in both realistic and animated content.The evaluation extends autoregressive video synthesis to 30 minutes at test time.Source: G 30-Minute AR Video Synthesis · p. 45
  • Rolling Sink preserves consistent subject identity, coherent structures and colors, and smooth dynamics throughout the rollout.The reported long-horizon drift failures avoided include over-saturation, texture degradation, and structural collapse.Source: G 30-Minute AR Video Synthesis · p. 45
  • The illustrations highlight cache maintenance as a principled and practical path to bridge limited-horizon training and open-ended testing.Source: G 30-Minute AR Video Synthesis · p. 45

H Abbreviation Legend & Prompt Lists

This supplement points to Tab. S11 for the abbreviation legend and documents prompt sets spanning quality, appearance, scene, motion, and temporal evaluation categories.

Abbreviation legend and quality prompts

  • Tab. S11 is identified as the abbreviation legend, and prompt lists are provided for aesthetic_quality, imaging_quality, and overall_consistency.Examples include a corgi playing a drum kit, a jellyfish in the ocean, and a spaceship landing on Mars.Source: H Abbreviation Legend & Prompt Lists · pp. 45–46

Appearance-style prompts

  • appearance_style prompts pair subjects and scenes with treatments such as animation, cyberpunk, watercolor, Ukiyo-e, surrealism, and Van Gogh-inspired styles.Examples include a Corgi in a park, a boat by the Eiffel Tower, a shark, and a couple caught in rain.Source: H Abbreviation Legend & Prompt Lists · p. 46

Scene, color, and motion prompts

  • background_consistency and scene prompts use settings such as a train station or coral reef, while color prompts vary the colors of birds, cars, and bicycles.dynamic_degree, motion_smoothness, and subject_consistency use animal and human action prompts.Source: H Abbreviation Legend & Prompt Lists · pp. 46–47

Action and object prompts

  • human_action, multiple_objects, object_class, and spatial_relationship prompts cover actions, object pairs, individual classes, and left/right arrangements.Examples include a person climbing a rope, a zebra and giraffe, a car, and a bird positioned left of a cat.Source: H Abbreviation Legend & Prompt Lists · pp. 47–48

Temporal-flickering prompts

  • temporal_flickering prompts describe tranquil tableaux and still frames featuring ruins, lampposts, furniture, and outdoor settings.The listed scenes include Chaco Canyon ruins, a dining table, a rocking chair, a phone booth, and benches.Source: H Abbreviation Legend & Prompt Lists · p. 48

Temporal-style prompts

  • temporal_style prompts combine subjects and scenes with camera movements, shaking, slow motion, or a steady and smooth perspective.Examples include panning across a Corgi, zooming out from a boat, and showing a panda in slow motion.Source: H Abbreviation Legend & Prompt Lists · p. 48

Related papers

Loading 2602.07775v6…