PaperlayerView exact revision on arXiv

Source-linked AI summary

Train Short, Inference Long: Training-free Horizon Extension for Autoregressive Video Generation

Jia Li, Xiaomeng Fu, Xurui Peng, Weifeng Chen, Youwei Zheng, Tianyu Zhao, Jiexi Wang, Fangmin Chen, Xing Wang, Hayden Kwok-Hay So

arXiv:2602.14027v3cs.CV

TL;DR

Autoregressive video diffusion models struggle when generating beyond their training horizons because positional embeddings and noise sampling provide inadequate long-range temporal support. FLEX is a training-free inference framework that combines frequency-aware RoPE modulation, Antiphase Noise Sampling, and inference-only attention sinks. It outperforms state-of-the-art methods at 30 seconds (6× extrapolation) and matches long-video fine-tuned approaches at 60 seconds (12× extrapolation).

  • Problem

    Autoregressive video diffusion models suffer temporal degradation when inference substantially exceeds their predefined training or self-rollout horizons.

  • Method

    FLEX combines frequency-aware RoPE modulation, Antiphase Noise Sampling, and inference-only Attention Sink to extend video horizons without training or architectural modification.

  • Results

    At 30 seconds (6× extrapolation), FLEX significantly outperforms state-of-the-art methods, while at 60 seconds (12× extrapolation) it performs comparably to long-video fine-tuned approaches.

  • Takeaways & Limitations

    FLEX serves as a plug-and-play augmentation for existing autoregressive inference pipelines and supports minute-level generation.

  • Takeaways & Limitations

    Video diffusion models are typically trained on short clips, such as 21 latent frames, leaving most temporal dimensions with limited rotation exposure.

Abstract

from arXiv · show

Autoregressive video diffusion models have emerged as a scalable paradigm for long video generation. However, they often suffer from severe extrapolation failure, where rapid error accumulation leads to significant temporal degradation when extending beyond training horizons. We identify that this failure primarily stems from the spectral bias of 3D positional embeddings and the lack of dynamic priors in noise sampling. To address these issues, we propose FLEX (Frequency-aware Length EXtension), a training-free inference-time framework that bridges the gap between short-term training and long-term inference. FLEX introduces Frequency-aware RoPE Modulation to adaptively interpolate under-trained low-frequency components while extrapolating high-frequency ones to preserve multi-scale temporal discriminability. This is integrated with Antiphase Noise Sampling (ANS) to inject high-frequency dynamic priors and Inference-only Attention Sink to anchor global structure. Extensive evaluations on VBench demonstrate that FLEX significantly outperforms state-of-the-art models at 6x extrapolation (30s duration) and matches the performance of long-video fine-tuned baselines at 12x scale (60s duration). As a plug-and-play augmentation, FLEX seamlessly integrates into existing inference pipelines for horizon extension. It effectively pushes the generation limits of models such as LongLive, supporting consistent and dynamic video synthesis at a 4-minute scale. Project page is available at https://ga-lee.github.io/FLEX_demo.

1. Introduction

Autoregressive video diffusion enables efficient long-video generation but degrades when inference exceeds training horizons. FLEX addresses this gap with training-free frequency-aware positional modulation, dynamic noise priors, and attention-based temporal anchoring.

  • Autoregressive diffusion models improve inference efficiency for long-video generation by generating video chunk-wise or frame-wise.
  • Extrapolating beyond trained horizons pushes models into out-of-distribution ranges, causing temporal drift, visual artifacts, and motion issues.
  • FLEX combines frequency-aware RoPE modulation, Antiphase Noise Sampling, and inference-only Attention Sink to preserve temporal detail, inject dynamics, and maintain global structure.
  • The identified failure modes are spectral bias from imbalanced 3D RoPE training exposure and insufficient dynamic priors in noise initialization.
  • At 30 seconds (6× extrapolation), FLEX significantly outperforms state-of-the-art methods; at 60 seconds (12× extrapolation), it performs comparably to long-video fine-tuned approaches.

2. Related Works

Prior work improves autoregressive video generation, positional extrapolation, and temporal coherence, but existing approaches remain constrained when inference exceeds training horizons.

  • Autoregressive Video Diffusion: Autoregressive video methods address long-sequence synthesis by avoiding bidirectional attention's quadratic complexity and improving causal generation efficiency.
  • Autoregressive Video Diffusion: Self-rollout sampling, Streaming Long Tuning, KVrecache, and joint denoising address exposure bias, extended generation, real-time interaction, and varied denoising levels.
  • Autoregressive Video Diffusion: Despite these advances, existing autoregressive approaches remain constrained when inference enters out-of-distribution temporal ranges beyond their training horizons.
  • Long Context Extension in RoPE: RoPE extension research includes linear Position Interpolation and non-uniform scaling methods inspired by Neural Tangent Kernel theory.
  • Temporal Priors and Noise Sampling: Noise-rescheduling, temporal-shifting, and group-based sampling methods inject temporal priors to improve spatio-temporal smoothness during inference.

3. Method

FLEX is a training-free inference framework for extending autoregressive video generation beyond short training horizons. It combines frequency-aware temporal RoPE modulation, Antiphase Noise Sampling, and inference-only attention sinks to preserve temporal detail, motion, and global structure.

  • Autoregressive video diffusion: Chunk-wise generation decomposes a length-L inference sequence into sequential latent-frame chunks conditioned on cached historical context.Each chunk is denoised using causal attention with a sliding window that retains only recent history.
  • Temporal RoPE: 3D RoPE divides the embedding into height, width, and temporal subspaces, with temporal dimensions represented by orthogonal rotary planes.Temporal rotations encode relative frame distances through frequency-dependent phases.
  • Frequency-aware RoPE modulation: FLEX modulates temporal RoPE by interpolating under-trained low-frequency components while extrapolating high-frequency components with stronger generalization.Training exposure measures completed rotation cycles, and the dynamic scale is S = max(1, L/Ltrain), leaving positions unchanged when L ≤ Ltrain.
  • Frequency-aware RoPE modulation: Standard position interpolation fails for video as scaling increases, producing nearly static videos and rapidly declining image quality during autoregressive inference.The method attributes this to uniform Fourier-space compression that diminishes fine-grained temporal discrimination.
  • Antiphase Noise Sampling: Antiphase Noise Sampling increases adjacent-difference energy by shifting noise power toward high frequencies, while remaining compatible with pretrained denoisers as a plug-and-play module.Its high-frequency initialization provides early temporal variation intended to avoid static results and encourage richer motion.
  • Temporal attention sink: Inference-only attention sinks fix the first N frames within the local attention window as persistent global semantic and structural anchors.This mechanism maintains global consistency without architectural modification or finetuning.

4. Experiments

Experiments evaluate FLEX quantitatively, qualitatively, through ablations, and on ultra-long generation. Across these settings, FLEX improves long-horizon consistency and dynamics while preserving visual quality.

  • Quantitative Comparison: 83.01 Quality Score at 30s, with imaging quality drift of −0.06, establishes FLEX’s strong performance on MovieGen prompts.At 60s, FLEX improves Quality Score from 80.51 to 82.68 and quality drift from −2.51 to −0.56 versus original Self Forcing.
  • Quantitative Comparison: 83.48 total score at 30s surpasses LongLive’s 82.88 and Rolling Forcing’s 82.83, while Dynamic Degree rises from 29.38 to 44.32 versus Self Forcing.FLEX also records the highest reported aesthetic score, 65.43, while maintaining competitive semantic alignment.
  • Qualitative Comparison: At 60s, FLEX preserves visual fidelity and subject consistency while CausVid and Self Forcing collapse visually and LongLive and Rolling Forcing exhibit identity drift.The comparison uses a complex multi-subject prompt under a 12× extrapolation setting.
  • Component Ablation: The full NR+ANS+AS configuration achieves an 83.07 Quality Score, while removing ANS reduces Dynamic Degree from 40.63 to 33.84.Removing NR degrades subject and background consistency, and removing AS lowers overall quality scores.
  • Hyper-parameter Ablation: Increasing α or β improves subject and background consistency but reduces dynamic performance, making intermediate settings such as α = 0.1 and β = 2.5 preferable.The results identify a trade-off between long-term consistency and content dynamics.
  • Noise Ablation: Increasing ρ from −1.0 to 1.0 improves subject/background consistency and aesthetic quality but constrains motion richness and temporal variation.The coefficient regulates temporal correlation in ANS’s within-chunk initial noise.
  • Ultra-long Generation: 82.48 quality score and image drift reduced from −2.19 to −1.17 when FLEX is integrated into LongLive, demonstrating model-agnostic gains.At 240s, FLEX preserves identity consistency and background dynamics relative to original LongLive.

5. Conclusion

FLEX extends autoregressive video generation beyond training limits by addressing positional-embedding spectral bias and insufficient noise dynamics. Its frequency-aware modulation, Antiphase Noise Sampling, and Attention Sink achieve strong extrapolation performance while remaining plug-and-play for existing inference pipelines.

  • Conclusion: FLEX addresses spectral bias in 3D RoPE and deficient dynamic priors in noise sampling to bridge short-term training and long-term inference.Frequency-aware modulation stabilizes global structure while preserving temporal detail.
  • Conclusion: 83.48 total score at 6× extrapolation surpasses training-based baselines, while the plug-and-play framework improves consistency and dynamics in minute-level generation.The framework combines frequency-aware modulation with Antiphase Noise Sampling and Attention Sinks.

A. Analysis of Position Interpolation in Video Extension

Dynamic Position Interpolation compresses temporal coordinates during extrapolation, causing static and chromatic collapse. Frequency-aware RoPE preserves high-frequency temporal distinctions for stable, dynamic generation.

  • PI dynamically compresses temporal indices beyond the training horizon, causing catastrophic long-range failure.The scaling factor is S = max(1, Linf/Ltrain), with compression applied only during extrapolation.
  • Temporal compression makes future frames nearly indistinguishable, abruptly eliminating motion and freezing the video at the training boundary.This occurs around the 5-second boundary when Ltrain = 21.
  • PI also produces saturated color patterns because compressed positional coordinates destabilize the autoregressive process and disrupt coherent structure.These artifacts appear in final frames beyond the training horizon.
  • Frequency-aware 3D RoPE extrapolates high-frequency components while preserving temporal resolution, enabling stable and dynamic generation as context length increases.

B.2. Spectral Analysis and Motion Energy

The spectral analysis shows that antiphase noise concentrates temporal energy at high frequencies. This aligns the noise with the temporal-difference filter, supplying variation for dynamic trajectories.

  • The PSD of the stationary AR(1) process is obtained from its autocovariance through the Wiener–Khinchin theorem.The autocovariance is R(k) = ρ^|k|Id.
  • As ρ approaches −1, the noise spectrum concentrates energy at the highest temporal frequency, ω = π.At the same limit, Sρ(π) approaches infinity while Sρ(0) approaches zero.
  • The temporal-difference operator acts as a high-pass filter with frequency response H(ω) = 1 −e^−jω.
  • For ρ < 0, the filter response aligns with the noise peak at ω = ±π, maximizing high-frequency temporal variance.This alignment provides gradients that initialize dynamic trajectories and prevent convergence to a static mean during early denoising.

C. Verification of Temporal Attention Sink

Inference-only attention sinks improve structural stability beyond Self Forcing’s 5-second training horizon, but do not eliminate all accumulated degradation. The intervention retains the first three frames in the local attention window.

  • Keeping the first N = 3 frames as attention sinks mitigates visual degradation and preserves structural stability over longer extrapolation lengths.The experiment extends Self Forcing generation to 30 seconds at 480 × 832 resolution.
  • Figure 8 compares original Self Forcing with inference using three latent frames as sinks.
  • Without attention sinks, Self Forcing develops rapid color over-saturation and prominent artifacts beyond its 5-second training horizon.
  • Attention sinks alone do not fully resolve quality degradation, with feature shifts, background noise, color bias, and over-saturation persisting.The passage attributes complete temporal consistency to collaboration with frequency-aware RoPE and designed sampling strategies.

D. Detailed VBench-Long Benchmark Evaluation

FLEX is evaluated on the 16-dimension VBench-Long benchmark using 30-second generated videos from 946 prompts and five random seeds. It consistently outperforms three baselines across key quality and semantic metrics.

  • VBench-Long evaluation covers all 16 quality and semantic dimensions using 946 prompts and five random seeds per prompt.Each generated video is 30 seconds long and is sliced before evaluation.
  • FLEX consistently outperforms three baselines on most key metrics, achieving superior comprehensive semantic and visual quality.
  • Figure 9 presents a normalized radar chart aggregating results across all 16 VBench-Long dimensions.

E. Additional Qualitative Comparisons

The supplementary comparisons evaluate 30-second and 60-second generations across diverse prompts. FLEX improves extrapolation stability beyond the 5-second training length and remains comparable to long-video-trained methods at 60 seconds.

  • Evaluation setup: The supplementary analysis compares 30-second and 60-second generations across diverse prompts and thematic categories.The 30-second cases are evaluated at t = 0s, 5s, 10s, 20s, and 30s.
  • 30-second results: FLEX prevents the rapid image collapse typically observed beyond Self-Forcing’s 5-second training length.The comparison covers Self-Forcing, LongLive, and Rolling Forcing across multiple scenarios.
  • 30-second results: FLEX provides competitive or superior character consistency and motion dynamics compared with LongLive across diverse 30-second scenarios.The qualitative comparisons span human portraits, scenes, sport and motion, and animals and nature.
  • Overall profile: The VBench-Long radar profile shows a more balanced and expansive performance across 16 dimensions than existing baselines.The profile is presented as evidence of robust handling of diverse long-horizon generation challenges.
  • 60-second results: At 60 seconds, FLEX stably enhances Self-Forcing’s extrapolation capability with performance comparable to LongLive’s long-video-optimized approach.The 60-second setting is described as challenging because traditional long-video paradigms accumulate errors.

F. Implementation Details on LongLive Integration

FLEX is integrated into LongLive for 240-second synthesis by replacing its 3D RoPE and noise sampling while retaining LongLive’s attention sink. The integration uses horizon-specific frequency parameters and ANS to support ultra-long generation.

  • Integration setup: FLEX is integrated into LongLive for ultra-long 240-second synthesis using LongLive’s official settings and inference pipeline.LongLive is fine-tuned on 60-second sequences, with Ltrain ≈240 latent frames, versus 21 frames for Self-Forcing.
  • Module integration: Because LongLive already includes an attention sink, integration replaces its original 3D RoPE and noise sampling with FLEX components.The attention sink is therefore retained rather than replaced.
  • RoPE adaptation: For LongLive’s extended training horizon, RoPE Modulation uses α = 1.0 and β = 15.0 for frequency-aware interpolation over 240 seconds.These parameters account for LongLive’s greater exposure to lower-frequency positional signals.
  • Qualitative behavior: At 60 seconds, FLEX preserves subject and background consistency in character and scene sequences, while baselines show collapse, color distortion, or background-light drift.The qualitative comparisons include CausVid, Self-Forcing, and Rolling Forcing.
  • Qualitative behavior: At 60 seconds, FLEX maintains species-specific fish appearance and coral-environment fidelity while avoiding the static freeze and identity fading seen in baselines.The comparison includes Self-Forcing and LongLive.
  • Noise adaptation: ANS uses ρ = −1.0 in place of LongLive’s default noise sampling to alleviate repetitive motion patterns and cyclic artifacts during 4× training-length extrapolation.This setting supplies the noise-sampling component of the LongLive integration.
Loading 2602.14027v3…