PaperlayerView exact revision on arXiv

Source-linked AI summary

RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers

Min Zhao, Guande He, Yixiao Chen, Hongzhou Zhu, Chongxuan Li, Jun Zhu

arXiv:2502.15894v3cs.CV

TL;DR

Extending video diffusion transformers beyond fixed training lengths remains difficult because existing methods cause temporal repetition or slowed motion. RIFLEx identifies and reduces an intrinsic positional-embedding frequency, achieving high-quality 2× extrapolation training-free and extending to 3× with minimal fine-tuning without long videos.

  • Problem

    Video models trained at fixed maximum sequence lengths struggle to generate longer, temporally coherent videos, while existing extrapolation methods produce repetition or slow motion.

  • Method

    RIFLEx analyzes positional-embedding frequencies and lowers the intrinsic frequency so it remains within a single cycle after extrapolation.

  • Results

    RIFLEx achieves high-quality 2× extrapolation training-free and extends to 3× with minimal fine-tuning using only original-length videos.

  • Takeaways & Limitations

    Reducing the intrinsic frequency suppresses temporal repetition while preserving motion consistency and also supports spatial and joint temporal-spatial extrapolation.

  • Takeaways & Limitations

    Intrinsic-frequency identification is primarily empirical, and 3× extrapolation is limited by diminished ability to discriminate sequential positions after excessive frequency reduction.

Abstract

from arXiv · show

Recent advancements in video generation have enabled models to synthesize high-quality, minute-long videos. However, generating even longer videos with temporal coherence remains a major challenge and existing length extrapolation methods lead to temporal repetition or motion deceleration. In this work, we systematically analyze the role of frequency components in positional embeddings and identify an intrinsic frequency that primarily governs extrapolation behavior. Based on this insight, we propose RIFLEx, a minimal yet effective approach that reduces the intrinsic frequency to suppress repetition while preserving motion consistency, without requiring any additional modifications. RIFLEx offers a true free lunch--achieving high-quality 2x extrapolation on state-of-the-art video diffusion transformers in a completely training-free manner. Moreover, it enhances quality and enables 3x extrapolation by minimal fine-tuning without long videos. Project page and codes: https://riflex-video.github.io/.

1. Introduction

Video diffusion transformers can generate coherent minute-long videos, but extending beyond their fixed training lengths remains difficult because existing methods produce repetition or slowed motion. RIFLEx analyzes positional-embedding frequencies and reduces an intrinsic frequency to enable training-free 2× extrapolation and fine-tuned 3× extrapolation.

  • Motivation: Fixed maximum training lengths limit models’ ability to generate longer videos with smoothly evolving temporal content.Length extrapolation seeks coherent videos without training on longer sequences.
  • Motivation: Existing extrapolation methods fail on video by producing temporal repetition or slow motion, revealing a gap in understanding positional encodings.These methods were originally developed for text and image generation.
  • Frequency analysis: Frequency analysis finds that high frequencies induce temporal repetition, low frequencies cause motion deceleration, and one intrinsic frequency primarily dictates repetition patterns.The intrinsic component is consistent across different videos from the same model.
  • RIFLEx: RIFLEx lowers the intrinsic frequency so it remains within a single cycle after extrapolation, suppressing repetition while preserving motion consistency without other modifications.The method is designed as a minimal solution based on the frequency analysis.
  • Results: The method is demonstrated on HunyuanVideo and CogVideoX-5B without using videos longer or larger than those used for pre-training.Figure 1 visualizes temporal, spatial, and combined 2× extrapolation.
  • Results: RIFLEx achieves high-quality 2× extrapolation training-free on state-of-the-art video diffusion transformers and extends to 3× with minimal fine-tuning using only original-length videos.The fine-tuning setup uses 20,000 original-length videos and 1/50,000 of pre-training computation; it also supports simultaneous spatial extension.

2. Background

Diffusion transformers model video as latent sequences, while positional embeddings encode location information for attention. The background introduces RoPE and explains how extrapolation methods modify its frequencies across sequence axes and diffusion sampling.

  • Video generation with diffusion transformers: Diffusion models perturb clean data with scheduled Gaussian noise and generate samples by reversing that process with a learned score function.Video systems commonly apply diffusion modeling to latent video sequences using transformer networks.
  • Position embeddings: Position embeddings map axis positions to representations and can enter transformers through additive or multiplicative interactions with input or hidden embeddings.They provide position information to the transformer.
  • Position embeddings: RoPE encodes relative positions through interactions between absolute position embeddings in attention, with frequencies controlled by a base-frequency hyperparameter.RoPE is applied to query and key vectors before their dot product.
  • Position embeddings: For video coordinates (t, h, w), multi-axis RoPE applies separate one-dimensional encodings to temporal, height, and width axes and concatenates them.This independent treatment allows extrapolation techniques to be applied per axis.
  • Length extrapolation with RoPE: Position Extrapolation extends sequences without changing positional encoding, whereas Position Interpolation rescales all RoPE frequencies by θPI = θ/s for inference length L′ relative to training length L.Both methods depend on positional encoding generalization beyond the training length.
  • Length extrapolation with RoPE: NTK adjusts the base frequency to combine extrapolation for high frequencies with interpolation for low frequencies, enabling training-free extrapolation.YaRN instead uses fine-grained frequency adjustments and can perform well training-free or with limited fine-tuning.
  • Length extrapolation with RoPE: Image diffusion transformers independently apply RoPE along height and width, and timestep-dependent adjustments such as TASR can combine different strategies during sampling.These methods target spatial extrapolation in image generation.
  • Failure patterns: Existing video extrapolation methods produce temporal repetition or slower motion, while image extrapolation produces spatial repetition or blurred content.Figure 2 compares these failure patterns for 2× extrapolation.

3. Method

The method analyzes how RoPE frequency components produce repetition and motion deceleration, identifies an intrinsic frequency governing extrapolation, and reduces it with RIFLEx. This enables training-free 2× extrapolation, while 3× extrapolation benefits from fine-tuning because the larger frequency shift creates a training-testing mismatch.

  • 3.1. Failure Patterns of Existing Methods: Existing video extrapolation methods produce temporal repetition or slow motion, revealing a gap in understanding positional encodings.PE and NTK cause repetition, while PI causes slow motion; related methods exhibit one or both failure patterns.
  • 3.2. Frequency Component Analysis in RoPE: High-frequency components capture short-term dependencies and can induce repetition when extrapolated beyond their periods.When the frame interval exceeds a component period, periodic positional encodings force generated content to repeat; high-frequency components can complete multiple cycles during extrapolation.
  • 3.2. Frequency Component Analysis in RoPE: Low-frequency components encode long-term dependencies but produce slower motion because their positional encodings change less between adjacent frames.Higher frequencies create larger positional shifts and sensitivity to rapid movements, whereas lower frequencies favor slow-motion dynamics.
  • 3.2. Frequency Component Analysis in RoPE: The intrinsic frequency is the component whose period is closest to the first observed repetition frame and primarily determines the repetition pattern.It remains consistent across videos generated by the same model; examples are k = 2 for CogVideoX-5B and k = 4 for HunyuanVideo.
  • 3.3. Reducing Intrinsic Frequency: A Minimal Solution: RIFLEx lowers the intrinsic frequency so it remains within a single period after extrapolation, suppressing repetition while preserving motion consistency.Ablations indicate that modifying only the intrinsic frequency is sufficient; changing higher frequencies disrupts fast motion, while changing lower frequencies has negligible impact.
  • 3.3. Reducing Intrinsic Frequency: A Minimal Solution: 2× extrapolation works training-free, whereas 3× extrapolation requires fine-tuning because the larger intrinsic frequency shift causes a training-testing mismatch affecting visual quality.Fine-tuning with 20,000 original-length videos uses 1/50,000 of pre-training computation and improves dynamic and visual quality.

4. Experiments

RIFLEx is evaluated on temporal, spatial, and joint extrapolation, with quantitative and qualitative results showing improved video quality and mitigation of slow motion and repetition. It supports 2× extrapolation without training and extends to 3× with minimal fine-tuning, although quality degrades beyond 3×.

  • Performance Comparison: RIFLEx achieves superior overall performance by generating new temporal content without compromising other aspects of video quality.For CogVideoX-5B, PI and YaRN show slow motion, while PE and NTK show repetition; RIFLEx addresses both challenges and ranks highest in user studies.
  • Performance Comparison: NTK avoids repetition at 2× HunyuanVideo extrapolation but fails at 2.3× because its intrinsic frequency reduction does not generalize to longer extrapolation.The failure is reflected in NTK’s low NoRepeat Score at 2.3× extrapolation.
  • Limitations: Quality degrades significantly beyond 3× extrapolation, including at 4×.Excessive frequency reduction may diminish RoPE’s effectiveness by making encoding changes minimal over the training length.
  • Additional Extrapolation: RIFLEx supports spatial and joint temporal-spatial extrapolation by adjusting the intrinsic frequency in the corresponding dimensions.The experiments include simultaneous extension of video duration and spatial resolution.

5. Conclusion and Discussion

The paper concludes that analyzing RoPE frequency components yields RIFLEx, which reduces intrinsic frequency to prevent repetition. It achieves training-free 2× extrapolation and enables 3× extrapolation through minimal fine-tuning, while intrinsic-frequency identification remains empirical.

  • Conclusion: RIFLEx reduces intrinsic frequency to prevent repetition and achieves high-quality 2× extrapolation without training.Minimal fine-tuning enables 3× extrapolation without long videos.
  • Discussion: Intrinsic-frequency identification primarily relies on empirical visual inspection rather than a theoretical foundation.A theoretical account would require studying how intrinsic frequencies emerge during pre-training.
  • Discussion: The 3× extrapolation boundary reflects diminished ability to discriminate sequential positions after excessive frequency reduction.Further extension would require investigating positional-encoding mechanisms during training.

Impact Statement

The paper advances video generation through a training-free length-extrapolation method for diffusion transformers. It also highlights responsible use because generated videos can create misleading fake content.

  • Responsible Use: Responsible use is important because video-generation technology can produce misleading fake videos.The statement identifies potential negative social impacts without evaluating RIFLEx-specific risks.
  • Impact and Scope: The paper contributes to video generation alongside diffusion-transformer, text-to-video, autoregressive, and long-video generation research.These areas are presented as related directions rather than as components of RIFLEx.
  • Impact and Scope: RIFLEx offers a length-extrapolation strategy tailored to video diffusion transformers while training only on the original sequence length.The work builds on insights into RoPE’s impact in video diffusion transformers.

B. Additional Results of RIFLEx

Additional visual results cover temporal, spatial, joint temporal-spatial, and longer-horizon extrapolation. They include 2× temporal extension from 129 to 261 frames, joint dimension extension, and 4× temporal extension from 49 to 193 frames.

  • Temporal Extrapolation: Additional temporal results show 2× extrapolation from 129 to 261 frames.
  • Spatial Extrapolation: Additional spatial results compare image-generation extrapolation and report better visual quality from generating new content.
  • Joint Extrapolation: Joint temporal-spatial results extend video dimensions from 480×720×49 to 960×1440×97.
  • Long-Horizon Extrapolation: Additional long-horizon results evaluate 4× temporal extrapolation from 49 to 193 frames.
  • Comparison: Existing methods can produce slower motion, temporal repetition, or both during 2× extrapolation.TASR performs resolution extrapolation but causes slower motion and temporal repetition in video generation, while YaRN leads to slower motion.

C. More Results of Failure Patterns of Existing Methods

Existing 2× extrapolation methods produce distinct video failures: YaRN slows motion, while TASR combines slower motion with temporal repetition.

  • YaRN results in slower motion during 2× video extrapolation.The reported setting uses α = 1 and β = 32.
  • TASR combines PI and NTK across timesteps, leading to slower motion and temporal repetition.TASR uses PI at larger timesteps and NTK at smaller timesteps.

D. Experimental Setup.

The experiments document code and licenses, specify intrinsic-frequency adjustments for spatial and temporal extrapolation, and define evaluation procedures and fine-tuning settings.

  • The paper lists all used code and its license in Table 2.
  • Spatial extrapolation adjusts intrinsic frequencies whose periods match repeating height- and width-pixel patterns.Spatial and temporal extrapolation jointly adjust intrinsic frequencies for time, width, and height dimensions.
  • The NoRepeat Score identifies possible repeated sequences using framewise L2 distances from the first frame.Videos receive higher non-repetition scores when average distances exceed a threshold.
  • Fine-tuning settings for all experiments, including simultaneous spatial and temporal extrapolation, are summarized in Table 3.
  • Figure 11 reports results for adjusting all frequency components below the intrinsic frequency.

E. Details about RIFLEx

RIFLEx relies on a consistent intrinsic frequency and reduces it to address repetition while preserving quality; ablations show that changing other frequencies has different effects.

  • Across 20 videos, HunyuanVideo consistently identifies k = 4 as the closest intrinsic frequency despite repetition-frame variation from 178 to 200.For this example, N_k = 200.
  • RIFLEx lowers the intrinsic frequency so it remains within a single cycle after extrapolation, suppressing repetition without other modifications.
  • Reducing higher-frequency components slows video motion, likely because they capture fast motion.
  • Reducing lower-frequency components has negligible impact because their encodings change little across the training length.
  • Figure 12 presents ablations for reducing frequencies other than the intrinsic frequency, using PE without frequency reduction as the reference.

F. More Results about Comparisons

CogVideoX-5B comparisons show that existing methods separately suffer from slow motion or repetition, whereas RIFLEx addresses both challenges and improves motion quality.

  • PI and YaRN suffer from slow motion, while PE and NTK experience repetition in CogVideoX-5B comparisons.
  • TASR suffers from both slow motion and repetition, whereas RIFLEx addresses both challenges and enhances motion quality.
  • Figure 13 visualizes CogVideoX-5B length-extrapolation results and reports better quality when slow motion and repetition are addressed.
Loading 2502.15894v3…