PaperlayerView exact revision on arXiv

Source-linked AI summary

MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens

Youngrae Kim, Qixin Hu, C. -C. Jay Kuo, Peter A. Beerel

arXiv:2603.12513v1cs.CV

TL;DR

Long-horizon autoregressive video generation needs context that evolves rather than being evicted or fixed to early tokens. MemRoPE combines EMA-based dual memory tokens with online RoPE indexing, and experiments report consistent improvements across durations and evaluation protocols. Its scope remains bounded by the frozen base model and lossy memory aggregation.

  • Problem

    Existing methods either evict past context or converge to stale token sets, failing to maintain a smoothly evolving representation as video unfolds.

  • Method

    MemRoPE uses dual long- and short-term EMA memory streams and stores unrotated keys, applying positional indices online for fixed-size, unbounded caching.

  • Results

    MemRoPE consistently outperforms existing training-free methods in visual fidelity and consistency across minute-scale to hour-long generation.

  • Takeaways & Limitations

    Long-horizon coherence depends on continuously remembering evolving context rather than simply retaining more frames.

  • Takeaways & Limitations

    Because it uses a frozen checkpoint and lossy EMA aggregation, MemRoPE’s per-frame quality and precise recall of distant content are bounded.

Abstract

from arXiv · show

Autoregressive diffusion enables real-time frame streaming, yet existing sliding-window caches discard past context, causing fidelity degradation, identity drift, and motion stagnation over long horizons. Current approaches preserve a fixed set of early tokens as attention sinks, but this static anchor cannot reflect the evolving content of a growing video. We introduce MemRoPE, a training-free framework with two co-designed components. Memory Tokens continuously compress all past keys into dual long-term and short-term streams via exponential moving averages, maintaining both global identity and recent dynamics within a fixed-size cache. Online RoPE Indexing caches unrotated keys and applies positional embeddings dynamically at attention time, ensuring the aggregation is free of conflicting positional phases. These two mechanisms are mutually enabling: positional decoupling makes temporal aggregation well-defined, while aggregation makes fixed-size caching viable for unbounded generation. Extensive experiments validate that MemRoPE outperforms existing methods in temporal coherence, visual fidelity, and subject consistency across minute- to hour-scale generation.

1. Introduction

Long-form video generation must preserve useful context and valid positional encoding beyond fixed horizons. MemRoPE addresses these challenges with evolving dual memories and online positional indexing, enabling training-free, unbounded generation.

  • Motivation: Fixed-length video diffusion models cannot directly support coherent generation over minutes to hours.Long-form generation is needed for continuous world simulation, cinematic long takes, and synthetic data generation.
  • Problem: Sliding-window FIFO eviction discards distant context, causing progressive error accumulation during extended generation.Existing autoregressive approaches enable variable-length generation but remain limited by finite temporal horizons.
  • Problem: Attention-sink methods preserve early frames, but Deep Forcing’s selected tokens converge to persistent tokens and cause abrupt shifts when the cache updates.Newly admitted tokens receive disproportionately high scores, producing temporal instability.
  • Problem: Current approaches lack a smoothly evolving representation that adapts as video content unfolds.The gap is between complete context eviction and stagnant token sets that change discontinuously.
  • Approach: MemRoPE uses EMA-based long- and short-term Memory Tokens plus Online RoPE Indexing to retain evolving context in a fixed-size cache.Keys are stored without positional embeddings and receive block-relative indices at attention time.
  • Contributions: MemRoPE is training-free, supports unbounded generation with constant memory, and outperforms state-of-the-art training-free methods across minute- to hour-long sequences.The reported contributions cover context retention, positional extrapolation, and visual fidelity and consistency.

2. Related Work

Related work addresses long video generation through token compression, streaming methods, context retention, and positional extrapolation. MemRoPE instead maintains a continuously evolving fixed-size memory while decoupling cached content from position.

  • Autoregressive video generation: Autoregressive video generation includes token-based prediction, chunk-level diffusion, and compressed-context or window-attention methods.These paradigms reduce generation or memory costs in different ways.
  • Autoregressive video generation: Frame-level methods improve causal generation, training-inference alignment, temporal extrapolation, and long-context streaming through progressively specialized techniques.Examples include Self-Forcing++, Rolling Forcing, Causal Forcing, LongLive, and FAR.
  • Core challenges: Extending autoregressive generation beyond the training horizon requires retaining useful past context and maintaining valid positional encoding.These are presented as the two central challenges.
  • Context retention: Existing cache-compression methods operate within fixed-length caches and cannot recover information after eviction.EMA temporal aggregation has been explored elsewhere but requires architectural and training modifications.
  • Context retention: Attention sinks preserve initial frames, while Participative Compression can collapse to a fixed token set during generation.Other approaches use streaming tuning, memory modules, compressed histories, or geometric priors.
  • MemRoPE: MemRoPE maintains a fixed-size, continuously evolving memory that adapts to new content without increasing memory cost with video length.This distinguishes it from static anchors and discrete selection strategies.
  • Positional extrapolation: Prior positional-extrapolation methods store keys with RoPE already applied, whereas MemRoPE separates content storage from positional assignment.Online RoPE Indexing enables temporal aggregation without conflicting rotary phases.

3. Background

Autoregressive video diffusion denoises sequential chunks while reusing cached key-value states, but finite caches require eviction. RoPE supplies relative positional attention, yet cache-time rotation permanently binds keys to their original positions.

  • Autoregressive inference: Autoregressive video diffusion generates video one frame chunk at a time using previously generated frames in the KV cache.The transformer denoises a noisy input through a small number of diffusion steps.
  • Autoregressive inference: Cached key-value pairs are appended for subsequent conditioning, while causal attention allows past states to be reused without redundant computation.Because storing all past states is infeasible, sliding windows evict older frames with FIFO.
  • Cache organization: MemRoPE organizes the cache into sink, memory, and recent-token tiers, absorbing old frames into long- and short-term memories when the local window fills.The cache is concatenated with the current noisy chunk before entering the DiT.
  • Rotary Position Embedding: 3D-RoPE encodes temporal and spatial coordinates through rotation matrices for query and key tokens.The formulation is designed so attention depends on relative token distance.
  • Rotary Position Embedding: Standard RoPE rotates keys when they enter the cache, permanently binding each key to its original spatiotemporal position.Values remain unrotated and are cached as-is.

4. Method

MemRoPE combines evolving dual-stream memory with position-free key caching and online RoPE assignment to retain context and support unbounded video generation in a fixed-size cache.

  • 4. Method: Unlike static sinks and discrete token selection, MemRoPE maintains a continuously evolving representation as the video unfolds.Existing approaches either discard context or converge to stale token sets that can shift abruptly during updates.
  • 4.1. Memory Tokens: MemRoPE uses Memory Tokens to continuously compress past keys into long-term and short-term streams via exponential moving averages.The long-term stream accumulates generation history, while the short-term stream tracks recent dynamics.
  • 4.1. Memory Tokens: The dual EMA streams maintain persistent identity and recent dynamics while keeping memory size constant regardless of video length.This fixed budget supports unbounded generation.
  • 4.2. Online RoPE Indexing: Conventional RoPE keys cannot be averaged across timesteps because incompatible rotary phases make the resulting key invalid for relative-position attention.RoPE does not distribute over addition, so aggregation of rotated keys breaks positional structure.
  • 4.2. Online RoPE Indexing: Online RoPE Indexing stores raw unrotated keys and dynamically assigns contiguous block-relative indices at each attention step.Keys and queries receive RoPE on the fly, while value states are not rotated.
  • 4.2. Online RoPE Indexing: Recomputing indices from zero keeps every position within the fixed cache size C = S+2M+L and avoids positional extrapolation beyond the training range.The cache contains sink, memory, and local-window regions, with memory summarizing evolving history through dual EMA.

5. Experiments

Experiments evaluate MemRoPE across durations, base models, automated metrics, qualitative comparisons, user preferences, VLM stability, and memory ablations. Results consistently favor MemRoPE for long-range visual quality and subject consistency, while its EMA hyperparameters remain robust.

  • Setup: MemRoPE is evaluated on Self-Forcing and LongLive using 480 × 832 videos at 16 fps, with VBench-Long metrics and durations extending to 1 hour.The protocol includes 128 prompts at 120 and 240 seconds, 20 at 480 seconds, and 10 at 1 hour.
  • Quantitative results: MemRoPE achieves the highest average score across 120- and 240-second settings on both Self-Forcing and LongLive.Deep Forcing falls 1.38 points below Self-Forcing at 240 seconds, while its gains on LongLive nearly vanish at that duration.
  • Qualitative results: MemRoPE preserves subject count, identity, and background consistency throughout 2-minute generations, whereas baselines show degradation, color instability, or structural collapse.The qualitative comparison covers both Self-Forcing and LongLive.
  • Ultra-long generation: At 480 seconds, MemRoPE exceeds ∞-RoPE by over 3 points on Self-Forcing and 1 point on LongLive in average score.At 1 hour on LongLive, MemRoPE leads across all six metrics, while Online RoPE Indexing enables generation beyond the 1024-frame limit.
  • User study and VLM evaluation: Participants consistently preferred MemRoPE across all six perceptual dimensions, and VLM evaluation gave it the highest stability scores.The user study included 30 participants comparing 20 pairs of 120-second videos per participant.
  • Ablations: Combining long-term and short-term memory yields the highest average and best Imaging Quality, while changing EMA decays alters the average score by less than 0.7.Temporally consistent regions attend more to long-term memory, whereas rapidly changing regions attend more to short-term memory.
  • Cache management: Continuous EMA aggregation absorbs evicted frames smoothly, unlike Deep Forcing’s discrete selection, which rarely updates and causes abrupt visual shifts.This comparison concerns training-free cache management beyond static attention sinks.

6. Conclusion

MemRoPE combines dual EMA memory streams with Online RoPE Indexing to preserve evolving context during unbounded video generation in a fixed-size cache. Its effectiveness is broad across tested durations and protocols, but quality remains bounded by the frozen base model and lossy memory compression.

  • Conclusion: MemRoPE smoothly preserves evolving context through dual EMA streams, while Online RoPE Indexing applies relative positions at attention time for fixed-size unbounded generation.The conclusion frames long-horizon coherence as remembering past frames better rather than retaining more frames.
  • Conclusion: Comprehensive experiments validate MemRoPE across all tested durations and evaluation protocols.
  • Limitations: Per-frame quality is bounded by the frozen base model, and lossy EMA aggregation may limit precise recall of distant content.The paper identifies learned memory compression as a possible direction for future work.

A. Quantitative Results on Shorter Videos

MemRoPE achieves the highest average scores on both Self-Forcing and LongLive at 30 and 60 seconds, outperforming competing methods even before substantial degradation accumulates.

  • A. Quantitative Results on Shorter Videos: The evaluation reports VBench-Long averages at 120, 240, 480 seconds, and 1 hour, with additional 30- and 60-second results.The shorter-duration results test effectiveness where cumulative degradation is minimal.
  • A. Quantitative Results on Shorter Videos: CausVid is included as an additional baseline but suffers from train-inference distribution mismatch and worsening over-exposure artifacts at longer durations.Self-Forcing is adopted as a base model for comparison.
  • A. Quantitative Results on Shorter Videos: MemRoPE already outperforms all competing methods at 30 seconds, where degradation has not yet accumulated significantly.This indicates benefits from the outset rather than only compensation for long-horizon drift.

B. Ablation on Position-Free Caching

Position-free EMA aggregation improves over base models, while retaining RoPE during aggregation yields smaller gains because different rotary phases conflict. MemRoPE therefore supports well-defined temporal memory compression by decoupling cached content from position.

  • B. Ablation on Position-Free Caching: Averaging keys with different rotary phases produces representations that do not correspond to a valid temporal position.RoPE does not distribute over addition, motivating position-free aggregation.
  • B. Ablation on Position-Free Caching: The ablation compares MemRoPE’s position-free EMA against an EMA variant that retains RoPE embeddings during aggregation.This isolates the effect of removing positional embeddings before temporal aggregation.
  • B. Ablation on Position-Free Caching: MemRoPE improves average performance by +0.84 on Self-Forcing and +0.53 on LongLive, compared with +0.71 and +0.37 for aggregation with RoPE.The deltas denote average improvement over the respective base models.
  • B. Ablation on Position-Free Caching: Both position-free EMA and EMA with RoPE-rotated keys improve over their respective base models.The comparison confirms that temporal aggregation is broadly beneficial.

C. VBench-Long Score Across Durations

Across five durations from 30 seconds to 480 seconds, both methods degrade as duration increases, but MemRoPE declines more slowly and its performance advantage widens monotonically.

  • C. VBench-Long Score Across Durations: MemRoPE exhibits a significantly flatter VBench-Long score slope than ∞-RoPE as duration increases.The comparison uses 30, 60, 120, 240, and 480 seconds on Self-Forcing with a consistent 20-prompt subset.
  • C. VBench-Long Score Across Durations: The performance gap between MemRoPE and ∞-RoPE widens monotonically across the evaluated durations.The widening gap demonstrates increasing benefits at longer generation horizons.
  • C. VBench-Long Score Across Durations: MemRoPE maintains higher VBench-Long average scores across all durations.Figure 8 reports the comparison on Self-Forcing.

D. Qualitative Comparisons

Qualitative comparisons show that MemRoPE preserves subject identity, visual fidelity, scene structure, and background appearance more consistently than competing methods from two minutes through one hour. Ablations further indicate that both memory streams are needed for scene integrity, while the compact cache adds negligible overhead in the reported setting.

  • D. Qualitative Comparisons: The qualitative two-minute comparison includes Self-Forcing, Deep Forcing, and ∞-RoPE as competing methods.The comparison is performed on two prompts.
  • D. Qualitative Comparisons: At two minutes, MemRoPE maintains consistent subject identity and visual fidelity on both Self-Forcing and LongLive.The competing methods show subject shape changes, color collapse, appearance drift, or color corruption.
  • D. Qualitative Comparisons: At one hour, MemRoPE preserves subject appearance and background color tone more stably than ∞-RoPE.Frames are sampled at 12-minute intervals from LongLive videos.
  • D. Qualitative Comparisons: Only the combined long-term and short-term memory configuration preserves background architecture and subject appearance throughout the sequence.The ablation is conducted on LongLive at 60 seconds.
  • D. Qualitative Comparisons: MemRoPE adds negligible inference overhead at the same cache size on LongLive.On Self-Forcing, its compact three-tier cache uses C = 12 instead of the original sliding window’s C = 21 and is faster.

F. VLM Evaluation and User Study Details

The paper evaluates generated videos using both VLM scoring and a user study. The VLM rates exposure stability on a five-point scale, while participants compare anonymized video pairs across six dimensions.

  • VLM Evaluation: Gemini 3.1 Pro evaluates generated videos using a 5-point exposure stability scale.The evaluation follows Self-Forcing++ and includes detailed reasoning.
  • VLM Evaluation: The exposure rubric ranges from catastrophic exposure to progressively less severe over- or under-exposure problems.The supplied rubric defines scores from 0 through at least 4, with higher scores indicating more limited exposure issues.
  • VLM Evaluation: Figure 12 presents example evaluations from the VLM-based assessment.
  • User Study: Participants compare 20 pairs of side-by-side videos generated from the same prompt by two anonymized methods.The interface includes a welcome page with instructions and an evaluation page for pairwise comparisons.
  • User Study: The user study asks about Color Consistency, Subject Consistency, Background Consistency, Text Alignment, Motion Smoothness, and Overall Preference.
Loading 2603.12513v1…