PaperlayerView exact revision on arXiv

Source-linked AI summary

Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation

Yuxuan Bian, Zeyue Xue, Songchun Zhang, Shiyi Zhang, Weiyang Jin, Yaowei Li, Junhao Zhuang, Haoran Li, Jie Huang, Haoyang Huang, Nan Duan, Qiang Xu

arXiv:2606.04527v1cs.MMcs.CVcs.GR

TL;DR

Long-horizon autoregressive video generation is constrained by unbounded KV-cache memory and temporal RoPE extrapolation. Echo-Infinity learns evolving Memory Queries and uses a unified relative-RoPE schedule, achieving state-of-the-art results and promising 24-hour, over-1.3-million-frame real-time generation.

  • Problem

    Long-horizon autoregressive video generation faces unbounded KV-cache growth and temporal RoPE indices exceeding the training range.

  • Method

    Echo-Infinity learns compact Memory Queries to filter, abstract, and compress evicted history at constant cost, alongside bounded relative RoPE during training and inference.

  • Results

    Echo-Infinity achieves state-of-the-art performance across long, short, and interactive video generation, with over 24-hour and 1.3M-frame real-time rollouts at 18.5 FPS.

  • Takeaways & Limitations

    The results demonstrate a promising path toward real-time infinite video generation with persistent memory.

  • Takeaways & Limitations

    Interactive-generation optimization remains open, and stability may decrease for hour-scale or longer videos with highly dynamic scenes.

Abstract

from arXiv · show

We present Echo Infinity, an autoregressive (AR) framework towards real-time infinite video generation that employs a learnable evolving memory to dynamically filter, abstract, and compress any-length history at constant cost. Existing methods mainly curate memory with predefined KV-cache schedules, fixed-ratio heuristic compression, or inference-time RoPE adaptation. These designs inevitably lose historical information and amplify compounding errors due to their limited cache window and ignorance of autoregressive generation noise. Inspired by human memory consolidation, Echo-Infinity replaces handcrafted memory curation with learnable Memory Query, which are updated by attention and a gating mechanism when past frames are evicted from the local window. The queries are optimized end-to-end with the video diffusion transformers (DiTs), forming an evolving memory that supports arbitrary compression ratios with constant computation independent of video length. They also act as a generalizable generation prior, improving quality even when only the optimized initial state is used. We further introduce Unified Relative RoPE Recipe, which anchors the sink frames to start from id 0 and lets the newest frame id grow at most to the DiTs' pretrained maximum temporal RoPE id throughout training and inference, freeing the model from the finite RoPE constraint and closing the train-test RoPE extrapolation gap. In long and short video generation, Echo-Infinity achieves state-of-the-art performance, and, to our knowledge, demonstrates promising 24-hour (>1.3 M frames) real-time rollouts for the first time, suggesting a practical path toward infinite video generation.

1 Introduction

Echo-Infinity addresses long-horizon autoregressive video generation by replacing handcrafted memory curation with trainable evolving Memory Queries and by applying a unified bounded relative-RoPE schedule during training and inference. It achieves state-of-the-art results across video settings and demonstrates 24-hour real-time generation exceeding 1.3 million frames.

  • Motivation: Long-horizon AR video generation faces unbounded KV-cache memory costs and temporal RoPE indices that exceed the training range, causing degradation and overflow.These are the two bottlenecks identified for real-time streaming video diffusion transformers.
  • Motivation: Existing approaches use window truncation or handcrafted KV-cache rules that discard distant history, obey fixed cache budgets, and ignore compounding historical error.The introduction groups prior methods into window truncation and hand-crafted KV-cache management strategies.
  • Method: Echo-Infinity uses end-to-end trainable Memory Queries to filter, abstract, and compress arbitrary-length history into an evolving memory state at constant cost.The queries update from evicted KV caches and use gated residuals to control memory overwriting.
  • Method: Unified Relative RoPE Recipe keeps active temporal RoPE ids within the pretrained range during both training and inference, avoiding train-test extrapolation and overflow.Older frames are rotated backward once the newest temporal id reaches the pretrained maximum fmax.
  • Results: 24 hours, >1.3M frames, and 18.5 FPS are achieved on a single NVIDIA H100 with only 10.6% throughput overhead over a memory-free baseline.Echo-Infinity also reports state-of-the-art performance on long, short, and interactive video generation.

2 Related Work

Related work has advanced video generation toward minute-scale and real-time streaming, but existing approaches still face memory and temporal-position bottlenecks. Prior methods truncate, heuristically curate, or compress history and use inference-time RoPE fixes, whereas Echo-Infinity introduces trainable evolving memory and unified relative RoPE.

  • Streaming video generation: Bidirectional video DiTs incur quadratic attention costs and bidirectional generation, making them unsuitable for streaming long videos.Autoregressive methods, Diffusion Forcing, DMD distillation, and Self-Forcing have pushed generation toward minute-scale and real-time streaming.
  • Memory management: Window-truncation methods retain a bounded local window and sink frames while discarding other history, causing inevitable information loss.This is one of the main strategies for addressing the unbounded KV cache.
  • Memory management: Hand-crafted KV-cache methods use rule-based schedules to decide which evicted key-value states to retain, including textual retrieval for memory-bank curation.These methods augment the local window with specially selected evicted KVs.
  • Echo-Infinity’s positioning: Echo-Infinity addresses these limitations with trainable Memory Queries that filter, abstract, and compress evicted history, plus relative RoPE used during both training and inference.Its relative RoPE anchors sink frames at id 0 and supports evolving compression of arbitrarily long histories.
  • Memory management: Compressed-history methods improve temporal context but remain tied to predefined compression ratios, schedules, or separate reconstruction and compression stages.Memorize-and-Generate compresses historical information into compact KVs consumed by a separate generator.
  • Temporal positional encoding: 20 is the cited maximum temporal RoPE index for Wan-2.1, after which autoregressive rollouts can overflow and suffer quality collapse.Training-free relative-RoPE methods mitigate inference-time overflow but leave train-test inconsistency unresolved.

3 Method

Echo-Infinity combines a three-tier KV cache with end-to-end learnable memory queries that filter, abstract, and compress evicted history at constant cost. A unified relative RoPE schedule keeps temporal identifiers within the pretrained range during training and inference.

  • Training Strategy: Training uses standard-tuning on 5s videos followed by streaming long-tuning on 60s videos with one prompt switch.During long tuning, Q and cached states are detached only across 5-s sub-clip boundaries while optimization continues within each sub-clip.
  • Three-Tier KV Organization: Echo-Infinity uses sink frames, a recent local window, and learnable evolving memory queries to organize each layer’s KV cache.The sink block provides a persistent global anchor, the local window retains recent frames, and queries summarize evicted history.
  • Memory Queries: The memory queries Q are optimized end-to-end with DMD-based autoregressive video generation to support arbitrary compression ratios.Q is a compact representation whose state is updated as the local window evicts frames.
  • Memory Query Update: When frames are evicted, a cross-attention encoder refreshes Q and a sigmoid-gated residual combines the previous and refreshed states.The learned update selects, abstracts, and compresses useful information from the evicted key-value cache into the evolving memory.
  • Unified Relative RoPE: The unified relative RoPE schedule anchors sink frames at 0 and reschedules other temporal ids so all remain within [0,fmax].This avoids overflow risks and closes the train-test mismatch caused by generating beyond the pretrained temporal RoPE range.

4 Experiments

Echo-Infinity achieves the strongest reported long-video results with real-time throughput, improving user preference and maintaining stable quality across 1- and 24-hour rollouts. Its memory and RoPE designs also improve short-video generation, interactive multi-prompt generation, and long-horizon ablations.

  • Long-Video Generation: Echo-Infinity performs best at 30s and 240s horizons while maintaining real-time throughput of 18.5 FPS.It is compared with LongLive, MemFlow, Memorize-and-Generate, and ∞-RoPE.
  • Long-Video Generation: 59.53 vs. 14.73 user preference at 30s demonstrates Echo-Infinity’s notable advantage over competing methods.The evaluation uses 30 users and 50 randomly sampled results, with selections based on overall quality and semantic alignment.
  • Infinite Video Generation: Echo-Infinity preserves stable visual quality during real-time inference over 1-hour and 24-hour videos generated from 15 MovieGen prompts.The evolving memory queries and unified relative RoPE recipe enable these long rollouts by removing fixed positional limits and train-inference RoPE mismatch.
  • Interactive Generation: Memory Queries and unified relative RoPE improve long-horizon quality in multi-prompt 60s interactive generation across 100 six-prompt narrative scripts.The evaluation uses VBench-Long metrics for visual quality and CLIP scores for clip-wise semantic adherence.
  • Short-Video Generation: On standard 5s VBench generation, memory updates raise total, quality, and semantic scores to 85.35, 86.32, and 81.49, respectively.Without memory updates, the optimized initial Q already reaches a total score of 84.57 and outperforms all compared baselines.
  • Ablation Studies: Removing Memory Queries substantially degrades consistency and dynamic degree, while replacing relative RoPE with absolute RoPE degrades most quality and consistency metrics.The ablation uses 240s generation on 128 randomly sampled MovieGen prompts; absolute RoPE causes failures from unseen positional indices.

5 Conclusion · A More Visualizations

Echo-Infinity combines learned evolving Memory Queries with Unified Relative RoPE to address unbounded memory growth and positional extrapolation in infinite video generation. The conclusion also identifies open challenges for interactive and dynamic-scene generation and provides additional visualization cases.

  • 5 Conclusion: Echo-Infinity presents an end-to-end memory-learning framework for real-time infinite video generation.
  • 5 Conclusion: Learned evolving Memory Queries filter, abstract, and compress arbitrary-length history at constant cost instead of using handcrafted schedules or fixed-ratio compression.
  • 5 Conclusion: Unified Relative RoPE keeps temporal indices within the trained range during training and inference.
  • 5 Conclusion: Together, the memory framework and RoPE recipe remove bottlenecks from unbounded memory growth and positional extrapolation.
  • Limitations and Future Works: Interactive generation remains open because the memory queries and unified relative RoPE recipe target long-video quality and consistency.
  • Limitations and Future Works: Reduced stability may occur for hour-scale or longer videos with dynamic scenes because of base-model scale and generative capacity.
  • A More Visualizations: Additional visualizations cover long interactive generation and 240s and 30s video generation in Figures 8–10.

B Implementation Details · C Related Works · C.1 Long Video Generation

Echo-Infinity is implemented by adapting Wan2.1-T2V-1.3B into a causal streaming model with learnable memory and unified relative RoPE, then tuning it for long-video generation. The related-work context motivates autoregressive chunk generation because bidirectional DiTs require full-clip denoising and incur quadratic attention costs.

  • B Implementation Details: Wan2.1-T2V-1.3B produces 5-second clips at 16 FPS and 832 × 480 resolution.Echo-Infinity is implemented on this pretrained backbone.
  • B Implementation Details: The pretrained model is adapted into a few-step causal-attention model using causal-forcing DMD on VidProM data, with Memory Queries and Unified Relative RoPE enabled.KV caches remain in pre-RoPE states so the relative RoPE recipe can rotate them to desired IDs during attention and memory updates.
  • B Implementation Details: 960 H200 GPU-hours support the main training run, comprising 400 adaptation iterations and 3,000 streaming long-tuning iterations.The two stages take about 3 hours and 12 hours, respectively, on 64 NVIDIA H200 GPUs.
  • B Implementation Details: Automatic evaluations average scores across five random seeds for each prompt, while baselines use the same backbone, checkpoints, resolution, sampling steps, prompts, and hardware.Benchmarks follow prior long-video-generation and VBench/VBench-Long protocols.
  • C Related Works · C.1 Long Video Generation: Modern video diffusion models mainly use bidirectional DiTs that jointly denoise entire short clips and have advanced diverse downstream applications.This summarizes the broader video-generation context in related work.
  • C.1 Long Video Generation: Quadratic attention and full-clip denoising prevent bidirectional DiTs from emitting frames in a streaming long-video setting.Autoregressive approaches instead generate video chunk by chunk under causal attention, following the paradigm of large-language-model inference.

C.2 Memory Mechanisms in Long Video Generation

Existing long-video memory mechanisms either discard history through bounded windows, rely on handcrafted KV-cache schedules, or retain unbounded memory under fixed-ratio compression. Echo-Infinity instead recurrently consolidates evicted history into a fixed-capacity Memory Query state, allowing the effective compression ratio to grow with video length while preserving constant computation and memory.

  • Existing memory mechanisms: Window truncation retains a bounded local window and sink frames but inevitably discards historical information.Handcrafted KV-cache management instead uses rule-based scheduling to select evicted keys and values for retention.
  • Fixed-ratio compression: ⌈T/C⌉ units remain after retaining one compressed unit per C historical units, so fixed-ratio compression still grows with sequence length T.Here, T is the number of generated frames and C is the fixed compression ratio.
  • Fixed-ratio compression: C = Ω(T/M) is required to keep fixed-ratio compression within a constant memory budget M, making compression increasingly aggressive over time.The compression ratio must grow with the generation horizon, changing the operating point as the video length increases.
  • Echo-Infinity memory: Echo-Infinity recurrently consolidates all evicted history into Q with TQ = NQS tokens under a fixed-capacity memory design.This avoids the unbounded-memory failure of fixed-ratio compression.
  • Echo-Infinity memory: The effective compression ratio grows with video length while the algorithm and memory budget remain unchanged.This follows from keeping the consolidated memory capacity fixed as the generated sequence length T increases.
  • Architecture comparison: Unlike VideoSSM’s architecture-level state-space memory, Echo-Infinity injects layer-shared Memory Queries as a plug-in KV source without changing DiT attention blocks.The mechanism is described as compatible with pretrained attention backbones and the unified Relative RoPE recipe.

C.3 Rotary Positional Embedding for Long Video Generation

Autoregressive video generation causes temporal RoPE indices to exceed the bounded range seen during pretraining, producing overflow and rapid quality collapse. Existing inference-only relative-RoPE patches leave the train-test mismatch embedded in the backbone, whereas Echo-Infinity eliminates it during both training and inference.

  • RoPE mismatch: Temporal RoPE indices can exceed the pretraining range during autoregressive rollouts, causing eventual overflow and rapid quality collapse.Wan-2.1, for example, has a pretraining maximum temporal index of 20.
  • RoPE mismatch: Inference-only relative-RoPE methods mitigate overflow risk without changing the backbone’s training index distribution, leaving train-test inconsistency unresolved.These methods operate on frozen backbones and apply relative RoPE only at inference time.
  • RoPE mismatch: Echo-Infinity eliminates the temporal RoPE mismatch during both training and inference.The supplied passage identifies this as Echo-Infinity’s distinguishing contribution, though the specific recipe is not included here.

D Ablation Studies · E User Study

The ablations show that Memory Queries and Unified Relative RoPE are essential to Echo-Infinity’s consistency, dynamics, quality, and generalization. A forced-choice study evaluates human preferences across 30s and 240s videos using anonymized comparisons focused on visual and temporal criteria.

  • D Ablation Studies: 128 randomly sampled MovieGen prompts are used to ablate Echo-Infinity on 240s video generation.Results are summarized in Table 5.
  • D Ablation Studies: Removing Memory Queries substantially degrades consistency and dynamic degree because historical information is inevitably lost.
  • D Ablation Studies: Replacing Unified Relative RoPE with original absolute RoPE degrades most quality and consistency metrics by requiring unseen positional indices at inference.The passage attributes this degradation to severe out-of-distribution generalization failure.
  • E User Study: 30 video generation researchers evaluate anonymized, randomized method outputs in forced-choice studies for 30s and 240s generation.Each duration uses 50 randomly sampled benchmark prompts, with one video per method for each prompt and duration.
  • E User Study: Participants select one best video by jointly considering overall visual quality, temporal consistency, motion naturalness, and caption alignment.Model names are hidden, and display order is independently randomized across trials and participants to reduce positional bias.
  • E User Study: 60 seconds must be watched from each 240s video, including the first 10 seconds and final 10 seconds spanning 230–240s.Participants may inspect the 10–230s middle segment freely but must watch at least 40 additional seconds before selecting.
  • E User Study: 1,500 total votes are collected for each duration from 50 prompt-level result groups and 30 participants.User preference rate is computed as the percentage of times each method’s result is selected.

F Limitations and Future Works.

Echo-Infinity improves long and infinite video generation, but interactive optimization, stability in highly dynamic hour-scale videos, and quantitative ultra-long-video evaluation remain open.

  • Limitations and Future Works.: Interactive video generation remains an open optimization target for the memory queries and unified relative RoPE recipe.These components were designed for long and infinite video generation rather than specifically interactive settings.
  • Limitations and Future Works.: Base model scale and generative capacity may reduce stability for hour-scale or longer videos with highly dynamic scenes.The limitation is attributed to the capacity of the underlying model.
  • Limitations and Future Works.: Quantitative evaluation of ultra-long videos remains an open challenge.The paper identifies ultra-long-video measurement as an unresolved evaluation problem.

G Broader Impacts

Echo-Infinity offers learnable memory and unified RoPE designs that support coherent hour-scale real-time video generation. Its capabilities also create misuse risks, motivating provenance safeguards, usage policies, and privacy-conscious deployment.

  • Positive impacts: Echo-Infinity enables coherent hour-scale long-video generation in real time through learnable Memory Queries and a unified Relative RoPE recipe.Memory Queries compress arbitrary-length history at constant per-step cost, while Relative RoPE aligns training and inference temporal indices.
  • Negative impacts: Coherent multi-hour video generation could amplify disinformation, non-consensual likeness generation, and impersonation.These risks parallel those of other capable video generation models but may scale with longer coherent outputs.
  • Mitigation: Recommended mitigations include interoperable watermarks, synthetic-media provenance signals, and policies prohibiting non-consensual personal-identity generation.Training uses publicly released prompt collections without personally identifying data, and the model does not decide about specific individuals or groups.
Loading 2606.04527v1…