Source-linked AI summary
FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion
Yu Lu, Junjie Yang, Piotr Koniusz, YuXin Song, Yi Yang
TL;DR
Long-video autoregressive generators face growing KV-cache storage and attention costs as history accumulates. FadeMem organizes history into a distance-aware temporal hierarchy within a fixed cache, improving long-range consistency while largely preserving visual quality across multi-minute and hour-long generation.
Problem
Autoregressive video generation incurs storage and attention costs that grow with video length, limiting long-video synthesis.
Method
FadeMem uses a unified power-law memory schedule that preserves recent KV blocks finely while progressively merging distant adjacent history into coarser spans under a fixed cache budget.
Results
FadeMem improves long-range consistency without architectural changes while largely preserving visual quality, remaining effective for four-minute and hour-long videos.
Takeaways & Limitations
The results indicate that long-video coherence depends on representing retained history at temporal resolutions appropriate to its distance.
Takeaways & Limitations
Its fixed allocation schedule may be suboptimal for abrupt scene transitions, fast motion, frequent semantic changes, or detail-sensitive objects.
Abstract
from arXiv · showhide
Autoregressive video generators synthesize long videos by generating successive temporal segments, but their historical KV cache grows with video length. Existing bounded-cache methods reduce this cost with local windows, sink tokens, or compressed memory states, yet they usually assign fixed roles to different parts of the history. We propose FadeMem, a distance-aware KV memory consolidation mechanism that organizes historical KV blocks into a temporal hierarchy under a fixed cache budget. This design is motivated by frequency-dependent temporal decay: fine details decorrelate quickly, while coarse scene structure and identity remain useful over longer horizons. During generation, new history is inserted as fine-grained entries, while older adjacent entries are progressively merged under a power-law temporal allocation schedule, yielding a dense-near, sparse-far memory within one cache. Without architectural changes, FadeMem improves long-range consistency while largely preserving visual quality, and lightweight adaptation further enhances motion dynamics and visual fidelity. Using the same unified schedule under a fixed cache budget, FadeMem also remains effective over multi-minute and hour long video generation and reduces peak memory under a matched KV budget.
Introduction · Related Work
Long-video diffusion generation faces storage and attention costs that grow with video length, while existing bounded-cache methods impose fixed temporal roles. FadeMem instead organizes one fixed-budget KV cache into a distance-aware hierarchy, preserving fine recent context and consolidating distant history to improve long-range consistency without architectural changes.
- Introduction: Introduction: Storage overhead and attention computation grow with video length, making long-video generation impractical.
- Introduction: Introduction: Existing bounded-cache methods retain local windows, sink tokens, or compressed memory states, but typically use manually specified cache partitions with fixed temporal roles.
- Introduction: Introduction: Frame correlations generally decrease with temporal distance, while high-frequency details decorrelate faster than lower-frequency scene structure.This frequency-dependent decay motivates allocating finer memory resolution near the present and coarser resolution farther back.
- Introduction: Introduction: FadeMem maintains a single bounded memory of representative historical KV blocks and temporal spans, using a power-law schedule to consolidate adjacent older entries.New KV blocks are initially stored individually, and consolidation occurs when the fixed entry budget is exceeded.
- Introduction: Introduction: The resulting hierarchy preserves recent context for short-term dynamics and compact long-range anchors for scene layout, appearance, and identity without architectural modification.FadeMem supports inference-time use and light fine-tuning.
- Introduction: Introduction: Without retraining, FadeMem improves long-range subject, background, and temporal consistency while largely preserving base-generator visual quality.Lightweight adaptation further improves motion dynamics and visual quality; the method remains effective for four-minute and hour-long videos and reduces peak memory under a matched KV budget.
- Autoregressive Video Generation: Autoregressive Video Generation: Recent approaches extend diffusion generation through long-context modeling, streaming generation, or next-token-style prediction for long-horizon and streaming synthesis.Related methods include causal KV-cache reuse, self-generated histories, error correction, rolling denoising windows, and streaming-oriented tuning.
- Memory in Long Video Generation: Memory in Long Video Generation: Memory-growth methods include sliding local windows, persistent early-frame or sink-token anchors, positional remedies, token compression, memory streams, and adaptive retrieval.
Method
FadeMem replaces the linearly growing full-history KV cache with a fixed-budget, distance-aware memory that preserves fine temporal detail nearby and consolidates distant history into progressively coarser entries. Its online update inserts each new KV block and merges adjacent entries according to a power-law distance schedule while maintaining temporal order.
- Bounded memory: FadeMem bounds cache size to M entries, storing at most MN token-level key/value vectors per layer while representing increasingly long history.The full-history cache grows linearly with video length, whereas the bounded memory satisfies |M_t| ≤ M.
- Motivation: Frequency-dependent temporal decay motivates denser memory for recent states and coarser representations for distant history.High-frequency details and local motion decorrelate rapidly, whereas low-frequency scene layout, identity, and global appearance remain correlated longer.
- Memory update: Each generation step inserts a new one-unit KV entry; when the budget is exceeded, FadeMem consolidates one adjacent pair.The update is online and local, adding one entry and performing one consolidation when necessary.
- Distance-aware scheduling: FadeMem maps temporal distance through a power-law warp and merges the adjacent pair with the smallest warped-space gap, producing dense-near, sparse-far coverage.Nearby entries remain relatively separated, while distant entries are brought closer together; adjacent-only merging preserves ordering.
- Consolidation operator: Consolidation replaces two entries with a span-weighted representative KV block, using fixed square-root span weighting by default.The merged metadata records total covered length and a span-weighted temporal center, while the replacement keeps memory size fixed.
- Boundary and positional handling: The newest entry is protected from immediate consolidation, and the first-frame KV block is preserved as a global anchor within the same ordered schedule.RoPE temporal-position handling and complete update pseudocode are provided as implementation details.
Experiments
Experiments show that FadeMem improves the balance among long-range consistency, visual quality, motion dynamics, and memory efficiency across 60-second and 240-second video generation. Ablations further support weighted-average consolidation, β = 0.3, and retaining a first-frame global anchor as the default design choices.
- 60-second quantitative results: In 60-second generation, FadeMem-TF raises the average score from 80.55 for LongLive to 80.93 while achieving the highest subject consistency, background consistency, and motion smoothness.FadeMem-TF maintains a competitive Dynamic Degree of 42.19 without additional training.
- 60-second quantitative results: FadeMem-FT reaches Dynamic Degree 47.06, Aesthetic Quality 61.41, and Imaging Quality 70.70, while exceeding MemFlow by 1.04 and 1.00 points, respectively.The fine-tuned variant combines the highest reported motion, aesthetic, and imaging scores.
- Long-duration results: At 240 seconds, FadeMem achieves the best aggregate consistency, Dynamic Degree, aggregate quality, and weighted average, demonstrating effectiveness as generated history grows.The evaluation uses the same protocol as the 60-second setting.
- Efficiency: Under a matched 15-slot KV budget, FadeMem uses the lowest peak memory at 22.72 GiB while achieving 11.24 FPS versus MemRoPE’s 11.85 FPS.Measurements use BF16, batch size 1, and a single RTX PRO 6000 GPU; core FPS excludes text encoding, VAE decoding, and disk I/O.
- VLM-based evaluation: FadeMem attains the highest Gemini 3.1-Pro visual stability score of 4.84 in the 60-second evaluation.This VLM-based result complements the VBench-Long metrics under the Self-Forcing++ protocol.
- Ablations: Ablations favor β = 0.3, weighted-average consolidation, and retaining the first-frame anchor for stable identity, scene preservation, global coherence, and motion progression.Max pooling over-compresses cached states, while removing the anchor sharply reduces Dynamic Degree despite improving some consistency metrics.
Conclusion
FadeMem is a distance-aware memory consolidation mechanism for long-horizon autoregressive video generation under a fixed KV cache budget. It uses a unified ordered memory whose temporal resolution fades with distance, preserving recent details while merging distant history into coarser anchors.
- FadeMem targets long-horizon autoregressive video generation under a fixed KV cache budget.
- Motivated by distance-dependent spectral decay, FadeMem replaces manually partitioned cache designs with a unified ordered memory.
- Recent entries remain fine-grained for local dynamics, while distant entries merge progressively into coarser span-level anchors for scene layout and appearance.
Limitations
FadeMem’s fixed allocation schedule may be suboptimal for abrupt scene transitions, fast motion, frequent semantic changes, and detail-sensitive objects. It can also exhibit transient motion stalls and inherits several limitations of the base generator, including prompt misalignment and semantic drift.
- Schedule limitations: The fixed allocation schedule may be suboptimal for abrupt scene transitions, fast motion, frequent semantic changes, and detail-sensitive objects.These conditions can challenge FadeMem’s distance-aware consolidation strategy.
- Generation instability: Stochastic initial noise can occasionally cause transient motion stalls, although rollout may recover as states are consolidated and prompt and first-frame influence regain strength.Recovery can occur progressively during rollout.
- Inherited limitations: FadeMem inherits base-generator limitations including prompt misalignment, implausible physical dynamics, weak action planning, and semantic drift.These limitations are not specific to the memory consolidation mechanism.
Supplementary Material · FadeMem Memory Update
FadeMem’s online update appends each new KV entry and, once the cache exceeds its budget, merges the adjacent pair with the smallest distance-aware growth cost. The update excludes the newest pair and treats a first-frame anchor specially when enabled.
- FadeMem Memory Update: With global anchors enabled, a selected first-frame anchor retains its original KV block and temporal position 0 while its size is combined.The anchor remains eligible for pair selection but is not averaged with its partner.
- FadeMem Memory Update: Algorithm S1 defines FadeMem’s complete online memory update.It specifies append, capacity checking, pair selection, merging, replacement, and return operations.
- FadeMem Memory Update: Each new entry is appended with its KV block, temporal position t, and size 1.The cache is returned unchanged while its size remains within budget M.
- FadeMem Memory Update: When the cache exceeds M, FadeMem evaluates every adjacent pair except the pair containing the newest entry.This preserves the newest history from immediate consolidation.
- FadeMem Memory Update: The pair with the smallest distance-aware growth cost g_i is selected for consolidation.The cost compares temporal distances using the power β schedule.
- FadeMem Memory Update: For ordinary pairs, the merged size is s_j+s_j+1, temporal position is size-weighted, and KV blocks use square-root-size interpolation weights.The merged entry replaces the selected adjacent pair.
Spectral Analysis Details
The spectral analysis measures frequency-dependent temporal persistence in latent video representations and finds that stable frequency bandwidth contracts approximately as a power law with temporal distance, motivating FadeMem’s allocation schedule.
- Data and preprocessing: The analysis uses 254 valid Pexels videos, divided into 374 non-overlapping clips of 708 RGB frames at 832 × 480 resolution.Native frame rates are retained, so temporal distances are reported in latent-frame steps rather than seconds.
- Latent frequency decomposition: Each 708-frame clip is encoded into 177 latent frames with 16 channels at 60 × 104 spatial resolution, then transformed using an orthonormal 2D DCT.The analysis averages latent channels before applying the DCT and groups coefficients by normalized radial frequency.
- Temporal correlation and stable bandwidth: Temporal correlations are averaged over all 374 clips separately for radial frequency bands, with visualization using low-, middle-, and high-frequency aggregates.For each band, the method identifies a decorrelation horizon as the onset of the near-flat regime after the main correlation decay.
- Temporal correlation and stable bandwidth: The fitted trend indicates that stable frequency bandwidth decreases approximately according to a power law as temporal distance increases.Boundary points pair each band’s decorrelation horizon with its center frequency radius before nonlinear least-squares fitting.
- Motivation: This empirical power-law relationship motivates FadeMem’s temporal allocation, while its memory-allocation exponent β is chosen through ablation rather than set directly to the fitted spectral exponent.Stable frequency bandwidth and memory-entry density are treated as different quantities.
VLM Evaluation Details
The evaluation uses Gemini 3.1-Pro Preview to score exposure stability from complete generated videos under a deterministic protocol. A rubric rates exposure from catastrophic failures to moderate issues, with qualitative examples comparing early and late frames.
- Evaluation protocol: The VLM score measures exposure stability using Gemini 3.1-Pro Preview (gemini-3.1-pro-preview).The protocol follows recent long-video work using VLM judges for exposure quality.
- Evaluation protocol: 128 complete 60-second videos per method are each evaluated once with temperature set to 0.The evaluator receives the complete generated video rather than selected frames.
- Qualitative example: Figure S1 visualizes rubric application with frames near 1 and 58 seconds, while evaluation uses each complete 60-second video.The early and late frames summarize one shared-prompt generation for readers.
- Scoring rubric: The rubric spans catastrophic exposure, severe exposure failure, noticeable exposure problems, and moderate exposure issues.The listed levels describe progressively less severe over- or under-exposure and visibility loss.
Implementation and Reproducibility Notes
FadeMem is implemented within LongLive using chunked latent-frame generation and a bounded KV memory configuration. Evaluation covers standard and long-duration video generation, with efficiency measured under a specified single-GPU BF16 setup.
- Implementation: FadeMem runs on Wan2.1-T2V-1.3B within the LongLive autoregressive generation framework, generating three-latent-frame chunks with four denoising steps.Each memory entry stores one generated latent frame’s KV block.
- Implementation: The default configuration uses 12 historical entries, three current latent-frame slots, 15 KV frame slots per attention computation, and β = 0.3.Memory is updated after each generation step, although the supplied passage ends before describing the update procedure.
- Evaluation: Primary evaluation uses 128 MovieGenBench prompts to generate 60-second videos at 480 × 832 resolution and 16 FPS, evaluated with VBench-Long metrics.Long-duration evaluation extends the same setup to 240 seconds.
- Efficiency: Efficiency is measured with batch size 1 and BF16 precision on a single NVIDIA RTX PRO 6000 Blackwell Server Edition GPU.Core FPS excludes text encoding, VAE decoding, and disk I/O; peak allocated memory is PyTorch’s maximum allocation during one generation request.
Additional Qualitative Results · time
Additional qualitative results show that FadeMem preserves recognizable content and coherent scene evolution across diverse 60-second videos, four-minute continuous rollouts, and one-hour generation under a fixed cache budget. The training-free and lightly tuned variants exhibit complementary long-horizon behaviors.
- Additional Qualitative Results: Additional qualitative evaluations span diverse 60-second generations, matched comparisons of training-free and lightly tuned variants, and continuous rollouts up to one hour.These evaluations provide qualitative evidence across progressively longer generation horizons.
- Additional Qualitative Results: Across diverse 60-second generations, FadeMem preserves recognizable visual content and coherent scene evolution across varied subjects, scenes, and motion patterns.Frames are sampled in temporal order from the corresponding generated videos.
- Additional Qualitative Results: Figure S2 samples frames at 10-second intervals, with each row representing one continuous generated video.The examples cover diverse subjects, scenes, and motion patterns.
- Additional Qualitative Results: FadeMem-TF preserves dandelion seed-head semantics through a macro zoom, whereas FadeMem-FT provides a smoother visual trajectory and higher visual fidelity.The variants are compared at matched timestamps.
- Additional Qualitative Results: The 240-second continuous rollout preserves subject identity, attire, train interior, and surrounding scene while maintaining natural, meaningful temporal evolution.The visualization covers four consecutive 60-second intervals in temporal order.
- Additional Qualitative Results: Figure S4 arranges representative frames temporally across consecutive 60-second intervals to visualize one continuous four-minute generation.The rollout demonstrates content preservation across a substantially longer temporal horizon.
- time: The one-hour rollout maintains a recognizable narrow-alley structure and intense tsunami dynamics throughout generation under a fixed cache budget.Each row covers one consecutive 12-minute interval.
- time: Figure S5 divides the hour-scale generation into consecutive 12-minute intervals, forming a qualitative stress test over an extremely long generated history.The generation uses a fixed cache budget.