Source-linked AI summary
LayerRecall: A State-Conditioned Memory Router for Long-Horizon Consistency in Video Generation
Yixuan Ding, Jiahao Kong, Wei Huang, Ruijie Quan, Yi Yang
TL;DR
Bounded recent context preserves local continuity but can evict historical cues needed for long-range consistency, and access to nonlocal history alone may not ensure effective use. LayerRecall routes current-relevant historical K/V states to backbone-specific memory-sensitive layers and trains this bounded-memory behavior with CHPM, achieving the best overall results on MemoBench and MovieBench while matching its backbone on VBench-Long.
Problem
Recency-based bounded memory can evict returning subjects, objects, scenes, and attributes, while existing access to nonlocal history does not guarantee effective use.
Method
LayerRecall jointly controls current-conditioned historical retrieval and backbone-specific layer-selective K/V injection, while CHPM matches a bounded-memory student to a frozen long-context teacher in prediction space.
Results
LayerRecall achieves the best overall scores on MemoBench and MovieBench while matching LongLive-2.0 on VBench-Long.
Takeaways & Limitations
Selective retrieval and layer-wise injection strengthen distant-memory use while preserving the local pathway, with qualitative self-correction of mismatched attributes during ongoing motion.
Abstract
from arXiv · showhide
Autoregressive video diffusion enables scalable long-video generation by producing chunks from a bounded recent context. While recency-based caching preserves local continuity, it evicts historical cues needed when subjects, objects, scenes, or attributes reappear. Existing memory mechanisms expose models to nonlocal history, but access alone does not ensure effective use. Our analysis reveals that video DiT layers exhibit distinct preferences for current, recent, and distant context, suggesting that long-range memory requires deciding both what to retrieve and where to use it. We introduce LayerRecall, a current-conditioned, layer-selective memory router that retrieves relevant historical K/V states and injects them only into backbone-specific memory-sensitive layers while preserving local attention elsewhere. To reduce reliance on scarce high-quality long-horizon videos and explicit memory-allocation labels, we further propose Cross-Horizon Prediction Matching (CHPM), which uses a privileged long-context reference to supervise the bounded-memory router in prediction space. Across 100 multi-shot evaluation prompts, LayerRecall achieves the best overall results on MemoBench and MovieBench while matching its backbone on VBench-Long, demonstrating stronger long-range recovery without sacrificing local continuity. Qualitative analyses further reveal memory-guided self-correction, whereby initially mismatched local attributes return to their historical appearance without resetting ongoing motion or scene structure. Additional analyses show cross-backbone portability and negligible inference overhead.
1 Introduction
Autoregressive video diffusion extends generation through bounded context, but recency-based memory can evict historical cues needed for later reappearance. LayerRecall addresses this by treating memory use as a joint decision over retrieved content and network depth.
- Bounded recent-context caching preserves local continuity but eventually evicts cues about returning subjects, objects, scenes, and attributes.
- Layer-wise temporal attention preferences motivate allocating historical memory across both content and DiT depth.
- Cross-Horizon Prediction Matching uses privileged long-context predictions to supervise bounded-memory routing without explicit memory-allocation labels or backbone finetuning.
- Across 100 evaluation prompts, LayerRecall achieves state-of-the-art results on both memory-oriented benchmarks while matching its backbone on VBench-Long.
- LayerRecall retrieves history conditionally on the current state and routes K/V states through backbone-specific memory-sensitive layers while preserving local attention elsewhere.
2 Related Works
Related work develops autoregressive video generation and several ways to preserve nonlocal history beyond a bounded local window. These approaches motivate selective historical retrieval while leaving open how retrieved information should be used within the network.
- Autoregressive Video Generation: Autoregressive video methods generate frames or chunks sequentially and reuse cached context for streaming inference and longer rollouts.
- Memory-Augmented Video Generation: LayerRecall complements these directions by routing retrieved historical K/V states selectively rather than exposing every layer uniformly to nonlocal history.
- Long-Horizon Video Generation: Long-horizon systems extend short-video models through rescheduling, overlapping windows, queue-based generation, global-local exchange, bounded context, and fixed anchors.
- Memory-Augmented Video Generation: Memory-augmented methods preserve nonlocal information through compressed states, online parameter memories, memory experts, and selective historical retrieval.
3 Method
LayerRecall combines current-conditioned historical retrieval with fixed, backbone-specific layer-selective injection. CHPM trains this bounded-memory router by matching predictions from a frozen long-context teacher and a frozen short-context student.
- LayerRecall: LayerRecall formulates memory use as deciding what to retrieve and where to use it, using current-conditioned retrieval and layer-selective routing.
- LayerRecall: The method preserves the pretrained local pathway while adding sparse access to historical K/V states.
- What to Retrieve: Current-Conditioned Historical Retrieval: Historical chunks are summarized for scoring, while selected full cached keys and values enter attention within a sink-protected sliding cache.
- What to Retrieve: Current-Conditioned Historical Retrieval: A shared query and current-conditioned residual rank resident historical records independently at each layer, with hard budgeted selection and differentiable straight-through training signals.
- Where to Use: Layer-Selective Memory Injection: Profiled memory-sensitive layers receive retrieved history, while all remaining layers retain the original sink-plus-sliding-window context.
- 3.2 CHPM: Cross-Horizon Prediction Matching: CHPM freezes the shared backbone and optimizes only router parameters to match bounded-memory student predictions to those of a long-context teacher.
- 3.2 CHPM: Cross-Horizon Prediction Matching: Teacher and student maintain independent rollout histories, while detached student rollouts expose the router to the context distribution encountered during autoregressive inference.
4 Experiments
Experiments evaluate LayerRecall on long-range recall, temporal stability, portability, efficiency, and routing ablations. Across these tests, selective historical-memory use improves memory-oriented consistency while preserving local temporal quality with negligible overhead.
- Qualitative Results: LayerRecall restores identities and fine-grained appearances after subjects, objects, or attributes leave the visible scene across three-shot sequences.The examples span people, animals, apparel, instruments, and scene-specific objects, while the intervening shot evolves independently.
- Quantitative Results: LayerRecall ranks first overall on both MemoBench and MovieBench while matching LongLive-2.0 on VBench-Long.The results indicate stronger memory-oriented consistency without compromising the backbone’s reported consistency and motion quality.
- Inference Efficiency: 305.9 to 309.4 seconds per video: selective memory routing introduces negligible end-to-end generation-time overhead under matched H100 settings.The comparison uses the LongLive-2.0 backbone.
- Temporal Stability: 0.60 to 0.38: LayerRecall reduces the high-frequency frame-change ratio and improves motion smoothness and adjacent-frame CLIP/DINO consistency.The temporal-stability analysis compares LayerRecall with all-layer memory routing.
- Effect of CHPM: 0.519 to 0.548: CHPM improves MemoBench overall and every reported subdimension against randomly initialized routing parameters.Both variants use the same ten memory-sensitive layers.
- Effect of Where-to-Use Routing: 0.538 to 0.570: profile-guided routing improves MemoBench overall and four of five reported dimensions over ten randomly selected layers.Identity is the only reported dimension favoring Random-10, at 0.507 versus 0.495.
5 Conclusion
LayerRecall addresses bounded-cache long-range consistency by separating historical retrieval from layer-specific memory use, with CHPM transferring long-context behavior to a bounded-memory student. Across evaluations, it improves memory-oriented benchmarks, matches VBench-Long, and retains portability and low overhead.
- Conclusion: LayerRecall jointly decides what historical content to retrieve and where to inject its K/V states, preserving the original local pathway elsewhere.The router is current-conditioned and backbone-specific, while CHPM supervises it from a privileged long-context reference without explicit allocation labels or backbone finetuning.
- Conclusion: LayerRecall achieves the best overall scores on MemoBench and MovieBench while matching LongLive-2.0 on VBench-Long.Ablations additionally validate layer-selective injection, CHPM, and cross-backbone transfer with negligible overhead.
A Additional Experiments
LayerRecall demonstrates recovery of mismatched historical attributes while preserving ongoing identity, motion, and scene structure. The routing mechanism explains how later, more discriminative representations can improve retrieval within the same shot.
- LayerRecall recovers a returning subject’s historically established garment after an initial mismatch, without resetting the surrounding generation.The example preserves identity, gray cardigan, action, and scene structure while correcting only the inner garment.
- Current-conditioned retrieval can improve as identity, body shape, and scene relations become more discriminative during reappearance.The evolving current representation updates the retrieval query toward the relevant historical chunk.
- Retrieved K/V states enter only memory-sensitive layers, while other layers retain the backbone’s local attention pathway.This selective injection supports localized correction rather than a global visual reset.
A.3 Complete Cross-Backbone Portability Results
The complete portability analysis evaluates LayerRecall transfer to LongLive and Self-Forcing under alternative insertion policies. The reported fine-grained results are restricted to these two target backbones.
- The portability claim is limited to the two evaluated target backbones, LongLive and Self-Forcing.The fine-grained scores underlie the main-paper portability analysis.
B Additional Method Details
This section supplies supplementary implementation and methodological details omitted from the main paper for space. It covers the architecture, retrieval, layer-policy, and CHPM training components.
- The supplementary section provides architecture details omitted from the main paper.
- The supplementary section provides retrieval details omitted from the main paper.
- The supplementary section provides layer-policy and CHPM training details omitted from the main paper.
B.1 LayerRecall Architecture
LayerRecall adds a small, state-conditioned router to a frozen chunk-autoregressive DiT, combining historical retrieval with selective layer insertion. Supplementary details specify its summaries, current-state conditioning, retrieval paths, and backbone-specific policies.
- B.1 LayerRecall Architecture: A 384-frame rollout is organized into 48 chunks of 8 latent frames in the LongLive-2.0 setting.
- B.1 LayerRecall Architecture: The router uses a shared global query and current-conditioned MLP gate, with a zero-initialized final projection and learnable residual scale.Only the layer scale γ_l is layer-specific; the remaining router components are shared across layers.
- B.1 LayerRecall Architecture: Historical summaries score retrieval, while the physical cache retains full RoPE-applied K/V payloads consumed by self-attention.The summary selects resident records; attention receives the complete cached payload rather than the compact summary.
- B.1 LayerRecall Architecture: Retrieval scores use cosine compatibility between the current query and resident historical summaries.The current query is derived from the detached pre-attention hidden state at the current chunk and layer.
- B.2 Hard and Soft Retrieval: CHPM couples hard and soft retrieval through a straight-through estimator: hard K/V payloads drive the forward pass, while soft mixtures provide gradients.The reported configurations retain the eight highest-scoring candidates and use τ = 1.0.
- B.3 Layer Policy: Layer policies are selected by profiling and controlled comparisons, then fixed per backbone rather than defined by a single raw attention ranking.Cross-backbone experiments reuse the trained router weights and change only the enabled insertion-layer mask.
B.4 CHPM Training Details
CHPM trains the LayerRecall router by matching bounded-memory student predictions to a privileged long-context teacher in denoised-latent space, without updating either backbone.
- CHPM matches bounded-memory student and privileged long-context teacher predictions in denoised-latent prediction space.Both use the same backbone checkpoint, current noisy latent, text condition, and diffusion timestep at supervised anchors.
C Prompt Construction, Baseline Adaptation, and Evaluation Protocol
The study constructs balanced multi-shot prompt banks, adapts baselines to preserve reappearance constraints, and evaluates generated videos with consistency, memory, temporal, VLM, and runtime protocols.
- C.1 Prompt Construction: 1,600 multi-shot prompts are balanced across subject, color, position, and scene axes, with 400 prompts per axis.The training bank includes 3-shot through 6-shot cases and targets identity, color, spatial-position, and scene-layout consistency.
- C.1 Prompt Construction: 100 held-out 3-shot prompts form the main evaluation bank, with 25 prompts per axis and disjoint profiling data for layer-policy analysis.Prompts encode salient elements that leave local context and later reappear, alongside metadata for targets, camera views, shot durations, and quality control.
- C.2 Prompt Filtering: The checked training and evaluation banks report empty quality-issue distributions after filtering for target axes, transitions, and visual clutter.Filtering also addresses overloaded consistency targets and excludes explicit scene-transition wording.
- C.3 Baseline Prompt Adaptation: Single-prompt baselines receive adapted descriptions that preserve subject, attribute, object, reappearance, and camera-return constraints without adding semantic targets.The examples retain detailed identity, appearance, ownership or location relations, and scene landmarks across shots.
- C.4 Reproducibility and Evaluation Protocol: The evaluation script requires 100 generated videos, extracts frames every 10 seconds through 60 seconds, and requires a minimum duration of 55 seconds.Frame extraction uses ffprobe and ffmpeg when available, with an OpenCV fallback.
- C.4 Reproducibility and Evaluation Protocol: CLIP/DINO, VBench, VBench-Long, Helios, MemoBench-style, and MovieBench-style metrics are run across all 100 evaluation videos.VBench and VBench-Long include consistency, motion, quality, and dynamic-degree dimensions.
- C.4 Reproducibility and Evaluation Protocol: The temporal diagnostic computes frame-change energy, abrupt-change residuals, and a normalized high-frequency spectrum ratio from grayscale decoded frames.The high-frequency ratio aggregates spectral power over 0.2≤f≤12 after Hann-windowed FFT processing.
- C.4 Reproducibility and Evaluation Protocol: VLM evaluation samples 12 evenly spaced frames per video, resizes the long side to 512 pixels, requests JSON scores, and averages video-level results.Reported table values are arithmetic means; VBench-Long Avg. averages subject consistency, background consistency, and motion smoothness, while MemoBench and MovieBench use evaluator-provided overall fields.