PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Retrieve: Internalizing Memory Retrieval for Video World Models

JiaKui Hu, Tailai Chen, Yuqi Pan, Xuerui Qiu, Jialun Liu, Xiao Cao, Zhenxin Zhu, Guang Chen, Hangjun Ye, Bing Wang, Yanye Lu

arXiv:2610.11444v1cs.CV

TL;DR

Video world models need to keep generated scenes consistent when camera trajectories revisit previously observed regions, while existing retrieval approaches rely on auxiliary pathways outside generation. L2R internalizes retrieval in the model’s persistent state, using a camera-conditioned gate and a 3D re-visibility-supervised trigger. Across camera-revisit benchmarks and multiple base models, it improves long-term scene consistency without requiring external memory retrievers or explicit 3D condition injection.

  • Problem

    Existing methods make retrieval decisions through auxiliary memory pathways outside the core generation process, rather than learning retrieval within the model’s internal dynamics.

    Source: §3.1 · p. 3

  • Method

    L2R uses the model’s persistent state as memory, with a camera-conditioned gate selecting historical context and a 3D re-visibility-supervised trigger activating retrieval.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    L2R improves long-range scene consistency across video world models and camera-revisit benchmarks, with reported gains across different underlying state-transition mechanisms.

    Source: §1 · p. 3, §4.3 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    The internal retrieval mechanism improves scene consistency without requiring external memory retrievers or explicit 3D condition injection.

    Source: §5 · p. 9

Abstract

from arXiv · show

Video world models aim to generate explorable, 3D-consistent scene videos conditioned on camera trajectories. Existing approaches often rely on external memory systems that explicitly retrieve previously observed content to mitigate scene drift during long-horizon generation. However, these auxiliary memory pathways operate outside the model's internal generative dynamics, preventing the model from intrinsically learning when and what historical information should be retrieved. We propose to internalize memory retrieval into the generation process, allowing retrieval to emerge as an intrinsic behavior of the video world model rather than relying on an external memory system. Based on this principle, we introduce \textbf{Learning-to-Retrieve (L2R)}, which repurposes the model's persistent internal state as a memory for historical context. A camera-conditioned retrieval gate selectively accesses relevant historical information from this state, determining \textit{what to retrieve}, while a retrieval trigger determines \textit{when to retrieve}. We further supervise the trigger with a 3D re-visibility signal, activating retrieval when previously observed content re-enters the current view while otherwise preserving the existing context. Together, these components enable the model to intrinsically acquire memory retrieval behavior and incorporate relevant historical observations into generation without a separate retrieval pathway. Across multiple base models and camera-revisit benchmarks, L2R improves long-term scene consistency while eliminating the need for an external memory bank or 3D conditions. https://jkhu29.github.io/l2r

1 INTRODUCTION

Video world models need to preserve scene consistency when camera trajectories revisit previously observed regions, but existing approaches use reconstruction-derived conditions or auxiliary memory pathways. L2R makes retrieval part of generation, and the paper reports improved consistency without degrading visual quality or camera controllability.

1 INTRODUCTION

  • Scene consistency requires generated content to remain consistent with earlier generations when the camera revisits previously observed regions.The models generate scene video conditioned on a user-specified camera trajectory and one or a small set of reference frames.Source: §1 · p. 2
  • Reconstruction-based consistency methods depend on reconstruction accuracy, and their errors can propagate to video generation and degrade long-term quality.A separate paradigm retrieves relevant past views as auxiliary context, but its memory-access pathway remains outside the model’s generative dynamics.Source: §1 · p. 2
  • L2R aims to make retrieval intrinsic to generation by jointly learning to preserve accessible historical information and select it for the current observation.This is intended to let memory access adapt to the model’s evolving generation dynamics.Source: §1 · p. 2
  • L2R integrates memory retrieval into generation, using a camera-conditioned gate to select historical information and a trigger to decide when to retrieve.The trigger is supervised by 3D re-visibility; retrieval is activated when previously observed content re-enters the view.Source: §1 · p. 2
  • Across re-appear and loop-closure trajectories, L2R improves scene consistency without degrading visual quality or camera controllability.The reported results also outperform existing approaches in consistency and show robust generalization to unseen camera trajectories.Source: §1 · pp. 2–3

2 RELATED WORK

Related work addresses long-term memory in video generation through internal representations and through mechanisms that retrieve or condition on historical content. Some approaches use external or auxiliary retrieval, while recent work also explores recurrent-state memory and sparse attention.

2 RELATED WORK

  • Video-generation systems represent historical information with sliding windows, attention sinks, positional extrapolation, state-space architectures, token compression, or hybrids.ARL2 uses the recurrent state as a memory bank but lacks conditioned retrieval.Source: §2 · p. 3
  • Video world models use 3D conditioning, external memory retrieval or compression, and external learnable retrieval modules, while Wonder uses sparse attention without guaranteeing geometrically relevant top-k tokens.External retrieval methods use camera-view overlap or geometric associations to select historical observations.Source: §2 · p. 3

3 METHOD

L2R frames retrieval as selective access to historical context within video generation, using a persistent model state rather than a separate memory pathway. It pairs camera-conditioned selection with a retrieval trigger supervised by 3D re-visibility, and supports different state-transition mechanisms.

3 METHOD

  • L2R is presented as a mechanism for retrieving historical context within the model to promote consistency in generated world videos.The method section covers preliminaries, implementation, and training.Source: §3 · p. 3

3.1 PRELIMINARIES

  • Existing retrieval makes two decisions—whether historical information is needed and which information to access—through an auxiliary pathway outside generation.The default context is used for generation, while prior approaches may retrieve trajectory-relevant information when target and reference views overlap.Source: §3.1 · p. 3

3.2 LEARNING TO RETRIEVE

  • L2R stores historical context in the accumulated model state; a retrieval trigger controls whether to access it, and a pose-conditioned gate selects the information.The camera ray and sliding-window size condition retrieval, while the state transition rule updates the state with the resulting context.Source: §3.2 · p. 4
  • Re-visibility labels are assigned when a previously observed 3D point leaves the frustum or becomes occluded and later reappears.Labels are computed by estimating depth, unprojecting tokens to 3D, and re-projecting earlier points into the current frame; the retrieval objective uses weighted BCE and masks invalid tokens and context frames.Source: §3.2 · p. 4, Figure 2 · p. 5
  • The same retrieval recipe is instantiated with Gated Linear Attention and Gated DeltaNet, with an additional delta-correction modification for Gated DeltaNet.The resulting variants are named L2R-GLA and L2R-GDN.Source: §3.2 · p. 5

3.3 TRAINING

  • Training supports scratch training or supervised fine-tuning, and combines the rectified-flow objective with re-visibility supervision.For most protocols, training starts from pretrained video-generation parameters; the denoising loss applies only to target-frame latents, and the default visibility-loss weight is λr = 0.05.Source: §3.3 · p. 5

4 EXPERIMENTS

Experiments evaluate memory capacity, video quality and camera control, design ablations, and inference efficiency. L2R improves revisit consistency across settings, with performance shaped by the retrieval design and attention-layer ratio.

4.1 SETUP

  • Memory capacity is tested through replay, loop closure, and open-domain return; video quality uses forth-and-back RealEstate10K and DL3DV paths with Wan2.2 and SANA-WM.The memory-capacity evaluation uses Wan2.1-T2V-1.3B and a shared training and inference setup across methods.Source: §4.1 · p. 6

4.2 RESULTS ON MEMORY CAPACITY

  • 14.74 replay PSNR (+1.03 dB over VMem’s 13.71), 0.375 loop-closure SSIM, and 72.57 O-V show L2R-GLA’s strong results across revisit protocols.Its O-V score exceeds VMem by 7.84 points.Source: §4.2 · p. 6, Table 1 · p. 7

4.3 RESULTS ON VIDEO QUALITY

  • Across Wan2.2 and SANA-WM, L2R improves revisit consistency while maintaining video quality and camera control; SANA-WM L2R-GDN raises PSNR from 15.52 to 17.29.On Wan2.2, L2R-GDN reaches the best revisit PSNR at 17.03, 0.19 dB above CaR.Source: §4.3 · pp. 7–8, Table 2 · p. 8

4.4 ABLATION

  • Camera-guided retrieval supplies the largest replay gain; the retrieval trigger and re-visibility supervision add further revisit improvements.Adding the trigger raises replay PSNR from 13.88 to 14.02; re-visibility supervision then improves replay PSNR by 0.72 dB over the trigger-only setting.Source: §4.4 · pp. 8–9, Table 3 · p. 8

4.5 EFFICIENCY

  • A 3:1 L2R-GLA/softmax ratio maximizes replay PSNR, while per-step latency is duration-invariant at 0.50 s for L2R-GLA and 0.49 s for L2R-GDN.The layer-ratio comparison also finds 1:1 maximizes loop-closure PSNR, while 7:1 degrades both replay and loop-closure PSNR.Source: Figure 4 · p. 9, §4.4 · p. 9, §4.5 · p. 9

5 CONCLUSION

The conclusion presents L2R as internalized, camera-guided retrieval with a trigger supervised by a 3D re-visibility target. It reports improved long-range scene consistency without external memory retrievers or explicit 3D condition injection.

5 CONCLUSION

  • L2R combines camera-guided retrieval, retrieval triggering, and 3D geometry-based re-visibility supervision in pretrained camera-controllable video world models.The conclusion frames this design through an analogy between external context retrieval and state-update dynamics.Source: §5 · p. 9
  • The authors report that L2R improves long-range scene consistency while eliminating external memory retrievers and explicit 3D condition injection.Source: §5 · p. 9
  • The paper warns against misuse of increasingly capable video generation, including misleading videos and malicious purposes.Source: ETHICS STATEMENT · p. 10
  • Implementation and dataset details are provided in Appendix A, and code and weights are planned for release after peer review.Source: REPRODUCIBILITY STATEMENT · p. 10

A.1 MORE EFFICIENT ANALYSIS

The appendix compares L2R's parameter and compute costs with softmax self-attention. L2R-GLA and L2R-GDN have linear context-length scaling, and replacing three of four blocks reduces the stated total forward-pass cost.

A.1 MORE EFFICIENT ANALYSIS

  • Replacing the first three of four blocks adds 5.25 M parameters when using Wan2.1 as the backbone.L2R-GLA has 11.20 M parameters, including 9.45 M inherited from the replaced self-attention.Source: §A.1 · p. 13
  • 0.49 TFLOPs per forward pass for L2R-GLA versus 2.45 TFLOPs for softmax self-attention on one 5 s Wan2.1 chunk.For N=18,480 and D=1536, the passage characterizes this as about 5× fewer FLOPs.Source: §A.1 · p. 13
  • 111 to 95 TFLOPs per forward pass (14%) when replacing the first three of four blocks in the 30-block DiT.Source: §A.1 · p. 13
  • L2R-GLA and L2R-GDN scale linearly with context length, unlike softmax attention's quadratic cost.L2R-GDN's delta correction adds only a constant-factor overhead over L2R-GLA.Source: §A.1 · p. 13

A.2 RE-VISIT BENCHMARK AND EVALUATION ON REALESTATE10K AND DL3DV

Revisit-Bench is a controlled benchmark for memory and loop-closure consistency, built from real camera trajectories and video frames. Its sequences return to the initial viewpoint at set times, enabling direct pixel-level evaluation under a static-scene assumption.

A.2 RE-VISIT BENCHMARK AND EVALUATION ON REALESTATE10K AND DL3DV

  • Revisit-Bench uses real camera trajectories and frames from RealEstate10K and DL3DV-10K to evaluate memory and loop-closure consistency.Benchmark generation is deterministic with a fixed random seed of 42.Source: §A.2 · p. 13

A.2.1 OVERVIEW

  • Each 81-frame sequence has one reference and 80 generated target frames, with a temporal-palindrome trajectory returning exactly to its initial viewpoint at a predefined revisit time.Under the static-scene assumption, each target viewpoint has a corresponding source-video frame for pixel-level evaluation.Source: §A.2.1 · p. 13
  • The two trajectory splits use the same 300 scenes but differ in revisit pattern: palindrome far returns at t = 80, while palindrome near returns at t = 48.The far split traverses the full trajectory before returning; the near split continues forward for 32 steps after the revisit to prevent a stationary-after-revisit solution.Source: §A.2.1 · p. 13
  • Each split contains 300 scenes—150 from RealEstate10K and 150 from DL3DV-10K—so each method produces 600 sequences across both splits.Source: §A.2.1 · p. 13

A.2.2 DATA SOURCES

The appendix specifies how Revisit-Bench is sourced and constructed, from scene filtering and camera-pose processing to palindrome trajectories and evaluation targets. It also reports prompt handling and trajectory statistics.

A.2.2 DATA SOURCES

  • Revisit-Bench samples RealEstate10K and DL3DV-10K clips from RealCam-Vid and uses trajectory constructions designed to evaluate revisit consistency.Source: §A.2.2 · pp. 13–14

A.2.3 FILTERING

  • Scene selection limits each source video to one window, requires finite and numerically stable camera poses, and filters motion to median translation steps of 0.02–0.8 and rotations no greater than 10°.The motion thresholds are applied after alignment scaling.Source: §A.2.3 · p. 14

A.2.4 CAMERA POSE PROCESSING

  • Camera poses are transformed using the training data loader's coordinate procedure and a per-window alignment factor; the appendix also recovers source-pixel intrinsics from normalized values.The same intrinsic matrix is used for all 81 frames in a sequence; benchmark and training clips have median translation steps of approximately 0.17 and 0.48, respectively.Source: §A.2.4 · p. 14

A.2.5 PALINDROME TRAJECTORY CONSTRUCTION

  • Palindrome trajectories mirror actual source viewpoints, with the camera exactly at the identity pose at t = 2f.The far split returns at t = 80; the near split revisits at t = 48 and then continues forward for 32 steps.Source: §A.2.5 · p. 15

A.2.6 GROUND-TRUTH FRAMES AND STATIC-SCENE ASSUMPTION

  • Under the static-scene assumption, each generated viewpoint uses its corresponding source frame as ground truth; the revisit frame matches the input reference image.Source: §A.2.6 · p. 15

A.2.7 TEXT PROMPTS

  • Sequence prompts use RealCam-Vid's short caption, falling back to the long caption or a fixed prompt when captions are unavailable.Prompts average 18.2 words and contain approximately 10–25 words.Source: §A.2.7 · p. 15

A.2.8 STATISTICS

  • Each method generates 600 sequences across the two splits, which share the same 300 source scenes.Original trajectories have a median per-frame translation step with mean 0.408 and median 0.171, ranging from 0.040 to 0.783; mean maximum single-frame rotation is 2.19°.Source: §A.2.8 · p. 15

A.2.9 EVALUATION PROTOCOL

The evaluation protocol covers revisit consistency, camera accuracy, and VBench dimensions. The supplied material also describes training settings and how L2R is combined with other state-update rules.

A.2.9 EVALUATION PROTOCOL

  • Revisit consistency uses PSNR and SSIM against the input reference image; camera accuracy compares estimated and ground-truth trajectories, and VBench reports six dimensions.The VBench dimensions include subject and background consistency, motion smoothness, temporal flickering, aesthetic quality, and imaging quality.Source: §A.2.9 · p. 16

A.3 TRAINING SETTINGS

B EXTENSION TO OTHER STATE UPDATE RULES

  • L2R defines retrieval condition rt and state-access gate γt, while the underlying state-space mechanism determines how retrieved state is updated.For GDN, γt controls accessed historical information and βt controls the subtractive delta update and corresponding write strength.Source: B EXTENSION TO OTHER STATE UPDATE RULES · pp. 16–17

C FAILURE MODE

A 7:1 memory-layer ratio loses observed structure on loop closure, unlike the 1:1 and 3:1 ratios. The paper identifies dynamic-scene retrieval as future work and states that AI was used only to polish grammar.

C FAILURE MODE

  • At a 7:1 memory-layer ratio, the return loses observed structure and fails to close the loop, while 1:1 and 3:1 close it.With only four memory layers, the recurrent state must summarize the trajectory at few network depths, without a high-bandwidth softmax fallback for overwritten content.Source: C FAILURE MODE · p. 17, Figure 5 · p. 18

D FUTURE WORK

  • Future work could address dynamic-scene retrieval and optimize triggers using more accurate geometric or external supervision.Source: D FUTURE WORK · p. 17

E AI ASSISTANCE

Loading 2610.11444v1…