Learning What to Recall: Adaptive Multi-Cue Episodic Memory for World Models

1 INTRODUCTION

• Episodic memory preserves individually accessible observations, complementing persistent-state memory that provides compact access to an evolving world state.Past observations can matter when scenes and objects persist or evolve outside the current field of view.Source: §1 · p. 2
• Fixed recency, pose-overlap, and visual-similarity criteria can misidentify useful memories because cue reliability varies across queries and environments.For example, pose may select a nearby memory across a wall, while similar corridor appearances can be ambiguous and spatial audio may be more informative.Source: §1 · p. 2
• FAR uses the observed future during training to estimate each memory’s predictive usefulness, then trains a retriever that operates without future access at inference.It learns cue-specific relevance and query-dependent weights for available cues such as time, pose, vision, and audio.Source: §1 · p. 2
• FAR uses negative diffusion prediction loss as a tractable surrogate for the conditional likelihood of the realized future given recalled context.The recalled context conditions the world model’s future prediction.Source: §1 · p. 2
• FAR improves episodic recall and downstream prediction over fixed retrieval strategies across three complementary environments.It uses the same retrieval cues as hand-designed rules in LoopNav, adapts to audio when spatial cues become ambiguous in SoundSpaces, and handles stale memories in changing-state AI2-THOR.Source: §1 · pp. 2–3

2 BACKGROUND AND RELATED WORK

• World models maintain long-range information through recurrent dynamics, diffusion, or masked generative modeling (Hafner et al., 2020; 2021; 2025; Alonso et al., 2024; Bruce et al., 2024; Kim et al., 2026).External episodic memory instead preserves past observations for selective recall using cues such as time, pose, vision, or audio.Source: §2 · p. 3
• Generator-internal methods maintain long-range information through attention, routing, context compression, or recurrent memory, while generator-external methods search explicit memory and pass a compact subset to the world model.These approaches are complementary: internal memory summarizes history, while external episodic memory preserves individually addressable observations for selective recall.Source: §2 · p. 3
• External world-model recall commonly uses fixed temporal, geometric, or embedding-based relevance, whereas FAR learns discrete retrieval from future predictive utility.The paper adapts the latent-variable retrieval principle used by EMDR2 (Sachan et al., 2021) to episodic recall from evolving interaction history, including query-dependent cue reliability.Source: §2 · p. 3

3 OUR METHOD: FUTURE-AWARE EPISODIC MEMORY

• FAR selects a compact set of past observations from episodic memory to condition next-observation prediction, using a future-blind retriever at inference.The retriever scores candidate contexts from historical cues and the current retrieval query; the recalled context, rather than the cues themselves, conditions the world model.Source: §3 · pp. 3–4

3.1 LEARNING WHICH MEMORIES TO RECALL

• A recalled context is relevant when it supplies information about the future beyond the current prediction query, captured by the world model’s likelihood of the realized future given that context.This predictive utility depends on the query: the same past experience may help one prediction and be largely irrelevant to another.Source: §3.1 · p. 4
• During training, FAR combines the future-blind relevance score with predictive credit from the realized future to form a teacher distribution, then trains recall to match that target.The resulting retriever estimates relevance from available cues and learns query-dependent weights because cue reliability can vary across queries.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 ADAPTIVELY LEARNING WHICH CUES TO TRUST

• FAR learns cue-specific memory relevance and uses query-dependent weights to fuse the available cues into context scores, selecting the highest-scoring K observations.The method can use available time, pose, visual, or audio signals; high-dimensional vision and audio keys can be cached using frozen memory-side encoders while query adapters and fusion remain trainable.Source: §3.2 · p. 5

3.3 VIDEO DIFFUSION WORLD MODEL INSTANTIATION

• For diffusion models, FAR approximates realized-future likelihood with negative diffusion prediction loss; temporal chunking selects top-scoring representatives to reduce redundant nearby frames.For high-dimensional cues, frozen memory-side encoders allow historical keys to be computed once and cached, while query-side adapters and cue fusion remain trainable.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

• Across LoopNav, SoundSpaces, and AI2-THOR, the agent predicts return-phase video using exploration-phase observations as its memory pool.The settings cover static Minecraft navigation, indoor navigation with spatial audio, and interactive household environments where objects can change state.Source: §4 · p. 6

4.1 LOOPNAV – FAR OUTPERFORMS HAND-DESIGNED RECALL

• At loop closure, learned visual retrieval yields 17% lower DreamSim than LongLive-RAG, and learned metadata retrieval yields 19% lower DreamSim than WorldMem using the same cues.WorldMem can select a high-FOV-overlap frame with an occluded goal view, whereas learned metadata retrieval selects a clearer predictive view.Source: §4.1 · p. 6
• LoopNav Multi-Cue rollout error follows the better metadata-only or visual-only variant across the horizon, suggesting adaptive fusion emphasizes the more informative cue.The experiments compare metadata cues, visual cues, and their adaptive fusion.Source: §4.1 · pp. 6–7

4.2 SOUNDSPACES – FAR ADAPTIVELY LEARNS WHICH CUES TO TRUST

• FAR outperforms temporal and WorldMem recall across both SoundSpaces corpora; audio gains grow on longer returns when memories are sparse.Metadata is strong with periodic scans, while audio complements it under endpoint-only scans, particularly for LPIPS and DreamSim.Source: §4.2 · p. 7, Figure 5 · p. 7

4.3 AI2-THOR – FAR RECALLS THE RIGHT STATE AS THE WORLD CHANGES

• Multi-Cue FAR substantially outperforms temporal and geometry-based recall across change magnitudes for both visible surface changes and container reveals.The advantage is especially pronounced for container reveals; all methods weaken when current and stale states differ only locally.Source: §4.3 · pp. 8–9
• With the current view and action held fixed, FAR predicts different refrigerator contents according to whether the history includes placing a tomato inside.Temporal and WorldMem fail to preserve the interaction-dependent state; WorldMem also hallucinates the tomato at its stale table location.Source: Figure 7 · p. 8, §4.3 · p. 9
• 94.6% accuracy with Multi-Cue FAR, versus 67.1% with metadata-only FAR, 41.9% Temporal, and 32.4% WorldMem, for off-scene A2 location prediction.Multi-Cue FAR recalls episodes containing motion information, while WorldMem retrieves a geometrically relevant but dynamically uninformative memory.Source: §4.3 · p. 9, Figure 9 · p. 9

4.4 ABLATION STUDY

• In the LoopNav ablation, adaptive fusion weights perform best across all metrics, supporting complementary metadata and visual cues over uniform fusion.Table 1 reports average loop-closure errors for different configurations.Source: Table 1 · p. 9, §4.4 · p. 10

5 CONCLUSION AND LIMITATIONS

• FAR learns cue-specific relevance and query-dependent cue weights to improve long-horizon prediction across navigation, multi-cue, and changing-state settings.The observed future supplies predictive credit during training; inference remains future-blind.Source: §5 · p. 10
• The work addresses recall from external episodic memory, leaving memory writing, compression, forgetting, and higher-order interactions among recalled memories to future work.Experiments use controlled simulations, motivating evaluation in richer real-world settings and integration with learned memory formation and persistent-state representations.Source: §5 · p. 10
• FAR requires informative retrieval cues and additional training-time computation for predictive-utility evaluation; diffusion-loss utility may underweight semantically important local changes.The authors identify alternative task-aware utility functions as a promising direction.Source: §5 · p. 10

A ADDITIONAL DISCUSSION OF RELATED WORK

• FAR learns external episodic relevance from predictive utility and query-dependent fusion over temporal, spatial, visual, and auditory cues, unlike retrievers based on fixed relevance notions.Table 2 compares episodic recall methods across prediction conditioning, external retrieval, prediction supervision, cue fusion, and cue options.Source: A ADDITIONAL DISCUSSION OF RELATED WORK · p. 13, Table 2 · p. 13
• Internal memory routes or compresses history within the generator, while external episodic recall preserves individually addressable observations for selective revisiting.The discussion presents these approaches as complementary.Source: A ADDITIONAL DISCUSSION OF RELATED WORK · p. 13
• Persistent-state methods update a representation of the evolving world, whereas episodic memory can recover specific past evidence when that state is incomplete or uncertain.The discussion describes the two approaches as complementary.Source: A ADDITIONAL DISCUSSION OF RELATED WORK · p. 13
• FAR applies predictive-credit retrieval to an agent’s episodic history and learns heterogeneous cue fusion, extending a principle also used in retrieval-augmented generation.EMDR2 uses the reader likelihood of an observed target to supervise discrete document retrieval.Source: A ADDITIONAL DISCUSSION OF RELATED WORK · p. 14
• Other long-horizon approaches address context scaling, drift, or external grounding, while FAR focuses on learning which observations from the agent’s history support the current prediction.The discussion cites FramePack, MilliVid, and Seoul World Model as examples.Source: A ADDITIONAL DISCUSSION OF RELATED WORK · p. 14

B ADDITIONAL THEORETICAL DETAILS

• The supplementary material provides the proof of Proposition 1 and summarizes the overall pseudocode as Algorithm 1.Source: B ADDITIONAL THEORETICAL DETAILS · p. 14

B.1 DERIVATION OF PREDICTIVE RELEVANCE BOUND

• The predictive-relevance bound links information between the future and recalled context, conditioned on the query, to an expected predictive log-likelihood term.The bound’s gap is characterized by a KL divergence and is tight when the predictive world model matches the conditional future distribution almost surely.Source: §B.1 · p. 14

B.2 LATENT-CONTEXT MAXIMUM-LIKELIHOOD VIEW OF THE RETRIEVER UPDATE

• The latent-context model yields a future-aware posterior proportional to the current recall distribution reweighted by predictive utility.When all contexts have equal predictive utility, the posterior leaves the current recall preference unchanged; a utility-only target instead imposes a uniform prior.Source: §B.2 · p. 15
• The retriever update alternates between computing posterior credit from the realized future and fitting the future-blind retriever to that fixed target.Stop-gradient implements the fixed-target step rather than adding a separate heuristic objective.Source: §B.2 · pp. 15–16
• With the exact posterior, posterior matching recovers the negative marginal-likelihood gradient with respect to the retriever.Source: §B.2 · p. 16
• For discrete Top-K recall, enumerating exact context sets is combinatorial, so the candidate-level approximation need not preserve the exact global gradient identity.It restricts the context space and utility evaluation to a finite candidate pool and singleton predictive utilities.Source: §B.2 · p. 16

B.3 RETRIEVER POSTERIOR APPROXIMATION

• Because exact posterior evaluation over all admissible context sets is combinatorial under discrete Top-K recall, FAR uses a tractable candidate-level approximation over historical observations.Source: §B.3 · p. 16
• The future-aware candidate posterior reweights each candidate’s future-blind relevance score with its singleton predictive utility, and the retriever is trained to match that target.The utility evaluates how well an individual candidate supports prediction of the observed future.Source: §B.3 · p. 16
• Posterior supervision updates both cue-specific relevance and query-dependent cue weights; with one cue it recovers the EMDR2-style candidate update.The approximation affects retrieval credit assignment, not the world model’s training on the complete recalled context, but it does not explicitly score higher-order selection effects.Source: §B.3 · p. 17

• The diffusion transformer predicts the next observation from the current observation, action, and recalled context frames, with retrieved context-frame tokens processed jointly with noisy target-frame tokens.The diffusion objective compares the model output with its training target using a noise-schedule weighting.Source: §C.1 · p. 17
• Visual and audio representations guide memory selection but are not passed to the world model as additional generation conditions.This lets the retriever use cues without changing the generator’s conditioning interface.Source: §C.1 · p. 18
• Time and pose use a lightweight scoring network, while vision and audio use action-conditioned query and memory-key encoders whose historical keys are cached after pretraining.The high-dimensional encoders are frozen after pretraining, while the query adapter remains trainable through FAR.Source: §C.1 · p. 18
• FAR standardizes cue scores over the current memory and derives query-dependent weights from cue statistics, normalizing across the cues available in that environment.Consequently, the relative importance of available cues can change between retrieval queries.Source: §C.1 · p. 18
• Negative diffusion prediction loss provides a tractable surrogate for each candidate memory’s predictive utility, while retrieval credit uses singleton contexts and the world model trains on the complete recalled context.Candidates are evaluated against the realized future; shared diffusion perturbations reduce variation in candidate comparisons.Source: §C.1 · pp. 18–19
• To avoid concentrating context slots in nearby redundant frames, FAR selects at most one observation per temporal chunk before choosing the K highest-scoring representatives.Retriever training uses global and local candidate pools to assign predictive credit across temporal regions and within a selected region; the retriever is updated periodically, and inference needs no future observation or utility evaluation.Source: §C.1 · p. 19

• LoopNav contains 19,200 episodes across 120 procedurally generated village scenes, with outbound observations stored as memory and return-leg prediction evaluated on an episode-level split.All scenes appear in training and test, so the split measures unseen trajectories within known worlds.Source: §C.2 · p. 20
• The paired SoundSpaces corpora each contain 14,425 episodes across 81 scenes with scene-disjoint splits; v2 adds deterministic scans beyond v1’s endpoint scans to test denser visual coverage.The variants share episode manifests and scene content, and no scans are inserted into the return leg.Source: §C.2 · p. 20
• AI2-THOR uses 8,125 single-room episodes across 119 scenes, where objects are manipulated during exploration and revisited from canonical viewpoints that can show different states at the same pose.Moved objects are hidden while carried, so their updated state must be inferred from interaction history.Source: §C.2 · p. 20
• AI2-THOR-dyn tests corridor-crossing history in seven rooms, where an agent’s final location depends on observed crossings despite the observer’s constrained view.The reported split shares rooms between training and test, measuring unseen trajectories within known rooms rather than held-out-room generalization.Source: §C.2 · pp. 20–21

C.3 TRAINING DETAILS

• Training uses frozen, precomputed frame latents and corpus-specific batch sizes, with learning-rate warmup, gradient clipping, EMA weights, and periodic EMA evaluation.FAR’s effective batch size is 128 on LoopNav and 64 on the other corpora.Source: §C.3 · p. 21
• Training queries and memory pools are drawn from outbound or exploration phases; AI2-THOR and AI2-THOR-dyn also use event- or reveal-anchored examples.High-dimensional retrievers are contrastively pretrained, then updated alongside the world model using shared diffusion perturbations and global plus within-chunk retrieval losses.Source: §C.3 · pp. 21–22
• AI2-THOR reweights diffusion loss around object changes, while AI2-THOR-dyn adds second-agent state prediction and includes agent-state supervision in the candidate reward.The diffusion model receives ground-truth goal agent state during training; inference uses the predictor output.Source: §C.3 · p. 22

C.4 INFERENCE DETAILS

• LoopNav, SoundSpaces, and AI2-THOR use replace-tail autoregressive rollouts along ground-truth camera paths while retaining real outbound or exploration memories.Retrieval is queried at every step; visual and audio cues rank memories but are not generator conditioning, and compared methods share noise seeds and target trajectories.Source: §C.4 · pp. 22–23
• AI2-THOR-dyn uses a one-step both-ends probe with matched noise seeds, and inference relies on the agent-state predictor rather than teacher forcing.A latent-space detector identifies rendered agents; on real test frames it has 0.906 recall and a 0.070 false-positive rate at the selected threshold.Source: §C.4 · p. 23