PaperlayerView exact revision on arXiv

Source-linked AI summary

Echo-Memory: A Controlled Study of Memory in Action World Models

Wayne King, Zeyue Xue, Yuxuan Bian, Jie Huang, Haoran Li, Yaowei Li, Yaofeng Su, Yuming Li, Haoyu Wang, Shiyi Zhang, Songchun Zhang, Yuwei Niu, Sihan Xu, Junhao Zhuang, Haoyang Huang, Nan Duan

arXiv:2606.09803v1cs.CVcs.GRcs.LG

TL;DR

Action-conditioned world models often lose scene and object identity when revisiting previously seen views, but memory mechanisms are difficult to compare under differing pipelines. Echo-Memory controls the backbone and evaluation while varying how history is stored and read, finding that structured recurrence is strongest for open-domain return and that replay quality alone is insufficient.

  • Problem

    Action world models must preserve geometry and object identity across camera-driven revisits, yet their characteristic failures are memory failures rather than local synthesis failures.

  • Method

    Echo-Memory fixes the generation and evaluation pipeline while comparing raw context, compression, spatial read-out, and block-wise state-space recurrence across replay, in-domain return, and open-domain return.

  • Results

    Block-wise State-Space memory reaches an open-domain VLM score of 69.00 despite lower replay PSNR, while replay and revisit quality are not monotonically aligned.

  • Takeaways & Limitations

    Memory evaluation should separate replay, in-domain return, and open-domain return, with structured recurrence treated as a first-class memory design axis.

  • Takeaways & Limitations

    The study uses a single dataset, VLM-based open-domain scoring, and imperfect proxies because revisit quality is not yet a cheap training-time signal.

Abstract

from arXiv · show

We present \textbf{Echo-Memory}, a controlled study of memory mechanisms in action-conditioned world models. These models generate multi-segment videos from a first frame, text prompt, and camera-action sequence, but their central failure is often memory rather than local image synthesis: after the camera leaves and returns, the scene or salient object may silently change. Existing memory designs are hard to compare because gains are entangled with backbone, training, retrieval, and evaluation differences. Echo-Memory fixes the action-to-video interface and varies only how history is stored and read by the generator. Under a shared video diffusion backbone, optimizer, camera-action representation, sampler, and evaluation pipeline, we compare raw context, compression-based memory, spatial summaries with different read-out paths, and state-space recurrence. This matched matrix separates four otherwise conflated axes: \emph{capacity}, \emph{compression}, \emph{read-out}, and \emph{recurrence}. We also evaluate memory through a three-branch protocol: replay quality, in-domain loop revisit, and open-domain return probes. The branches routinely disagree, showing that replay fidelity is not a sufficient proxy for remembering a world. Three findings follow. Raw context is a strong capacity baseline and improves open-domain return far more than it improves replay metrics. Compactness is not a free substitute for capacity: aggressive spatial and hybrid-compression memories lose the salient evidence needed for return. Finally, block-wise state-space recurrence is the strongest open-domain return mechanism in our matrix, showing that the structure of implicit memory matters as much as the decision to use it. These results provide a compact protocol for studying memory in action world models beyond isolated replay metrics.

1 Introduction

Echo-Memory studies memory in action world models under a fixed video-generation interface, varying only memory representation and separating capacity, compression, read-out, and recurrence. Its three-branch evaluation shows that replay quality alone does not measure whether a world remains consistent after revisits.

  • Motivation: Action world models must preserve geometry, object identity, and camera obedience across chunk-wise generation and revisits, where returning views may silently change.The model receives a first frame, text prompt, and camera-action sequence, then generates successive video chunks.
  • Controlled study: Echo-Memory fixes the backbone, data interface, camera-action encoding, sampler, and evaluation pipeline while varying only the memory representation.This controlled interface tests what information is stored, how it is compressed, how the generator reads it, and whether it survives return motion.
  • Evaluation: The study compares four memory families across replay quality, in-domain loop revisit, and open-domain return, because familiar trajectories can overstate memory quality under distribution shift.The design addresses confounds from larger raw context, hidden retained evidence in compression operators, and plausible reconstruction despite lost object identity.
  • Findings: K=1 to K=20 context increases the replay image bundle and raises open-domain VLM return from 12.25 to 58.63, establishing raw context as a strong capacity baseline.The result indicates that additional visual evidence can substantially improve return behavior, even beyond replay gains.
  • Findings: Spatial Memory remains competitive on replay PSNR but is weak on open-domain return, while hybrid compression loses much of the signal preserved by simpler length compression.These results show that compactness does not automatically preserve semantic evidence needed for revisits.
  • Findings: Block-wise State-Space memory reaches an open-domain VLM score of 69.00 despite lower replay PSNR, while read-out variants show that stored tokens must be usable at return.The findings identify structured recurrence and generator access as distinct determinants of memory quality.

2 Related Work

Related work frames action world models as systems that must preserve scene geometry, object identity, and action consistency across camera revisits. Existing memory interfaces span retrieval, compression, and implicit recurrence, while evaluation increasingly supplements standard video metrics with loop-closure and return measures.

  • Action world models: Action world models generate visual rollouts conditioned on prompts, camera motion, or agent actions, requiring consistency after revisiting regions.The relevant memory challenge includes preserving scene geometry, object identity, and action consistency after the camera leaves and returns.
  • Memory mechanisms: Existing long-horizon video systems use retrieval, compression, or implicit memory interfaces to carry history across generation.Retrieval appends selected frames or features; compression reduces history cost through weighting, pooling, packing, or compact summaries; implicit methods use recurrent or state-space computation.
  • Evaluation: Standard metrics remain useful health checks, but loop-closure, return, and VLM-as-judge protocols better target revisit consistency and semantic preservation.Echo-Memory combines these evaluation ideas in a controlled matrix, with Replay measuring camera-following quality and in-domain and open-domain return measures addressing return behavior.

3 Method: The Echo-Memory Design Space

Echo-Memory defines a controlled memory design space for action world models by varying how history is stored and read while keeping the backbone, action interface, training loop, sampler, and evaluation pipeline fixed. Its shared interface compares Context, Compression, Spatial, and State-Space memory profiles, with targeted ablations isolating individual mechanisms.

  • Design space: The design space factorizes memory into Context tokens, Compression operators, Spatial summaries, and a State-Space state without introducing a new backbone.All rows share the same training and evaluation paths; only the memory/context profile changes.
  • Context representation: The model represents context as one VAE latent token group per frame, preserving temporal alignment and making compression operations transformations over a fixed-length latent stack.This per-frame representation keeps the context timeline semantically aligned with target tokens.
  • Context retrieval: A current segment receives an anchor observation plus K−1 retrieved historical observations, with pdrop = 0.1 dropping retrieved frames to simulate cold starts.The context set contains K historical observations, and the retriever supplies the non-anchor elements.
  • Shared interface: Every variant uses the same video diffusion backbone and relative camera-action interface, differing only in how historical information is stored and read.The action representation is a 12-dimensional relative camera RT with 9 rotation and 3 translation entries.
  • Controlled comparison: Focused ablations vary Spatial read-out, Compression type, recurrence structure, or raw context length one mechanism at a time under the shared interface.Representative rows include anchor-only I2V, raw Context at different lengths, Compression, Spatial Memory, and State-Space recurrence.

4 Training Protocol

Training holds the backbone, optimization, action interface, sampler, data interface, and evaluation path fixed so differences isolate memory profiles. Replay sampling supplements scalar loss by exposing continuity, identity, and action-alignment failures before final return evaluation.

  • Controlled protocol: All variants share the backbone, optimizer, schedule, sampler, data interface, action representation, and evaluation path; only the memory profile changes.This control separates storage, compression, read-out, and recurrence effects from hidden training-recipe changes.
  • Data and actions: Training uses 81-frame targets at 352×640 resolution with per-frame 12-dimensional relative-RT actions and fixed field-of-view context retrieval.The Context-as-Memory dataset supplies long camera trajectories, poses, prompts, and source videos, using the first target frame as the reference frame.
  • Replay diagnostics: Replay samples use the evaluation generation path and supplement scalar denoising loss with visual checks for boundary continuity, identity drift, and action-visual alignment.Fixed replay cases are sampled at matched intervals with the same context retriever, RT-relative action constructor, and memory profile.
  • Replay diagnostics: Replay sampling is treated as a fixed in-training protocol because nearly identical denoising losses can mask sharply different memory behavior.Slightly worse loss can coexist with better identity preservation when the memory path carries useful evidence across segments.
  • Training–evaluation bridge: Replay exposes where memory mechanisms first become unreliable but does not replace final return metrics, which follow dataset-backed camera trajectories.Observed failures include delayed context drift, lost object identity despite preserved layout, and scene evidence retained at local reconstruction cost.

5 Evaluation

The evaluation separates camera-action following from memory after leaving and revisiting a scene through replay, in-domain return, and open-domain return branches. These branches use pixel, perceptual, and semantic checks suited to their available ground truth.

  • Evaluation branches: Evaluation reports three branches: Replay, in-domain return, and open-domain return, preventing camera-action following and scene preservation from collapsing into one score.Replay follows dataset trajectories; in-domain return uses a GT-backed 180° loop; open-domain return uses held-out edited first-frame probes.
  • Evaluation branches: Open-domain return avoids pixel differences as geometric truth and instead scores held-out first-frame probes through semantic verification.This split has no pixel-level ground-truth trajectory, so it uses a visual-return PSNR proxy plus Qwen3-VL-30B-A3B semantic scores for Scene and Special Identity.
  • Open-domain probes: The open-domain pool contains 20 game-style environment prompts with 8 edited variants each, inserting distinctive identity anchors while preserving surrounding scenes.The objects are designed to be easy to name, localize, and revisit after the camera returns.
  • Evaluation branches: Replay averages PSNR, SSIM, and LPIPS over three generated chunks, while in-domain return averages pairwise image scores on mirrored outgoing and returning frames.The in-domain split provides ground-truth poses and images for direct loop-closure PSNR and VLM checks.
  • Open-domain probes: The open-domain VLM score weights appearance and presence most heavily, followed by view and scene, then rescales the weighted sum from [0,5] to [0,100].The weighting is 0.45sappearance + 0.25spresence + 0.20sview + 0.10sscene.

6 Additional Evaluation Details

A judge-robustness sanity check re-scores open-domain recognition cases with alternative VLM judges and a human anchor. Their scores remain closely aligned with Qwen3-VL, preserving conclusions and memory-profile rankings.

  • Judge robustness: The sanity check re-scores stratified open-domain cases using Qwen3-VL-30B-A3B, two stronger closed-source judges, and a human anchor.All judges receive the same images and prompt while evaluating salient-object, viewpoint, and scene preservation.
  • Judge robustness: Alternative judges remain correlated above 0.90 with Qwen3-VL and stay within a few points on average.This indicates only a small scoring offset across judges.
  • Judge robustness: Changing the judge preserves the subset’s relative reading rather than reordering the memory profiles.The alternative judges therefore support the same comparative conclusion as Qwen3-VL.

7 Results

Results show that replay fidelity and semantic memory at revisit time are distinct: open-domain return is the most discriminative stress test. Raw context provides the capacity baseline, while block-wise recurrence currently offers the strongest compact-memory bias.

  • Diagnostic protocol: Open-domain return is the most discriminative memory stress test, whereas replay and in-domain return remain necessary filters.Replay evaluates known camera trajectories; open-domain return tests whether the model reconstructs the same world state after leaving visible support.
  • Capacity: Raw Context is the key capacity baseline: K=5 raises open-domain VLM from 12.25 to 50.75, while K=20 reaches 58.63.The replay image bundle improves much less dramatically, indicating that raw history primarily benefits semantic return rather than local pixel fidelity.
  • Spatial summaries: Spatial summaries improve replay efficiency but remain unreliable as semantic memory because low-bandwidth storage and read-out under-specify salient-object identity.Dedicated cross-attention improves open-domain VLM to 17.12, but the family remains far below raw Context and block-wise State-Space.
  • Compression: Compression must preserve return-critical evidence: length compression with r=4 can outperform weight-only compression, while hybrid compression destroys identity evidence needed for revisit.Pooling removes temporal evidence before weighting can select which views matter, so the returned object is often under-specified.
  • Recurrence: Block-wise State-Space recurrence is the strongest current open-domain memory bias, with nearly double the open-domain VLM of the legacy hybrid row.Its structural integration makes the state harder for the network to bypass when the camera leaves visible support, even when replay metrics are weaker.
  • Evaluation standard: Specialized memories should be judged by how much of the K=20 open-domain benefit they retain, not by how much they beat I2V.Only block-wise State-Space exceeds raw Context on open-domain VLM under this comparison bar.

8 Discussion

Echo-Memory isolates memory/context as the main experimental variable and shows that preserving a revisitable world differs from merely following camera actions or replaying trajectories. Its findings motivate revisit-aware memory design while highlighting evaluation and scope limitations.

  • Contribution: Controlled comparisons separate camera-action following from preserving the world after actions leave it by fixing the backbone, optimizer, sampler, and evaluation protocol.This makes the memory/context profile the main experimental variable.
  • Main findings: Raw context improves open-domain revisit more reliably than replay metrics, while compact Spatial Memory and hybrid compression lose identity evidence without object-aware retention and read-out.The findings distinguish capacity from compactness and emphasize the importance of retaining salient object evidence.
  • Evaluation lesson: Replay quality and revisit quality are not monotonically aligned, so model selection should report replay, in-domain return, and open-domain return separately.Choosing models by Replay alone can favor the wrong family for interactive, revisitable world generation.
  • Limitations: The study is limited by a single dataset, possible sensitivity to poses, camera statistics, compute, and curricula, VLM-judged open-domain scores, and the lack of a cheap training-time revisit signal.The passage notes that broader human calibration could strengthen the comparison.
  • Outlook: Future memory designs should preserve high-value object evidence, provide object-level read-out, evaluate structured recurrence explicitly, and measure whether models return to the same world after leaving it.The outlook frames revisit awareness as a central design and evaluation objective.

A Action World Models: Preliminaries and Related Work

Action world models combine video generation, action conditioning, and memory to preserve scene and object evidence across long-horizon camera revisits. Echo-Memory studies these factors through a controlled interface, matched comparisons, and a three-branch evaluation stack rather than introducing a new memory mechanism.

  • Action World Models: Generated visual rollouts make memory essential because the model must preserve scene and object evidence after the camera leaves and revisits a region.This extends world-model research from latent or predictive states to prompt-, camera-, and action-conditioned video rollouts.
  • Action and Camera Conditioning: Action representation is orthogonal to memory, spanning world-frame extrinsics, Plücker embeddings, optical flow, discretized tokens, and other camera parametrizations.The study treats action conditioning as a separate design axis from the memory subsystem.
  • Memory for Long-Horizon Video Generation: Long-horizon systems store history through retrieval or recurrent and learned states, but effective memory also depends on whether the generator can read information back after viewpoint changes.Related variants include recurrent gates, test-time fast weights, flow-equivariant states, and plug-and-play memory tokens.
  • Evaluation and Benchmarking: Standard image and video metrics do not capture long-horizon revisit consistency, motivating specialized revisit measures and model-as-judge evaluation protocols.The related literature reports rotation-loop metrics and uses strong VLMs such as GPT-4V or Qwen3-VL-30B-A3B.
  • How Echo-Memory differs: Echo-Memory contributes a common plug-in interface, matched ablations under a single-stage training budget, and a three-branch stack linking replay, in-domain revisit, and open-domain revisit.The protocol avoids changing the training loop, sampler, or evaluation protocol and does not introduce a new storage mechanism, retriever, or conditioning trick.

B Training and In-training Diagnostics … C Empirical Analysis

The study fixes the training and inference protocol while varying only memory modules, then uses replay sampling and a three-branch evaluation stack to diagnose whether models preserve the world at revisit time. The setup covers controlled data handling, single-stage optimization, hard-fail invariants, and empirical comparisons across Context, Compression, Spatial, and State-Space families.

  • B Training and In-training Diagnostics: Backbone, optimizer, schedule, and inference path remain fixed, so variant differences isolate storage, compression, read-out, or recurrence effects.Only the memory module varies across the comparison rows.
  • B.1 Data: Training uses real-world camera trajectories with per-frame poses, textual prompts, 81-frame segments at 352×640 resolution, 12-dim relative-RT actions, and K−1 retrieved historical frames.Historical frames are selected using pre-computed field-of-view overlap labels keyed on the segment mid-frame.
  • B.2 Single-stage Optimization: All variants use identical-length single-stage training with shared optimizer settings, avoiding multi-stage curricula because pilot experiments found no consistently superior schedule.The simpler protocol makes ablations easier to interpret.
  • B.3 Replay Sampling Methodology: Replay diagnostics supplement flow-matching loss with per-frame and per-chunk PSNR/SSIM/LPIPS, revealing drift onset and quality decay as memory carries information farther from the anchor.The scalar flow-matching loss can miss visually different boundary behavior or camera-action misconfiguration.
  • B.3 Replay Sampling Methodology: Replay sampling uses fixed held-out first frames, 81-frame generated segments, concatenated multi-segment sequences, evaluation-matched actions, and the shared generation path.This keeps replay close to the actual inference regime rather than a short single-clip proxy.
  • B.3 Replay Sampling Methodology: Multi-segment replay preliminarily stresses the later revisit failure mode by testing boundary continuity, identity preservation, and action–visual alignment after generated chunks replace immediate visual context.These samples provide qualitative evidence before the full evaluation bundle is interpreted.
  • B.4 Protocol Invariants: Evaluation enforces matching context handling, noise levels, K, supervision, flow-noise shift, camera-action loading, and mutually exclusive memory configurations.These are treated as hard-fail protocol invariants.
  • C Empirical Analysis: Empirical analysis compares Context, Compression, Spatial, and State-Space families through replay fidelity, in-domain revisit, and open-domain VLM evidence.Replay measures observed-trajectory fidelity, in-domain revisit tests controlled return to known scenes, and open-domain VLM tests salient-object and scene preservation without pixel-aligned reference.

C.1 Memory Families under Matched Evaluation

Matched evaluation shows that replay, in-domain return, and open-domain revisit can select different memory winners, so replay quality alone is insufficient. Raw context and block-wise State-Space recurrence preserve semantic return better than compact spatial memories, despite weaker or middling replay metrics.

  • Metric-bundle interpretation: Spatial Memory leads replay PSNR, long raw Context leads replay SSIM/LPIPS, and block-wise State-Space leads in-domain LPIPS and open-domain VLM.These inversions show that memory quality is not equivalent to image quality under ground-truth motion.
  • Raw context: Increasing raw Context history improves semantic return more clearly than it improves the full replay or in-domain image-metric bundle.Replay follows dataset trajectories, while open-domain revisit tests whether object identity and scene layout survive a generated excursion.
  • Metric-bundle interpretation: Spatial Memory wins replay PSNR but collapses on open-domain VLM, whereas block-wise State-Space loses the replay bundle yet wins open-domain VLM by a wide margin.The comparison holds the backbone, optimizer, sampler, and evaluation pipeline fixed.
  • Raw context: K=20 Context reaches 29.40 in-domain VLM and 58.63 open-domain VLM while remaining mid-pack on Replay.Raw retrieval uses no learned compression or specialized read-out and outperforms every compact memory mechanism on open-domain VLM in the matrix.
  • Metric-bundle interpretation: Replay is a dense, cheap health signal for trajectory coherence, but mean replay quality cannot serve as the final memory score because open-domain return can disagree.In-domain return provides an intermediate test under familiar, pixel-aligned statistics, while open-domain VLM probes distribution-shifted memory survival.

C.2 Spatial Storage and Read-out · C.3 Compression and Temporal Resolution

Spatial memory results show that read-out access is decisive: dedicated cross-attention improves open-domain returns, while withheld spatial tokens can paradoxically maximize replay. Compression likewise is not monotonic: temporal length reduction can outperform weighting, but hybrid compression backfires and remains a cost-control trade-off.

  • C.2 Spatial Storage and Read-out: Spatial Memory separates storage from access by comparing unreachable tokens, text-KV concatenation, and dedicated cross-attention read-out.These variants test whether gains come from storing a stable scene summary or from providing it with an effective access path.
  • C.2 Spatial Storage and Read-out: Dedicated cross-attention raises open-domain VLM to 17.12, compared with 6.00 for the default spatial baseline.The improvement shows that stronger read-out can help, although the spatial family remains weak as semantic memory relative to raw context.
  • C.2 Spatial Storage and Read-out: Inject-none achieves 14.66 Replay PSNR, 0.417 SSIM, and 0.541 LPIPS, outperforming active spatial read-out variants despite withholding stored tokens.The result indicates that better replay does not necessarily mean the summary is functioning as usable memory.
  • C.2 Spatial Storage and Read-out: Spatial read-out ranges from unreachable tokens to text-KV concatenation and gated dedicated cross-attention, making storage inseparable from how the generator accesses it.Dedicated read-out gives spatial tokens their own query mechanism and a zero-initialised gate that optimization can open.
  • C.2 Spatial Storage and Read-out: A G×G spatial grid captures rough locations but has limited capacity for distinctive object identity, weakening open-domain return after camera revisits.The probe uses salient objects distinguished by shape, colour, and category, making identity preservation a high-entropy memory requirement.
  • C.3 Compression and Temporal Resolution: Compression compares token weighting, which preserves temporal positions, with length reduction, which discards temporal resolution; hybrid compression combines both operations.These mechanisms make different assumptions about whether history is redundant across time or unevenly informative across observations.
  • C.3 Compression and Temporal Resolution: 43.25 open-domain VLM for length r=4 exceeds 22.38 for weight-only compression, showing that shorter context can outperform full-length weighted history.Length r=4 discards three out of every four temporal positions, whereas weight-only retains every temporal position.
  • C.3 Compression and Temporal Resolution: Hybrid compression reaches only 8.75 and 9.00 open-domain VLM, below both pure weighting and pure length pooling, while compression remains a cost-control trade-off rather than a capacity substitute.Length r=4 improves the Replay PSNR/LPIPS bundle over weight-only, but absolute replay values remain below the strongest Context and Spatial rows.

C.4 Implicit State-Space Memory … C.7 Cross-cutting Observations

The controlled comparisons show that memory quality depends on recurrence structure, raw capacity, storage/read-out choices, and evaluation axis rather than replay fidelity alone. Block-wise State-Space recurrence and raw context provide the strongest revisit robustness, while compactness can sharply reduce open-domain return.

  • C.4 Implicit State-Space Memory: Block-wise State-Space recurrence dominates open-domain VLM despite legacy hybrid recurrence achieving the best in-domain PSNR and strong replay PSNR.The two variants use the same evaluation bundle, while their replay and in-domain metrics remain close or favor the legacy hybrid.
  • C.4 Implicit State-Space Memory: Block-wise recurrence sacrifices replay fidelity because its non-bypassable recurrent state allocates capacity to summarizing the past.Its reported values are 9.59 Replay PSNR and 0.698 Replay LPIPS.
  • C.5 Raw Context Capacity: Raw context improves both replay and return, with open-domain VLM improving more steeply than replay metrics as retrieved history increases.The ablation compares anchor-only I2V with K=5 and K=20 retrieved historical observations.
  • C.5 Raw Context Capacity: Replay PSNR rises 2.51 points from K=1 to K=20, whereas open-domain VLM rises 46.38 points over the same range.This slope mismatch indicates that raw history contributes more to restoring the object and scene after the camera leaves than to pixel-level reconstruction.
  • C.5 Raw Context Capacity: Most open-domain VLM improvement arrives by K=5, which adds +38.50 points over the I2V baseline, while K=20 adds 7.88 points.The marginal value of additional retrieved frames therefore declines after the cold-start regime.
  • C.6 Quality–Time Trade-off: The quality–time trade-off has three distinct corners: Spatial Memory favors replay efficiency, block-wise State-Space favors semantic return, and raw context provides capacity.Raw Context at K=20 is described as the safest default when neither cost nor open-domain robustness is binding.
  • C.6 Quality–Time Trade-off: A single efficiency scalar would obscure a more than 11× open-domain VLM gap between Spatial Memory and block-wise State-Space: 69.00 vs. 6.00.The scalar could favor Spatial Memory because of cost while missing semantic-return performance.
  • C.7 Cross-cutting Observations: Across the matrix, replay and revisit are different axes, storage and read-out are separable, and compactness is not free.Spatial Memory and hybrid compression score 6.00 and 8.75 on open-domain VLM, versus 58.63 for raw context at K=20 and 69.00 for block-wise State-Space; small differences within roughly five points should not be over-interpreted.
Loading 2606.09803v1…