PaperlayerView exact revision on arXiv

Source-linked AI summary

Does Video Memory Use What It Retrieves? A Causal Audit of Memory Specificity

Aditi Tiwari, Akshit Bhalla, Darshan Prasad, Heng Ji

arXiv:2609.12090v1cs.CV

TL;DR

Memory evaluations commonly test whether memory helps, not whether gains depend on the correct retrieved content. This paper uses read-time memory substitution to distinguish those questions, finding that dependence varies from generic support to broader context to exact stored content across model families.

  • Problem

    Standard memory ablations compare performance with and without memory, leaving untested how much of the gain depends on the retrieved content.

    Source: §1 · p. 1

  • Method

    Read-time memory substitution changes the consumed memory value while preserving the rest of the computation, distinguishing memory benefit from memory specificity.

    Source: §1 · p. 2, §7 · p. 9

  • Results

    Memory dependence ranges from limited exact-content dependence in frozen world models, through graded context dependence in WorldMem, to strong spatial-memory dependence in SAM 2; on DAVIS, the mean region and boundary segmentation score falls from 0.926 with correct spatial memory to 0.182 with wrong spatial memory.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    Memory gains can depend on generic representation support, broader context, or exact episodic content, implying different memory-design choices across these regimes.

    Source: §1 · p. 2, §7 · p. 8

    The audit requires access to the memory-read interface, and results across model families and tasks support comparing dependence patterns rather than a common scale.

    Source: §7 · pp. 8–9

Abstract

from arXiv · show

Video models increasingly use memory to preserve information over long sequences, with the assumption that gains come from retrieving and using the correct past content. Standard memory ablations test whether memory helps, but not whether the retrieved content is responsible. We test this directly with read-time memory substitution, which replaces the consumed memory value while leaving the rest of the computation unchanged. This separates memory benefit from memory specificity, the extent to which the gain depends on retrieved content. Across frozen video world models, identity-free controls containing no evaluation-specific content recover essentially the full benefit on Ego-Exo4D and 7-Scenes and about 70% on TUM. In the Ego-Exo4D dose response, recovery falls from 102% to 1% as these values move away from observed training-memory representations, supporting representation repair as the best-supported explanation in this setting. WorldMem shows graded dependence. A wrong memory from the same trajectory recovers 94.1% of the PSNR benefit relative to zero content, while a donor from a disjoint trajectory and biome recovers 43.7%. SAM 2 shows strong content dependence. On DAVIS, replacing the correct spatial memory with a valid wrong memory reduces mean region and boundary score from 0.926 to 0.182. At MOSEv2 reappearance, it falls from 0.459 to 0.000. These results show that memory gains can depend on generic representation support, broader context, or exact episodic content. Read-time substitution provides a direct way to distinguish them.

1. Introduction

The introduction asks whether memory gains require the correct retrieved content, a question standard with/without-memory comparisons do not answer. It proposes read-time substitution and reports different dependence patterns across model families.

1. Introduction

  • Standard memory-versus-no-memory evaluations measure how much memory helps, but not how much of that gain depends on retrieved information.An unrelated but plausible memory could produce a similar improvement without supplying the correct past content.Source: §1 · p. 1
  • Memory specificity measures how much of the memory benefit depends on retrieved content; dependence may require generic support, broader context, or the exact past event.Identity-free memory can preserve benefit through representation repair without supplying episode-specific information.Source: §1 · p. 1
  • Retrieval can improve only the part of memory gain that depends on retrieved content; when broader context matters, preserving it may matter more than recovering the exact past state.If identity-free memory preserves the gain, retrieval is not the active ingredient in that portion of the improvement.Source: §1 · p. 1
  • Memory dependence ranges from limited exact-content dependence in frozen world models, through graded context dependence in WorldMem, to strong correct-spatial-memory dependence in SAM 2.Identity-free memory recovers essentially the full benefit on Ego-Exo4D and 7-Scenes and about 70% on TUM; on Ego-Exo4D, the interventions support representation repair as the best-supported explanation. A same-trajectory wrong memory recovers about 94% of the PSNR benefit relative to zero content, while a disjoint-trajectory-and-biome donor recovers about 43%; on DAVIS, the mean region and boundary segmentation score falls from 0.926 with correct spatial memory to 0.182 with wrong spatial memory.Source: §1 · p. 2
  • Read-time memory substitution replaces the consumed memory value while keeping the rest of the computation fixed, separating memory benefit from memory specificity.The audit compares correct memory with unrelated-scene or object memories, identity-free values, and values farther from observed training-memory representations.Source: §1 · p. 2

2. Related Work

Related work studies memory architectures and uses, as well as causal interventions that test whether information affects model outputs. This paper focuses on what information in retrieved video memory is responsible for gains.

2. Related Work

  • External-memory and retrieval-augmented methods show that retrieval can help, but not how much the gain depends on retrieved content.These lines of work include learned read/write operations, explicit addressing, key-value access, and large non-parametric stores.Source: §2 · p. 2
  • Video memory supports object identity, temporal context, and scene state, while much prior work asks how memory should be stored or retrieved.Related work also studies compressed history, alternative readout mechanisms, and state-space memory designs.Source: §2 · p. 2
  • Causal mediation, interchange interventions, and causal tracing modify internal representations while preserving surrounding computation to test what affects outputs.The paper applies this principle directly to the consumed memory value.Source: §2 · p. 2
  • Denoising, feature alignment, and test-time adaptation motivate identity-independent representation repair as an alternative to supplying instance-specific information.The audit separates generic representation support from broader contextual dependence and exact retrieved content.Source: §2 · p. 2

3. A Causal Audit of Memory Specificity

The audit separates the gain from having a memory pathway from dependence on retrieved content. Controlled read-time substitutions and task-specific readouts test which information must remain.

3.1. Memory Benefit and Memory Specificity

  • Memory benefit compares correct-memory performance with the content-free reference; benefit recovery measures how much of that gain survives substitution.Recovery of 1 means full recovery and 0 means none; the remaining fraction is the content-specific residual relative to that control.Source: §3.1 · p. 3

3.2. Read-Time Memory Substitution

  • Read-time substitution changes only the consumed memory value, preserving the query, selected locations, model, and earlier computation.The intervention measures the direct effect of the consumed value at the target step, not effects from changing earlier reads.Source: §3.2 · p. 3

3.3. The Substitution Ladder

  • The substitution ladder contrasts correct, wrong, identity-free, content-free, and strongly perturbed values to probe dependence on identity and context.Wrong memories can retain broader trajectory or scene context; identity-free values contain no evaluation-specific identity but preserve targeted memory-like statistics.Source: §3.3 · p. 4

3.4. Testing the Representation-Repair Hypothesis

  • Representation repair predicts lower recovery as identity-free values move farther from training-memory representations and as injection direction changes at fixed magnitude.Low specificity alone does not establish representation repair; the paper tests these additional predictions.Source: §3.1 · p. 3, §3.4 · p. 4

3.5. Family-Specific Readouts

  • The audit uses different readouts by model family: revisit consistency error for latent world models, PSNR and LPIPS for generative models, and J & F for segmentation.Higher PSNR and segmentation scores are better, while lower LPIPS and revisit consistency error are better.Source: §3.5 · p. 4

4. Experimental Setup

The experiments apply read-time substitutions across frozen video world models, WorldMem, and SAM 2. Their model interfaces, data, controls, and evaluation conditions differ by setting.

4.1. Frozen Video World Models

  • Identity-free and wrong memories retain nearly the full benefit on Ego-Exo4D and 7-Scenes, while TUM shows a mixed regime and recovery decreases as dreal increases.Figure 2 summarizes DINO-WM substitutions, training-memory mean recovery across datasets, and the Ego-Exo4D dose response.Source: Figure 2 · p. 5
  • DINO-WM experiments freeze the encoder and predictor while training a memory pathway, and test revisit events across TUM, Ego-Exo4D, and 7-Scenes.Training-memory controls exclude evaluation trajectories; 7-Scenes uses disjoint training and evaluation scenes.Source: §4.1 · p. 5

4.2. WorldMem

  • WorldMem uses its frozen pretrained Minecraft model and native memory bank, tested beyond a 600-frame context on 1,000 cases across four biomes.Controls include another place in the same trajectory, a donor from a disjoint trajectory and biome, same-scene mean content, and zero content.Source: §4.2 · pp. 5–6

4.3. SAM 2

  • SAM 2 is evaluated with its pretrained native memory on DAVIS visible-target frames and MOSEv2 reappearance events, with substitutions applied to spatial memory and object pointers.MOSEv2 uses training-split masks; memory is substituted only at the first reappearance frame, leaving the preceding trajectory unchanged.Source: §4.3 · p. 6

5. World Models Show Limited Exact-Content Dependence

World models vary in how much their memory gains depend on exact retrieved content: frozen hosts often retain gains with identity-free memory, while WorldMem retains more benefit as trajectory context is preserved.

5. World Models Show Limited Exact-Content Dependence

  • Across the studied world models, large memory gains often persist without exact retrieved content, while WorldMem shows graded dependence on broader context and exact place.This contrast includes frozen latent world models with weak exact-content dependence and WorldMem, where broader context remains important.Source: §5 · p. 6, §5.5 · p. 7

5.1. Identity-Free Memory Recovers Most of the Gain

  • Identity-free controls recover 102% of the correct-memory benefit on Ego-Exo4D and 101% on 7-Scenes, but 70% on TUM.The TUM interval is [59, 94], based on only two evaluation takes, so the result suggests rather than precisely estimates mixed content dependence.Source: §5.1 · p. 6

5.2. Recovery Falls Away from Observed Training Memories

  • On Ego-Exo4D, identity-free recovery falls monotonically from 102% at dreal = 0.002 to 1% at dreal = 0.97, supporting representation repair in this setting.Evaluation-specific identity is absent throughout; the changing factor is proximity to memory representations observed during training.Source: §5.2 · p. 6

5.3. Direction Matters at Fixed Magnitude

  • With injected norm fixed, recovery on Ego-Exo4D falls from 101% to 29%, 2%, and 0%; TUM shows the same qualitative pattern, falling from 65% to 0%.This indicates injection strength alone is insufficient to explain the identity-independent gain.Source: §5.3 · p. 6, Table 1 · p. 7

5.4. Weak Content Dependence Transfers to a Second Host

  • On the second host V-JEPA 2, wrong memory gives revisit consistency error 0.0455 versus 0.0442 with correct memory, a gap equal to about 1.5% of the total memory benefit.This is a second-host check only; the full substitution ladder and representation-repair analysis remain restricted to DINO-WM.Source: §5.4 · p. 7

5.5. WorldMem Shows Graded Content Dependence

  • 94.1% of the reference-relative benefit is recovered by a wrong same-trajectory place in WorldMem, versus 43.7% by a donor from a disjoint trajectory and biome.Mean same-scene content recovers 62.3%; LPIPS follows the same ordering, and the experiment does not establish the representation-repair mechanism.Source: Figure 3 · p. 7, §5.5 · p. 7

6. SAM 2 Shows Strong Content Dependence

SAM 2's segmentation depends strongly on spatial-memory content, especially when a target reappears after disappearance. Substitution sharply lowers scores, while the object pointer has a smaller, setting-dependent effect.

Spatial Memory Is Strongly Content Dependent on DAVIS

  • On DAVIS 2017, replacing correct spatial memory reduces J &F from 0.926 to 0.182; replacing only the object pointer leaves performance at 0.927.The spatial-memory collapse occurs clearly in eight of ten sequences; in the other two, current-frame evidence preserves the target.Source: §6 · p. 7

Correct Spatial Memory Is Critical at Reappearance

  • At MOSEv2 reappearance, wrong spatial memory lowers the mean J and F score from 0.459 to 0.000, while an identity-free prototype gives 0.003.Performance partially recovers over later frames because only the first memory read at return is changed; replacing the object pointer lowers the return score to 0.385.Source: §6 · p. 8, Figure 4 · p. 8
  • At MOSEv2 reappearance, the object pointer also contributes: replacement lowers return-frame J and F from 0.459 to 0.385, unlike its negligible effect on DAVIS target frames.Target transfer remains at or near zero across MOSEv2 conditions, indicating that spatial-memory corruption usually loses the target rather than transferring it to the donor object.Source: §6 · pp. 7–8, Figure 4 · p. 8

7. Discussion and Conclusion

Memory gains vary in their dependence on retrieved content: frozen world models often retain gains without exact identity, WorldMem relies on broader context, and SAM 2 relies strongly on correct spatial memory. The paper argues that memory benefit and memory specificity should be evaluated separately because they point to different design priorities.

Impact Statement

  • Read-time substitution is proposed as a diagnostic before investing in complex retrieval, with controls that distinguish exact retrieval, broader context, and generic memory support.The desired specificity is described as task dependent; future work could test what task ambiguity, objectives, memory bottlenecks, temporal distance, and current evidence contribute to its emergence.Source: §7 · p. 8

7. Discussion and Conclusion

  • The conclusions are bounded by the studied models and interventions; the full substitution ladder and mechanism tests are limited to DINO-WM, and TUM's split has only two evaluation takes.WorldMem controls change several forms of context at once, and SAM 2 is one native-memory example rather than a representative sample of all end-to-end systems.Source: §7 · pp. 8–9
  • Frozen world models show limited exact-content dependence, WorldMem depends more on broader context than exact place, and SAM 2 depends strongly on correct spatial memory.Ego-Exo4D supports representation repair as the best-supported explanation in that setting.Source: §7 · pp. 8–9

Impact Statement

  • If identity-free memory preserves most gains, retrieval capacity may add limited value for that part; when exact content is load bearing, retrieval quality matters for reliability.The paper frames these as implications for evaluating and designing memory systems.Source: Impact Statement · p. 12

B. Intervention and Control Details

  • Read-time substitution replaces the consumed memory value while leaving the surrounding computation unchanged, measuring dependence on memory content rather than memory-pathway removal.The query, selected memory locations, number of entries, temporal and positional information, weights, and earlier computation remain fixed.Source: B. Intervention and Control Details · p. 12, §B.1 · p. 12

B.2. Construction of DINO-WM Memory Controls

  • Training-derived identity-free controls use training data only and contain no evaluation-specific identity, while random and zero controls do not preserve the same memory structure.The controls remove different information, so recovery is reported for each rather than collapsed into one score.Source: §B.2 · p. 12

C.1. Expanded Ego-Exo4D Substitution Ladder

  • 93% of the Ego-Exo4D memory benefit is recovered by a wrong-scene memory, while the learned constant recovers 40% and random and zero controls recover almost none.The training-memory mean recovers the full benefit and identity-free controls recover nearly all of it.Source: §C.1 · p. 12

C.2. Leakage-Safe Recovery Across Datasets

Across datasets and DINO-WM diagnostics, identity-free memory often retains substantial benefit, but recovery declines as its representation moves away from training memories or its injection direction changes. Similar gates and signal magnitudes do not explain the recovery pattern, which persists across repeated reads.

C.2. Leakage-Safe Recovery Across Datasets

  • Ego-Exo4D and 7-Scenes show little dependence on evaluation-specific identity, while TUM suggests mixed dependence but is not a precise estimate because its leakage-safe split has only two evaluation takes.Values slightly above 100% are un-clipped finite-sample estimates around full recovery.Source: §C.2 · p. 13, Table 7 · p. 13

C.3. Dose Response and Internal Diagnostics

  • 102% recovery falls to 32% at α = 0.50 while the gate remains fully open at 1.000, so differential gate activation does not explain the early decline.Recovery falls as the training-memory mean moves farther from observed training-memory representations; dreal is an empirical support measure, not a formal manifold distance.Source: §C.3 · p. 13, Figure 5 · p. 14

C.4. Direction at Fixed Injection Magnitude

  • Recovery still collapses when injection magnitude is fixed and only direction changes, showing that injection strength alone is insufficient to explain the identity-independent gain.The fixed norms are 335.0 on Ego-Exo4D and 346.6 on TUM.Source: §C.4 · pp. 13–14, Table 8 · p. 14

C.5. Gate and Injection Diagnostics

  • Correct, wrong-scene, training-mean, barycenter, and identity-free sample memories keep the gate fully open and produce injected norms of approximately 328 to 334.Random and zero inputs produce much smaller signals; the separate fixed-magnitude direction control shows magnitude alone is insufficient.Source: §C.5 · p. 14

C.6. Specificity Under Repeated Memory Reads

  • The training-memory mean retains approximately full recovery through ten substituted reads, while wrong-scene recovery falls from 102% to 89% and zero memory remains at 0%.The repeated-read experiment includes effects accumulating after earlier substitutions, unlike the main single-read audit.Source: §C.6 · p. 14

C.7. Robustness to Retrieval Width and Memory Budget

The correct-versus-wrong memory gap remains near zero on Ego-Exo4D and larger on TUM across tested retrieval widths and memory budgets. A separate scenario-level split finds similarly small gaps for distinctive and generic Ego-Exo4D scenes.

C.7. Robustness to Retrieval Width and Memory Budget

  • Ego-Exo4D retains a near-zero correct-versus-wrong gap across tested retrieval widths and memory budgets, while TUM retains a larger gap.The dataset difference is not tied to one retrieval width or memory budget.Source: §C.7 · pp. 14–15, Figure 6 · p. 15

C.8. Robustness to Scene Appearance

  • Distinctive and generic Ego-Exo4D scenario groups show similarly small correct-versus-wrong gaps, so scene appearance does not explain weak dependence under this partition.The groups comprise 111 distinctive and 108 generic takes.Source: §C.8 · p. 15

D. V-JEPA 2 Second-Host Check

The V-JEPA 2 second-host check finds weak dependence on retrieved identity: correct and wrong memories produce the same RCE, and training-derived identity-free controls recover approximately 98% of the correct-memory benefit. This ablation does not reproduce DINO-WM's full substitution ladder or mechanism tests.

D. V-JEPA 2 Second-Host Check

  • Correct and wrong memory give the same RCE, while the training-memory mean and identity-free barycenter recover approximately 98% of the correct-memory benefit.Random memory recovers 26.1%, and zero performs slightly worse than the no-memory reference.Source: D. V-JEPA 2 Second-Host Check · p. 15
  • The second-host ablation does not reproduce DINO-WM's full substitution ladder or mechanism tests.The learned-constant and identity-free-sample controls, dose response, and fixed-magnitude analysis remain restricted to DINO-WM.Source: D. V-JEPA 2 Second-Host Check · p. 15

E. WorldMem Content Dependence

WorldMem shows graded dependence on memory context, while SAM 2’s spatial-memory results show strong content dependence in visible-frame evaluation. The SAM 2 intervention also changes internal representations even when visible-frame predictions remain stable.

E. WorldMem Content Dependence

  • WorldMem has no true memory-off condition in this evaluation, so content-dependence recovery is assessed relative to a zero-content reference.The reference is an active zero-content condition, not memory removal.Source: E. WorldMem Content Dependence · pp. 15–16

F.1. DAVIS Visible-Frame Substitution

  • On ten DAVIS sequences, spatial-memory substitution removes most segmentation performance, while object-pointer substitution has almost no effect on generally visible targets.The spatial collapse is clear in eight of ten sequences; current-frame evidence preserves the target in the other two.Source: §F.1 · p. 16

E. WorldMem Content Dependence

  • 94.1% of the PSNR benefit is retained with a wrong same-trajectory memory, versus 43.7% with a disjoint-trajectory, different-biome donor.The ordering indicates graded dependence on broader context; exact place identity adds little beyond the same-trajectory control.Source: E. WorldMem Content Dependence · p. 15, Table 12 · p. 17

F.2. SAM 2 Intervention Verification

  • A valid wrong pointer changes SAM 2’s internal representation despite stable visible-frame predictions, while a very large random pointer can collapse the prediction.Spatial-memory substitution causes a much larger internal change and collapses the mask on the diagnostic sequence, so the weak visible-frame pointer effect is not due to an inactive intervention.Source: §F.2 · p. 17, Table 14 · p. 17

F.3. MOSEv2 Reappearance

At MOSEv2 reappearance, correct spatial memory is critical, and the strong spatial-memory effect persists across bootstrap samples and disappearance durations. Object-pointer importance varies by setting: it is greater for reacquired targets than for generally visible DAVIS frames.

F.3. MOSEv2 Reappearance

  • Wrong spatial memory and the identity-free prototype reduce MOSEv2 return-frame performance to essentially zero, while the object pointer also contributes but less.Only the first reappearance read is substituted; later performance partially recovers when predictions return to the intact memory state, and target transfer remains near zero.Source: §F.3 · p. 17

F.4. Bootstrap Uncertainty on MOSEv2

  • Wrong-spatial return-frame performance remains at zero across 10,000 bootstrap resamples, so the spatial-memory effect is not driven by a small subset of events.Wrong-pointer performance remains substantially higher, preserving the channel-specific difference.Source: Table 15 · p. 17, §F.4 · p. 18

F.5. Object Pointer on Re-Detected Events

  • On 49 events where SAM 2 re-detects the object with correct memory, replacing the pointer reduces J &F from 0.910 to 0.757, an absolute drop of 0.153.The pointer is load bearing when the target must be reacquired, despite its small effect on generally visible DAVIS frames.Source: §F.5 · p. 18

F.6. Dependence Across Disappearance Durations

  • Spatial-memory substitution reduces return-frame performance to approximately zero across all three disappearance-duration groups.Correct-memory performance is lower for longer gaps, but spatial-memory dependence remains strong across the range; the pointer contributes in every group without a monotonic duration pattern.Source: §F.6 · p. 18
Loading 2609.12090v1…