PaperlayerView exact revision on arXiv

Source-linked AI summary

When Memory Lies: An Empirical Study of Spatial Memory Staleness in VLM Agents

Yushi Sun, Yanjie Zhang

arXiv:2608.04574v1cs.CL

TL;DR

Spatial-memory claims can become stale as environments change, creating a need to reconcile memory with current observations before acting. This paper evaluates staleness auditing and navigation in dynamic FrozenLake, finding that stale-memory safety depends on reliable visual grounding and action selection.

  • Problem

    Existing evaluations rarely test whether changed spatial memories are reconciled with current observations before safety-relevant actions.

  • Method

    The study pairs text- and image-based staleness detection with downstream navigation using persistent cell-level memory in a dynamic FrozenLake testbed.

  • Results

    Vision F1 ranges from 0.887 to 0.067, while raw stale memory raises GPT-4o deaths from 28.0% to 74.4%; filtering helps inconsistently and oracle labels add no significant gain.

  • Takeaways & Limitations

    Spatial-memory staleness is a safety failure mode, with reliable visual auditing and safe action selection remaining open challenges.

  • Takeaways & Limitations

    The controlled, fully observed 8×8 FrozenLake setting yields lower-bound numbers, while some mechanistic and vision analyses use narrower model and seed coverage.

Abstract

from arXiv · show

Memory-augmented VLM agents act on persistent spatial knowledge, yet that knowledge silently goes stale as the environment changes. We ask what happens when an agent must reconcile a confident memory claim with a contradicting observation, and whether current models can catch the conflict before it becomes a safety-relevant mistake. Using a dynamic FrozenLake testbed, we pair a staleness-detection task with a downstream navigation task across three closed-source models and three open-weight VLMs under both text and image inputs (1,800 detection runs, and 12,000 text-mode navigation episodes over four LLM navigators at a shared 50-seed scale). Three findings emerge. First, text solvability does not imply visual grounding: models that flag stale entries reliably from text nonetheless span vision F1 from 0.887 down to 0.067 on the identical grids, and the weakest keeps making fluent, confident decisions that ignore the image. Second, consuming stale memory without an audit is a safety liability: in our primary GPT-4o setting, an agent that trusts raw memory dies more than twice as often as the same agent given no memory at all. Third, auditing helps but does not close the gap: a transparent read-time filter removes much of the safety cost in text mode, yet even oracle stale labels bring no further significant gain on the current grid size, and when visual auditing is unreliable, filtering yields no consistent benefit. Together these results frame spatial-memory staleness as a safety failure mode and isolate reliable visual grounding and action selection under memory--observation conflict as the central open challenges for memory-augmented agents.

1 Introduction

The paper studies how stale spatial memories conflict with current observations in dynamic environments, and shows that unreliable visual auditing can turn persistent memory into a safety liability. It evaluates auditing and filtering across modalities, models, and navigation tasks.

  • Core Findings: Raw stale memory is a safety liability when consumed without an audit, whereas filtering contradictory entries can enable safer detours.The failure mechanism is an agent following a remembered safe cell after it becomes a lethal hole; the figure reports OMCD reducing death from 74.4% to 31.6%.
  • Problem and Testbed: SpatialSTALE tests whether agents reconcile textual cell-level memory claims with changed FrozenLake observations before navigating.Nonterminal cells can switch between safe frozen ice and lethal holes, while current observations are presented as text or images.
  • Core Findings: Vision F1 ranges from 0.887 for Qwen to 0.067 for GLM on identical grids, despite reliable stale-entry detection from text.The weakest case continues issuing fluent stale decisions with little detectable sensitivity to image contradiction.
  • Study Design: The study links entry-level text/vision auditing to safety-sensitive navigation across three closed-source and three open-weight VLMs.It uses paired auditing and navigation evaluations to characterize modality-dependent behavior and stale-memory safety costs.
  • Core Findings: The paper identifies reliable visual auditing and action selection over filtered memory as the remaining challenges for reducing stale-memory harm.Its contribution characterizes where transparent filtering helps and where it ceases to help because upstream visual auditing is unreliable or downstream action selection fails.

2 Related Work

Prior work spans persistent-memory agents and methods for managing stale factual knowledge, alongside benchmarks for spatial reasoning, visual hallucination, reflection, embodied planning, and environmental change detection. These studies provide complementary foundations for analyzing spatial-memory staleness in changing environments.

  • Memory agents and knowledge staleness: Persistent-memory agents study how systems accumulate, retrieve, reflect on, and reuse past experience, while complementary work examines factual-knowledge validity and updating outdated associations.Examples include Voyager, GITM, Reflexion, MemGPT, temporal-validity benchmarks, and weight-editing methods.
  • Spatial reasoning and hallucination in VLMs: Spatial benchmarks evaluate VLM perception, reasoning, planning, and low-level visual capability on fixed observations, while hallucination benchmarks probe errors under static visual inputs.The cited benchmarks include Spatial-Eval, SpatialVLM, SpatialRGPT, VSR, What’s Up, SpatialBot, VSP, BLINK, POPE, and HallusionBench.
  • Reflection, embodied planning, and change detection: Reflection methods revise outputs from feedback, embodied planners combine language reasoning with actions or grounded feedback, and classical benchmarks study image differences or navigation in unknown environments.This work connects feedback-driven revision and grounded action with change detection and navigation settings.

3 Study Design

The study uses controllable dynamic 8 × 8 gridworlds to separate stale-memory detection from its consequences for navigation. It varies when and where cells change, audits textual memory against text or image observations, and compares raw, filtered, and absent memory under shared-seed evaluation.

  • Experimental pipeline: Each instance generates an original 8 × 8 grid and memory snapshot, changes selected cells, audits memory, then navigates with raw, filtered, or no memory.Detection measures recognition of invalid beliefs, while navigation measures whether recognition changes behavior.
  • Memory construction: Memory coverage is fixed at one entry per cell, yielding N = 64 entries after 20 capped random walks on the original grid.The original grid has approximately 25% holes, fixed start and goal cells, and at least one valid path.
  • World changes and evaluation: After the snapshot, nonterminal frozen and hole cells are flipped while preserving a route to the goal, and each entry receives a binary valid-or-stale judgment against the current observation.The same changed instances support downstream navigation, linking recognition to safety-relevant action.
  • Change regimes: The three regimes vary change timing, requested count, and spatial localization: L1 samples across the grid, L2 uses radius 2 neighborhoods, and L3 introduces online events every fifth step.L1 and L2 detect once before navigation, whereas L3 detects after each realized event.
  • Modalities and statistical design: Text mode presents a coordinate-labeled list, vision mode a 384×384 color-coded rendering, and both retain textual memory across 50 shared deterministic seeds.The shared seeds enable paired Wilcoxon signed rank tests and correlations between detection F1 and navigation success.

4 Memory Strategies and the OMCD Intervention

The section compares four memory-presentation strategies for navigation under conflicting memory and observations. OMCD filters stale entries through batched, event-aware audits, while Oracle tests whether better stale labels would improve remaining navigation performance.

  • Memory Strategies: Four strategies isolate the effects of no memory, raw memory, one-shot self-verification, and OMCD filtering under identical navigation conditions.All strategies share the same current full-grid observation, action space, and step budget.
  • Memory Strategies: NoMemory chooses actions from the current grid without a persistent belief store, providing the reference for measuring memory’s effects.The prompt includes the model’s position, goal, and remaining step budget.
  • Memory Strategies: NoFilter passes all 64 original memory entries unchanged, measuring the safety cost of consuming persistent beliefs that may contradict the visible grid.Its paired comparison with NoMemory isolates the effect of trusting raw stale memory.
  • Memory Strategies: SelfVerify performs one full-store consistency check before navigation and removes entries judged inconsistent, testing whether a simple read-time filter mitigates raw-memory costs.It performs no further audit during the episode.
  • OMCD Intervention: OMCD audits memory in batches of B = 10, removes detector-flagged stale entries, and under L3 re-audits after realized environmental changes.Each batch receives the current text-grid or rendered-image observation and returns binary stale judgments with brief reasons.
  • OMCD Intervention: Oracle supplies ground-truth stale labels through OMCD’s filtering interface to test whether residual navigation error is limited by stale-label quality at the current scale.The Oracle–OMCD contrast identifies whether better labels would provide additional benefit.

5 Results

Results show that text-based staleness detection is strong for capable models, but safety depends on whether stale memory is audited before navigation. In primary GPT-4o text navigation, blindly trusting stale memory is substantially more dangerous than using no memory, while oracle labels provide no detectable downstream advantage over the learned audit.

  • Staleness detection: Above 0.88 average text F1 is achieved by all three closed-source models and GLM-5.1, whereas InternVL3-2B and InternVL3-8B average about 0.22 and 0.31, respectively.InternVL3-8B exceeds InternVL3-2B in both modalities across every regime, and capable models degrade on L3 as changes arrive incrementally.
  • Staleness detection: Incremental L3 changes shift viable text detectors from over-flagging in L1/L2 to under-flagging, making missed stale entries the dominant error.Change direction is safety-relevant because thaws turn claimed-safe cells deadly, whereas freezes make claimed-danger cells safe.
  • Navigation safety: 14.4% success and 74.4% death occur for blindly trusted stale memory on primary L2 GPT-4o, versus 28.8% success and 28.0% death with no memory.Stale memory is 2.7 times deadlier than no memory in this setting.
  • Audit effectiveness: No detectable difference appears when OMCD’s learned text labels are replaced with Oracle labels in any regime (|∆| ≤0.02, p > 0.2).The result is interpreted as failure to detect a benefit at the current sample size, consistent with the learned audit being close to label-saturating in the fully observed 8×8 text setting.
  • Failure mechanisms: 8.5 and 7.8 steps are the NoFilter death lengths for GPT-4o and Claude, versus 14.5 and 14.6 steps for successful trajectories, indicating a rush into mislabelled lethal cells.Claude’s NoMemory deaths instead terminate at 16.1 steps, reflecting exhaustive exploration before failure; OMCD lies between these mechanisms.

6 Conclusion … C Navigation Decision Procedure

The study shows that stale spatial memory can make VLM navigation less safe than using no memory, while auditing helps only when perception reliably detects contradictions. It isolates visual grounding and action selection under memory–observation conflict as central challenges, within a controlled fully observable benchmark.

  • 6 Conclusion: Raw stale memory is less safe than no memory because stale beliefs can route the agent directly into a hole.A controlled text audit removes much of this safety cost, but its value depends on reliable perception.
  • 6 Conclusion: The benchmark uses an 8×8 FrozenLake with symbolic changes and full observability, making each death traceable to a specific stale entry.The authors describe the resulting numbers as lower bounds from a clean setting rather than field estimates.
  • A Difficulty Level Configuration: Each instance contains exactly N=64 memory entries, with a guaranteed valid path from S = (0, 0) to G = (H−1, W−1).The common grid parameters include an initial hole fraction of 0.25 and rejection sampling for path validity.
  • B Game Setup and Rendering: The grid is presented either as a coordinate-labeled text list or a 384 × 384 RGB image, while memory entries remain text in every condition.Vision images use 48 px per cell and coordinate labels, with distinct colors for frozen cells, holes, start, goal, and the agent.
  • B Game Setup and Rendering: Memory is initialized from 20 random-walk exploration episodes and completed with synthetic entries for unvisited cells, eliminating coverage variance.This process guarantees exactly 64 entries per instance, each anchored to a cell’s correct prechange type.
  • C Navigation Decision Procedure: All strategies use the same per-step LLM navigator and differ only in the memory text supplied and whether staleness detection runs.The navigator receives the current full-grid observation, position, goal, and remaining step budget, then returns one of four moves.
  • C Navigation Decision Procedure: NoMemory omits memory, NoFilter retains all 64 entries, SelfVerify removes self-judged inconsistencies, and OMCD or Oracle removes detected stale entries before navigation.Under L3, OMCD and Oracle re-audit after each realized event, whereas the other strategies keep their memory views fixed.

D Binary vs. Continuous Staleness Output

On GPT-4o text-mode evaluation, thresholded continuous staleness scores did not outperform binary classification. Because score meanings varied across models, the paper uses binary output for OMCD while leaving the benchmark output-agnostic.

  • D Binary vs. Continuous Staleness Output: On GPT-4o, the best continuous result at τ=0.7 achieved F1 = 0.881, trailing binary classification by 3 points.The comparison used L1, N=50 seeds, and text mode.
  • D Binary vs. Continuous Staleness Output: At s=0.6, true-stale rates were ∼80% for GPT-4o and ∼35% for Claude, showing that continuous scores have heterogeneous semantics across models.The cross-model comparison used N=50 seeds per model, so no global threshold works for all models.
  • D Binary vs. Continuous Staleness Output: Binary judgments avoid the cross-model calibration problem by asking the same yes/no question and aggregating yes/no outcomes.This contrasts with interpreting numerical continuous scores using a shared threshold.
  • D Binary vs. Continuous Staleness Output: The main paper therefore uses binary output for OMCD, while the benchmark admits any detector producing ˆyi ∈ {stale, valid}.This output choice affects OMCD alone; the benchmark itself is agnostic to output type.

E Batch Size Ablation

The ablation finds that increasing OMCD batch size monotonically reduces detection F1, while B=10 preserves nearly the quality of per-entry queries at 9× lower inference cost. B=10 is therefore recommended as a cost–quality trade-off for N=64, with scaling left for future work.

  • E Batch Size Ablation: On GPT-4o in text mode with N=50 seeds and 64 memory entries, OMCD requires ⌈64/B⌉ LLM calls per detection.The ablation evaluates batch size B and reports detection quality alongside inference cost.
  • E Batch Size Ablation: F1 decreases monotonically with batch size as attention dilutes across more concurrent entries.The result indicates a quality cost from processing more entries concurrently.
  • E Batch Size Ablation: A 2.7pp F1 drop from B=1 to B=10 accompanies a 9× reduction in inference cost.B=1 uses per-entry queries and is expensive, whereas B=10 retains nearly comparable detection quality at substantially lower cost.
  • E Batch Size Ablation: B=20 incurs a further 3.6pp F1 drop from B=10, suggesting the LLM begins losing track of entry identities within the batch.This sharper decline marks a regime where concurrent batching more strongly harms attribution across entries.
  • E Batch Size Ablation: B=10 is a reasonable cost–quality trade-off for N=64, while whether batch size should grow sublinearly with larger memory stores remains future work.The authors expect the same trend for larger memory stores but do not provide a scaling study.

F Trajectory Length Decomposition · G Navigation Trace Studies · H Detection Diagnostics

The appendix shows that navigation failures reflect stale-memory routing rather than longer successful paths, while traces illustrate how filtering can avert an early death that memory-free exploration may replace with timeout. Additional diagnostics refine, but do not extend, the aggregate detection results.

  • F Trajectory Length Decomposition: 8.5 and 7.8 steps: GPT-4o and Claude NoFilter fatal trajectories are shortest, consistent with stale memory directing agents into holes early.Qwen differs: its few OMCD deaths occur earliest, at 3.1 steps, so the signature is not universal.
  • F Trajectory Length Decomposition: |∆| < 0.5 steps: successful trajectory lengths vary little across strategies within every model.This rules out longer, luckier NoMemory routes as the explanation for its reversal against OMCD on Claude and Qwen.
  • G Navigation Trace Studies: NoFilter dies at step 9 after memory steers GPT-4o toward a stale SAFE entry at (5,4), which had thawed to H.The trace starts at (0,0), follows the memory-suggested route, and ends with death and reward −1.
  • G Navigation Trace Studies: OMCD succeeds at step 26 after flagging mem_044 at (5,4) as stale and detouring around the misleading entry.The filtered trace avoids (5,4) and reaches the goal with reward +1.
  • G Navigation Trace Studies: 74.4%: NoFilter deaths across 50 L2 seeds occur almost always within the first 15 steps, while OMCD succeeds 32.8% of the time.OMCD failures mainly involve missed stale entries causing late deaths or long detours that exhaust the step budget.
  • G Navigation Trace Studies: ∼32%: NoMemory has the highest timeout rate but the lowest death rate, at 28.0%, because it explores slowly without confident stale claims.In the representative trace, the agent avoids a visibly thawed hole but times out at step 50 near the goal.
  • H Detection Diagnostics: Three appendix diagnostics refine the aggregate F1 results in Figure 3 without serving as additional primary endpoints.They support the main-text interpretation of detection performance.

H.1 Precision–Recall Error Regimes · H.2 Recall by Change Direction · H.3 Detection F1 and Navigation Success

Error patterns depend on regime and modality: snapshot settings over-flag stale cells, viable online text settings miss them, and GLM vision collapses on both precision and recall. Change-direction recall is asymmetric for Claude vision, while GPT-4o text detection F1 shows no association with navigation success.

  • H.1 Precision–Recall Error Regimes: Snapshot regimes L1/L2 generally have P < R, indicating aggressive stale-cell recovery with more false alarms.This pattern appears in both text and vision markers in Figure 7.
  • H.1 Precision–Recall Error Regimes: Viable L3 text series instead have P > R, shifting the dominant residual error toward missed stale entries.Figure 7 identifies viable L3 text points as under-flagging.
  • H.1 Precision–Recall Error Regimes: GLM vision collapses below 0.15 precision and 0.10 recall, unlike the other error regimes.Both precision and recall fall near the lower-left region of Figure 7.
  • H.2 Recall by Change Direction: Claude vision recall favors the deadly class by 17.2 pp on L1 and 30.9 pp on L2.Qwen is near ceiling on both classes, whereas GLM vision lacks enough recall for meaningful directional comparison.
  • H.2 Recall by Change Direction: On Claude vision L2, thaw recall is 0.945 (N=436) versus 0.636 for freeze (N=121).A thaw is F→H and lethal when trusted; a freeze is H→F and overcautious when trusted.
  • H.3 Detection F1 and Navigation Success: All three GPT-4o text correlations have p > 0.67, and the wider L3 F1 spread yields no navigation-success gradient.The null relationship indicates that, after reliable text auditing, label accuracy is not the observable source of residual navigation variation.
  • H.3 Detection F1 and Navigation Success: GPT-4o text shows no per-seed association between detection F1 and OMCD success: Pearson r ranges from +0.005 to +0.060.All three p values exceed 0.67 across N = 50 seeds per regime.

I Additional Detection Results

GPT-4o’s visual detection degrades substantially relative to text mode, with errors concentrated in missed Thaw events and adjacent-cell false positives. Because missed Thaw-stale memories can kill the agent, detectors should prioritize Thaw recall.

  • GPT-4o Vision Failure Profile: GPT-4o’s vision detection drops by average ∆F1 = −0.598 relative to text mode, significantly more than Claude’s −0.134.This places GPT-4o between Claude’s moderate degradation and GLM-5.1’s total collapse.
  • GPT-4o Vision Failure Profile: GPT-4o often refuses or hedges when enumerating 8×8 image cells, with roughly 30% of L1 vision responses containing reliability disclaimers.It also produces partially correct enumerations that systematically swap nearby cells’ F/H labels.
  • Detection Error Breakdown: 63.2% of all false negatives occur on Thaw events, where memory says SAFE but the cell is now H.These misses are especially safety-critical because they can cause the agent to die.
  • Detection Error Breakdown: 36.8% of all false negatives occur on Freeze events, where memory says DANGER but the cell is now F.False positives predominantly affect cells adjacent to true stale cells, consistent with propagation behavior.
  • Safety Implication: Thaw-stale memories are the most safety-critical misses, motivating future detectors to up-weight Thaw recall.A Thaw miss leaves the agent trusting a memory labeled safe when the cell is actually a hole.

J GLM-5.1 Vision Failure: Verbatim Cases · K Planned Benchmark Extensions

GLM-5.1 vision-mode detection consistently accepted stale spatial memories with maximum confidence, apparently resolving image–text conflicts in favor of textual priors. Planned SpatialSTALE extensions target larger, noisier, more realistic, and less text-dependent memory settings.

  • J GLM-5.1 Vision Failure: Verbatim Cases: All three reproduced GLM-5.1 vision cases labeled ground-truth stale memories as is_stale: false with maximum confidence.Each response confidently described the image as agreeing with the stale memory.
  • J GLM-5.1 Vision Failure: Verbatim Cases: Case 1 confidently called the dark-hole cell (2, 5) frozen ice, reversing the observed H state and accepting stale SAFE memory.The response reported confidence 1.0 and claimed the image showed light-blue F.
  • J GLM-5.1 Vision Failure: Verbatim Cases: Case 2 confidently described hole cell (4, 3) as frozen, again treating stale SAFE memory as valid.The response reported confidence 1.0 despite the observed dark-hole tile.
  • J GLM-5.1 Vision Failure: Verbatim Cases: Case 3 confidently called thawed cell (0, 2) a hole, preserving stale DANGER memory instead of recognizing the H-to-F flip.The response reported confidence 1.0 and asserted that the image confirmed the outdated state.
  • J GLM-5.1 Vision Failure: Verbatim Cases: Reported confidence = 1.0 on 100.0% of judgments, while the modal hallucination pattern described images as matching memory rather than their actual content.The aggregate covers all 6,400 GLM vision judgments and attributes the conflict resolution to trusting the textual prior.
  • J GLM-5.1 Vision Failure: Verbatim Cases: Standard confidence-based filtering would discard zero of these failures, making SpatialSTALE a setting for studying silent vision hallucinations.Token-level uncertainty might catch some failures, but current commercial APIs do not expose the relevant logit entropy.
  • K Planned Benchmark Extensions: Planned extensions increase difficulty through 16 × 16 or 32 × 32 grids, distractor entries, and multi-step memory edits.Larger grids stress localization and may use ∼1,000 tokens for a 16 × 16 text observation; edits shift the task from classification to generation.
  • K Planned Benchmark Extensions: Further extensions introduce other agents’ actions, continuous-control environments with perception noise, and vision-only memory crops for direct visual matching.These variants broaden staleness beyond environment dynamics, grids, and textual memory.
Loading 2608.04574v1…