PaperlayerView exact revision on arXiv

Source-linked AI summary

MemoWM: How World Models Change What Agents Need to Remember

Bingfan Zeng, Zhisheng Chen, Chenbo Sang, Zhengwei Xie, Jinpeng Wang, Xiangchen Guan, Rui Qian, Zheng Lu, Jingwei Song

arXiv:2610.10778v1cs.LGcs.AI

TL;DR

Long-term agents need to retain experience over time, but storing every observation is costly and prediction error alone does not identify what matters for future tasks. MemoWM uses a shared world-model prior and task-aware allocation to compress retained information and reconstruct omissions. Across five benchmarks, it reports 42.42% average answer accuracy and lower experience-specific storage than baselines.

  • Problem

    Long-term agents face growing storage demands, and memory must distinguish reusable world-model regularities from experience-specific information that remains important to future tasks.

    Source: §1 · pp. 1–2

  • Method

    MemoWM uses a shared predictive prior and task-impact estimates to decide which information to store and which content can be reconstructed from the world model.

    Source: §1 · pp. 2–3

  • Results

    42.42% average answer accuracy across five benchmarks, 2.62 points above the strongest baseline, with average experience-specific storage 53.9% lower than MIRIX.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The storage-optimal balance between world-model capacity and episodic memory depends on the agent’s memory horizon.

    Source: §1 · p. 2, §5 · p. 10

    The health-related experiments assess memory and visual reasoning, not clinical validity; predictive reconstruction can also omit evidence or introduce incorrect details, so consequential recall needs verification.

    Source: ETHICS STATEMENT · p. 10

Abstract

from arXiv · show

Long-term agents face growing storage demands as they accumulate experience. World models capture reusable regularities that can reduce the information stored for each experience. We formulate the problem of memory allocation conditioned on a world model and introduce MemoWM, a framework that uses shared predictions to compress retained information and reconstruct omitted content. Its task-aware allocation rule balances the expected impact of reconstruction errors against storage cost, retaining information with downstream value beyond the predictive prior. Across five long-term agent-memory benchmarks, MemoWM achieves 42.42\% average answer accuracy, exceeding the strongest baseline by 2.62 percentage points, while reducing average experience-specific storage by 53.9\% relative to MIRIX, the most storage-efficient baseline. Further analysis shows that stronger world models reduce per-experience storage at comparable task quality. Accounting for model parameters reveals a trade-off between shared model capacity and recurring storage costs, with the capacity that minimizes total storage increasing as more interactions are retained. Our code is available at https://github.com/Feld-maxiu/MemoWM.

1 INTRODUCTION

MemoWM treats long-term memory as a division of labor: a shared world model supplies reusable predictions, while episodic memory retains information that cannot be safely recovered and matters to future tasks. Its task-aware allocation balances recoverability, downstream impact, and storage cost, with model capacity trading off against recurring episodic storage.

1 INTRODUCTION

  • Long-term memory must balance reusable predictions against task value: predictable information may still matter, while unpredictable changes may be irrelevant.Storing every observation makes persistent storage grow with an agent’s lifetime.Source: §1 · p. 2
  • A world model can supply both the coding distribution for retained information and the reconstruction for omitted content, making prediction part of the memory representation.Episodic memory carries information that the shared model cannot recover and that still matters for future reasoning or action.Source: §1 · p. 2
  • MemoWM combines a query-independent state, shared predictive prior, and offline task-impact estimates to retain important information that cannot be recovered.The allocation uses prediction risk and expected coding cost without future queries or a transmitted state-specific mask; reconstructed states feed back into subsequent prediction.Source: §1 · pp. 2–3
  • 42.42% average answer accuracy across five benchmarks, with the lowest experience-specific storage on every benchmark and 53.9% lower average storage than MIRIX.MemoWM exceeds the strongest baseline by 2.62 points.Source: §1 · p. 2
  • Stronger world models reduce recurring storage per experience, but their parameter cost makes storage-optimal model capacity grow with the memory horizon.The paper distinguishes selecting which experiences matter from determining how many bits each selected experience needs.Source: §1 · p. 3

2 RELATED WORK

MemoWM connects world models to memory in a complementary direction: it uses predictions to determine what information an experience needs to retain. Its related work distinguishes this approach from prediction-guided memory construction and from conditional compression.

World models and memory

  • MemoWM studies how predictions from an available world model change the information needed to retain an experience, complementing work that uses memory to improve world-model capabilities.The passage contrasts MemoWM with R2I, R-WoM, MemWM, and WorldEvolver (Wang et al., 2026; Zhang et al., 2026).Source: §2 · p. 3

Selective and compressed agent memory

  • Unlike approaches where prediction guides distillation or updates, MemoWM also uses the predictive distribution during decoding for conditional coding and reconstruction.Its allocation separately weighs prediction-error probability and task impact against expected coding cost.Source: §2 · p. 3

Predictive and conditional compression

  • MemoWM couples task-relevant omission with a retention rule reproduced from shared reconstructed history, avoiding a per-state mask while synchronizing successive-state decoding.The task distortion is the effect of an omission on future answers.Source: §2 · p. 3

3 MEMOWM

MemoWM treats long-term memory as joint coding by a shared world model and episodic store, retaining task-relevant information while reconstructing omitted content. Its method combines a query-independent state representation, shared-history predictions, and offline fitting and calibration.

3 MEMOWM

  • MemoWM retains a coordinate when its estimated task impact exceeds its predicted storage cost; otherwise, the shared world model supplies its reconstruction.The writer and decoder use reconstructed history, allowing the decoder to determine retention decisions without a per-state mask.Source: §3 · p. 3, Figure 2 · p. 4

3.1 PROBLEM DEFINITION

  • A trajectory is encoded as a sequence of discrete states, with one query-independent memory supporting different future tasks.The formulation distinguishes episodic storage per episode from shared-parameter costs that can be amortized across episodes.Source: §3.1 · p. 4

3.2 TASK-RELEVANT RECOVERABILITY

  • For each coordinate, MemoWM weighs predicted omission risk against storage cost, using offline error-impact and entropy-to-error estimates to guide retention.The coordinate-wise rule is a proxy: joint omissions can interact through retrieval and future predictions.Source: §3.2 · pp. 4–5

3.3 SHARED-HISTORY RECONSTRUCTION

  • Omitted coordinates are reconstructed from shared history, so identical initial states, actions, parameters, and computation rules let writer and decoder make the same predictions and retention decisions.This synchronization avoids transmitting a per-state retention mask.Source: §3.3 · p. 5

3.4 OFFLINE FITTING AND CALIBRATION

  • Offline fitting estimates task impact and prediction reliability, trains retrieval to handle reconstructed states, and selects a retention threshold through full-episode calibration.Calibration evaluates candidate thresholds on independent episodes and accounts for interactions among omissions, retrieval changes, and later prediction histories.Source: §3.4 · p. 6

4 EXPERIMENTS

The experiments evaluate MemoWM across five long-term agent-memory benchmarks and examine how predictive coding, task-aware omission, world-model capacity, and reconstruction affect storage and task quality. MemoWM reduces experience-specific storage while preserving answer performance, and the storage-optimal model capacity depends on the memory horizon.

4.1 EXPERIMENTAL SETUP

  • The evaluation uses five web- or GUI-based memory benchmarks, compares twelve baselines, and reports native answer accuracy averaged over four random seeds.Methods share the same backbone, reader, and LLM judge per benchmark protocol; experience-specific storage excludes shared model weights.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • 42.42% mean answer accuracy across five benchmarks accompanies the lowest experience-specific storage on every benchmark, with 53.9% lower average storage than MIRIX.MemoWM also improves accuracy by 7.24 points relative to MIRIX.Source: §4.2 · p. 7

4.3 A COMPOSABLE PRINCIPLE FOR MEMORY SYSTEMS

  • 30.6%–67.6% lower storage rate than quantization alone comes from predictive coding and task-aware omission, with answer-accuracy changes of at most 0.31 percentage points.The pipeline also integrates with M2A by replacing its raw evidence store with encoded states while leaving accuracy nearly unchanged across three benchmarks.Source: §4.3 · pp. 7–8

4.4 THE STORAGE TRADE-OFF BETWEEN WORLD MODELS AND EXPERIENCE

  • The storage-optimal model capacity rises with retained transitions: about 0.54M parameters at 10K transitions toward 29M at long horizons.Larger models reduce recurring storage; counting amortized FP16 weights yields a finite optimum around 5–7M parameters in the evaluated setting.Source: §4.4 · p. 8, Figure 3 · p. 8

4.5 TASK-AWARE RISK IMPROVES THE RATE–UTILITY TRADE-OFF

  • At matched distortion, MemoWM has the lowest rate; Static-e and Uniform-V incur 17.0% and 25.5% overhead, while Random and Uncertainty incur 33.1% and 42.0%.The comparison supports combining state-dependent prediction risk with coordinate-specific task impact when allocating storage.Source: §4.5 · p. 9, Table 3 · p. 9

4.6 RECONSTRUCTION QUALITY AND CLOSED-LOOP STABILITY

  • With omitted coordinates and storage held fixed, world-model filling reduces answer-NLL distortion from 0.1280 to 0.0867 bit versus previous-state copying and reaches 60.18% accuracy.Only 14.2% of omitted codes are recovered exactly, while semantic agreement reaches 97.0%.Source: §4.6 · p. 9, Figure 4 · p. 9
  • Across rollouts up to 40 steps, closed-loop reconstruction adds 0.006–0.021 bit of answer-NLL distortion and 4.4–23.1 bits/state of storage versus a teacher-forced oracle.The reported results indicate limited error accumulation over the evaluated horizon.Source: §4.6 · p. 9, Figure 4 · p. 9

5 CONCLUSION

The conclusion frames long-term memory as a division of labor between reusable world knowledge and episodic experience, with MemoWM storing information that is hard to recover and task-relevant. The paper also identifies boundaries around its evaluation setting and the risks of persistent memory and predictive reconstruction.

5 CONCLUSION

  • MemoWM uses a shared world-model prior to reduce recurring memory cost while preserving task performance, with the optimal balance depending on the agent’s memory horizon.The framework stores information that is both hard to recover and important for downstream tasks.Source: §5 · p. 10
  • The authors report using generative AI for editing and formatting, and language models for data collection, supervision, answering, and evaluation, while retaining responsibility for the paper.They state that citations, equations, settings, and results were checked and AI-assisted text reviewed.Source: AI USE STATEMENT · p. 10
  • The evaluation uses benchmark tasks, self-hosted WebArena trajectories, and synthetic data; the health-related experiments do not establish clinical validity.Persistent memory can retain sensitive information, and compression alone does not provide a privacy guarantee.Source: ETHICS STATEMENT · p. 10
  • The reproducibility statement locates experimental details in Section 4 and Appendix A, data construction and overlap checks in Appendix B, and storage accounting in Appendix C.Main-table answer accuracies are reported as means and standard deviations over four random seeds.Source: REPRODUCIBILITY STATEMENT · p. 10

A DETAILED METHODS AND EVALUATION PROTOCOL FOR MEMOWM

The appendix details how MemoWM constructs a discrete state, predicts its coordinates, and keeps writer and decoder reconstruction synchronized. It also specifies storage accounting and evaluation protocols.

A.1 STATE REPRESENTATION AND DISCRETIZATION

  • MemoWM encodes observations as discrete reference states, with a query-independent representation and frozen OPQ quantization defining the coordinates used for prediction and reconstruction.The reference state is z_t = E(o_t); reconstructed states are written as ˜z_t. The reported OPQ reference uses 1,024 coordinates and 6,144 bits per observation.Source: A DETAILED METHODS AND EVALUATION… · p. 14, §A.1 · p. 14
  • The state encoder is trained with overall-observation and local-detail readout supervision, while questions supervise readout but are not inputs to memory formation.The query-independent encoder computes Fρ(o); other observations from the same session serve as negatives in a symmetric InfoNCE objective.Source: §A.1 · p. 14

A.2 WORLD MODEL AND SHARED-HISTORY RECONSTRUCTION

  • The action-conditioned world model predicts all 1,024 next-state coordinates from reconstructed history, while writer and decoder feed back the same reconstructed states.The writer feeds back ˜z_t before processing the next step, allowing the decoder to reproduce later predictions; retention masks are recomputed rather than transmitted.Source: §A.2 · pp. 14–15

A DETAILED METHODS AND EVALUATION PROTOCOL FOR MEMOWM

  • Storage accounting includes per-episode auxiliary content needed for decoding, counts shared-model storage separately, and excludes regenerable retrieval vectors and indices from the main table.The appendix also specifies data construction, storage accounting, and evaluation protocols.Source: A DETAILED METHODS AND EVALUATION… · p. 14, §A.2 · p. 15

A.3 RISK ESTIMATION, RETRIEVAL TRAINING, AND THRESHOLD CALIBRATION

This section describes task-impact estimation, retrieval and calibration procedures, and the storage trade-off between shared model capacity and episodic memory. It also states the conditions and scope of its retention-rule analysis.

A.3.1 RISK ESTIMATION AND CALIBRATION; A.3.2 RETRIEVAL REPRESENTATION LEARNING

  • Task-impact labels compare answer loss for complete and single-coordinate-prediction states, while retrieval training pairs complete and reconstructed views with the same frozen teacher embedding.Teacher-alignment and in-batch negative terms encourage observation-discriminative keys, but are not a formal guarantee against collapse.Source: §A.3.1 · p. 15, §A.3.2 · p. 16

A.3.3 RECONSTRUCTED-MEMORY READING; A.3.1 RISK ESTIMATION AND CALIBRATION

  • The section evaluates complete and reconstructed memories using the same retrieval and reading components, while retrieved states may differ and alter both retrieval and Reader evidence.Online predictive error frequency uses pooled entropy bins, with a coordinate-specific marginal-rate fallback for empty bins.Source: §A.3.3 · pp. 16–17, §A.3.1 · p. 16

A.3.4 THRESHOLD CALIBRATION

  • For each threshold candidate, separate closed-loop reconstructions measure answer-loss change and episodic storage, then select the minimum-storage candidate satisfying the average distortion constraint.Separate reconstructions matter because early omissions can change later histories, code lengths, and masks.Source: §A.3.4 · p. 17

A.3.5 FORMAL ANALYSIS OF THE RETENTION RULE

  • The coordinate-level proxy is minimized by retaining a value when its retain-versus-omit difference is nonpositive, but this does not establish episode-level global optimality.Simultaneous omissions may change retrieval and subsequent predictive histories, motivating threshold calibration with complete recurrent reconstruction.Source: §A.3.5 · p. 17

A.4.1 TOTAL STORAGE AND COST PER TRANSITION; A.4.2 DIMINISHING RETURNS AND A FINITE OPTIMUM; A.4.3 BREAK-EVEN BETWEEN TWO MODELS

  • The storage analysis combines shared-model cost and episodic rate, identifies a finite capacity optimum under its assumptions, and gives a break-even condition for comparing two capacities.The horizon analysis holds the measured episodic-rate curve and episode-length distribution fixed while varying retained transitions; if episodic rate does not fall with the larger model, the smaller model is cheaper at every positive retention scale.Source: §A.4.1 · p. 18, §A.4.2 · p. 19, §A.4.3 · p. 19

A.4.5 IMPLICATIONS FOR LONG-TERM MEMORY; A.4.4 DEPLOYMENT HORIZON AND PREFERRED CAPACITY

  • The measured storage optimum is near 5–7M parameters, shifting from roughly 0.54M at short horizons toward about 29M at long horizons for the reported candidate set and deployment distribution.Under the stated assumptions, preferred capacity increases with deployment horizon; for discrete candidates, the optimum grows in steps.Source: §A.4.5 · p. 19, §A.4.4 · p. 19

A.5 STATE-DEPENDENT GATING AND A SHARED TASK-IMPACT PRIOR

MemoWM uses a common coordinate space for a shared task-impact prior and evaluates selected coordinates by their answer-loss impact when prediction is wrong.

A.5.1 COMMON COORDINATE SPACE AND CONDITIONAL DAMAGE

  • The shared task-impact prior assigns a 16-dimensional subspace to each coordinate, and an orthogonal projection onto that subspace is denoted P_j.The representation has 1,024 coordinates, with 32 slots and 32 subspaces per slot.Source: §A.5.1 · p. 19

2 Aerr

  • A sufficient bound identifies a shared set of low-utility coordinates that remains low-impact across the specified family of trajectory-conditioned distributions.MemoWM estimates combined effect directly with counterfactual answer loss.Source: §2 · p. 20
  • Selected coordinates have approximately 1/166–1/176 of the error-conditional damage of same-size random coordinates in the reported training–held-out split and comparison setting.The comparison is empirical for that setting, not a distribution-free guarantee.Source: §2 · p. 20

A.5.2 ENTROPY CALIBRATION AND GATING STABILITY

MemoWM’s calibration tests support task impact and entropy as reusable signals for retention decisions: task impact transfers to unseen trajectories, while entropy tracks prediction-error risk. The gate is stable when risk-estimation error stays below its distance from the threshold.

A.5.2 ENTROPY CALIBRATION AND GATING STABILITY

  • The gate is stable when risk-estimation error is smaller than the decision’s distance from its threshold.The analysis separates context transfer, task-impact estimation, and error-rate calibration; populated entropy bins use pooled error estimates, while empty bins fall back to coordinate-specific calibration.Source: §A.5.2 · pp. 20–21
  • 166–176× as much answer-NLL damage comes from matched random omissions as from the bottom 5%–20% of task-impact-ranked coordinates on unseen trajectories.Damage remains 0.323–0.565 across low-task-impact subsets for independently constructed unseen-query variants, far below matched random omission.Source: Table 4 · p. 21, §A.5.2 · p. 21
  • 8.1%–88.8% observed prediction error rises monotonically across entropy bins, and the frozen lookup preserves this ordering on held-out trajectories.This supports using entropy to adapt the prediction-error estimate to the current predictive context.Source: Table 4 · p. 21, §A.5.2 · p. 21

B TRAINING-DATA CONSTRUCTION AND EVALUATION-OVERLAP CHECKS

The appendices describe how benchmark data are collected and checked for overlap, then specify storage accounting across memory systems. MemEye Health uses synthetic training data whose task structure reflects its evaluation set, limiting claims about generalization to unseen interface styles.

B.1 UNIFIED COLLECTION FRAMEWORK

  • WebArena provides a common browser-collection framework, recording each step as a state-action-next-state transition with task, page, interface, action, and outcome information.The collection retains both original and normalized semantic actions; the structured world-model encoder uses 17 action types.Source: §B.1 · p. 21

B.2 FOUR TRAINING-DATA SOURCES

  • The four training sources preserve different trajectory evidence: AMA-Bench keeps successes and failures; WorldMemArena uses visual GUI actions; LongMemEval-V2 retains failures and rollouts; WebQuest requires multi-frame QA.WorldMemArena sessions are annotated after collection, while WebQuest items are filtered to require multiple answer-bearing frames.Source: §B.2 · pp. 21–22

B.3 DEDUPLICATION

  • Evaluation tasks are excluded from training through normalized exact matching, supplemented by near-duplicate checks using goals, URLs, environments, and initial observations.Manual checks verify matched tasks and confirm that training goals and collection trajectories do not copy evaluation items; the website environment is still shared.Source: §B.3 · p. 22

B.4.1 VISUAL STRUCTURE AND INSTANCE SEPARATION

  • MemEye Health has no training split, so synthetic data follows the public benchmark’s task structure and does not establish zero-shot generalization to unseen interface styles.The generated examples use new virtual patients, screens, values, and questions without reusing public evaluation images, questions, or patient records.Source: §B.4.1 · p. 22

B.4.2 CROSS-SESSION TRAJECTORIES AND QUESTIONS

  • MemEye trajectories contain 8–12 dated sessions, and seven question templates test cross-session evidence, trends, and other screen details.Questions have multiple-choice and open-ended forms; items with conflicting answers or ambiguous screen identities are removed.Source: §B.4.2 · p. 22

C.1 UNIFIED ACCOUNTING RULES

  • The main storage table counts persistent logical content and observation-specific metadata, while excluding regenerable retrieval structures, temporary state, physical layout overhead, and shared model weights.Functional model weights are amortized separately over retained experience; physical deployment overhead may be reported separately when relevant.Source: §C.1 · p. 23

C.2 PERSISTENT CONTENT COUNTED FOR EACH BASELINE

  • Baseline-specific accounting counts retained records and, where applicable, their media, identifiers, metadata, links, and associations.The rules cover systems including RAG, A-Mem, SimpleMem, MemGPT, Omni-SM, M2A, ViLoMem, MIRIX, and AUGUSTUS.Source: §C.2 · p. 23

C.3 QUERY-DEPENDENT MEMORY STATES

  • For reusable memory, query-dependent systems are charged for the original observations, media, actions, order, and session metadata needed to construct states for future questions.Temporary states for one query are excluded unless explicitly archived; single-query storage may be reported separately when the original trajectory is deleted.Source: §C.3 · p. 23

D PHYSICAL ENTROPY-CODED REALIZATION OF THE REPORTED STORAGE RATE

An actual entropy-coded stream closely matches MemoWM’s ideal storage estimate and can be independently decoded without changing reconstructed states. The implementation recomputes retention decisions from shared reconstructed history rather than storing a mask.

D PHYSICAL ENTROPY-CODED REALIZATION OF THE REPORTED STORAGE RATE

  • The writer and decoder recompute predictions, entropies, retention decisions, and modal reconstructions from shared history; retained values are entropy-coded and omitted coordinates use modal predictions.No explicit per-coordinate or per-state retention mask is stored.Source: D PHYSICAL ENTROPY-CODED REALIZATION… · p. 24
  • 1.50% overhead separates the serialized entropy-coded rate of 4,388.70 bits/state from the ideal-code rate of 4,323.84 bits/state.The ideal rate is reported for the same 817-transition WorldMemArena Web out-of-domain set used in the rate analysis.Source: D PHYSICAL ENTROPY-CODED REALIZATION… · p. 24
  • 100.00% of writer-side reconstructed states are reproduced exactly by independent decoding from initial states, actions, shared model parameters, and coded payload.This confirms that retention decisions can be recomputed from shared reconstructed history without a hidden state-specific mask.Source: D PHYSICAL ENTROPY-CODED REALIZATION… · p. 24
  • The physical stream supports the use of ideal conditional code lengths in storage analysis: measured rate rises only 1.50% and reconstructed memory remains unchanged.The authors conclude the reported storage reduction is not an artifact of ideal-code accounting or an uncounted retention mask.Source: D PHYSICAL ENTROPY-CODED REALIZATION… · p. 24

E EVALUATION SETTINGS

The evaluation settings specify benchmark-specific readers, judges, answer protocols, and metrics. Storage and reconstruction diagnostics use persistent bits per observation and teacher-forced NLL change per answer.

E EVALUATION SETTINGS

  • WorldMemArena Web uses a frozen Qwen3.5-9B Reader and LLM judge with greedy decoding and benchmark-defined QA-C accuracy; AMA-Bench uses its open-ended QA configuration with Qwen3-32B.The AMA-Bench judge makes a binary correctness decision from the question, reference answer, and prediction.Source: E EVALUATION SETTINGS · p. 24
  • LongMemEval-V2 uses the Small setting with 100 WebArena and 100 ServiceNow trajectories, a 200K-token context limit, and Qwen3.5-9B Reader and judge.The setup retains benchmark-specific prompts, answer parsing, and semantic scoring for trap and false-premise questions.Source: E EVALUATION SETTINGS · p. 24
  • WebQuest reports Relaxed Accuracy, while storage rate is persistent bits per episode divided by observations and reconstruction distortion is teacher-forced NLL change per answer.WebQuest numeric answers use ±0.05 tolerance; integers and strings use normalized exact matching.Source: E EVALUATION SETTINGS · p. 24
Loading 2610.10778v1…