Source-linked AI summary
MemGuard: Persisting Verifier Signals for LLM-Agent Memory Governance
Haoyu Wang, Guangyuan Dong, He Liang, Zijing Zhang, Jiachen Luo, Chuang Liu, Chao Xue, Hao Tang
TL;DR
Long-running LLM agents need reliable memory, but unreliable admission and memory drift can undermine stored experience. MemGuard persists verifier signals as lifecycle metadata to govern memory, and outperforms ReasoningBank across all 16 backbone–benchmark settings.
Problem
Long-running LLM agents need memory to reuse experience, but unreliable admission and accumulating duplicate, stale, or conflicting records can degrade memory reliability.
Method
MemGuard attaches persistent reward, confidence, verifier-label, uncertainty, usage, and conflict metadata to candidate experiences, using it for admission, retrieval, consolidation, summarization, and archival.
Results
MemGuard achieves the best success metric in all 16 backbone–benchmark settings, outperforming ReasoningBank with gains of 5.3–7.9 SR points on WebArena and 3.5–5.6 SSR points on Mind2Web.
Takeaways & Limitations
The verifier-only control improves over ReasoningBank but underperforms MemGuard in every setting, indicating that persistent verifier signals contribute beyond one-time filtering.
Takeaways & Limitations
MemGuard is evaluated on controlled benchmarked task streams and is not a substitute for domain-specific safety evaluation, user consent, privacy review, or operational monitoring.
Abstract
from arXiv · showhide
LLM agents are moving from single-prompt use to long task streams in which reusable memory becomes a core capability for terminal, software-engineering, and web tasks. Such memory is useful only when stored experience remains reliable across hundreds of interactions, but two failure modes break that assumption in practice. The first is unreliable admission: failed trajectories,accidental successes, and misleading observations enter memory because they appear relevant, then mislead later decisions. The second is memory drift: long-running banks accumulate duplicate, stale, and conflicting records that retrieval alone cannot repair. MemGuard's key distinction is to treat verifier output not as a one-shot filter, but as persistent lifecycle metadata. It converts multi-criteria score-token verification into reward, confidence, label, and uncertainty descriptors that are attached to every candidate before activation and reused during retrieval, conflict resolution, summarization, and archival. We evaluate MemGuard on Terminal-Bench 2.0, SWE-Bench Verified, WebArena, and Mind2Web across four backbones, comparing against four memory baselines plus a verifier-only control under matched runtime budgets. Averaged over five seeds, MemGuard achieves the best success metric and lowest average steps in all 16 backbone-benchmark settings, improving over ReasoningBank, the strongest prior baseline among the memory methods we evaluate, with a largest gain of 7.9 success-rate points on WebArena, 5.6 step-success-rate points on Mind2Web, and 2.4-3.5 points on terminal and software-engineering benchmarks. Code is available at https://github.com/whyyyyy123/MemGuard.
1 Introduction
MemGuard addresses unreliable memory admission and memory drift by making verifier evidence persistent throughout the memory lifecycle. Across four benchmarks and four backbones, it outperforms prior memory baselines and a verifier-only control.
- Agent memory quality determines whether long task streams become more reliable or degrade unpredictably.
- Unreliable admission stores failed trajectories, accidental successes, misleading observations, invalid patches, and outdated commands when they appear relevant.
- Memory drift accumulates duplicate, conflicting, stale, and over-generalized records that remain retrievable over many tasks.
- MemGuard attaches reward, confidence, verifier labels, and uncertainty-related metadata to candidates before activation, routing weak experience to provisional or failure-guard storage.
- MemGuard achieves the best success metric in every backbone–benchmark setting, with up to 7.9 SR-point gains on WebArena and 5.6 SSR-point gains on Mind2Web.
- The verifier-only control improves over ReasoningBank in most cells but underperforms MemGuard in every setting, supporting persistent signals over one-time filtering.
2 Related Work
Prior work establishes persistent conversational memory, task-level experience reuse, and verifier-guided decision support. MemGuard differs by making verifier-derived metadata a persistent control field for memory governance after writing.
- Earlier memory systems store conversational or contextual information, whereas MemGuard targets task-level experience whose correctness is judged after environment interaction.
- Task-level systems reuse reflections, skills, workflows, reasoning strategies, or trajectories from prior attempts.
- Verifier-guided methods score or select answers, reasoning paths, trajectories, or candidate sets for current decisions.
- MemGuard's contribution is not a new verifier but persistent verifier-derived governance metadata attached to each memory record.
- Unlike retrieval or parameter editing, MemGuard governs an external bank of self-generated experiences that may be relevant yet harmful when stale or accidental.
- MemGuard connects memory and verifier research by using descriptor dm to adjust retrieval, block risky injection, prioritize conflict resolution, and control summarization or archival.
3 Method
MemGuard converts task trajectories into compact experience records and governs them using persistent verifier descriptors. Its pipeline combines multi-criteria admission, precision-oriented retrieval, and budgeted consolidation through merging, conflict resolution, summarization, and archival.
- 3.1 Overview: MemGuard retrieves reliable memories, injects them into the agent context, verifies completed trajectories, induces candidates, and updates the memory bank.
- 3.1 Overview: The system distills noisy histories into transferable decomposition patterns, environment observations, tool-use rules, and failure-avoidance principles.
- 3.1 Overview: Each memory carries dm = (Rm, cm, ℓm, νm), representing reward, confidence, verifier label, and verification time.
- 3.1 Overview: Verifier metadata remains attached after admission and changes retrieval, merging, conflict resolution, summarization, and archival.
- 3.3 Verifier-Guided Memory Admission: Candidates become active only when Rt ≥0.70 and ct ≥0.60; otherwise they are rejected, provisional, or routed to failure-avoidance storage.
- 3.4 Precision-Oriented Retrieval: Retrieval first removes archived, low-relevance, or verifier-unsafe records, then ranks remaining procedural and tool-usage memories by a normalized soft score.
- 3.5 Memory Consolidation and Budgeted Governance: Consolidation compares structured signatures and conflict links, discards duplicates, merges complementary records, resolves conflicts, summarizes repeated successes, and archives stale or low-quality records.
4 Experiments
MemGuard is evaluated across four benchmarks and four backbones under matched budgets, with results showing consistent gains over memory baselines and verifier-only filtering. Ablations and audits attribute these gains to lifecycle governance, verifier-aware retrieval, and protection against risky memories.
- Main Results: MemGuard achieves the highest primary success metric and lowest average steps in all 16 backbone–benchmark settings.Results use five seeded runs under matched task order, runtime, step, retrieval, decoding, and memory-context budgets.
- Main Results: 2.4–3.5 success-metric points are gained on Terminal-Bench 2.0 and SWE-Bench Verified over ReasoningBank.The largest gains occur on interactive web benchmarks.
- Main Results: The advantage emerges later in continuous task streams rather than only under isolated per-task resets.On 2,294 SWE-bench Full issues, MemGuard improves Resolve Rate by 2.96 points and reduces average steps by 2.10 over ReasoningBank.
- Verifier Control and Overhead: Verifier-only filtering improves over ReasoningBank in 15 of 16 settings but remains below MemGuard in all 16.MemGuard adds 1.0–2.5k tokens and 5.1–15.0 seconds per task over no memory while remaining below other memory and verifier baselines on these overhead measures.
- Ablation Study: Removing governance causes the largest success drop, while removing failure memories mainly increases average steps.Semantic-only retrieval performs worse in every setting, and a simplified label-confidence descriptor trails the full descriptor by 0.7–1.5 success-metric points.
5 Conclusion
MemGuard frames reliable agent memory as a lifecycle-governance problem rather than a retrieval-only problem. Persisting verifier signals improves performance across benchmarks, while residual verifier risk and broader transfer remain open concerns.
- Conclusion: MemGuard attaches reward, confidence, verifier labels, usage statistics, and conflict links to memories throughout a task stream.These metadata guide admission, retrieval, merging, summarization, and archival after storage.
- Conclusion: MemGuard outperforms ReasoningBank in all 16 backbone–benchmark settings and lowers deployment cost through fewer steps, tokens, and latency.Reported reductions are 3.0–8.4k total tokens and 6.5–19.0 seconds per task.
- Conclusion: Reliable agent memory is a lifecycle problem because harmful memories can remain semantically relevant even when retrieval finds them.Future work includes verifier transfer, broader memory-drift evaluation, and multimodal or privacy-sensitive settings.
Limitations
MemGuard’s evaluation is limited by verifier dependence, restricted benchmark coverage, and uncertainty about generalization to longer, more heterogeneous deployments. These constraints leave false accepts, recalibration needs, and production-scale memory issues unresolved.
- Verifier dependence: Verifier labels agreed with author inspection in 86% of Qwen-3.5-Plus cases and 83% of Qwen-3.5-Flash cases, with disagreements driven mainly by incomplete logs or ambiguous success criteria.Benchmark outcome checks operate at larger scale but do not replace human assessment of evidence consistency, execution validity, or generalizability.
- Verifier dependence: False accepts can promote unsupported experience into active memory, and lifecycle governance does not guarantee protection against calibrated, systematic, or adversarial verifier errors.Stricter thresholds, provisional states, repeated verification, and failure-guard separation reduce risk but do not eliminate it.
- Generalizability beyond reported streams: MemGuard uses fixed thresholds, weights, and memory budgets selected on a small held-out development stream, so new domains may require recalibration.Sensitivity analyses support nearby settings on tested web benchmarks, but not necessarily unseen domains.
- Generalizability beyond reported streams: The 2,294-issue SWE-bench Full stream remains shorter and less heterogeneous than open-ended production deployments, leaving budget saturation, lossy summaries, and stale failure guards open.Larger multi-seed sweeps are also needed for tighter effect-size estimates, especially on Terminal-Bench 2.0 and SWE-Bench Verified.
- Scope of evaluation: MemGuard governs external memory rather than backbone parameters and was evaluated only in terminal, software-engineering, and web-agent environments.It does not address multimodal, open-world embodied, or high-stakes production settings, and it cannot compensate for missing base-agent tool-use or reasoning capabilities.
- Scope of evaluation: The study omits some reflection and skill-library systems because their interaction loops differ and does not yet provide a complete privacy layer for stored information.The comparison covers trajectory-, workflow-, and reasoning-memory baselines, while privacy remains outside the current system’s completeness.
Ethical Considerations
MemGuard’s ethical considerations center on the privacy and safety risks of persistent agent memory. The paper frames benchmark gains as reliability evidence for research settings, not as evidence that autonomous agents are safe for unconstrained deployment.
- Ethical Considerations: Persistent agent memory introduces ethical risks that require consideration alongside its intended reliability benefits.The paper treats ethical risk as a distinct concern of long-running memory governance.
- Privacy of stored trajectories: Agent trajectories may contain sensitive information from web pages, repositories, terminal outputs, or user instructions.The experiments use benchmark tasks rather than private user data, but real deployments would expose broader privacy concerns.
- Privacy of stored trajectories: The current system lacks a complete privacy-preserving memory layer, so deployments should add redaction, access control, audit logs, and deletion mechanisms.These controls are recommended before persistent memory is deployed for real users.
- Residual verifier risk: Verifier-guided admission can falsely accept unsupported trajectories, falsely reject useful ones, or encode biases from its model family and prompt.MemGuard mitigates these risks through lifecycle controls, but high-risk settings should require human review and rollback mechanisms.
- Scope of safety claims: Controlled benchmark gains should not be interpreted as evidence that autonomous agents are safe for unconstrained deployment.MemGuard is presented as a reliability layer for research agents and benchmarked task streams, not a substitute for domain-specific safety evaluation, consent, privacy review, or monitoring.
A.5 Detailed Web Results
The detailed results define the WebArena evaluation and compare governed memory against baselines under controlled interfaces and budgets. MemGuard’s advantage grows over continuous streams, while persistent verifier metadata improves success, efficiency, and protection against harmful false accepts.
- Evaluation scope: AS measures the average number of agent–environment interaction steps per task.The benchmark caption defines AS as interaction steps rather than API calls.
- Evaluation scope: WebArena reports site-level results because navigation and memory failures differ across shopping, administration, code-hosting, forum, and multi-site tasks.Mind2Web is additionally split by official domains to stress different forms of memory transfer.
- Stream behavior: As task streams grow, MemGuard separates more clearly from ReasoningBank and reduces action steps while success curves improve.The paired trend indicates later gains are associated with avoiding repeated failed actions and locating useful actions earlier.
- Stream behavior: At the full-stream checkpoint, MemGuard gains 2.96 Resolve Rate points and reduces AS by 2.10 steps over ReasoningBank.The cumulative gap widens through the stream, reaching 5.88 points in the final nonoverlapping window as memory approaches its fixed 384-record capacity.
- Governance comparison: MemGuard’s lifecycle metadata is reused in retrieval, conflict resolution, summarization, and archival rather than applied only during admission.This distinguishes the framework from a one-time verifier-only filter.
- Cost and reliability: MemGuard adds verifier and governance calls, but its lower AS keeps total tokens and latency below other memory and verifier baselines.Its cost remains above the no-memory lower-cost reference.
- Cost and reliability: Governance blocks harmful false accepts from becoming active positive memories and reduces their chance of later injection compared with one-time filtering.The risk decomposition identifies admission and injection as additional gates after verifier acceptance.
C.8 Runtime Feedback, Weight, and Cross-Family Verifier Results
MemGuard remains effective when runtime feedback is masked, weights are varied, and verifier families are switched, indicating robustness beyond privileged or same-family evaluation. Its governance and verifier-aware retrieval components are more important than semantic-only retrieval.
- Runtime feedback: Masking runtime feedback modestly decreases performance, but MemGuard remains above ReasoningBank without consuming hidden benchmark labels.
- Weight sensitivity: Uniform and random retrieval weights remain within a narrow performance band, suggesting the result is not driven by a fragile hand-specified ranker.
- Cross-family verification: Cross-family verification slightly lowers absolute scores but preserves MemGuard’s advantage over matched ReasoningBank results in all eight agent–benchmark cells.
- Ablations: Governance is the most important single-component removal across all four backbones, while semantic-only retrieval performs worse by removing verifier- and conflict-aware ranking.
- Persistent governance: Each memory record stores verifier-derived reward, confidence, labels, lifecycle state, conflict links, and retrieval-relevant metadata for governance and reuse.
D.5 Diagnostic Analysis Protocols
The diagnostic protocols audit verifier reliability, failure-memory governance, and induction behavior through controlled corruption, annotation, and case-based inspection. These procedures distinguish active constrained guards from records governed before action use.
- Verifier calibration: Verifier calibration audits 100 completed trajectories per benchmark-model slice, stratified by verifier label and confidence band for independent author inspection.
- Noise sensitivity: Verifier-label corruption randomly swaps success, failure, and uncertain labels while adjusting reward and confidence toward the nearest threshold interval.
- Failure-memory audit: The failure-memory audit examines 200 WebArena and Mind2Web cases, classifying primary risks and governance outcomes with two annotators.
- Memory induction: Memory induction extracts evidence spans, abstracts instance-specific details into reusable conditions, and emits structured records for governed storage.
- Case study: A web-admin failure is represented as a provisional, low-confidence failure-avoidance record rather than a reusable action recipe.
E.1 Memory Health and Governance Analysis
MemGuard’s memory-health analysis treats long-term reliability as an active governance problem rather than simple memory growth. The results show domain-sensitive filtering, failure-memory composition, and a budget trade-off between usefulness and retrieval overhead.
- Health metrics: Governance statistics track active counts, rejection, merging, archival, and uncertainty to measure whether the memory bank maintains a compact active set.
- Governance outcomes: Web benchmarks produce more rejected, uncertain, and conflict-resolved candidates than terminal or software-engineering benchmarks.
- Governance actions: Uncertain records stay out of active memory, repeated evidence is merged into summaries, and weak or conflicting entries are archived.
- Active-memory composition: Procedural hints remain the largest active-memory category, while web benchmarks contain more failure-avoidance memories and tool-usage memories remain comparatively stable.
- Memory-budget sensitivity: Very small budgets discard useful domain-specific patterns, whereas very large budgets increase retrieval latency and slightly reduce precision; the default is near the best trade-off.
E.3 Failure-Memory Risk Analysis
The failure-memory analysis tests whether negative experience can reduce repeated mistakes without becoming misleading action guidance. Audits and case studies show that governance restricts risky records to constrained guards or removes them before action use.
- Audit scope: The audit covers 200 failure-derived WebArena and Mind2Web memories, focusing on risks including over-generalized guards and stale page-state assumptions.
- Active failure guards: Only 31 risky records become active, and they are injected as constraints rather than direct action plans.
- Governance outcomes: Other risky records are rejected, provisional, merged into safer summaries, or archived after conflict detection.
- Case study: In the representative WebArena task, the earlier rule fails because table ordering is unstable despite superficial relevance.
- Case study: MemGuard marks that candidate uncertain with low reward and keeps it provisional because evidence consistency and generalizability are low.
- Statistical validation: Paired bootstrap testing finds all 16 comparisons significant after Benjamini–Hochberg correction, while 7/16 remain significant under Holm correction.
F Reproducibility Statement
The paper reports matched evaluation settings and separates action-step measurements from verifier, token, and storage costs. It also documents AI assistance while retaining author responsibility for scientific content and reported results.
- Evaluation controls: Evaluation uses matched step budgets, decoding settings, retrieval budgets, and memory-context budgets.The appendix specifies memory schemas, prompts, model versions, verifier settings, governance thresholds, and table/figure procedures.
- Evaluation controls: Average agent action steps per task are reported separately from verifier calls, token counts, and storage overhead.
- AI use: LLMs serve as backbone agents, verifiers, and memory-induction modules in the experiments.
- AI use: AI assistance was used for language polishing, organization, and LaTeX formatting, while authors retain responsibility for scientific claims, experiment design, benchmark descriptions, and reported numbers.