Source-linked AI summary
MemHarness: Memory Is Reconstructed, Not Replayed
Rong Wu, Daocheng Fu, Licheng Wen, Xuemeng Yang, Shu Zou, Jianbiao Mei, Yuxin Wang, Hairong Zhang, Yu Yang, Tao Hu, Cong Zhang, Botian Shi, Pinlong Cai
TL;DR
Memory-augmented agents often replay retrieved experiences verbatim despite mismatches with the current state. MemHarness reconstructs memories into state-conditioned guidance and outperforms pure RL and static memory baselines on ALFWorld and WebShop.
Problem
Most memory-augmented agents replay retrieved experiences verbatim, leaving their action-level applicability to the current interaction state unresolved.
Method
MemHarness inserts critique and reconstruction between retrieval and action, learning a unified policy end-to-end with GRPO to produce context-sensitive guidance.
Results
MemHarness consistently outperforms pure RL and static memory-augmented methods, achieving 85.2% success on ALFWorld and 75.6% on WebShop.
Takeaways & Limitations
State-conditioned reconstruction enables agents to leverage historical experience while mitigating the noise introduced by verbatim replay.
Abstract
from arXiv · showhide
Retrieving past experiences has become a common strategy to enhance large language model agents. However, most existing memory-augmented agents treat retrieved experiences as static records to be replayed verbatim, injecting them into the context regardless of whether they align with the agent's current situation. This ``replay'' paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer. In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context. Inspired by this, we propose MemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context. At each decision step, a unified policy model critiques and reconstructs the retrieved experience conditioned on the current state, producing context-grounded guidance before acting. This reconstructive ability emerges naturally through end-to-end training with GRPO. Experiments on ALFWorld and WebShop show that MemHarness substantially outperforms pure RL and static memory-augmented baselines, demonstrating strong robustness in out-of-distribution (OOD) scenarios. Furthermore, our analyses reveal that this reconstruction objective not only prevents negative transfer but also serves as latent guidance during training, fundamentally improving the agent's intrinsic reasoning capabilities.
1 Introduction
MemHarness reframes agent memory as context-dependent reconstruction rather than verbatim replay, inserting critique and reconstruction between retrieval and action to produce state-aligned guidance. Across ALFWorld and WebShop, it outperforms pure RL and static memory baselines, while ablations identify reconstruction as the primary source of gains.
- Motivation: Verbatim replay treats retrieved trajectories or principles as static records, conflating retrieval relevance with action-level applicability and risking state misalignment.This applicability gap can cause retrieved experiences to conflict with the agent’s current situation.
- Motivation: Human-like reconstruction reorganizes retrieved experience using present cues and prior knowledge before behavior, motivating context-sensitive memory use.The paper contrasts direct retrieval-to-action pipelines with an intermediate process that evaluates and adapts memories.
- Framework: MemHarness decomposes memory-guided decision-making into observation, retrieval, critique, contextual reconstruction, and action generation.Critique identifies state mismatches, while reconstruction preserves transferable knowledge and revises or discards incompatible content.
- Framework: MemHarness inserts explicit critique and reconstruction between retrieval and action, transforming memory from a static prompt fragment into context-sensitive guidance.A single policy conditions critique and reconstruction on the current observation before generating the executable action.
- Evaluation: MemHarness consistently outperforms pure RL baselines and state-of-the-art static memory-augmented methods on ALFWorld and WebShop.The evaluation covers two challenging agent decision-making benchmarks.
- Evaluation: Removing reconstruction reduces performance to naive memory replay, showing that adaptive reconstruction—not retrieval alone—drives the observed gains.This result comes from ablation studies.
2 Related Works
Prior LLM-agent methods use prompting, supervised learning, or reinforcement learning, but training-free approaches remain limited by frozen capabilities and standard RL lacks cross-episode experience reuse. Memory-augmented agents address this gap, while MemHarness combines traceable explicit memory with state-conditioned reconstruction instead of verbatim replay.
- Prompting and Agent Training: Prompting paradigms such as ReAct, planning, and self-reflection support interactive-task navigation but remain bounded by frozen model capabilities.
- Prompting and Agent Training: Supervised learning and reinforcement learning acquire task-specific skills from interaction, but standard RL lacks explicit cross-episode experience accumulation and reuse.
- Memory-Augmented Agents: Memory-augmented agents retrieve past interactions or distilled insights, yet verbatim replay can produce misaligned or misleading guidance under high state variance.
- Memory-Augmented Agents: MemHarness retains traceable explicit memory banks while end-to-end training enables state-conditioned critique and rewriting into context-specific guidance before acting.
3 Method
MemHarness models decision-making as a partially observable process and reconstructs retrieved experiences into state-specific guidance before generating actions. A unified policy performs retrieval-conditioned critique, reconstruction, and action generation, trained end-to-end with GRPO from task-level rewards.
- Sequential Decision-Making: The agent models interaction as a POMDP and conditions its policy on a recent history window of observations and actions.The POMDP is defined by state, action, observation, transition, and reward spaces.
- Experiential Memory: Experiential memory stores distilled natural-language strategies with source observations, while retrieval uses a context-derived query and top-k selection.Entries are semantically deduplicated, tracked for empirical utility, and periodically pruned when utility is low.
- Action Generation: The same policy separately performs reconstruction and action generation, conditioning executable actions on adapted memory or independent reasoning.Both stages share parameters θ and are jointly optimized, treating reconstruction guidance as a latent reasoning process without ground-truth annotations.
- Critique and Reconstruction: Rather than replaying retrieved experiences verbatim, MemHarness compares them with the current history and retains, revises, or rejects mismatched content as state-specific guidance.If no retrieved experience applies, the policy outputs <EMPTY>, enabling action generation to rely on independent reasoning.
- Optimization: GRPO trains the retrieve–reconstruct–act pipeline end-to-end without a separate value network, using task outcomes plus format-based signals.The primary trajectory reward is 0 for failure or 10 for success, augmented by 0.1 × Rformat; trajectory-level advantages credit thinking, reconstruction, and action generation jointly.
4 Experiments
MemHarness outperforms strong baselines on ALFWorld and WebShop, including under OOD shifts, by reconstructing retrieved memories in a state-conditioned manner. Ablations show that RL and policy-internal reconstruction are essential, while reconstruction also improves intrinsic reasoning and learned memory decisions.
- Overall Performance: MemHarness achieves 85.2% success on ALFWorld and 75.6% on WebShop, outperforming closed-source and memory-augmented baselines.The evaluation covers embodied household tasks and goal-directed online shopping, using task success rate and WebShop average task score.
- Overall Performance: MemHarness exceeds Gemini-2.5-Pro by +23.1% on AlfWorld and +39.7% on WebShop despite using 7B parameters.The comparison is against the Qwen2.5-7B-Instruct policy backbone.
- Overall Performance: State-conditioned reconstruction lets MemHarness outperform standard GRPO by +8.8% and +Naively combining GRPO with external memory severely degrades performance, whereas reconstruction mitigates state-mismatch noise.
- Ablation Studies: In ablations, cold-start training alone reaches only 7.6% on ALFWorld, while raw memory hurts performance and removing reconstruction causes state-mismatch noise.These findings establish the necessity of RL and the limitations of verbatim memory replay.
- Ablation Studies: Disabling test-time memory still yields 83.0% versus 76.4% for RL Only on ALFWorld, indicating reconstruction training improves intrinsic reasoning.Replacing internal reconstruction with Qwen2.5-7B-Instruct also reduces ALFWorld performance from 85.2% to 77.7%.
- OOD Robustness: On ALFWorld OOD settings, MemHarness reaches 85.9% average success versus 76.3% for RL + Raw Memory, while reconstruction filters mismatched guidance.Removing reconstruction lowers performance to 82.4%, compared with 83.0% without test-time memory.
5 Conclusion · Appendix
MemHarness shifts memory-augmented agents from verbatim replay to state-conditioned reconstruction, using end-to-end GRPO training to produce state-specific guidance before acting. Its 7B model outperforms closed-source models and RL baselines, particularly in out-of-distribution environments.
- 5 Conclusion: MemHarness is an RL-driven framework for state-conditioned memory reconstruction.
- 5 Conclusion: The framework replaces verbatim replay with reconstruction of historical experiences.
- 5 Conclusion: A single unified policy critiques and rewrites historical experiences into state-specific guidance before acting.
- 5 Conclusion: The unified policy is optimized end-to-end through GRPO.
- 5 Conclusion: The 7B model significantly outperforms closed-source models and RL baselines.
- 5 Conclusion: The performance advantage is especially pronounced in out-of-distribution environments.
A Implementation Details · A.1 Environment and Evaluation Setup
MemHarness is evaluated on ALFWorld and WebShop under bounded interaction, prompt, and generation horizons. Both environments use sparse task-success rewards supplemented by formatting incentives and protocol constraints.
- A.1 Environment and Evaluation Setup: The evaluation covers two interactive benchmarks: ALFWorld and WebShop.ALFWorld is an embodied text-based environment, while WebShop simulates an e-commerce website.
- A.1 Environment and Evaluation Setup: ALFWorld contains six sub-task categories, including Pick, Look, and Clean.The benchmark uses an embodied text-based interaction setting.
- A.1 Environment and Evaluation Setup: ALFWorld uses a maximum interaction horizon of 50 steps, a 2048-token prompt limit, and a 512-token generation limit.These bounds define the evaluation budget for each ALFWorld interaction.
- A.1 Environment and Evaluation Setup: WebShop is an e-commerce website simulation containing over 1.1 million products.Its interaction horizon is restricted to 15 steps.
- A.1 Environment and Evaluation Setup: Across both environments, successful task completion receives a sparse reward of +10, whereas failure receives 0.The reward function is outcome-driven and combined with a format penalty.
- A.1 Environment and Evaluation Setup: A formatting bonus weighted by 0.1 supplements the sparse outcome reward to encourage adherence to the prescribed interaction protocol.The format score checks valid action structure, memory retrieval blocks, a think block, and strictly English generated content.
A.2 Training Hyperparameters · A.3 Memory Bank and Retrieval Setup
MemHarness trains Qwen2.5-7B policies with GRPO under fixed reference-policy regularization and specified rollout, optimization, batching, and objective settings. Its memory bank uses online-distilled experiences, vector retrieval, and deduplication-based curation.
- A.2 Training Hyperparameters: MemHarness uses GRPO with veRL, initializes Qwen2.5-7B instruction-tuned policies, and freezes πref to compute the KL-divergence penalty.
- A.2 Training Hyperparameters: Rollouts sample G = 8 trajectories per prompt at temperature 1.0, validate at 0.4, and use 16 groups across 128 parallel environments.
- A.2 Training Hyperparameters: Optimization uses Adam with learning rate 1 × 10^-6, PPO-style clipping ϵ = 0.2, and KL penalty coefficient β = 0.01.
- A.2 Training Hyperparameters: Batching uses mini-batch size 256 for Alfworld and 64 for Webshop, tensor parallel size 1, and vLLM with GPU memory utilization ratio 0.75.
- A.3 Memory Bank and Retrieval Setup: The memory store B is implemented in Milvus, with each entry pairing a distilled experience ei with its source interaction state osrc.
- A.3 Memory Bank and Retrieval Setup: BGE-M3 embeds retrieval queries and stored entries, while cosine similarity retrieves the top-k experiences with k = 3 at each step.
- A.3 Memory Bank and Retrieval Setup: The memory bank is populated online from the policy’s interactions, distilling successful and failed trajectories into concise principles paired with source states.
- A.3 Memory Bank and Retrieval Setup: Curation retains 50% of GRPO trajectories for balanced success-failure distillation when possible, and write-time deduplication skips entries exceeding θsim default 0.85.The provided passage describes the first of three curation stages and states that deduplication uses an embedding-based nearest-neighbor probe; it notes activation for WebShop but is truncated thereafter.
A.4 Cold-Start Dataset Construction · B Counterfactual Probe Construction · B.1 Generation Prompt
The paper builds a cold-start dataset to align MemHarness with memory retrieval and trajectory summarization protocols, then constructs a counterfactual probe dataset to test fine-grained state comparison. Counterfactual states are minimally edited to preserve structure while changing memory applicability.
- A.4 Cold-Start Dataset Construction: The cold-start dataset uses AgentGym seed trajectories to teach memory retrieval initiation and reusable trajectory-memory summarization.This stage familiarizes the policy with the interaction protocol rather than task-specific behavior.
- A.4 Cold-Start Dataset Construction: 200 examples per benchmark form memory-augmented interaction trajectories with inserted retrieval turns and GPT-5.1-generated context-appropriate queries and distilled memories.Retrieval is inserted at a randomly sampled intermediate decision point using the task description and interaction prefix.
- A.4 Cold-Start Dataset Construction: A second subset contains 200 trajectory-to-memory pairs teaching compact memories with situational preconditions and reusable principles.Each pair uses a completed AgentGym trajectory and its GPT-5.1-generated summary as the target.
- A.4 Cold-Start Dataset Construction: The two cold-start subsets are jointly fine-tuned for 2 epochs at a learning rate of 1 × 10−5 and an effective batch size of 2.GPT-5.1 is used only for offline data construction, while the stage targets protocol and format alignment rather than task-skill acquisition.
- B Counterfactual Probe Construction: The counterfactual probe dataset contains 2,000 examples, split evenly between ALFWorld and WebShop, pairing applicable real states and retrieved memories with minimally altered counterfactual states.The edits make the memory inapplicable or misleading, enabling evaluation of state comparison rather than superficial pattern matching.
- B.1 Generation Prompt: GPT-5.1 generates counterfactual states whose edits remain minimal and structurally identical to the original observations.This prompt design prevents agents from rejecting memories based on formatting artifacts.
B.2 Examples of State Edits … D.1 ALFWorld: Abstracting and Transferring Skills
MemHarness reconstructs or rejects retrieved memories according to minimal changes in the current state, preventing invalid guidance in ALFWorld and WebShop. Its prompts operationalize retrieval, reconstruction, fallback reasoning, trajectory summarization, and state-grounded skill transfer.
- B.2 Examples of State Edits: MemHarness accepts a memory in the original state but adaptively rewrites or rejects it after a minimal factual edit changes the situation.This behavior is demonstrated across both ALFWorld and WebShop examples.
- B.2 Examples of State Edits: In ALFWorld, revealing a mug invalidates a search heuristic, so the agent rejects the memory and directly picks up the mug.With no mug visible, the agent instead redirects its search to open surfaces.
- C Prompt Templates: The prompt templates cover memory retrieval, state-conditioned reconstruction, agent–environment interaction, trajectory summarization, and fallback reasoning when no adapted memory applies.Runtime placeholders are enclosed in braces, and retrieval and environment actions are prevented from appearing in the same response.
- C Prompt Templates: When no retrieved principle is applicable, the fallback prompt directs the agent to rely on the current observation and its own reasoning.The reconstruction prompt returns <EMPTY> when the retrieved principle does not apply.
- C Prompt Templates: After each episode, trajectory summarization extracts concise, reusable memories grounded in the interaction trajectory and constrained by a required JSON schema.Additional prompts construct counterfactual probes for the analysis in Section 4.5.
- D Case Studies: The reconstruction cases show why verbatim replay can fail and how MemHarness rewrites historical experience into guidance grounded in the present interaction state.The system prompt requests concise reusable guidance or exactly <EMPTY>, without chain-of-thought or a step-by-step action plan.
- D.1 ALFWorld: Abstracting and Transferring Skills: In ALFWorld, MemHarness avoids hallucinated “credit card” and “coffee table” objects by abstracting the procedural skill and grounding it with “cup” and “sidetable.”It preserves the skill of placing a held item at the final destination after a prerequisite step.
D.2 WebShop: Grounding Heuristics into Actionable Targets
In WebShop, MemHarness reconstructs abstract retrieved heuristics against the current noisy observation instead of replaying them as static prompt rules. This grounding produces directly actionable guidance, such as the exact target ID to select.
- Heuristic Grounding: MemHarness scans the current noisy WebShop observation, aligns it with the user’s constraints, and outputs the exact target ID for action.The retrieved memory may advise crosschecking specific constraints before clicking, but reconstruction executes that heuristic against the present state.
- Heuristic Grounding: Naive replay appends an abstract heuristic to the prompt, providing no concrete operational help for the current WebShop state.MemHarness instead converts the abstract rule into context-grounded guidance before acting.