Source-linked AI summary
ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
Siru Ouyang, Jun Yan, I-Hung Hsu, Yanfei Chen, Ke Jiang, Zifeng Wang, Rujun Han, Long T. Le, Samira Daruki, Xiangru Tang, Vishy Tirumalashetty, George Lee, Mahsan Rofouei, Hangfei Lin, Jiawei Han, Chen-Yu Lee, Tomas Pfister
TL;DR
Persistent LLM agents often fail to learn from accumulated interaction histories, motivating a memory framework that extracts reusable reasoning from experience. ReasoningBank learns from self-judged successes and failures, while MaTTS scales task exploration to strengthen memory; across web and software-engineering benchmarks, the combination improves performance and efficiency.
Problem
Persistent agents encounter continuous task streams but largely fail to learn from accumulated experience, leaving transferable reasoning patterns and failure lessons underused.
Method
ReasoningBank distills actionable strategies from self-judged successful and failed experiences, retrieves them for new tasks, and reintegrates new learnings; MaTTS adds parallel or sequential memory-aware exploration.
Results
Across WebArena, Mind2Web, and SWE-Bench-Verified, ReasoningBank improves effectiveness by up to 20% relative and efficiency by up to 16% fewer interaction steps, with MaTTS amplifying gains.
Takeaways & Limitations
Memory-driven experience scaling offers a pathway toward adaptive, lifelong-learning agents that develop increasingly complex reasoning strategies over time.
Takeaways & Limitations
The study deliberately keeps retrieval and consolidation simple, leaving more sophisticated techniques as future enhancements.
Abstract
from arXiv · showhide
With the growing adoption of large language model agents in persistent real-world roles, they naturally encounter continuous streams of tasks. A key limitation, however, is their failure to learn from the accumulated interaction history, forcing them to discard valuable insights and repeat past errors. We propose ReasoningBank, a novel memory framework that distills generalizable reasoning strategies from an agent's self-judged successful and failed experiences. At test time, an agent retrieves relevant memories from ReasoningBank to inform its interaction and then integrates new learnings back, enabling it to become more capable over time. Building on this powerful experience learner, we further introduce memory-aware test-time scaling (MaTTS), which accelerates and diversifies this learning process by scaling up the agent's interaction experience. By allocating more compute to each task, the agent generates abundant, diverse experiences that provide rich contrastive signals for synthesizing higher-quality memory. The better memory in turn guides more effective scaling, establishing a powerful synergy between memory and test-time scaling. Across web browsing and software engineering benchmarks, ReasoningBank consistently outperforms existing memory mechanisms that store raw trajectories or only successful task routines, improving both effectiveness and efficiency; MaTTS further amplifies these gains. These findings establish memory-driven experience scaling as a new scaling dimension, enabling agents to self-evolve with emergent behaviors naturally arise. Our code can be found at https://github.com/google-research/reasoning-bank.
1. Introduction
ReasoningBank addresses agents’ limited learning from accumulated experience by distilling actionable reasoning strategies from both successful and failed interactions. Combined with MaTTS, it improves effectiveness and efficiency across web browsing and software engineering benchmarks while supporting increasingly complex strategies.
- Persistent agents often approach each task in isolation, failing to learn from accumulated interaction experience.
- Existing memory methods mainly store raw trajectories or successful routines, limiting transferable reasoning and underusing lessons from failures.
- ReasoningBank distills actionable, generalizable principles from self-judged successful and failed experiences, then retrieves and consolidates memories in a continuous loop.
- MaTTS scales each task through diverse exploration in parallel or sequential settings, producing contrastive signals that improve memory synthesis and guide later exploration.
- 20% relative improvement and 16% fewer interaction steps are reported for ReasoningBank across WebArena, Mind2Web, and SWE-Bench-Verified benchmarks.
- The approach is presented as a memory-driven scaling dimension that enables agents to develop increasingly complex, emergent reasoning strategies over time.
2. Related Work
Prior agent-memory research organizes stored information using varied representations and retrieval or management mechanisms, often emphasizing personalization and long-context handling. Agent test-time scaling has improved end-to-end problem solving, but prior efforts did not consider memory’s role in guiding scaling.
- Agent-memory systems store past information as plain text, latent embeddings, or structured graphs, with retrieval and memory-management strategies.
- Recent reinforcement-learning approaches also use RL for memory management, while much prior work emphasizes personalization and long-context management.
- Test-time scaling methods such as best-of-N, beam search, and verifiers have been applied to coding, mathematics, and other end-to-end problem-solving tasks.
- Prior test-time-scaling efforts did not consider agents learning from past experiences to guide future decisions; MaTTS introduces memory-aware scaling to address this gap.
3. Methodology
ReasoningBank turns streaming test-time interactions into structured, reusable reasoning memory and retrieves it to guide future agent actions. MaTTS extends this loop by scaling exploration within each task through parallel self-contrast or sequential self-refinement.
- Problem setup: At test time, agents process streaming queries without future queries or ground-truth labels, using past trajectories and self-verification to evolve.The agent policy interacts sequentially with web or software-engineering environments, with memory supplied as additional system instruction.
- ReasoningBank: ReasoningBank distills past experiences into structured memory items containing a title, description, and content.The items abstract transferable reasoning patterns while preserving decision rationales and operational insights.
- ReasoningBank: ReasoningBank retrieves relevant memories, extracts new items from experience, and consolidates them into a closed-loop memory process.Retrieved items guide current actions, while new items are formed from both successful and failed trajectories using LLM-as-a-Judge signals.
- MaTTS: Vanilla test-time scaling independently converts additional trajectories into memory items but does not leverage guidance across those trajectories.This motivates memory-aware scaling methods that use richer interactions to improve memory construction.
- MaTTS: Parallel MaTTS compares multiple trajectories through self-contrast, whereas sequential MaTTS iteratively refines one trajectory and uses intermediate notes as memory signals.The scaling factor k denotes the number of parallel trajectories or sequential refinement steps.
- MaTTS: The memory-aware scaling strategies are designed so additional test-time computation produces more transferable and higher-quality memory for future tasks.MaTTS therefore couples exploration with memory formation rather than treating scaling as independent trajectory generation.
4. Experiments
Experiments across web browsing and software engineering benchmarks evaluate ReasoningBank and MaTTS for effectiveness, efficiency, generalization, and scaling behavior. ReasoningBank improves success rates and reduces interaction steps, while MaTTS further benefits from memory-aware scaling.
- Benchmarks and setup: Experiments cover WebArena, Mind2Web, and SWE-Bench-Verified using success-rate and interaction-step metrics.Evaluations use multiple LLM backbones and compare memory-free, trajectory-based, and workflow-based baselines.
- ReasoningBank results: ReasoningBank consistently outperforms baselines across LLM backbones and datasets.On WebArena, overall success rate improves by +8.3, +7.2, and +4.6 versus memory-free agents across three backbones.
- Generalization: +4.6 averaged SR over the strongest baseline is achieved on WebArena’s Multi subset, while Mind2Web gains persist across increasingly demanding generalization settings.The strongest gains occur in Mind2Web’s cross-domain setting, which requires the highest level of generalization.
- Efficiency: Up to 1.4 fewer average steps than No Memory and 1.6 fewer than other memory baselines are reported on WebArena.ReasoningBank also reduces average steps on SWE-Bench-Verified.
- MaTTS scaling: Increasing the MaTTS scaling factor generally improves success rate in both parallel and sequential settings.With ReasoningBank, parallel scaling rises from 49.7 at k=1 to 55.1 at k=5, while sequential scaling rises from 49.7 to 54.5.
- Memory-aware scaling: At k=5, MaTTS reaches 55.1 in parallel scaling versus 52.4 for vanilla TTS, and 54.5 versus 51.9 in sequential scaling.Memory-aware coordination and aggregation synthesize contrastive signals from multiple trajectories or interaction steps.
5. Analysis
Analysis examines failure-trajectory use, emergent memory strategies, efficiency by outcome, and robustness to judge calibration. The findings associate stronger memory with better scaling and targeted reductions in interaction steps.
- Emergent behaviors: ReasoningBank’s strategies evolve from procedural rules to adaptive self-reflection and systematic completeness checks.A case study tracks the evolution of a User-Specific Information Navigation memory item during test-time learning.
- Failure trajectories: ReasoningBank benefits from incorporating failure trajectories, unlike success-only baselines whose performance is limited or degraded when failures are added.Figure 7 compares success-only and success-plus-failure memory construction on WebArena-Shopping.
- Judge robustness: The analysis evaluates robustness to verification noise from an LLM-as-a-Judge, whose baseline accuracy is 72.7% against ground-truth labels.The calibration study is conducted on the WebArena-Shopping subset with Gemini-2.5-flash as judge.
- Efficiency: ReasoningBank reduces interaction steps across domains, with successful cases improving by up to 2.1 steps, or 26.9%, relative to failed cases.The analysis attributes this pattern to following effective reasoning paths rather than simply truncating failed trajectories.
6. Conclusion
The paper concludes that ReasoningBank distills strategy-level signals from successful and failed experiences and integrates them with MaTTS. Experiments support improved performance, reduced redundant exploration, and a synergy in which memory and diverse rollouts improve one another.
- Conclusion: ReasoningBank distills strategy-level reasoning signals from both successes and failures and integrates them into MaTTS.The framework is presented as a memory-based approach for adaptive agent systems.
- Conclusion: Experiments show consistent performance improvements while reducing redundant exploration.The conclusion summarizes evidence across the paper’s benchmark studies.
- Conclusion: Memory guides scaling toward promising rollouts, while diverse rollouts enrich memory with contrastive signals.The paper also reports analyses of individual components and emergent behaviors.
- Implementation details: Implementation details cover extraction prompts, self-evaluation, and handling successful and failed web-navigation trajectories.The appendix describes an LLM-based pipeline for converting trajectories into memory items.
A.2. Implementation Details
Implementation stores extracted strategy items, retrieves relevant memories, and consolidates new experiences. MaTTS uses either parallel self-contrast across trajectories or sequential re-checking, with Best-of-N selecting among candidate answers.
- Memory extraction: An LLM extractor converts trajectories into memory items with a title, description, and content.The extractor processes each trajectory and can produce at most three memory items.
- Memory retrieval: Query embeddings support cosine-similarity retrieval of the top-k relevant experiences for insertion into the agent prompt.The default retrieval setting is k=1, using gemini-embedding-001 embeddings.
- Memory consolidation: Memory items are appended directly to the pool after each query without additional pruning.The minimal consolidation strategy isolates ReasoningBank from more complex consolidation algorithms.
- Parallel scaling: Parallel MaTTS compares successful and failed trajectories to identify generalizable strategies and mistakes through self-contrast reasoning.The procedure seeks observable differences between outcomes to produce transferable insights.
- Sequential scaling: Sequential MaTTS repeatedly re-examines a trajectory with check instructions to improve consistency and correction without external judgment.The sequential setting is contrasted with parallel multi-trajectory comparison.
- Best-of-N: Best-of-N feeds N trajectories to the same backbone LLM, which selects the single best answer.This selection procedure is used to calculate the BoN metric.
B. Details for Experiment Settings
The experiments evaluate ReasoningBank on web-browsing and software-engineering benchmarks using effectiveness and efficiency metrics, with comparisons against memory baselines and ablations of retrieved experiences. Additional analyses examine inference cost and exclude AWM from SWE-Bench because its open-ended action space does not fit fixed-workflow extraction.
- Datasets: Experiments cover WebArena, Mind2Web, and SWE-Bench-Verified for web navigation, generalization, and repository-level issue resolution.WebArena contains 684 instances, Mind2Web 1,341, and SWE-Bench-Verified 500 manually verified issues.
- Baselines: ReasoningBank is compared with Vanilla, Synapse, and AWM, which respectively provide no memory, trajectory memory, and reusable workflows.The comparison isolates differences in memory formulation, particularly memory extraction.
- Metrics: WebArena effectiveness is measured by success rate, requiring the final output or state to match the benchmark ground-truth goal.Success rate is the number of successful tasks divided by the total number evaluated.
- Metrics: Average steps measure efficiency as the interaction cost of completing tasks, with each step defined as one complete agent-environment interaction cycle.The ReAct cycle typically includes observation, thought generation, and a subsequent action.
- Ablation: One retrieved experience raises WebArena-Shopping success from 39.0 without memory to 49.7, while adding more experiences lowers it to 46.0, 45.5, and 44.4.The results suggest that memory relevance and quality matter more than quantity because excessive experiences may introduce conflicts or noise.
- Implementation: AWM is excluded from SWE-Bench because mini-SWE-Agent permits arbitrary Bash commands, making fixed workflow extraction difficult.The SWE-Bench implementation uses a Bash-only environment with a simple ReAct loop.
C.3. Pass@k Analysis
The Pass@k analysis studies parallel scaling on WebArena-Shopping and shows that memory-aware scaling can improve sample efficiency rather than merely increasing the number of trials needed for success.
- Parallel scaling: Pass@k under parallel scaling evaluates memory-aware test-time learning on the WebArena-Shopping subset.The analysis uses Gemini-2.5-Flash and compares MaTTS without aggregation against other scaling behavior.
- Sample efficiency: At k=2, MaTTS without aggregation achieves 50.8 compared with 47.6 for the referenced alternative.The passage characterizes this as improved sample efficiency through guided exploration.
- Interpretation: MaTTS without aggregation makes test-time learning behave similarly to reinforcement-learning training by guiding exploration instead of simply inflating Pass@k at larger k.The stated effect concerns exploration efficiency under parallel scaling.
C.4. Results with Smaller Open-source LLMs
ReasoningBank remains effective on a smaller open-source model and illustrates how retrieved reasoning hints improve both task correctness and navigation efficiency in representative shopping tasks.
- Smaller open-source LLMs: On WebArena-Shopping with Gemma-3-12B-Instruct, ReasoningBank raises success from 17.1% without memory to 24.1%.It outperforms Synapse at 16.0% and AWM at 21.4%.
- Smaller open-source LLMs: ReasoningBank requires the fewest average interaction steps on Gemma-3-12B-Instruct, using 11.8 steps.The passage presents this as an efficiency advantage across the compared baselines.
- Effectiveness: In a first-purchase case, ReasoningBank recalls past reasoning hints, searches the full purchase history, and identifies the earliest order correctly.The baseline checks only Recent Orders and returns the most recent purchase date.
- Efficiency: In a navigation-heavy shopping task, ReasoningBank completes the task in 10 steps versus 29 for the baseline.Stored reasoning about category filtering lets the agent reach the relevant items directly.
- Reflection on Successes and Failures: ReasoningBank reflects on failed trajectories to identify causes of errors and extract strategies intended to prevent similar mistakes.The example attributes the original failure to an imprecise search query that returned numerous irrelevant items.
D. Future Directions
The paper identifies modular memory composition and richer memory architectures as future directions for extending ReasoningBank beyond independent item reuse and embedding-based retrieval.
- Modular and Compositional Memory: Future work could compose modular memory items into higher-level strategies, reusable macros, and complementary combinations for long-horizon generalization.Suggested modules include planning, tool-use, operational, and user-centric memory.
- Modular and Compositional Memory: The current framework retrieves and reuses memory items independently, without composing them into higher-level strategies.The proposed extension is composition-aware retrieval and consolidation.
- Advanced Memory Architectures: A future layered memory stack could combine episodic traces, short-term working memory, and consolidated long-term knowledge with decay or refresh policies.The paper also proposes reasoning-intensive retrieval controllers and learned routing or consolidation policies.
E. Limitations
The framework’s limitations concern its focus on memory content, intentionally simple retrieval and consolidation, and reliance on an LLM judge for correctness signals.
- The study focuses on memory content rather than comparing episodic or hierarchical memory architectures.Combining these memory forms remains a future direction.
- ReasoningBank uses simple embedding-based retrieval and straightforward consolidation to isolate the effect of memory-content quality.Adaptive retrieval and hierarchical consolidation are compatible but outside this work’s focus.
- Success and failure signals depend on an LLM judge, which may introduce noise on ambiguous tasks or when the judge errs.The authors suggest stronger verifiers, human feedback, or ensemble judging as future alternatives.