PaperlayerView exact revision on arXiv

Source-linked AI summary

Escaping the Self-Confirmation Trap: An Execute-Distill-Verify Paradigm for Agentic Experience Learning

Shiding Zhu, Yudi Qi, Yajie Wang, Jiaze Li, Chao Song, Yaorui Shi, Yibo Miao, Hanqi Gao, Kai Zhang

arXiv:2606.24428v1cs.CL

TL;DR

Existing single-agent experience learning can accept flawed trajectories as valid memory when execution and evaluation share one perspective. EDV decouples execution, distillation, and verification through heterogeneous multi-agent collaboration, and consistently outperforms strong baselines on three challenging long-horizon benchmarks.

  • Problem

    Single-agent experience learning couples task execution and evaluation, allowing flawed trajectories to be mistaken for valid experience in open-world environments without explicit ground-truth feedback.

  • Method

    EDV constructs experience through heterogeneous parallel execution, third-party comparative distillation, and consensus-based validation before memory insertion.

  • Results

    EDV consistently outperforms strong baselines on τ2-bench, Mind2Web, and MMTB, improving the reliability of written memory and downstream task performance.

  • Takeaways & Limitations

    The findings suggest that reliable, reusable experience matters more for agent self-evolution than simply accumulating more experience.

  • Takeaways & Limitations

    EDV may suffer from consensus bias, interference from low-quality agents, and difficulty attributing failures in decentralized multi-turn interactions.

Abstract

from arXiv · show

Experience-driven self-evolution is critical for large language model (LLM) agents to improve through open-world interaction. However, existing experience learning methods mostly rely on single-agent loops, where the same agent executes tasks, summarizes outcomes, and determines memory content. This setup makes agents vulnerable to the Self-Confirmation Trap: wrong-but-self-consistent trajectories are misidentified as successful experience, leading to cumulative errors during retrieval and reuse. To address this issue, we propose EDV, an Execute-Distill-Verify framework for reliable experience learning. In the Execute stage, multiple heterogeneous agents explore the same task space in parallel to generate diverse candidate trajectories. In the Distill stage, a dedicated third-party agent comparatively analyzes these trajectories to produce candidate experiences, reducing executor-centric summarization bias. In the Verify stage, the execution group validates candidates via a consensus mechanism, and only approved experiences are written into shared or private memory. By decoupling the three stages, EDV transforms experience learning from isolated self-reflection into collaborative construction, filtering erroneous and noisy content before memory insertion. We evaluate EDV on three challenging long-horizon benchmarks: tau2-bench, Mind2Web and MMTB. Results show EDV consistently outperforms strong baselines, validating that reliable experience construction is essential for robust agent self-evolution. Our code is available at https://github.com/shidingz/EDV.

1 Introduction

The introduction identifies the Self-Confirmation Trap in single-agent experience learning and proposes EDV, which separates execution, distillation, and verification to construct more reliable memories. EDV uses heterogeneous exploration, third-party comparative analysis, and consensus validation, and is reported to outperform strong baselines on challenging long-horizon benchmarks.

  • Motivation: Experience learning enables agents in persistent, open-world environments to distill reusable lessons from historical executions and improve through accumulation and reuse.Memory-based self-evolution has been shown to improve long-horizon reasoning and decision-making.
  • Problem: Single-agent loops couple task execution, outcome interpretation, lesson distillation, and memory writing, making them brittle without explicit ground truth.Execution and evaluation remain coupled within the same reasoning process.
  • Problem: The Self-Confirmation Trap occurs when agents mistake wrong-but-self-consistent trajectories for successful experience and amplify errors through memory accumulation.The problem is especially severe in long-horizon tasks, where intermediate decisions are difficult to verify directly.
  • Method: EDV improves experience-construction reliability through heterogeneous execution, third-party distillation, and consensus-based validation before memory insertion.The framework also routes new tasks to appropriate solvers and retrieves relevant experience during inference.
  • Method: EDV reduces single-agent bias through diverse parallel trajectories, comparative cross-trajectory analysis, and a higher threshold for long-term memory insertion.The introduction attributes these benefits respectively to heterogeneous execution, third-party distillation, and consensus-based validation.
  • Results: EDV consistently outperforms strong baselines on multiple challenging long-horizon agent benchmarks while suppressing erroneous and noisy experience before memory insertion.The introduction presents these results as evidence of EDV’s effectiveness.

2 Related Work

Related work establishes experience-driven adaptation, agentic memory, and multi-agent collaboration as complementary foundations for improving LLM agents. However, existing multi-agent research rarely uses division of labor to improve the reliability of experience construction itself.

  • Experience-driven adaptation: AgentEvolver and FLEX established learning from execution trajectories as a paradigm for continual agent adaptation.Subsequent work extends this paradigm across diverse scenarios, including continual improvement in general environments.
  • Agentic memory: ReasoningBank, General Agentic Memory, and CoPS study distilled reasoning memory, hierarchical memory organization, and cross-task experience sharing for agents.These studies collectively demonstrate the importance of externalizing past interactions into reusable knowledge.
  • Multi-agent collaboration: Collaborative LLM systems can outperform single agents on complex reasoning and decision-making tasks by combining diverse perspectives and complementary capabilities.X-MAS further highlights the value of model collaboration.
  • Multi-agent collaboration: Xolver combines multi-agent collaboration with experience accumulation for distributed capability improvement through shared team-level experience.Yet most multi-agent studies target single-task performance or experience sharing and transfer, while few improve the reliability of experience construction itself.

3 The Self-Confirmation Trap in Experience Learning

The Self-Confirmation Trap occurs when one agent both executes tasks and evaluates trajectories, allowing plausible but flawed experience to enter memory without ground-truth feedback. Because these errors are repeatedly retrieved and reused, decoupling execution from evaluation is necessary for reliable experience construction.

  • Failure mode: The Self-Confirmation Trap arises when the same agent executes tasks and evaluates trajectories, causing flawed trajectories to be accepted as valid experience in open-world settings.Explicit ground-truth feedback is often unavailable, making self-validation especially vulnerable to mistaken acceptance.
  • Failure mode: Coupling execution and evaluation makes their errors statistically dependent, substantially elevating the likelihood that flawed trajectories are incorrectly endorsed as successful.The same policy πθ performs both execution and evaluation in the standard single-agent loop.
  • Failure mode: Once erroneous experience enters memory, repeated retrieval and reuse cause persistent error accumulation.Wrong-but-self-consistent trajectories can therefore reinforce future behavior rather than being corrected.
  • Motivation: EDV addresses this problem by decoupling experience construction roles so candidate experience is examined from independent perspectives before memory insertion.This design directly targets the reinforcement of systematic mistakes caused by relying on one perspective for both generation and validation.

4 Method

EDV constructs reliable experiences through decoupled Execute, Distill, and Verify stages, using heterogeneous exploration, third-party comparison, and consensus validation before memory insertion. At inference time, it selects solvers via an Ability Matrix and hierarchically retrieves shared and private memories.

  • Framework overview: EDV separates experience construction into Execute, Distill, and Verify, followed by inference-time retrieval and application of validated experience.Only validated experiences are written into memory.
  • Execute: In Execute, randomly selected heterogeneous agents independently explore each task in parallel, generating diverse trajectories that expand solution-space coverage.The agent pool spans different foundation models or prompting strategies, while random sampling avoids persistent role bias.
  • Distill: In Distill, a third-party agent compares execution trajectories and extracts reusable candidate experiences rather than restating a single best trajectory.This reduces executor-centric bias and identifies useful differences across multiple trajectories.
  • Verify: In Verify, the original execution agents validate each candidate using consensus, applying default rejection before writing approved experiences to shared or private memory.Unanimous approval writes to shared memory, subset approval writes to corresponding private memory, and other candidates are discarded.
  • Inference-time usage: At inference time, EDV selects a solver through the Ability Matrix, queries shared memory first, then the selected solver’s private memory when needed, and appends retrieved memories to context.The two memory banks respectively store generally reusable and agent-specific experience.

5 Experiments

EDV is evaluated on three long-horizon benchmarks and consistently improves performance, memory quality, and inference efficiency. Ablations and contamination tests attribute these gains to heterogeneous execution, third-party distillation, consensus verification, and hierarchical memory.

  • Benchmarks and baselines: EDV is evaluated on τ2-bench, Mind2Web, and MMTB, covering constraint solving, web interaction, and multi-tool task execution.The evaluation compares EDV with No Memory, Reasoning Bank, Judge, and Router baselines.
  • Main results: 86.6 average Pass@1 on τ2-bench exceeds Router (83.5) and Judge (81.5), while No Memory achieves 76.4–79.6.EDV also maintains strong generalization under Mind2Web’s cross-task, cross-website, and cross-domain settings.
  • Memory quality: EDV improves memory groundedness/correctness from 3.72 to 4.41 and reduces noise/hallucination from 1.21 to 0.63 versus Reasoning Bank.Actionability rises from 3.58 to 4.32, specificity from 3.64 to 4.27, and potential harm if reused falls from 1.08 to 0.51.
  • Memory contamination: Injecting erroneous experiences into 10% of memory lowers Reasoning Bank’s τ2-bench RETAIL Pass@1 from 82.5 to 77.2.The result simulates the Self-Confirmation Trap using internally coherent but incorrect payment rules.
  • Component analysis: An independent verifier paired with one agent improves performance by only 1.2 points, whereas EDV benefits from diverse trajectories and third-party distillation.Unanimous execution-group verification further filters residual errors, making the overall advantage synergistic rather than attributable to one module.
  • Memory design and efficiency: Shared and private memory contribute 2.3% and 0.6% to overall performance, respectively, for a combined contribution of 2.9%.Shared memory is retrieved in 72.3% of tasks, while private memory is retrieved in 31.8% and complements shared memory on personalized edge cases.

6 Conclusion · A Additional Experimental Results

EDV presents a collaborative Execute–Distill–Verify framework that decouples experience execution, distillation, and validation to mitigate the Self-Confirmation Trap. The findings emphasize constructing reliable, reusable experience before memory insertion, making memory quality more important than quantity.

  • 6 Conclusion: EDV is an Execute–Distill–Verify framework for reliable agentic experience learning.The framework targets experience construction for agent self-evolution.
  • 6 Conclusion: EDV mitigates the Self-Confirmation Trap associated with single-agent experience learning in open-world environments.The trap arises when experience learning relies on isolated single-agent processes.
  • 6 Conclusion: EDV decouples execution, distillation, and validation into separate stages.This design separates the processes involved in constructing experiences for memory.
  • 6 Conclusion: EDV transforms memory construction from isolated self-reflection into collaborative experience construction and filtering.The collaborative process is intended to filter experience content before memory use.
  • 6 Conclusion: The experimental findings show that EDV improves the reliability of written memory.The supplied conclusion states this improvement without reporting a numerical result.
  • 6 Conclusion: Effective agent self-evolution depends on constructing reliable and reusable experience, not merely accumulating more experience.The conclusion explicitly prioritizes experience quality before memory insertion.
  • 6 Conclusion: Memory quality matters more than memory quantity for effective agent self-evolution.This is the passage’s broader interpretation of the findings.
  • 6 Conclusion: EDV is presented as a practical step toward stronger long-horizon decision-making and more robust self-improvement.The passage frames these capabilities as broader goals for agent development.

A.1 Training Convergence and Stability … B Prompts Details

EDV improves training convergence, memory-retrieval scaling, and recall-threshold robustness over ReasoningBank, while its prompt system uses comparative distillation and strict verification to retain grounded, high-utility memories. Across the reported analyses, EDV reaches 0.909 at Epoch 4, 0.886 with 3 retrieved experiences, and 0.886 at τ = 0.8.

  • A.1 Training Convergence and Stability: Approximately 7%–9% in Pass@1 score separates EDV from ReasoningBank at every training stage, with EDV reaching 0.909 at Epoch 4.RB stagnates or declines from 0.830 to 0.815 after the second epoch, whereas EDV trends upward.
  • A.1 Training Convergence and Stability: EDV shows a steady upward training trajectory, while ReasoningBank stagnates or slightly declines after the second epoch.The reported RB change is 0.830 →0.815; EDV peaks at Epoch 4.
  • A.2 Scaling with Number of Retrieved Memories: 0.886 is EDV’s Pass@1 score with 3 experiences, after both methods begin at 0.781 with no experiences.ReasoningBank peaks at 0.825 with 1 experience before degrading to 0.793 as more experiences are retrieved.
  • A.2 Scaling with Number of Retrieved Memories: EDV improves monotonically as retrieved experiences increase, whereas ReasoningBank degrades after 1 experience, likely because lower-quality or less relevant memories cause distraction.The contrast is attributed to EDV’s stronger filtering and integration mechanism.
  • A.3 Sensitivity Analysis of Recall Threshold: 0.886 is EDV’s optimal Pass@1 score at τ = 0.8, while performance remains above 0.860 across a wide range of recall thresholds.The slight decline at τ = 0.9 suggests that overly strict filtering can remove useful experiences.
  • B Prompts Details: The prompt system combines an Distill Prompt for contrastive multi-trajectory memory generation with an Verify Prompt for post-hoc auditing of retained memories.The modular design supports diverse environments, including DOM-based navigation and tool-use settings.
  • B.1 Distill Prompt: The Distill Prompt selects among four heterogeneous candidates and requires evidence-based memories structured as “Context →Risk →Action,” with tool outputs injected for data-integrity checks.It is designed to replace simple self-reflection with strict multi-trajectory comparison.
  • B.2 Verify Prompt: The Verify Prompt uses a default-reject audit to exclude redundant or low-information memories unless concrete trajectory evidence supports a non-obvious, high-utility lesson.Schema alignment is identified as a key design principle for tool-use benchmarks.

C Details for Experiment Settings … C.4 Parallel Inference and Reproducibility

The experimental setup combines multi-reasoner trajectory sampling, staged temperature control, structured incremental memory management, semantic retrieval, and deterministic parallel inference. These choices support diverse yet reproducible experience construction across public and private memory repositories.

  • C.1 Generative Modeling and Temperature Scheduling: Two reasoning models each generate four candidate trajectories per step, with generations capped at 8,192 tokens.This multi-reasoner sampling design is used to enhance reasoning robustness before selection and summarization.
  • C.1 Generative Modeling and Temperature Scheduling: Execute, Distill, and Verify use stage-wise temperatures to balance exploration during acquisition with stability during consolidation.Higher temperatures encourage diverse exploration, while zero-temperature bank generation supports stable, reproducible consolidation.
  • C.2 Memory Formation and Integration Strategy: Memory training generates at most five items per step, serializing each as JSON with a title, description, and 1–3 sentence content summary.The fields standardize memory storage around a concise identifier, abstract, and summary of strategies or empirical insights.
  • C.2 Memory Formation and Integration Strategy: Public and Private memories are stored separately, with new entries appended incrementally without clustering, deduplication, or pruning.The minimalist update strategy reduces confounding from complex memory-management modules.
  • C.3 Embedding and Retrieval Configuration: Qwen3-Embedding-4B produces fixed 2560-dimensional vectors, and retrieval ranks entries by cosine similarity.The fixed representation maintains consistency across experimental settings.
  • C.3 Embedding and Retrieval Configuration: Retrieval applies asymmetric thresholds of 0.80 for Public Memory Bank and 0.85 for Private Memory Bank, using task descriptions plus active page context as the query.The higher private-memory threshold suppresses noise and improves the relevance of personalized insights.
  • C.4 Parallel Inference and Reproducibility: Task-level parallel inference uses Nworkers = 16, deterministic pre-allocated model indices, k = 3 retries, and checkpoints every 10 steps.These mechanisms target evaluation throughput, prevent thread-scheduling routing artifacts, tolerate transient anomalies, and preserve intermediate results and memory snapshots.

D Benchmark Specifications and Evaluation Protocols … D.3 MMTB: Hierarchical Metrics for Tool Usage

The evaluation spans dynamic multi-step interaction, web-agent generalization under retrieval constraints, and hierarchical tool-use reliability. It uses task-specific success and progress metrics, including pass@1, Mind2Web’s element/action/step/task measures, and MMTB’s mission-type and path-quality rates.

  • D.1 τ2-Bench: Adaptation in Dynamic Environments: τ2-Bench evaluates agents on dynamic, multi-step tasks that require adaptation as scenarios evolve across multiple domains.Performance is reported using pass@1.
  • D.2 Mind2Web: Retrieval Constraints and Theoretical Bounds: Mind2Web tests web agents under Cross-Task, Cross-Website, and Cross-Domain generalization settings.The settings contain 252 tasks from 69 websites, 177 tasks on held-out websites within seen domains, and held-out domains, respectively.
  • D.2 Mind2Web: Retrieval Constraints and Theoretical Bounds: Mind2Web measures element selection with Element Accuracy (EA), operation agreement with Operation F1 (AF1), and strict execution with Step Success Rate (SSR).EA is binary, AF1 is token-level F1, and SSR requires both correct element selection and a fully correct operation.
  • D.2 Mind2Web: Retrieval Constraints and Theoretical Bounds: Task Success Rate (SR) requires every constituent step in a Mind2Web task to succeed.A single unsuccessful step makes the entire task unsuccessful.
  • D.2 Mind2Web: Retrieval Constraints and Theoretical Bounds: 24.21%, 20.34%, and 18.31% are the SR upper bounds for CROSS-TASK, CROSS-WEBSITE, and CROSS-DOMAIN, respectively, because Top-10 retrieval can exclude the ground-truth element.When the correct element is absent from the retrieved candidates, the step is unsolvable and receives metric scores of 0.
  • D.3 MMTB: Hierarchical Metrics for Tool Usage: MMTB evaluates reliability on interconnected, evolving missions across real-world domains using overall, single-tool, chat-only, clarification, and multi-tool success rates.Its mission-type metrics include All, Asingle, Achat, Aclarity, AP multi, AS multi, and AS+P multi.
  • D.3 MMTB: Hierarchical Metrics for Tool Usage: For multi-tool missions, MMTB additionally reports Optimal Path Rate (Opt.Path) and Accomplished Progress (Acc.Prog).Opt.Path measures optimal invocation paths, while Acc.Prog gives partial credit for completed mission progress.

E Case Study · F Memory Study · F.1 Derivation I: From Operational Inertia to Dynamic Adaptation

Case studies across τ2-Bench, Mind2Web, and MMTB show that EDV addresses single-agent exploration and reasoning biases through collaboratively distilled memories. The memory study derives a dynamic-adaptation category that replaces rote procedural behavior with state-aware execution, improving token efficiency and latency.

  • E Case Study: EDV’s qualitative cases span τ2-Bench, Mind2Web, and MMTB, covering tool use and open-ended web navigation.The samples are selected to assess effectiveness and robustness across different domains.
  • E Case Study: In τ2-Bench, the baseline loops on an invalid travel-certificate payment, while EDV addresses the single-agent exploration upper bound.The flight-modification case attributes failure to an implicit environmental rule and inherent behavioral biases.
  • E Case Study: In Mind2Web, the baseline selects a filter by keyword matching, whereas collaborative memory steers logic-driven information retrieval.The retrieved constraint requires actions to lead to actual information retrieval, functioning as a meta-cognitive intervention.
  • E Case Study: In MMTB, shared memory corrects natural-language tool arguments to ISO 639-1 codes “en” and “ru”, enabling successful translation execution.The memory accumulates executable precision for rigid external-tool requirements.
  • F Memory Study: The memory study organizes extracted memories into three classes derived from single-agent limitations and EDV’s corresponding methodological advances.The taxonomy is presented as a logical progression from identified deficits to memory categories.
  • F.1 Derivation I: From Operational Inertia to Dynamic Adaptation: Single agents exhibit operational inertia by rote adherence to procedural heuristics, even when environmental state makes standard actions redundant or counter-productive.The stated example is “always apply filters.”
  • F.1 Derivation I: From Operational Inertia to Dynamic Adaptation: The Breaking Inertia & Dynamic Adaptation category uses diverse rollouts to encode state-aware shortcuts, such as skipping filters when the target is visible.This category optimizes the Execution Layer, maximizing token efficiency and reducing latency; Table 6 compares single-agent and EDV-induced memories with rollout evidence.

F.2 Derivation II: From Local Optima to Strategic Elevation · F.3 Derivation III: From Epistemic Failure to Deep Attribution

F.2 shows EDV overcoming single-agent local optima by aggregating diverse strategies into globally efficient SOP-level plans. F.3 shows EDV diagnosing grounding failures through heterogeneous consensus and tool-grounded contradictions, producing robust correction memories.

  • F.2 Derivation II: From Local Optima to Strategic Elevation: Single agents often pursue greedy, bounded-horizon behavior, yielding inefficient valid trajectories or complete stagnation on complex tasks.Examples include sequential processing instead of parallelization.
  • F.2 Derivation II: From Local Optima to Strategic Elevation: EDV aggregates diverse strategies and evaluates their global efficacy to synthesize an SOP that outperforms individual local policies.The framework compares competing strategies such as Sorting versus Filtering to identify the optimal path.
  • F.2 Derivation II: From Local Optima to Strategic Elevation: The resulting category, Strategic Elevation & Global Optimization, upgrades the Planning Layer from sub-task completion to global strategy orchestration.An example SOP prioritizes sorting over filtering for price minimization, raising the system’s methodological ceiling.
  • F.3 Derivation III: From Epistemic Failure to Deep Attribution: Single agents struggle to distinguish stochastic failures from fundamental grounding errors, causing hallucinatory or superficial memories.A representative error is targeting a <label> instead of an <input>.
  • F.3 Derivation III: From Epistemic Failure to Deep Attribution: EDV uses heterogeneous model consensus for comparative root-cause analysis, isolating technical or logical fallacies when models fail identically or tools contradict static knowledge.These provenance signals support attribution beyond executor self-diagnosis.
  • F.3 Derivation III: From Epistemic Failure to Deep Attribution: The resulting category, Deep Attribution & Robust Correction, fortifies the Grounding Layer with high-fidelity memories addressing fundamental misconceptions.One example recommends targeting DOM structures by type rather than text to improve environmental interaction robustness.

G Limitations · H Future Work

EDV’s decentralized design introduces consensus bias, low-quality-agent interference, and attribution difficulty. Future work targets long-term memory management and adaptive agent scaling, while qualitative cases show retrieved experience improving tool-use decisions across benchmarks.

  • G Limitations: EDV faces consensus bias when heterogeneous agents share a failure mode, allowing the consensus mechanism to validate noise.This limitation arises despite EDV’s reported success and reflects its decentralized complexity.
  • G Limitations: A significantly underperforming agent can obstruct consensus, creating interference from low-quality agents.The limitation is distinct from shared failure modes among otherwise heterogeneous agents.
  • G Limitations: Multi-turn critic–executor interplay makes attribution difficult within the decentralized framework.The supplied passage identifies attribution difficulty as a further inherent challenge of EDV.
  • H Future Work: Future work will investigate a memory value function for pruning obsolete data and consolidating similar long-term memories.The goal is to maintain long-term efficiency as the Memory Bank continuously grows.
  • H Future Work: Future work will explore inference-time adaptive agent scaling that adjusts the number of agents according to task complexity.The passage also states that the scaling behavior will be analyzed further, but the excerpt truncates that discussion.
  • H Future Work: Qualitative cases report that retrieved memory steers correct tool-use and decision making on τ2-Bench, validates ranking criteria on Mind2Web, and supports parameter extraction on MMTB.The cases contrast memory-augmented behavior with no-memory baselines, including avoiding premature actions and redundant clarification.
Loading 2606.24428v1…