PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Exp: Experience-Driven Software Issue Resolution

Silin Chen, Shaoxin Lin, Yuling Shi, Heng Lian, Xiaodong Gu, Longfei Yun, Dong Chen, Lin Cao, Jiyang Liu, Nu Xia, Qianxiang Wang

arXiv:2507.23361v2cs.SEcs.CLcs.LG

TL;DR

Existing software-engineering agents treat issues as isolated problems, limiting reuse of prior repair knowledge. SWE-Exp builds an experience bank from earlier trajectories and retrieves actionable guidance for new repairs, achieving a 73.0% Pass@1 resolution rate on SWE-Bench Verified with Claude 4 Sonnet.

  • Problem

    Existing agents primarily rely on internal knowledge and address issues in isolated, stateless workflows rather than reusing structured knowledge from prior resolution attempts.

  • Method

    SWE-Exp distills successful and failed repair trajectories into a multi-faceted experience bank and uses retrieved experiences with dual-agent strategic planning and tactical execution.

  • Results

    73.0% Pass@1 resolution rate on SWE-Bench Verified using Claude 4 Sonnet, significantly outperforming prior results under other agent frameworks.

  • Takeaways & Limitations

    SWE-Exp supports a shift from trial-and-error exploration toward automated agents that systematically accumulate and leverage repair knowledge.

  • Takeaways & Limitations

    Effectiveness depends on the quality and relevance of retrieved experiences, while applicability assessment in novel contexts remains insufficiently robust.

Abstract

from arXiv · show

Recent advances in large language model (LLM) agents have shown remarkable progress in software issue resolution, leveraging advanced techniques such as multi-agent collaboration and Monte Carlo Tree Search (MCTS). However, current agents act as memoryless explorers - treating each problem separately without retaining or reusing knowledge from previous repair experiences. This leads to redundant exploration of failed trajectories and missed chances to adapt successful issue resolution methods to similar problems. To address this problem, we introduce SWE-Exp, an experience-enhanced approach that distills concise and actionable experience from prior agent trajectories, enabling continuous learning across issues. Our method introduces a multi-faceted experience bank that captures both successful and failed repair attempts. Specifically, it extracts reusable issue resolution knowledge at different levels - from high-level problem comprehension to specific code changes. Experiments show that SWE-Exp achieves a Pass@1 resolution rate of 73.0% on SWE-Bench Verified using the state-of-the-art LLM Claude 4 Sonnet, significantly outperforming prior results under other agent frameworks. Our approach establishes a new paradigm in which automated software engineering agents systematically accumulate and leverage repair expertise, fundamentally shifting from trial-and-error exploration to strategic, experience-driven issue resolution.

1 Introduction

SWE-Exp addresses the limitation of memoryless issue-resolution agents by accumulating and retrieving structured knowledge from prior repairs. Its multi-faceted experience bank and dual-agent design support continuous learning, with 73% Pass@1 on SWE-Bench Verified using Claude 4 Sonnet.

  • Problem: Current agents treat issues independently and fail to reuse insights from previous repair attempts.This produces redundant exploration and limits knowledge transfer across similar issues.
  • Approach: SWE-Exp distills prior resolution attempts into structured experiences and retrieves them to guide future repairs.The approach transforms issue resolution from isolated problem-solving into continuous learning.
  • Approach: The experience bank covers problem comprehension, fault localization, and modification strategies.Retrieved experiences provide actionable guidance for new issues.
  • Approach: A dual-agent architecture separates high-level strategy formation from low-level execution.The Instructor formulates strategies while the Assistant performs operations.
  • Results: 73% Pass@1 resolution rate was achieved on SWE-Bench Verified using Claude 4 Sonnet.The benchmark contains 500 human-verified GitHub issues.
  • Contribution: The framework captures repair knowledge from multiple issue contexts and reports state-of-the-art resolution performance among open-source agent frameworks.

2 Motivation

The motivating example contrasts symptom-focused repair with experience-guided root-cause analysis. Historical experience directs the agent toward a defensive-copy fix for the shared attrs mutation underlying the Django checkbox bug.

  • Problem: The Django issue causes all checkbox widgets to appear checked because CheckboxInput.get_context() mutates a shared attrs dictionary.
  • Without Experience: Without experience, the agent patches the composite widget’s rendering context and misses the underlying CheckboxInput design flaw.The resulting solution is narrow and symptom-focused.
  • With Experience: With relevant experience, the agent examines parameter mutation and identifies CheckboxInput.get_context() as the root-cause location.
  • With Experience: The experience-guided fix defensively copies attrs before modification, addressing unintended side effects across usage contexts.The example presents this as more robust than a symptom-level patch.
  • Motivation: The example motivates experience accumulation as a way to move agents from reactive symptom fixing toward pattern-based root-cause resolution.

3 Methodology

SWE-Exp builds an experience bank by collecting repair trajectories, extracting transferable knowledge, and retrieving it during new issue resolution. The bank supports both conceptual diagnosis and concrete code modification across a staged agent workflow.

  • Pipeline: The framework collects successful and failed repair trajectories, extracts structured knowledge, retrieves relevant experiences, and guides subsequent resolution.This constitutes a four-stage pipeline from trajectory collection to experience-informed execution.
  • Trajectory Collection: Each trajectory records directives, actions, resulting repository states, and environment feedback.Success or failure annotations preserve both effective workflows and failure patterns for later analysis.
  • Experience Extraction: An Experiencer agent converts lengthy, noisy, problem-specific trajectories into structured, reusable knowledge from both successful and failed attempts.
  • Experience Representation: Each experience pairs a perspective describing abstract problem understanding with a modification describing a generalized repair strategy.
  • Experience Bank: Experiences are embedded and stored in a vector database called the Experience Bank for semantic similarity retrieval.Issue type and description serve as metadata attributes for indexing.
  • Comprehension Experiences: Comprehension experiences guide issue interpretation, diagnostic hypotheses, and early exploration using contextual and structural cues.They help agents prioritize relevant information and narrow the search space.
  • Modification Experiences: Modification experiences encode responsibility assignment, violated behavioral contracts, and strategies for managing safety, scope, and side effects.Examples separate validation from default assignment and handle optional parameters without overriding valid inputs.
  • Agent Architecture: A dual-agent architecture separates Instructor strategy formation from Assistant tool execution, enabling instruction-level control over operations.The separation is intended to make repair trajectories more focused and interpretable.

4 Experimental Setup

The experiments evaluate SWE-Exp on 500 verified SWE-Bench issues against established repair agents and configure it by extending SWE-Search with experience-driven components. The setup also specifies trajectory collection, retrieval, reranking, and leakage prevention procedures.

  • Research Questions: The evaluation addresses effectiveness against other approaches, component contributions, and hyperparameter effects.
  • Dataset: SWE-Bench-Verified contains 500 verified issues and provides human-verified ground-truth patches for more reliable assessment.
  • Baselines: SWE-Exp is compared with Agentless, SWE-Agent, SWE-Search, and OpenHands.
  • Implementation: The implementation extends SWE-Search and uses DeepSeek-V3-0324 and Claude-4-Sonnet as agent models.DeepSeek uses temperature 0.7 and a maximum of 20 iterations; Claude follows the SWE-Search setup.
  • Experience Collection: Experience collection runs SWE-Search on SWE-Bench-Verified and extracts structured experiences from its trajectories.
  • Experience Retrieval: Retrieval selects the top N=10 error-type-similar issues by cosine similarity, then reranks them to choose k=1 applicable experience.Embeddings come from Multilingual-E5-Large, and same-repository experiences are excluded to prevent leakage.

5 Results

SWE-Exp improves issue-resolution performance across model settings and agent frameworks, while ablations show that experience extraction, retrieval, comprehension, modification, and coordination each contribute. Hyperparameter and case-study analyses further indicate that selective, cross-repository experience reuse supports effective repairs.

  • RQ1: Effectiveness: 73.0% Pass@1 is achieved with Claude-4 Sonnet, outperforming strong agentic baselines under the same model.With DeepSeek-V3-0324, SWE-Exp also reaches 42.0% Pass@1 and outperforms reported methods using that backbone.
  • RQ1: Effectiveness: SWE-Exp attains a higher resolution rate than SAGE and EvoCoder under identical model conditions.The paper attributes this advantage to cross-trajectory, cross-repository experience reuse rather than experience localized to one repair instance.
  • RQ2: Ablation Study: Removing comprehension, modification, and dual-agent components reduces performance by 3.2%, 2.6%, and 2.2%, respectively.These components support diagnostic guidance, safe code changes, and separation of strategic reasoning from tactical execution.
  • RQ2: Ablation Study: Removing experience extraction causes the largest single-component drop, reducing performance by 6.0%.Replacing abstraction with direct problem-statement and golden-patch demonstrations weakens knowledge transfer across repositories.
  • RQ3: Impact of Hyperparameters: 42.0% Pass@1 is reached with one retrieved experience, compared with 37.8% without experience; additional experiences can introduce cognitive burden or conflicting information.Across settings, 131 issues are consistently resolved, while each experience also uniquely resolves some instances.
  • RQ3: Impact of Hyperparameters: Pass@1 improves as the experience bank grows, with the largest gains up to approximately 300 experiences before diminishing returns.In the case study, comprehension experience redirects the repair from an incorrect field-handling location to the enum’s __str__ method.

6 Discussion

SWE-Exp introduces leakage controls and experience-quality safeguards while maintaining modest computational and monetary overhead. Its effectiveness depends on retrieving relevant, correctly extracted experiences, and future work targets more reliable filtering, applicability assessment, and verification.

  • 6.1 Data Leakage: Repository boundaries and temporal ordering restrict retrieval to prior experiences and exclude later trajectories from the target repository.This design addresses repository-specific leakage and supports evaluation of cross-repository generalization.
  • 6.2 Quality of Extracted Experience: Manual verification covered 75 instances, but explicitly citing comprehension experiences caused excessive dependence and misleading strategies during later exploration.Providing experience as message context without forcing its use was more effective.
  • 6.2 Quality of Extracted Experience: One relevant experience performed best because additional trajectories introduced irrelevant or conflicting information that impaired focus and generalization.Modification experiences were more robust because the Instructor had already determined the modification direction.
  • 6.3 Cost Analysis: 203.3K vs. 189.1K average tokens and $0.13 vs. $0.12 average API cost indicate modest overhead relative to SWE-Search.The comparison is reported for the DeepSeek-V3-0324 setting.
  • 6.3 Cost Analysis: 15min49s vs. 12min37s total wall-clock time reflects a marginal increase despite approximately 37 seconds of retrieval overhead per instance.The retrieval component constitutes a relatively small fraction of end-to-end execution time.
  • 6.4 Limitations and Future Directions: SWE-Exp’s effectiveness depends on experience quality and relevance, while its current applicability assessment may permit inappropriate reuse in novel contexts.Future work proposes improved extraction, retrieval and alignment, confidence estimation, applicability scoring, and formal verification.

7 Threats to Validity

The validity discussion identifies threats from model dependence, possible training-data overlap, and limited evidence beyond Python repositories. Cross-language evaluation remains an open direction.

  • Internal: Reliance on a single underlying language model may introduce model-specific bias and limit generalizability across LLM architectures.The authors state that the architecture can be adapted to other state-of-the-art models.
  • Internal: Potential overlap between SWE-bench and undisclosed model training data cannot be verified for DeepSeek-V3-0324.The authors note consistent gains over baselines using the same underlying models.
  • External: Generalizability beyond Python repositories and the evaluated programming-language setting remains an external validity concern.The paper identifies cross-language evaluation as a promising direction for future work.

8 Related Work

Related work spans agentic and non-agentic repository-level repair, systematic search, experience-enhanced learning, and SWE-Exp’s extension of historical experience reuse across the full workflow.

  • Repository-level repair methods include agentic frameworks, specialized non-agentic pipelines, graph-guided localization, and MCTS-based exploration.Examples include SWE-Agent, AutoCodeRover, SpecRover, OpenHands CodeAct, Agentless, LocAgent, and SWE-Search.
  • The broader repository-level setting requires resolving issues across multiple files while understanding dependencies and maintaining code consistency.
  • Existing evaluations often rely on static offline datasets, raising concerns about memorization and configuration-specific optimization.
  • EvoCoder and SAGE demonstrate experience-based adaptation through reflection, plan abstractions, and conditioning later decisions on prior task experience.
  • SWE-Exp extends experience-driven learning to the complete repository-level issue-resolution workflow.It captures and leverages structured experiences across multiple resolution stages.
  • Experience-enhanced agent architectures aim to accumulate and reuse knowledge from past interactions.
  • Experience-enhanced agent research captures procedural knowledge through natural-language insights, weighted experience management, and record-and-replay systems.Examples include ExpeL and AgentRR.

9 Conclusion

SWE-Exp reframes repository-level issue resolution as experience-driven rather than isolated exploration by distilling successful and failed repair knowledge for future guidance. On SWE-bench, it achieves a Pass@1 score of 73.0%.

  • SWE-Exp captures and distills knowledge from successful and failed repair trajectories at multiple levels, including comprehension and modification experiences.Its dual-agent architecture uses historical insights for strategic planning and tactical execution.
  • 73.0% Pass@1 is achieved on SWE-bench.
  • SWE-Exp presents automated issue resolution as a process that systematically accumulates and leverages knowledge rather than relying on trial-and-error exploration.

A Hyperparameters of MCTS

This appendix identifies the hyperparameters used by the Monte Carlo Tree Search algorithm in the study.

  • The study’s Monte Carlo Tree Search algorithm uses hyperparameters listed in Table 4.

B Prompt Templates

The prompt templates define a structured instructor workflow that understands the task, retrieves grounded code context, generates one next action, and iterates until resolution.

  • B Prompt Templates: The appendix documents prompts for the dual-agent workflow, experience extraction, and experience reuse to support reproducibility.
  • B.1 Instructor: The instructor first determines what the task establishes, what remains unclear, and which causes or changes require attention.
  • B.1 Instructor: The instructor narrows code exploration step by step, locating exact regions before directing modifications.
  • B.1 Instructor: Generated modifications must remain task-focused, minimal, localized, and based on retrieved code rather than guesses.
  • B.1 Instructor: The workflow iterates only over unresolved issues and avoids unnecessary changes to already correct code.
  • B.1 Instructor: The instructor uses prior observations to choose efficient next actions and avoids repeating similar exploration or modification steps.
  • B.1 Instructor: Each output contains thoughts, instructions, and context, with instructions expressed as one specific actionable objective for the assistant.
  • B.1 Instructor: The context records environment-grounded paths, classes, functions, code identifiers, line ranges, and span identifiers, while action types distinguish search, view, modify, and finish.

B.2 Assistant

The assistant workflow enforces structured, evidence-based issue resolution through explicit reasoning, targeted code exploration, minimal modifications, and strict output formats. Separate agents classify issues and distill abstract insights from successful and failed repair trajectories.

  • Execution format: Agents must analyze the task, environment, interaction history, and code context before selecting retrieval or modification actions.For code changes, they identify the target path, exact snippet, surrounding logic, and dependencies rather than guessing.
  • Execution format: The workflow requires each response to contain a Thought explaining reasoning and an Action specifying one concrete next step.Thoughts should incorporate prior observations, justify the chosen action, state expected outcomes, and identify risks.
  • Action policy: Code exploration prioritizes class, function, snippet, and semantic searches, using ViewCode only for additional context.Modifications primarily use exact StringReplace operations, while CreateFile is reserved for new functionality.
  • Action policy: Agents are instructed to implement only the requested change, avoid unrelated edits, and finish only when the task is fully resolved.The workflow limits changes to visible code and prohibits extending the task with unsolicited reviews or additional changes.
  • Issue analysis: Issue classification produces a JSON object containing a generalized error category and a brief description of that category.The required category format is a descriptive name ending in Error.
  • Experience extraction: Successful trajectories yield an abstract perspective on how the issue was understood, while failed trajectories yield reflections on perspective and modification mistakes.Both forms of experience avoid specific object names and implementation details, emphasizing generalized reasoning.

B.5 Modification ExpAgent

The modification-oriented agents distill transferable reasoning from successful patches, then retrieve and adapt relevant experiences to strengthen future debugging instructions. This process connects past issue-solving approaches with current code exploration and likely code entry points.

  • Modification experience: The modification experience agent compares a successful generated patch with the issue and trajectory to identify why the patch resolved the bug.It also determines which code modification was vital and abstracts the reasoning into principles applicable to future debugging.
  • Modification experience: Extracted modification experiences may contain at most three abstract insights and must omit specific function names, variable names, and code strings.The intended output captures transferable principles or patterns rather than reproducing implementation details.
  • Experience retrieval: The reranking agent compares the current problem with past issue descriptions and experiences, selecting up to k clearly relevant trajectories while requiring at least one selection.It prioritizes past approaches whose problem-solving perspective aligns with the current issue.
  • Experience reuse: The comprehension reuser adapts a selected perspective or reflection to the current issue and identifies the most likely critical codebase entry point.Its output is a new experience statement tailored to the present problem.
  • Experience reuse: The modification reuser rewrites the agent’s next instruction using relevant modification experiences and the current exploration history.The revised instruction must remain related to the original goal while adding more robust reasoning or coverage.
  • Experience reuse: The modification reuser is restricted to modification-type experiences and must not add content unrelated to solving the current issue.Its output is a single enhanced instruction in JSON format.
Loading 2507.23361v2…