PaperlayerView exact revision on arXiv

Source-linked AI summary

EXPEREPAIR: Dual-Memory Enhanced LLM-based Repository-Level Program Repair

Fangwen Mu, Junjie Wang, Lin Shi, Song Wang, Shoubin Li, Qing Wang

arXiv:2506.10484v2cs.SE

TL;DR

Repository-level repair methods often ignore prior repair experience and use static prompts. ExpeRepair combines episodic and semantic memories with dynamic prompting, and reports the best performance among evaluated open-source methods on SWE-Bench Lite and Verified.

  • Problem

    Current repair methods often treat issues in isolation and use static prompts that do not adapt to project or issue context.

    Source: §1 · p. 2

  • Method

    ExpeRepair accumulates and reuses historical repair knowledge through dual memories, using concrete demonstrations and abstract repair patterns to compose context-aware prompts.

    Source: §1 · p. 3, §2.3.1 · p. 7

  • Results

    ExpeRepair outperforms all evaluated state-of-the-art open-source baselines on SWE-Bench Lite and Verified, achieving 60.3% pass@1 on SWE-Bench Lite using Claude 4 Sonnet.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors conclude that leveraging past repair knowledge improves performance on challenging software issues.

    Source: §7 · p. 20

    The approach places less emphasis on bug localization, whose correctness lacks an automated oracle.

    Source: §5.4 · p. 18

Abstract

from arXiv · show

Automatically repairing software issues remains a fundamental challenge at the intersection of software engineering and AI. Although recent advances in Large Language Models (LLMs) have demonstrated potential for repository-level repair tasks, current methods exhibit two notable limitations: (1) they often address issues in isolation, neglecting to incorporate insights from previously resolved issues, and (2) they rely on static, rigid prompting strategies that constrain their ability to generalize across diverse and evolving contexts. We propose ExpeRepair, a novel LLM-based program repair framework inspired by the dual-memory systems of human cognition, where episodic and semantic memory synergistically support learning and decision-making. Unlike existing methods, ExpeRepair continuously learns from historical repair experiences via dual-channel knowledge accumulation, enabling it to adaptively reuse past knowledge during inference. Specifically, ExpeRepair organizes prior repair knowledge into two complementary memories: an episodic memory that stores concrete repair demonstrations, and a semantic memory that encodes abstract, reflective insights. At inference time, ExpeRepair activates both memory systems by retrieving relevant demonstrations from episodic memory and recalling high-level repair insights from semantic memory. It further enhances adaptability through dynamic prompt composition, integrating both memory types to replace static prompts with context-aware, experience-driven prompts. We evaluate ExpeRepair on two benchmarks: SWE-Bench Lite and SWE-Bench Verified. Experimental results show that ExpeRepair achieves pass@1 scores of 60.3% and 74.6% on the two benchmarks, respectively, achieving the best performance among the evaluated open-source methods. We have open-sourced ExpeRepair at https://github.com/ExpeRepair/ExpeRepair.

1 Introduction

Repository-level repair must handle large codebases, complex issue descriptions, and cross-file dependencies, yet current methods often ignore prior repairs and rely on static prompts. ExpeRepair combines episodic and semantic memories to adapt repair prompts, and its reported results outperform open-source baselines on two SWE-Bench benchmarks.

1 Introduction

  • Current repository-level repair methods often treat issues in isolation and use static prompts that do not adapt to project conventions or changing task context.Recurring bug patterns can involve syntactically similar edits or different edits with similar underlying repair logic.Source: §1 · p. 2
  • ExpeRepair continuously accumulates and reuses historical repair experience through episodic demonstrations and semantic insights that inform repairs.The test and patch agents collaborate on test generation, patch generation, and patch validation; episodic memory stores concrete demonstrations, while semantic memory distills natural-language insights from repair trajectories.Source: §1 · pp. 2–3
  • 60.3% pass@1 on SWE-Bench Lite and 74.6% pass@1 on SWE-Bench Verified, outperforming state-of-the-art open-source baselines with Claude 4 Sonnet.Ablation studies also validate the dual-memory mechanism's effectiveness in enhancing repair performance.Source: §1 · p. 3
  • Dynamic, experience-driven prompts tailor repair strategies to issue and repository characteristics, addressing the limits of static handcrafted prompts.The framework draws on concrete repair demonstrations and abstract repair patterns.Source: §1 · p. 3

2 Method

ExpeRepair's workflow centers on a repair module that collaborates with a memory module to capture and reuse repair experience. The repair module handles test generation, patch generation, and validation, while later repairs draw on stored demonstrations and insights.

2 Method

  • The Program Repair Module uses a test agent and a patch agent to perform test generation, patch generation, and patch validation.The Memory Module captures the repair trajectory and extracts concrete demonstrations and high-level insights for storage.Source: §2 · p. 3
  • After an issue is processed, the Memory Module stores concrete demonstrations in episodic memory and high-level repair insights in semantic memory.These are extracted from the corresponding repair trajectory.Source: §2 · p. 3
  • For subsequent issues, ExpeRepair dynamically retrieves relevant demonstrations and insights to inform repair strategies.The section presents the end-to-end workflow before detailing the two modules.Source: §2 · p. 3

2.1 Workflow of ExpeRepair

ExpeRepair first builds memories from seed-issue repair trajectories, then uses both memories during inference to compose context-aware prompts. The resulting repairs can add further demonstrations and insights to those memories.

2.1 Workflow of ExpeRepair

  • ExpeRepair uses a two-phase workflow: it first builds memories from seed issues, then activates both memory systems during inference to compose context-aware prompts.The initial phase operates without prior memories and uses static prompts; inference retrieves episodic demonstrations and semantic insights.Source: §2.1 · pp. 3–4

2.1.1 Initial Phase.

  • In the initial phase, ExpeRepair attempts to reproduce the first five issues from each repository and stores trajectories for successfully reproduced and fully resolved issues.No external data is used as seed issues, and the initial phase operates without accessing prior memories.Source: §2.1.1 · p. 4

2.1.2 Inference Phase.

  • During inference, each agent appends semantic insights to its task prompt and retrieves top-k episodic demonstrations using the current issue context and latest feedback as the query.After repairs, the memory module records trajectories, extracts new demonstrations and insights, and updates both memories.Source: §2.1.2 · pp. 4–5

2.2 Program Repair Module

The Program Repair Module generates reproduction tests, localizes faults, iteratively proposes patches, and validates candidates with expanded tests. Memory retrieval and feedback inform test and patch generation, while selection follows execution results and LLM review.

2.2.1 Test Generation.

  • The test agent iteratively refines reproduction tests using feedback, retrieved demonstrations, and semantic insights to address execution failures and broaden tests beyond reported symptoms.It continues until it produces a valid reproduction test or reaches the iteration limit.Source: §2.2.1 · p. 5

2.2.2 Patch Generation.

  • After successful issue reproduction, ExpeRepair localizes suspicious files and faulty lines, then uses episodic demonstrations and semantic insights to iteratively generate and refine patches.Multiple high-temperature samples produce candidate patches until one passes the reproduction test or the iteration limit is reached.Source: §2.2.2 · p. 6

2.2.3 Patch Validation.

  • After a candidate passes the reproduction test, ExpeRepair revises and augments it for edge cases and regression risks, then selects a final patch using expanded-test results and LLM review.The expanded suite includes reproduction and validation tests, with selection criteria including correctness and adherence to best practices.Source: §2.2.3 · p. 7

2.3 Memory Module

ExpeRepair stores concrete repair demonstrations and reflective insights from completed repair trajectories, then retrieves both forms of experience to guide future repairs.

2.3.1 Memory Storage.

  • Completed repair trajectories supply issue descriptions, reproduction scripts, candidate patches, and test results for memory updates.Episodic memory stores successful input-output pairs and failed attempts with feedback and corrected outputs.Source: §2.3.1 · p. 7
  • Semantic memory distills trajectory analysis into insights about repair strategies, failure causes, and effective heuristics.Insights are added, removed, or edited to address novelty, contradiction, redundancy, or overlap; the memory has a per-agent entry limit.Source: §2.3.1 · p. 7

2.3.2 Memory Activation.

  • Episodic memory retrieves the top-k relevant demonstrations using a query built from current issue context and available feedback.Retrieval may use embedding search or BM25, and the demonstrations become in-context examples for test-script or patch generation.Source: §2.3.2 · p. 8
  • Semantic-memory insights are added to both agents’ prompts to adapt their strategies to the current issue.The paper characterizes this as lightweight auto-prompting that draws on accumulated semantic experience.Source: §2.3.2 · p. 8

3 Experimental Setup

The evaluation asks how well ExpeRepair resolves issues, how its components and model choice affect performance, and how memory hyperparameters matter. Experiments use two SWE-Bench subsets, open-source baselines, and repair, cost, and intermediate-stage metrics.

3.1 Research Questions

  • Four research questions examine issue resolution against other approaches, component contributions, underlying LLM choice, and memory-hyperparameter sensitivity.Source: §3.1 · p. 9

3.2 Benchmark

  • SWE-Bench Lite contains 300 real-world GitHub issues from Python projects, while SWE-Bench Verified is curated with developer validation that issues are solvable.Source: §3.2 · p. 9

3.3 Baselines

  • The comparison includes representative open-source systems such as SWE-agent, Aider, AutoCodeRover, Agentless, OpenHands, PatchPilot, and DARS.The listed systems include both agentic and procedural approaches.Source: §3.3 · p. 9

3.4 Metrics

  • The primary metrics are pass@1 for first-attempt issue resolution and average inference cost per instance.Ablations additionally use execution success rate and reproduction success rate to assess intermediate repair stages.Source: §3.4 · pp. 9–10

3.5 Implementation

  • Claude 3.5/4 Sonnet is the primary model, with o4-mini used only in rare fallback cases under 1% of LLM calls.Test generation runs for up to three rounds; patch generation runs for up to three iterations with four candidate patches per iteration, followed by validation tests and augmented patches.Source: §3.5 · p. 10
  • Episodic memory retrieves the top three demonstrations with BM25, while semantic insights are appended as repair guidelines.Stored semantic insights are capped at 15 entries and updated using ADD, REMOVE, or EDIT.Source: §3.5 · p. 10

4 Results

ExpeRepair outperforms the evaluated open-source baselines on both SWE-Bench benchmarks, while ablations show contributions from both memories and from retrieval and maintenance choices. BM25 and updates that include REMOVE perform best among the reported variants.

4.1 Overall Performance

  • 60.3% on SWE-Bench Lite and 74.6% on SWE-Bench Verified are ExpeRepair’s pass@1 results with Claude 4 Sonnet, surpassing evaluated open-source approaches.With Claude 3.5 Sonnet, ExpeRepair resolves 48.3% and 57.2% of issues on the respective benchmarks; Table 1 compares methods’ pass@1 and average inference cost.Source: Table 1 · p. 11, §4.1 · p. 11

4.2.1 Ablations on Episodic Memory and Semantic Memory Components.

  • Removing the full Memory Module drops resolution from 48.3% to 42.3% on SWE-Bench Lite and from 57.2% to 50.4% on SWE-Bench Verified.Execution and reproduction success rates also fall substantially; the table reports ablations of the memory components.Source: §4.2.1 · p. 12, Table 2 · p. 12
  • Disabling episodic memory lowers pass@1 to 44.7% on SWE-Bench Lite and 52.8% on SWE-Bench Verified, while disabling semantic memory lowers it to 47.0% and 56.2%.The reported component hierarchy places the full Memory Module first in impact, episodic memory next, and semantic memory after it; the combination yields the highest pass@1, ESR, and RSR on both benchmarks.Source: §4.2.1 · p. 12

4.2.2 Ablations on Episodic Memory Retrieval.

  • 48.3% on SWE-Bench Lite and 57.2% on SWE-Bench Verified are BM25’s best or tied-best pass@1 results among the tested episodic retrieval strategies.Embedding retrieval underperforms and hybrid retrieval does not surpass BM25; the authors attribute this to lexical overlap in failure signals such as error messages and test feedback.Source: §4.2.2 · p. 12

4.2.3 Ablations on Semantic Memory Update Mechanisms.

  • The update-strategy comparison includes ADD-only, ADD+EDIT, ADD+REMOVE, and the full ADD+EDIT+REMOVE strategy.When insights exceed the model’s input limit, the oldest entries are truncated to stay within budget.Source: §4.2.3 · p. 13
  • 3.6 pp is the pass@1 improvement on SWE-Bench Verified for updates with REMOVE over ADD-only.Variants without REMOVE underperform; combining EDIT with REMOVE achieves the best performance across both benchmarks, with REMOVE controlling growth and noise and EDIT refining remaining entries.Source: §4.2.3 · p. 13, Figure 2 · p. 13

4.3 Results on Different LLMs

ExpeRepair’s performance varies consistently with the underlying foundation model across both benchmarks. Claude 4 Sonnet ranks highest, while even less capable models achieve competitive results.

4.3 Results on Different LLMs

  • Claude 4 Sonnet ranks first, followed by Claude 3.5 Sonnet, DeepSeek-R1, and o1-mini across both benchmarks.ExpeRepair pass@1 ranges from 41.7% (o1-mini) to 60.3% (Claude 4 Sonnet) on SWE-Bench Lite, and from 49.0% to 74.6% on SWE-Bench Verified.Source: §4.3 · p. 14, Figure 3 · p. 14
  • ExpeRepair with o1-mini attains pass@1 performance comparable to SpecRover built on Claude 3.5 Sonnet.The passage also describes o1-mini and DeepSeek-R1 pairings as competitive despite being relatively less capable models.Source: §4.3 · p. 14
  • ExpeRepair is broadly compatible with diverse LLM backbones, and stronger base models consistently achieve higher performance.The paper identifies model selection as important when deploying automated code repair systems in practice.Source: §4.3 · p. 14

4.4 Sensitivity Analysis of Hyperparameters

The sensitivity analysis varies retrieved episodic demonstrations and stored semantic insights independently on SWE-Bench Lite and Verified with Claude 3.5 Sonnet. Results peak at moderate memory sizes, with performance remaining near-optimal across nearby settings.

4.4 Sensitivity Analysis of Hyperparameters

  • 15 stored insights and top-3 demonstration retrieval achieve 48.3% on Lite and 57.2% on Verified, with near-optimal performance at 10–20 insights and k=2–4.The analysis describes ExpeRepair as not overly sensitive to minor hyperparameter variations.Source: Figure 4 · p. 14, §4.4.2 · p. 15

4.4.1 Number of Retrieved Demonstrations.

  • Increasing retrieved demonstrations from k=0 to k=1 raises performance from 44.7% to 48.0% on Lite and from 52.8% to 56.8% on Verified.Performance peaks at k=3 on both datasets, then falls at k=5 to 45.3% on Lite and 55.6% on Verified, suggesting retrieval breadth trades off against contextual interference.Source: Figure 4 · p. 14, §4.4.1 · p. 15

4.4.2 Number of Stored Insights.

  • Stored insights peak at 15 on Lite (48.3%) and at 15–20 on Verified (57.2%), then decline slightly at 25 insights.The text suggests redundancy or lower-utility information in the constrained context window may explain the later plateau or decline.Source: Figure 4 · p. 14, §4.4.2 · p. 15

5 Discussion

The discussion uses SymPy and Django cases to illustrate the distinct contributions of semantic and episodic memory, then examines overlap with other repair methods, performance over time, and study limitations.

5.1.1 Semantic Memory.

  • With semantic memory, the patch addresses both Max and its counterpart Min, avoiding the asymmetric, incomplete fix produced without that memory.The stored insight advises systematically handling symmetric or related operations.Source: §5.1.1 · pp. 15–16

5.1.2 Episodic Memory.

  • Episodic-memory examples help the test agent build a functional Django setup that reproduces the regression, unlike its incomplete initial test script.The retrieved example sets up a temporary application, models, and test data, then executes queries to expose the fields included in generated SQL.Source: §5.1.2 · p. 17

5.2 Intersection Analysis

  • ExpeRepair uniquely resolved 8 SWE-Bench Lite issues, while DARS and PatchPilot uniquely resolved 6 and 7, respectively; 65 issues were solved by all methods.Six of ExpeRepair’s eight unique fixes followed recurring patterns derived from accumulated experience. The authors describe the methods’ capabilities as complementary and suggest hybrid approaches as future work.Source: §5.2 · pp. 17–18
  • Six of ExpeRepair’s eight unique fixes followed recurring experience-derived patterns, while four of DARS’s six unique fixes required novel solutions.The authors note that DARS’s structured branching and iterative refinement better suit exploratory reasoning when repository analogues are scarce.Source: §5.2 · pp. 17–18

5.3 Longitudinal Study

  • After roughly 20 issues, cumulative resolution rates begin to stabilize and gradually improve across the three studied repositories.Django stabilizes around issue 80 at approximately 60%; the authors suggest diminishing returns from repeated memory and greater heterogeneity among remaining issues may explain this.Source: Figure 7 · p. 17, §5.3 · p. 18

5.4 Limitation

  • The framework places less emphasis on bug localization because its correctness lacks an automated oracle comparable to execution-based checks.Accumulating localization experiences only from successful repairs could leave memory with limited diversity and coverage.Source: §5.4 · p. 18

5.5 Threats to Validity

  • The authors flag possible benchmark training-data overlap and evaluation across only four models from three providers as threats to validity.They use SWE-Bench as a newer benchmark considered less susceptible to contamination and note that results across evaluated models were consistently improved.Source: §5.5 · pp. 18–19
  • Evaluation covers open-source Python projects, so the study does not demonstrate cross-language effectiveness.The authors describe the approach as language-agnostic and suggest it may extend to languages such as C# and Ruby.Source: §5.5 · p. 19

6 Related Work

Prior work explores persistent and interaction-derived memories for LLM agents, alongside agentic and procedural approaches to repository-level repair. ExpeRepair applies dual memory and dynamic prompts to collaborative software maintenance.

6.1 Memory-Enhanced LLM Agents

  • Memory-enhanced agent work ranges from dialogue continuity to procedural knowledge and architectures that combine semantic and episodic memory.Examples include dialogue-oriented persistent memory, interaction-derived insights, record-and-replay, linked notes, and a dynamic knowledge graph world model.Source: §6.1 · p. 19

6.2 Repository-Level Automatic Program Repair

  • Repository-level repair includes tool-using agents, fixed procedural workflows, and training-based pipelines.The cited examples cover search and exploration strategies, localization and collaboration, reproduction through validation phases, and continued training on repair data.Source: §6.2 · p. 20
  • ExpeRepair addresses the gap between agent reasoning and practical repository maintenance by accumulating repair trajectories and reusing them through dynamic prompts.The paper states this adaptation does not rely on costly fine-tuning or complex agent orchestration.Source: §6.1 · p. 19, §6.2 · p. 20

7 Conclusion

ExpeRepair accumulates and reuses repair experience through episodic and semantic memories to adapt prompts to issues and repositories. It reports pass@1 scores of 60.3% on SWE-Bench Lite and 74.6% on SWE-Bench Verified using Claude 4 Sonnet.

7 Conclusion

  • ExpeRepair combines episodic and semantic memories to accumulate and reuse repair experience through dynamic, context-aware prompt adaptation.The conclusion says this enhances repair effectiveness and efficiency.Source: §7 · p. 20
  • 60.3% pass@1 on SWE-Bench Lite and 74.6% pass@1 on SWE-Bench Verified were reported using Claude 4 Sonnet.Source: §7 · p. 20
  • The authors conclude that leveraging past repair knowledge significantly improves performance on challenging program repair tasks.Source: §7 · p. 20

Related papers

Loading 2506.10484v2…