PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillForge: Self-Distilling Agents for Project-Specific Issue Resolution

Silin Chen, Han Li, Xiaodong Gu, Yuling Shi, Haibing Guan

arXiv:2608.18933v1cs.SEcs.AI

TL;DR

Agents often lack project-specific knowledge when resolving issues in a specific repository, creating a cold-start challenge. SkillForge proactively distills such knowledge from synthesized repository issues into reusable, entity-grounded skills, improving issue-resolution performance across SWE-bench Verified and Pro.

  • Problem

    Agents resolving issues in a specific repository often lack project-specific knowledge and must repeatedly rediscover repository-specific conventions.

  • Method

    SkillForge synthesizes project-specific issues from repository tests and code, resolves them, and distills trajectories into reusable entity-grounded skills.

  • Results

    SkillForge yields absolute Pass@1 gains of +5.8%/+5.6% on SWE-bench Verified and +5.8%/+4.1% on SWE-bench Pro for DeepSeek-V3.2 and GPT-5-mini, respectively.

  • Takeaways & Limitations

    Proactively acquiring project-specific knowledge before real issue resolution is an effective and scalable alternative to reactive acquisition from repository history or online exploration.

  • Takeaways & Limitations

    SkillForge may be less effective for code regions that are rarely exercised by runnable tests, because they provide fewer learning signals.

Abstract

from arXiv · show

Large language model (LLM) based agents have demonstrated remarkable proficiency in automated software issue resolution, yet they often struggle to resolve issues in a specific repository because they lack project-specific knowledge. Existing self-evolving approaches acquire such knowledge from repository history or online repair trajectories, but they either depend on available historical issue-resolution signals or incur substantial per-issue test-time exploration cost. In this paper, we propose SkillForge, a self-distillation framework that proactively acquires project-specific knowledge from the repository itself. Instead of waiting for real issues to expose project-specific knowledge gaps, SkillForge synthesizes project-specific issues by re-implementing test-covered core functionalities of the repository. By resolving these synthetic issues, SkillForge distills reusable project-specific knowledge into entity-grounded skills and associates them with relevant repository entities for future issue resolution. Extensive experiments using both open-source and closed-source models show that SkillForge consistently improves issue resolution performance over strong baselines. These results demonstrate that proactively acquiring project-specific knowledge before solving real issues substantially improves downstream software issue resolution.

I. INTRODUCTION

SkillForge addresses the cold-start problem in project-specific issue resolution by proactively distilling repository knowledge into entity-grounded skills before real issues arise. It improves downstream performance over Mini-SWE-Agent and evaluated history-driven and online baselines.

  • Motivation: Project-specific regularities, API contracts, and implicit execution paths force knowledge-lacking agents to repeatedly rediscover conventions and risk recurring repository-specific pitfalls.This creates a cold-start problem in which capable agents operate as generic repository explorers.
  • Approach: SkillForge synthesizes project-specific issues from test-covered core functionalities instead of relying on historical issue-resolution trajectories or costly per-issue online exploration.It follows execution traces to identify code regions jointly implementing functionality and rewrites them into synthetic issues for resolution.
  • Approach: SkillForge stores project-specific knowledge as global diagnostic skills and local intervention skills, enabling entity-grounded retrieval and just-in-time guidance during downstream resolution.Global skills support diagnosis and navigation, while local skills provide entity-specific modification guidance and pitfall-avoidance lessons.
  • Results: 72.2% and 60.6% Pass@1 on SWE-bench Verified improve over Mini-SWE-Agent by +5.8 and +5.6 percentage points for DeepSeek-V3.2 and GPT-5-mini, respectively.SkillForge is implemented with Mini-SWE-Agent and evaluated using both DeepSeek-V3.2 and GPT-5-mini.
  • Results: +5.8 and +4.1 percentage points over Mini-SWE-Agent on SWE-bench Pro are achieved by SkillForge for DeepSeek-V3.2 and GPT-5-mini, respectively, while outperforming evaluated history-driven and online baselines.The contribution summary reports the same improvements as +5.8%/+5.6% on SWE-bench Verified and +5.8%/+4.1% on SWE-bench Pro.

II. METHODOLOGY · A. Project-Specific Issue Synthesis

SkillForge proactively acquires project-specific knowledge from a repository by synthesizing issues from test-covered functionality, resolving them, and distilling the resulting trajectories into reusable skills. Its issue-synthesis pipeline uses coverage-guided segment selection, constrained code rewriting, failure-based instance assembly, and SWE-bench-style resolution.

  • II. METHODOLOGY: SkillForge addresses cold-start project-specific issue resolution by deriving knowledge from the repository instead of waiting for historical issue-resolution trajectories.The framework synthesizes project-specific issues, resolves them with a SWE agent, and distills the resulting knowledge into reusable skills.
  • II. METHODOLOGY: The framework generates synthetic issues, resolves them through iterative patching and test feedback, then organizes distilled knowledge as reusable skills in a dual-level repository.These are three of the four stages presented in the framework overview.
  • A. Project-Specific Issue Synthesis: The framework treats test failures from rewritten segments as targets, constructing a buggy repository snapshot with buggy and reference patches and a user-facing problem statement derived from failure evidence.The problem statement excludes implementation details and repair hints.
  • A. Project-Specific Issue Synthesis: SkillForge identifies passed tests exercising core functionality and uses coverage instrumentation to trace the source files and line ranges they execute.The execution traces define candidate regions for subsequent rewriting.
  • A. Project-Specific Issue Synthesis: An LLM ranks a small top-k subset of traced segments using the test’s purpose, scenario, and candidate-segment summaries.The selection step focuses the subsequent rewriting phase on segments most likely to represent the tested functionality.
  • A. Project-Specific Issue Synthesis: For each selected segment, an LLM re-completes the functionality without seeing the original implementation, using limited context and a high-level test goal.The rewritten code is intended to preserve the API while inducing realistic implementation mistakes under constrained context.
  • A. Project-Specific Issue Synthesis: Figure 2 summarizes the process for synthesizing project-specific issues.The figure is identified as “Synthesize project-specific issues.”
  • A. Project-Specific Issue Synthesis: Each synthesized instance is presented in an isolated repository environment, where a SWE-bench-style agent receives the problem statement and buggy codebase and attempts a repair patch.SkillForge records file edits, shell commands, and test runs as the action trajectory for later skill distillation.

B. Skill Distillation

SkillForge distills synthetic issue-resolution trajectories into reusable, entity-grounded project-specific skills. It separates diagnostic knowledge for navigating and understanding repository entities from intervention knowledge for modifying them.

  • Skill repository: The extraction pipeline transforms each synthetic issue-resolution trajectory into two complementary skill sets, Mext and Mint, stored in a dual-level skill repository.The stage’s objective is to organize project-specific knowledge from synthetic issue-resolution trajectories as reusable skills.
  • Trajectory normalization: Trajectory normalization parses code-access events and aligns accessed files and line ranges with the repository’s structural index before distillation.The process identifies shell-command accesses and maps their coordinates to repository entities.
  • Global diagnostic skills: Mext records global diagnostic skills that explain an entity’s role, reusable reasoning strategies, debugging entry points, jointly involved APIs, and repository-specific behaviors.These structured records are associated with repository entities for future retrieval and guide repository navigation and diagnosis.
  • Project-specific knowledge: Unlike repository summarization, SkillForge distills actionable project-specific knowledge that emerges through attempts to solve project-specific issues rather than static code semantics.The distinction applies to both diagnostic and intervention knowledge organized around repository entities.
  • Local intervention skills: Mint records local intervention skills that guide how individual repository entities should be modified using concrete repair knowledge distilled from issue-resolution trajectories.Successful trajectories reinforce effective repair patterns, while failed trajectories help agents avoid recurring modification mistakes.

C. Skill Adaptation

SkillForge adapts acquired project-specific knowledge through a two-stage, context-aware retrieval process: global diagnostic guidance initializes reasoning, while local intervention skills are injected just in time based on repository entities the agent accesses. This entity-grounded design delivers knowledge when the agent reaches relevant code, reducing retrieval ambiguity and aligning guidance with the code under inspection.

  • Two-stage skill retrieval: SkillForge retrieves distilled skills associated with relevant repository entities and injects them into the agent’s reasoning through macro-level initialization and micro-level just-in-time intervention.The two-stage mechanism provides context-aware adaptation during downstream issue resolution.
  • Macro-level Initialization: Before issue resolution, SkillForge uses the issue description with BM25 to retrieve top-k records from the global skill set Mext and prepends them to the initial prompt.The records contain API paths, their purposes, and associated playbooks.
  • Micro-level JIT Injection: During interaction, SkillForge monitors executed shell commands, extracts accessed file paths, and dynamically injects matching local intervention skills from Mint instead of flooding the context with all local skills.This JIT strategy is intended to avoid significant noise from injecting every local intervention skill at once.
  • Entity-grounded retrieval: Unlike centralized semantic retrieval, SkillForge triggers retrieval through the agent’s interaction with corresponding repository entities, grounding knowledge directly to the code context.Diagnostic and intervention knowledge is delivered when the agent reaches relevant repository context, reducing retrieval ambiguity and maintaining tight alignment with the code under inspection.

III. EXPERIMENTAL SETUP … C. Baseline Methods

SkillForge is evaluated through four research questions covering effectiveness, component ablations, hyperparameters, and performance across diverse repositories. The evaluation uses SWE-bench Verified and SWE-bench Pro, comparing history-driven, online, and controlled SkillForge variants.

  • A. Research Questions: The study asks four research questions on SkillForge’s effectiveness, component contributions, hyperparameter effects, and efficacy across diverse repositories.These correspond to RQ1 through RQ4.
  • B. Datasets: SWE-bench Verified provides 500 human-validated tasks from popular Python repositories on GitHub.It is described as a widely adopted standard for evaluating software engineering agents.
  • B. Datasets: SWE-bench Pro adds 731 instances across Python, JavaScript, Type-Script, and Go repositories, emphasizing challenging long-horizon, multi-file tasks.The full benchmark is intended to better reflect realistic software engineering tasks.
  • C. Baseline Methods: History-driven baselines reuse project-specific knowledge distilled from prior issue-resolution cases, including SWE-Exp, EvoCoder, and MemGovern.SWE-Exp distills diagnostic patterns and repair strategies; EvoCoder uses trajectory reflection; MemGovern structures human debugging traces into memory.
  • C. Baseline Methods: Online baselines distill project-specific guidance from substantial test-time reasoning trajectories while solving the current issue, including SAGE, SWE-Debate, and Live-SWE-agent.They respectively use plan learning, competitive debate, and runtime scaffold updates.
  • C. Baseline Methods: The study compares controlled SkillForge variants while keeping the same downstream reuse mechanism to isolate the source of project-specific knowledge.The variants replace functionality-level repository probing or trajectory-distilled knowledge with alternative acquisition methods.
  • C. Baseline Methods: SkillForge w/ SWE-Smith replaces functionality-level repository probing with SWE-Smith single-function rewriting, whereas SkillForge w/ LLM Summary uses static LLM-generated repository summaries.These variants test alternative sources of project-specific knowledge.

D. Metrics · E. Implementation Details

The paper evaluates SkillForge by first-attempt effectiveness and amortized end-to-end cost, then specifies its agent implementation, temporally isolated synthesis procedure, variants, and fixed experimental settings.

  • D. Metrics: Pass@1 measures the percentage of issues resolved successfully on the first attempt without repeated repair iterations.It evaluates whether the framework generates correct patches immediately.
  • D. Metrics: Avg Cost measures average end-to-end monetary cost per evaluated issue in U.S. dollars, including offline pre-computation and all pipeline stages.Included stages are synthetic issue generation, trajectory collection, skill distillation, and online issue resolution.
  • E. Implementation Details: SkillForge is implemented with Mini-SWE-Agent, a bash-based agent scaffold, and evaluated using DeepSeek-V3.2 and GPT-5-mini with default inference settings.The implementation uses the named scaffold and two evaluated language models.
  • E. Implementation Details: For SWE-bench Verified, synthesis rolls repositories back before the golden patch and test patch, then filters semantically relevant tests before generating 577 synthesized issues.The filtered-test procedure avoids extracting every snapshot’s full test suite because of time and compute costs.
  • E. Implementation Details: Commits used for synthesis predate each target instance’s gold commit, preventing use of current-instance or future information.This temporal constraint supports the stated isolation of synthesis from target-instance information.
  • E. Implementation Details: The two variants extract relevant functions from the baseline agent trajectory and use either single-function rewriting or an LLM summary of joint function usage.Both variants retain SkillForge’s skill format and injection interface.
  • E. Implementation Details: All stages use temperature 0, a 250-step action budget, BM25 retrieval of the top five skills, and Pass@1 averaged over three runs.These settings apply across the reported experimental stages.

IV. RESULTS · A. RQ1: Effectiveness of SkillForge

SkillForge improves software issue resolution across SWE-bench Verified and Pro, outperforming Mini-SWE-Agent and other project-specific knowledge acquisition baselines. Its gains persist across backbones and reflect a proactive offline skill-distillation process, while simplified variants underperform the full method.

  • A. RQ1: Effectiveness of SkillForge: SkillForge reaches 72.2%/60.6% Pass@1 with DeepSeek-V3.2/GPT-5-mini on SWE-bench Verified, exceeding Mini-SWE-Agent by +5.8%/+5.6%.It also surpasses MemGovern by +3.0%/+2.6%.
  • A. RQ1: Effectiveness of SkillForge: SkillForge’s main cost difference is offline pre-computation, which synthesizes project-specific issues and distills skills from their resolution trajectories.Reported Avg Cost includes amortized offline pre-computation; after repository-level skill construction, online real-issue resolution cost remains close to that of comparison methods.
  • A. RQ1: Effectiveness of SkillForge: SkillForge achieves 34.1%/51.7% Pass@1 with DeepSeek-V3.2/GPT-5-mini on SWE-bench Pro, improving over Mini-SWE-Agent by +5.8%/+4.1%.Both gains are statistically significant at p-value < 0.05.
  • A. RQ1: Effectiveness of SkillForge: On SWE-bench Pro, SkillForge exceeds Live-SWE-agent by +1.7%/+2.6% and SWE-Exp by +4.7%/+3.2% under DeepSeek-V3.2/GPT-5-mini.These comparisons cover the strongest available Pro baselines reported in the passage.
  • A. RQ1: Effectiveness of SkillForge: SkillForge w/ SWE-Smith attains 68.0%/$0.088 and 56.4%/$0.071, while SkillForge w/ LLM Summary attains 68.7%/$0.069 and 54.4%/$0.065.The pairs correspond to DeepSeek-V3.2 and GPT-5-mini, respectively, and both variants underperform the full method.

B. RQ2: Ablation Study · C. RQ3: Impact of Hyperparameters

SkillForge’s ablations show that effective issue resolution requires both repository-level diagnostic and entity-level intervention knowledge, while distilled knowledge is LLM-specific. Hyperparameter studies find the best performance at moderate retrieval and synthesis scales, with excessive settings slightly degrading results.

  • B. RQ2: Ablation Study: Removing Mext causes drops of 3.8% and 3.0%, demonstrating the importance of repository-level diagnostic knowledge.Removing Mint causes larger drops of 4.4% and 3.4%, indicating that intervention knowledge is equally critical.
  • B. RQ2: Ablation Study: Effective issue resolution requires both repository-level diagnostic knowledge and entity-level intervention knowledge.The two knowledge types provide complementary support for issue resolution.
  • B. RQ2: Ablation Study: Distilled project-specific knowledge is LLM-specific rather than universally transferable across backbones.SkillForge distills skills with one LLM and injects them during issue resolution by the same or a different LLM.
  • C. RQ3: Impact of Hyperparameters: With kr = 0, the agent achieves 62.3%, while performance peaks at kr = 5 with 69.7%.The study varies retrieval from kr = 0 to 7 and also evaluates a full-retrieval condition on Django and Sphinx with GPT-5-mini.
  • C. RQ3: Impact of Hyperparameters: With ks = 0, performance is 62.3%, increasing to 63.2% at ks = 1 and peaking at ks = 5.Increasing ks to 7 slightly decreases performance because larger perturbations tend to generate overly broad rewrites.
  • C. RQ3: Impact of Hyperparameters: SkillForge consistently outperforms the baseline across all settings of kr and ks.This indicates that proactively acquired project-specific knowledge remains effective across a broad range of retrieval and synthesis configurations.
  • C. RQ3: Impact of Hyperparameters: Moderate synthesis and retrieval scales achieve the best performance, whereas excessive knowledge generation or retrieval slightly degrades results.The degradation is attributed to redundant information.

D. RQ4: Efficacy across Diverse Repositories

SkillForge generalizes across diverse repositories, improving Pass@1 over the baseline on all seven evaluated repositories under both DeepSeek-V3.2 and GPT-5-mini without regression. Its entity-grounded skills capture repository-specific APIs, implementation patterns, and architectural organization for precise downstream retrieval.

  • SkillForge improved baseline performance on all seven evaluated repositories under both DeepSeek-V3.2 and GPT-5-mini without regression.The comparison included Baseline, SWE-Exp, and SkillForge across the seven repositories with the largest numbers of evaluation instances.
  • +13.6% was the largest per-repository gain under DeepSeek-V3.2, achieved on Sphinx, while +15.6% was the largest under GPT-5-mini, achieved on Scikit-learn.
  • SWE-Exp behaved inconsistently and regressed on three repositories, including −11.8% on Matplotlib and −4.5% on Astropy under DeepSeek-V3.2.Under GPT-5-mini, SWE-Exp also regressed by −9.1% on Astropy and −2.0% on Pydata.
  • SkillForge distills repository-specific APIs, implementation patterns, and architectural organization into entity-grounded skills that enable precise retrieval during downstream issue resolution.

E. Case Study · V. DISCUSSION · A. Quality of Distilled Skills and LLM-Generated Problem Statements

SkillForge’s project-specific skills reshape repair trajectories by replacing brittle local heuristics with repository-faithful reasoning and test-driven investigation. Manual inspection further indicates that distilled skills are actionable and synthesized issues expose model-specific weaknesses beyond the categories of real evaluation issues.

  • E. Case Study: Django issue #11206 involves formatting extremely small Decimal values with fixed decimal places, where scientific notation should instead become a fixed-point representation.The buggy call returns "1.00e-200" for Decimal("1e-200") with decimal_pos=2, while users expect a representation such as "0.00".
  • E. Case Study: The baseline GPT-5-mini agent identifies the relevant condition and applies an exponent-based zero heuristic that fixes the observed example but fails all FAIL TO ...The passage contrasts this trajectory with one using retrieved project-specific knowledge.
  • E. Case Study: Skill-enhanced repair preserves the existing Decimal formatting pipeline and applies repository precision semantics instead of introducing alternative representations or simple exponent heuristics.The retrieved skills provide two repository-specific insights: preserve the formatting pipeline and reason about numerical equivalence using repository precision semantics.
  • E. Case Study: Skill injection promotes value-based thresholding, Decimal-level operations, preservation of the original formatting path, and tests targeting subtle edge cases.The baseline instead converges quickly on a plausible but brittle local heuristic that fails systematic evaluation.
  • A. Quality of Distilled Skills and LLM-Generated Problem Statements: Manual inspection found distilled skills grounded in repository entities, resolution trajectories, and golden patches, providing actionable guidance without hallucinated APIs or misleading advice.The same inspection checked whether synthesized problem statements faithfully describe observable failures caused by functionality-level rewriting.
  • A. Quality of Distilled Skills and LLM-Generated Problem Statements: Synthesized and real SWE-bench issue-type distributions do not overlap, indicating that synthesized issues do not merely reproduce evaluation issue categories.The comparison used real SWE-bench issues from the same repositories.
  • V. DISCUSSION: Synthesized issues expose finer-grained model biases in using project-specific functions, APIs, and cross-function interactions.The passage frames SkillForge’s benefit as less about teaching the age...

B. Threats to Validity … VII. CONCLUSION

SkillForge addresses project-specific issue resolution through proactive synthetic-issue-based knowledge acquisition, while acknowledging limitations from test coverage and LLM-generated problem statements. Related work frames its distinction from history- or trajectory-based self-evolution and training-oriented issue synthesis, and experiments report consistent performance gains.

  • B. Threats to Validity: Limited runnable-test coverage may reduce SkillForge’s effectiveness because rarely exercised code regions provide fewer learning signals.This threatens external validity, especially in repositories with limited test coverage.
  • B. Threats to Validity: LLM-generated problem statements were manually checked against executable rewriting-induced bugs and observable behaviors, but may still differ from developer-written descriptions.The passage notes that this follows standard SWE-bench-style synthetic-instance synthesis pipelines.
  • A. Project-Specific Knowledge for SWE Agents: Recent SWE agents increasingly self-evolve by distilling reusable guidance from repository histories, prior trajectories, and execution feedback.The related work groups these methods by the source of their evolution signal.
  • A. Project-Specific Knowledge for SWE Agents: History-based methods accumulate interaction trajectories, structured memories, or community debugging traces to guide future issue resolution.Examples include EvoCoder, SWE-Exp, ExpeRepair, and MemGovern.
  • B. SWE-Style Instance Synthesis: Existing software-engineering issue-synthesis methods primarily treat synthesized instances as scalable training data, including rewritten API functions and executable environments.The passage names SWE-Smith, R2E-Gym, and BugPilot as examples of this direction.
  • B. SWE-Style Instance Synthesis: SkillForge instead uses synthetic issues for proactive project-specific knowledge acquisition, targeting knowledge not readily inferable from static repositories and using joint function rewriting.This differs from maximizing the number or diversity of tasks and from rewriting isolated API-level functions.
  • VII. CONCLUSION: SkillForge proactively derives repository knowledge by synthesizing project-specific issues and distilling resolution trajectories into a dual-level, entity-grounded skill repository.The framework is presented as addressing the cold-start problem without relying on repository history or costly per-issue test-time exploration.
  • VII. CONCLUSION: +5.8%/+5.6% absolute Pass@1 gains on SWE-bench Verified and +5.8%/+4.1% on SWE-bench Pro were reported for DeepSeek-V3.2 and GPT-5-mini, respectively.The results support proactive knowledge acquisition as an effective and scalable alternative to reactive acquisition from repository history or online exploration.
Loading 2608.18933v1…