PaperlayerView exact revision on arXiv

Source-linked AI summary

Skill Dreaming: Evolving Repository Skills through Pull Request Streams

Xinyuan Song, Hasibul Haque, Jay Somasundaram, Ryan Turner, Bowen Zhu, Liang Zhao

arXiv:2610.36188v1cs.AIcs.SE

TL;DR

Repository-specific skills can become outdated as repositories and conventions change, while PR histories record implementation and review knowledge. Skill Dreaming turns validated PR records into evolving skills that guide repository development and maintenance. It reports improved resolved rates on SWE-bench Pro and SWE-bench Verified.

  • Problem

    Repository-specific skills have limited support for staying aligned with changing repository structures, APIs, and conventions, while PR knowledge may remain scattered across history.

    Source: §1 · p. 1

  • Method

    Skill Dreaming extracts reusable knowledge from validated PRs into a repository-specific skill library, uses the skills to guide development, and updates them from subsequent PR information.

    Source: §1 · p. 2

  • Results

    Skill Dreaming improves the resolved rate from 79% to 88% on SWE-bench Pro and from 83% to 92% on SWE-bench Verified.

    Source: §7 · p. 9

  • Takeaways & Limitations

    The evolving skills capture repository-specific implementation and maintenance knowledge to guide coding agents as repositories change.

    Source: §1 · p. 2, §7 · p. 9

    The staged process approximates the constrained objective without guaranteeing global optimality, and its probabilities express desired reliability rather than guarantees from available tests.

    Source: §3 · p. 4

Abstract

from arXiv · show

Large language model (LLM) agents increasingly rely on reusable skills for complex software engineering tasks, making skill libraries essential for repository development and maintenance. However, repository-specific skills must evolve with changing code structures, implementation conventions, and maintenance requirements. Pull requests (PRs) provide a natural source of such knowledge by recording how repositories evolve through implementation and review. Meanwhile, new PRs must follow repository-specific implementation patterns, modify related components consistently, and ensure that existing tests continue to pass while adding coverage for new functionality. This creates a mutually reinforcing cycle: PRs provide knowledge for updating skills, while these skills improve future PRs, maintain repositories, and support further skill evolution. To enable this co-evolution of PRs, repositories, and skills, we propose Skill Dreaming, a framework that learns from PR streams to guide continuous repository development and maintenance. Given a repository and its PR history, Skill Dreaming builds a repository-specific skill library by extracting executable knowledge from validated PRs and continuously refines the skills as new PR information becomes available. These skills guide coding agents not only in implementing requested features but also in maintaining associated tests, documentation, compatibility, and repository conventions. By integrating PR analysis, skill management, code execution, and continuous integration, Skill Dreaming provides an end-to-end workflow for applying learned skills to repository changes and incorporating subsequent PR outcomes into future skill updates.

1 Introduction

Repository-specific skills can become outdated as code and conventions change, while PR histories contain implementation and review knowledge that could keep them aligned. Skill Dreaming turns validated PRs into reusable skills and uses subsequent PRs to evolve those skills as they guide repository work.

1 Introduction

  • Existing skill-learning approaches primarily learn from past experience, with limited support for aligning repository-specific skills as structures, APIs, and conventions change.Outdated skills may become incomplete or inconsistent with current development requirements.Source: §1 · p. 1
  • PR streams record implementation patterns, review, and validation, but retrieval-augmented generation retrieves historical information for individual tasks without preserving repository-specific knowledge as it evolves.Successive PRs can document conventions changing over time, while new requirements may remain scattered across history.Source: §1 · pp. 1–2
  • Skill Dreaming extracts reusable knowledge from validated PRs into a dynamic library that guides development and maintenance, while resulting PRs provide information for further skill updates.Its three-stage process constructs an initial library, retrieves skills for new tasks, and evolves skills through reuse, extension, revision, or replacement.Source: §1 · p. 2
  • Skill Dreaming improves the resolved rate from 79% to 88% on SWE-bench Pro and reports consistent cross-repository gains and benefits from skill reuse, evolution, and maintenance.The experiments cover repository-level software engineering tasks across three aspects.Source: §1 · p. 2
  • The paper presents contributions spanning a continuous PR-to-skill learning process, an end-to-end framework that does not update LLM parameters, and experiments on issue resolution, cross-repository performance, and maintenance.These contributions are summarized by the authors as the paper’s main contributions.Source: §1 · p. 2

2 Related Work

Prior work studies repository knowledge extraction, experience-based skill learning, hierarchical reuse, and repository editing, often as separate capabilities. Skill Dreaming connects PR records, evolving repository-specific skills, and subsequent repository changes in one continuous process.

Learning repository conventions from history

  • Learning to Commit [11] extracts repository knowledge from historical commits and supports skill revision and deprecation, while Skill Dreaming uses PR records that include implementation, review feedback, and validation outcomes.Skill Dreaming aligns new information with existing skills and preserves version-specific implementation and maintenance requirements.Source: §2 · pp. 2–3

Experience distillation and skill management

  • Trace2Skill [16], ReasoningBank [19], and CODESKILL [14] draw on agent experiences or skill management, whereas Skill Dreaming uses repository development records without updating LLM parameters.Its construction and alignment processes account for repository conventions, implementation dependencies, and maintenance obligations.Source: §2 · p. 3

Skill-guided task generation and hierarchical reuse

  • Socratic-SWE [27] uses skills for task synthesis and SkillLens [29] supports hierarchical reuse, while Skill Dreaming decides whether PR information adds, refines, specializes, or replaces skills.These decisions use repository changes and existing skill relationships rather than indiscriminate accumulation.Source: §2 · p. 3

Repository editing and PR analysis

  • Repository editing and evaluation work provides foundations for multi-file changes, repair, PR analysis, and benchmarks, while Skill Dreaming turns validated PR records into skills for implementation and maintenance.Subsequent PR outcomes provide new evidence for skill evolution.Source: §2 · p. 3

Comparison with existing methods

  • Skill Dreaming integrates PR record construction, evidence-grounded skill distillation, skill relationship management, and skill-guided implementation into a continuous process.Table 1 compares approaches across PR-driven learning, skill evolution, repository maintenance, and training-free adaptation.Source: §2 · p. 3, Table 1 · p. 3

3 Problem Formulation

The formulation models repository development as successive requests in which a repository-specific skill set guides PR generation and evolves using PR records. It balances task success and preservation of existing functionality against process costs, subject to correctness, safety, and resource constraints.

3 Problem Formulation

  • At round t, the system has repository R_t and skills S_t learned from previous PRs; the skills encode implementation patterns, coding conventions, and maintenance requirements.A new request τ_t prompts generation of a PR P_t, while new PR information may lead to skill reuse, refinement, extension, or replacement.Source: §3 · p. 3
  • The PR-record process detects relevant historical PRs, repairs incomplete or unsuccessful implementations, and synthesizes candidate changes before verifying them against repository requirements.Verification identifies successful changes and informative failures for skill updates.Source: §3 · pp. 3–4
  • The skill update extracts reusable engineering knowledge from PR records, then reuses, extends, revises, or replaces skills to guide subsequent development.The updated skill set guides a fixed-weight agent to generate a new PR.Source: §3 · p. 4
  • In historical evaluation, each request uses its recorded base repository and only the skill set evolves across requests, without chaining generated repository changes.This distinguishes skill evolution from a continuously updated repository state in that evaluation setting.Source: §3 · p. 4
  • The utility of a generated PR measures request completion, preservation of existing functionality, and repository-specific requirements against token, time, verification, and skill-maintenance costs.A nonnegative weight vector sets penalties for those costs, balancing implementation success and efficiency.Source: §3 · p. 4
  • The objective is constrained by minimum task-completion and functionality-preservation requirements and a resource budget, but its probabilities express desired reliability rather than guarantees from available tests.The staged process approximates the constrained objective without updating model weights or guaranteeing global optimality.Source: §3 · p. 4

4 Method: From PR Records to Aligned Skills

Skill Dreaming builds repository-specific skills from historical PR records, applies them to new requests, and uses each round's outcomes to update both the repository and skill library.

4 Method: From PR Records to Aligned Skills

  • Each round initializes PR and skill stores, then diagnoses a request, constructs and verifies PR records, evolves skills, implements and verifies a PR, and records feedback.The stores draw on the repository, available historical PRs, and optional seed skills.Source: §4 · p. 4

4.1 PR Record Construction

  • PR record construction localizes relevant code and maintenance needs, retrieves or reconstructs historical PRs, and classifies them by relevance, freshness, compatibility, and validation evidence.Candidates reuse, repair, or synthesize changes and are retained as successful records only after verification; failed candidates contribute failure records.Source: §4.1 · p. 4

4.2 Distill PR Information into Skills

  • A fixed-weight LLM writer distills PR code changes and review or CI feedback into skills specifying applicability, implementation steps, and validation.Failed implementations and review corrections contribute negative constraints and repair strategies; the skills span five families covering PR handling, implementation, maintenance, and evolution.Source: §4.2 · pp. 4–5

4.3 Align New and Existing Skills

  • Skill alignment relates candidate skills to existing ones and chooses whether to reuse, modify, or replace knowledge while preserving valid patterns and maintenance requirements.Skills retain links to supporting PR records, repository versions, dependencies, and related skills so obsolete procedures can be superseded without discarding still-valid practices.Source: §4.3 · p. 5

4.4 Skill-Guided Implementation and Pool Maintenance

  • Retrieved skills guide feature implementation and repository maintenance; generated PRs are verified, repaired within bounds if needed, and recorded with outcomes for subsequent skill evolution.The active pool adds and revises skills while archiving superseded versions with their supporting PR records.Source: §4.4 · p. 5

5 Experimental Design

The experiments use SWE-bench tasks and compare retrieval and skill-learning baselines. They report task-resolution, maintenance-coverage, localization, and efficiency measures, with stated scope limits for some metrics.

5.1 Benchmarks and Baselines

  • SWE-bench Pro tasks cover Ansible, qutebrowser, and Open Library, while evaluated Verified instances span 12 repositories and use historical base commits without future PR information.Comparisons include BM25, dense and hybrid retrieval, and recent skill-learning methods.Source: §5.1 · pp. 5–6
  • Experiments use Claude Opus-5 and Sonnet-4.6, mini-swe-agent, official SWE-bench procedures, and dense-cosine retrieval with top eight results and a 0.20 threshold.Embeddings use text-embedding-3-small; execution uses Docker.Source: §5.1 · p. 6

5.2 Metrics and Statistical Interpretation

  • The evaluation reports official resolved rate, task success, safe rate, and Maintenance Surface F1, which measures coverage of required repository changes rather than correctness of individual changes.Maintenance categories include code, tests, documentation, configuration, examples, and changelogs; logging and compatibility are not evaluated separately.Source: §5.2 · p. 6
  • Localization measures include Top-k Hit Rate, Recall@k, and MRR on tasks with nonempty gold locations; efficiency is measured by per-task token consumption and wall-clock time.Offline skill-construction costs are reported separately when excluded.Source: §5.2 · p. 6

6 Results

Skill Dreaming improves official benchmark resolution, generalizes across repositories, and supports broader maintenance outcomes. Ablations and continuous-evolution results examine how skill management contributes, while the efficiency comparison exposes added resource costs.

6.1 End-to-End Evaluation

  • 88% on SWE-bench Pro and 92% on SWE-bench Verified are Skill Dreaming’s official resolved rates, each 9% above the base agent.On SWE-bench Verified, it also achieves a 92% resolved rate, Safe Rate of 1.00, and Maintenance Surface F1 of 0.74, outperforming all baselines, including append-only skill accumulation.Source: §6.1 · p. 6, Table 3 · p. 6, Table 4 · p. 7

6.2 Cross-Repository Generalization

  • Task Success gains range from 0.07 to 0.20 across six repositories in four domains, alongside improved Maintenance Surface F1.The evaluation constructs repository-specific skills from historical PRs and applies evolving skills to evaluation PRs.Source: §6.2 · p. 6, Table 5 · p. 7

6.3 PR Record Analysis and Implementation Evaluation

  • Localization performs best at module level, then file level, while function-level localization is harder; all five PR-quality categories exceed 0.86 F1, with irrelevant PRs at 0.945.The localization comparison covers module, file, and function levels; PR classification uses 1,000 SWE-bench Pro Ansible PRs.Source: §6.3 · pp. 6–7, Table 6 · p. 7, Table 7 · p. 7
  • The reported method achieves a Maintenance Surface F1 of 0.871, outperforming Single Sample and both skill-based baselines, while matching Best-of-K on shared candidate outputs.Source: §6.3 · p. 7

6.4 Ablation Studies

  • The full method leads relation-management ablations, with TS of 0.93 and 0.94 and Surface F1 of 0.75 and 0.77; disabling filtering reduces Surface F1 by 0.111.Among updating strategies, the full method reaches TS 0.93 and outperforms variants, while existing PR retrieval reaches 1.00.Source: §6.4 · p. 8, Table 10 · p. 8, Table 11 · p. 8

6.5 Continuous Skill Evolution

  • With 500 historical PRs and 50 additional PRs per update through 1,000 PRs, task performance and maintenance quality show an overall upward trend despite fluctuations.The experiment uses Claude Opus 5 under the SWE-bench Pro headline setting.Source: §6.5 · p. 8, Figure 2 · p. 8

6.6 Efficiency Analysis

  • TS is 0.93 and Surface F1 is 0.80 versus SkillLens at 0.61 and 0.71, respectively, with 26,894 tokens and 36.00 seconds versus 14,300 tokens and 32.90 seconds.Skill Dreaming improves task success and maintenance quality while using more tokens and requiring a modestly longer execution time.Source: Table 12 · p. 8, §6.6 · p. 9

7 Conclusion

Skill Dreaming continuously evolves repository-specific skills from pull request streams to guide coding agents in extending and maintaining repositories. Experiments report improved benchmark resolution, cross-repository effectiveness, and better task success and maintenance quality with modest additional computational costs.

7 Conclusion

  • Skill Dreaming continuously evolves repository-specific skills from pull request streams, using verified PR records, distilled skills, and maintained skill relationships to guide coding agents.The conclusion reports official resolved-rate improvements from 79% to 88% on SWE-bench Pro and from 83% to 92% on SWE-bench Verified.Source: §7 · p. 9

Related papers

Loading 2610.36188v1…