Source-linked AI summary
Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills
Jianlyu Chen, Yuyang Hu, Hongjin Qian, Jiawei Liu, Wenqing Wei, Xiaolong Chen, Defu Lian, Zhicheng Dou, Chaozhuo Li, Qiwei Ye, Zheng Liu
TL;DR
ML research agents lack operational knowledge: practical domain know-how that models and harnesses do not provide, despite its presence in repositories and papers. DisCo distills that knowledge into verified skills through task-agnostic and task-oriented processes, producing the AREX-Skill Library. With the backbone, harness, and downstream budget fixed, the skill-equipped agent scores higher across MLE-bench, PaperBench, FrontierCS, and PassNet.
Problem
ML research agents’ models and harnesses provide reasoning and procedure but leave domain-specific operational knowledge—how to select, configure, and correctly use methods and tools—unspecified.
Method
DisCo distills repository, paper, and task knowledge into compact verified skill graphs that agents can load as operating context without changing the model backbone or harness.
Results
134.3% higher MLE-bench, 34.4% higher PaperBench, 9.2% higher FrontierCS, and 14.0% higher PassNet scores were achieved with skills under matched GPT-5.5 settings and budgets.
Takeaways & Limitations
Distilled skills add reusable operating context, allowing research agents to improve by augmenting what they know rather than changing how the harness controls research.
Takeaways & Limitations
The repository collection is a curated snapshot of commonly used ML software, selected using open-source visibility and practical-use signals rather than an exhaustive ecosystem partition.
Abstract
from arXiv · showhide
Autonomous agents are beginning to carry out machine-learning (ML) research end to end. These agents combine a model backbone with a harness for planning, execution, memory, and verification, but this architecture still leaves domain-specific know-how outside the agent. We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work. That knowledge is not absent from the field. It appears in repositories and papers, but in forms written for human readers and too large to load during a task. Once distilled into compact, verified skills, this knowledge can be reused across tasks rather than rediscovered during each run. We present DisCo, a skill-powered research agent that creates skills and uses them during research. Its distillation runs in two complementary forms: task-agnostic, condensing the field's widely used repositories into reusable skills, and task-oriented, producing the skills a concrete task calls for. The former, applied across the open ecosystem, yields the AREX-Skill Library, with 5,000+ verified skills distilled from 1,000 widely used ML repositories and organized into 20 areas and 178 capability families. With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped research agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet than the same agent without skills. These gains come from adding distilled operating context under that fixed setup.
1 Introduction
ML research agents can execute substantial parts of the research pipeline, but their model-and-harness architecture lacks operational knowledge: the practical know-how needed to make methods work. DisCo distills this knowledge into verified skills and improves a fixed agent across four benchmarks.
- Motivation: Operational knowledge is the missing layer between an agent’s model and harness, covering method selection, tool use, configuration, and implementation pitfalls.The model supplies broad but fixed prior knowledge, while the harness supplies procedure without domain content.
- Motivation: Repositories and papers contain relevant know-how, but their human-oriented scale and organization make them difficult to load during tasks.The methodological challenge is to produce compact, operational, verified knowledge automatically and at scale.
- Approach: DisCo creates and uses skills through task-agnostic distillation of reusable repository knowledge and task-oriented distillation for concrete research tasks.Candidates are checked, repaired where possible, and recorded with unresolved gaps before entering the skill layer.
- Scale: 5,000+ verified skills were distilled from 1,000 widely used ML repositories into 20 areas and 178 capability families in the AREX-Skill Library.A library-level router narrows requests to relevant repository skill graphs.
- Evaluation: 134.3% higher MLE-bench, 34.4% higher PaperBench, 9.2% higher FrontierCS, and 14.0% higher PassNet scores were obtained with skills under matched settings.The GPT-5.5 backbone, research harness, and downstream execution budget were held fixed, with skills as the runtime variable.
2 Operational Knowledge for Autonomous Research
Autonomous research agents are commonly modeled as a backbone and harness, but this leaves domain-specific operational knowledge unspecified. Operational knowledge converts declarative facts about methods and tools into executable, task-level choices and procedures.
- The conventional agentic system combines an LLM backbone for reasoning and execution with a harness for orchestration, memory, verification, and refinement.
- The two-component view leaves domain-specific operational knowledge unspecified for research tasks.The backbone’s prior is broad but fixed, while the harness controls procedure without supplying domain content.
- Research agents must choose suitable methods and packages, configure data correctly, select appropriate settings, and avoid pitfalls that can invalidate runs.Trial-and-error inference consumes the same bounded environment budget used for evaluation.
- Operational knowledge is represented as explicit operating context K, extending the agent from (Mθ, H) to include the knowledge available during action selection.With K, the policy conditions actions on the task, history, harness, and operational knowledge.
- Declarative knowledge states facts about methods, APIs, or design choices, whereas operational knowledge translates those facts into task-level actions.The paper states that operational knowledge must be derived from declarative sources.
- Current derivation of operational knowledge is manual, requiring experts to read sources, build tools, and write usage instructions for particular domains, stacks, and releases.Its cost scales with expert labor and the setting for which the knowledge is written.
3 DisCo: Producing and Using Operational Knowledge
DisCo treats operational knowledge as verified skills that a research agent can both create and selectively use. It distills source- or task-grounded material through scoping, grounding, construction, and verification, then retrieves relevant skill branches during research.
- Skills and Skill Graphs: DisCo instantiates operational knowledge as self-contained skills that become operating context without changing the model backbone or harness.Skills are designed to work across compatible harnesses and accumulate across tasks.
- Skills and Skill Graphs: AREX-Skill separates each skill into SKILL.md as the knowledge interface, references as the knowledge substrate, and scripts as the execution interface.SKILL.md provides the entry point, operating procedure, pointers, examples, and known failure modes.
- Skills and Skill Graphs: Each source’s skills form a graph with an entry skill, component skills, and links encoding routing, dependency, or composition relations.Progressive disclosure lets the agent follow only the links required by the current problem.
- Skill Distillation: Skill distillation transforms declarative source knowledge into operational knowledge through capability scoping, evidence gathering, skill construction, and verification.The construction record retains supporting evidence, performed checks, and unresolved gaps.
- Task Distillation: Task-agnostic distillation starts from a repository, paper, or tutorial and produces reusable skills ahead of time for later tasks.It includes source understanding, capability identification, knowledge extraction, tool encapsulation, skill packaging, and verification.
- Task Distillation: Task-oriented distillation starts from a concrete problem, identifies capability gaps, discovers supporting sources, and generates skills on demand.The resulting skills remain reusable for the class of problems they address.
- Verification: Verification distinguishes distillation from summarization by requiring checks before admission and recording any surviving gaps rather than hiding them.
- DisCo: Creator mode deposits accepted graphs into the library, while researcher mode retrieves task-relevant branches as operating context.Creator costs are paid once per source, whereas researcher mode opens only what the task needs.
4 The AREX-Skill Library
The AREX-Skill Library stores verified skill graphs from a curated repository snapshot and routes researchers to relevant capabilities. Its overlapping taxonomy and progressive disclosure support selective loading rather than full-library access.
- Repository Collection: The repository collection covers 1,000 ML repositories selected using open-source visibility and practical-use signals, including GitHub stars.It is a curated snapshot rather than an exhaustive partition of the ecosystem.
- Verification: Repository graphs are verified using assertion-backed cases and repository-native examples, tests, CLI checks, tiny fixtures, or smoke scripts when available.Graph-attributed failures trigger local repair and reruns of relevant checks.
- Repository Collection: 5,353 skills are organized across 1,000 repository graphs, 20 areas, and 178 capability families.The counts exclude paper-derived and task-oriented graphs.
- Repository Collection: Repository membership is overlapping: 700 repositories appear in more than one family, producing 2,209 exact area-to-family assignments.
- Taxonomy: The taxonomy is induced before final assignment from frozen repository summaries and evaluated through locator and judge calls across stable repository batches.Stars, URLs, and pre-existing category fields are excluded from the model input.
- Assignment: Each verified graph is assigned only to exact area-to-family paths supported by repository evidence, rationale, and confidence.Keyword-only, dependency-only, optional-integration, and example-only matches are rejected.
- Routing: The router narrows requests from area to family to repository graph before exposing relevant skills, references, or scripts.When no family is a close fit, the system does not force a route.
- Other Collections: Paper-derived collections contain 636 skills from 153 source papers, with module-level checks and bounded recovery experiments excluding the original implementation repository.
5 Experiments
Across four benchmarks, distilled skills improved the same research agent under fixed backbone, harness, and execution conditions. Gains were broad, strongest on lower-baseline or harder tasks, while a small number of regressions appeared.
- MLE-bench: 134.3% higher MLE-bench performance came from skills, raising overall Any-Medal from 31.11% to 72.89% without changing the backbone.Gains were 43.94, 37.72, and 48.89 points on Low, Medium, and High tasks, respectively.
- MLE-bench: 8.45 percentage points above the strongest public MLE-bench baseline shows gains without a custom harness, orchestration strategy, or modified control loop.The comparison rose from 64.44% to 72.89%.
- PaperBench: 34.4% higher PaperBench replication resulted from skills, increasing the average score from 29.45% to 39.59% across 20 papers.Skills improved 18 of 20 tasks and degraded only two.
- Cross-benchmark patterns: Skills produced larger relative gains on low-baseline or high-difficulty tasks, but two PaperBench tasks regressed under skills.The regressions were samplespecific-masks (−5.07) and stay-on-topic (−4.52), consistent with a possible retrieval-precision trade-off.
- FrontierCS: 9.22% higher FrontierCS Score raised performance from 70.63 to 77.14 across 188 tasks, with a paired-bootstrap 95% interval of [3.41, 9.83] points.Skills improved 74 tasks and left 66 effectively unchanged.
- PassNet: 14.0% higher PassNet AS Score increased from 1.343 to 1.5313, while failed samples fell from 14 to 5 and correctness rose from 81.35% to 90.76%.The geometric-mean speedup also increased from 1.5891 to 1.6688.
6 Related Work
Prior work advances autonomous ML research agents, reusable agent knowledge, and agent-skill representations, while DisCo addresses the cross-task reuse and verification gap left by these approaches.
- Autonomous-discovery systems chain ideation, implementation, experimentation, and writing into end-to-end research pipelines.
- ML engineering agents commonly solve tasks through propose–run–evaluate loops, treating each task largely in isolation.
- DisCo’s Position: DisCo complements new control loops by distilling reusable operational knowledge that compatible agents can consume across tasks.
- Agent Skills: Agent skills provide portable, explicitly modifiable procedural knowledge centered on SKILL.md files, without modifying model parameters.
- Reusable Knowledge: Experience-based methods derive reusable code, workflows, or insights from interaction traces, but skill quality can be difficult to verify.
7 Conclusion
DisCo fills a missing operational-knowledge layer for ML research agents while leaving their model backbone and harness unchanged. Under matched GPT-5.5 Codex settings and budgets, skills improve performance across four benchmarks.
- DisCo distills source knowledge into reusable operational-knowledge skill graphs loaded as operating context without modifying the model backbone or research harness.
- The AREX-Skill Library contains 5,000+ skills distilled from 1,000 widely used ML repositories.
- 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet are achieved by the skill-equipped agent.
- These results support improving autonomous research agents by adding operational knowledge rather than relying only on stronger control loops.
A DisCo Implementations
DisCo is instantiated differently for repository, paper, and benchmark settings, adapting its source anchor and skill construction process to each evaluation context.
- The public repository collection distills versioned software sources into reusable task-agnostic skill graphs.
- The PaperBench implementation distills prior papers into source-anchored module skills and assembles a target-conditioned pool for each reproduction.
- MLE-bench, FrontierCS, and PassNet anchor task-oriented construction on a competition or benchmark interface.
A.1 Task-Agnostic ML Repository Skill Construction
Task-agnostic repository construction scopes and inspects versioned evidence, builds self-contained skill graphs, verifies them, classifies repositories, and routes only relevant knowledge downstream.
- Construction: Each selected ML repository is distilled into a self-contained operational-knowledge skill graph using a stored repo-to-skill procedure.
- Scoping and Grounding: DisCo fixes the evidence boundary before installation or execution by mapping source roots, documentation, examples, tests, scripts, and configuration files.
- Scoping and Grounding: Live inspection checks import names, package versions, public signatures, CLI entry points, optional backends, and small runtime behaviors.
- Graph Construction: A repository-level entry skill routes agents to bounded component workflows such as data preparation, training, evaluation, serving, troubleshooting, or maintenance.
- Graph Construction: Each graph bundles needed examples, helpers, and validators with provenance including source commits, package versions, dirty state, and evidence paths.
- Verification: Verification uses assertion-backed usability cases and safe native checks, while static gates test metadata, links, self-containment, provenance, routing, and path leakage.
- Classification and Routing: The taxonomy pipeline proposes a two-level tree, evaluates 1,000 repositories in 100 stable batches, and uses locator and independent-judge reviews.
- Classification and Routing: After approval, the library router exposes only repository operational knowledge selected for the current ML research task, not the full library.
A.2 Paper-Derived and Task-Oriented Skill Construction
DisCo applies paper-derived and task-oriented distillation to construct skills for the four autonomous-research evaluations, with construction completed before downstream evaluation.
- PaperBench uses source-anchored paper skills, while MLE-bench, FrontierCS, and PassNet use task-oriented distillation.
A.2.1 MLE-bench
MLE-bench constructs a task-level skill graph for each competition through planned scoping, evidence gathering, construction, and execution-feedback verification before benchmark running.
- MLE-bench performs task-oriented distillation once per competition and separates skill construction from downstream benchmark execution.
- Research planning decomposes each competition into capabilities and identifies operational knowledge needed for productive trials.
- Execution logs and observed results drive iterative skill refinement through Codex’s research-plan-refine loop.The loop revises the plan, gathers additional evidence when necessary, and evaluates the next skill version through another trial.
- Exploration prioritizes rapid diagnostic iterations and learning progress rather than requiring medal-level benchmark performance.
- Final skills provide descriptive guidance without runnable training or inference scripts, which Codex uses to generate the implementation during each benchmark run.
- Each task receives at most 24 GPU-hours, with the no-skill condition using the same backbone and running budget.The comparison isolates downstream access to finalized operational knowledge; exploration is a separate one-time construction cost.
- PaperBench instead distills selected related-work papers into reusable module skills assembled into target-conditioned pools for reproduction.
A.2.3 FrontierCS
FrontierCS builds a benchmark-wide recovery-oriented skill graph and refines it through paired trials against fixed no-skill and preceding-version baselines.
- FrontierCS uses one shared skill graph across Agent Track settings rather than separate graphs for individual tasks.
- The graph is grounded in expert guidance on algorithms, applicability conditions, heuristic search, and resource allocation.
- An entry router identifies the earliest implicated layer and progressively discloses one of eight recovery modules after a focused attempt.
- Matched trials compare each graph version with both a fixed no-skill baseline and the preceding graph version under identical evaluation conditions.These comparisons measure accumulated graph effects and the latest revision’s effect separately.
- Refinement retains recurrent, causally supported mechanisms such as failure-layer routing, evaluator evidence gates, champion preservation, and plateau escape.These revisions transformed the initial monolithic workflow into the final routed graph.
A.2.4 PassNet
PassNet distills task-specific optimization procedures, scales refinement through screened training instances, and verifies graph revisions with paired metrics and failure analysis.
- PassNet treats each benchmark instance as a task anchor and pools evidence across instances when revising a shared skill graph.
- Capability scoping covers FX-graph inspection, fusion selection, Triton replacement, semantic validation, and evaluator-failure interpretation.
- The training split contains more than 4k instances, so screening and dispatch reduce the cost of finding informative optimization or failure patterns.
- The PassNet construction-stage table reports results from 50 sampled training tasks using the Claude Code harness and DeepSeek v4 pro backbone.
- The geometric mean increases from 0.5352 for the no-skill baseline to 0.6941 for the second-version graph.
- Match failures decrease from 10/50 to 1/50, while average solve time decreases from 40 minutes to 22 minutes.
- Failure analysis identifies an overly absolute warning against reimplementing vendor-optimized operators as the cause of an unreproduced optimization miss.
- The repository-skill catalog contains 1,000 repository skill roots, 2,209 exact repository assignments, 20 areas, and 178 families.Repositories may appear multiple times when they provide distinct capabilities assigned to different families.