PaperlayerView exact revision on arXiv

Source-linked AI summary

Retrieval-Augmented Skill Optimization via Cross-Harness Adaptation

Jaewon Chu, Ji Soo Lee, Jihwan Park, Dohwan Ko, Jeehye Na, Seunghun Lee, Taehoon Lee, Minseo Yoon, Minseok Joo, Yunyang Xiong, Hyunwoo J. Kim

arXiv:2609.38024v1cs.AI

TL;DR

Existing skill optimization largely overlooks the procedural knowledge accumulated in public skill collections and instead relies on costly agent rollouts. RASO retrieves external knowledge, adapts it to the target task and harness, and uses it for both rollout-free initialization and feedback-guided updates. Across four benchmarks and two model backbones, it consistently outperforms retrieval-free skill-optimization baselines.

  • Problem

    Despite millions of publicly shared skills containing procedural knowledge across tasks and harnesses, skill optimization has largely relied on costly agent rollouts rather than external skill-corpus knowledge.

    Source: §1 · p. 1

  • Method

    RASO retrieves external skill knowledge and adapts it to the target domain and harness, using it for rollout-free initialization and execution-feedback-guided skill updates.

    Source: Retrieval-augmented skill… · p. 2, §3 · p. 4, §4.3 · p. 5

  • Results

    Across four agent benchmarks and two model backbones, RASO consistently outperforms retrieval-free skill-optimization baselines; with GPT-5.6-Luna, it improves over the strongest competing method by +6.31 on SpreadsheetBench (63.33 vs. 57.02).

    Source: §5.1 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The reported gains are associated with adapting diverse external knowledge rather than relying on particular source documents.

    Source: Retrieval-augmented skill… · p. 2

Abstract

from arXiv · show

An agent skill is a reusable, actionable natural-language artifact that guides an agent to perform a task effectively under a given harness. Recent studies have explored the optimization of agent skills, contributing to a growing collection of publicly available skills spanning diverse tasks, domains, and harnesses. Despite millions of publicly shared skills, existing skill optimization methods largely overlook this accumulated knowledge, instead relying solely on expensive agent rollouts to iteratively refine skills for a target task. To address this, we propose \textbf{Retrieval-Augmented Skill Optimization (RASO)}, a framework that leverages an external skill corpus as prior knowledge throughout skill optimization. RASO retrieves relevant knowledge from existing skills and adapts it to the target task and harness via Cross-Harness Adaptation, accounting for mismatches in both domain and harness. RASO comprises two complementary stages: \textbf{Retrieval-Augmented Skill Initialization (RASI)} constructs a knowledge-grounded initial skill without requiring agent rollouts, while \textbf{Retrieval-Augmented Skill Update (RASU)} iteratively refines the skill by retrieving external knowledge guided by execution feedback. Across four agent benchmarks and two models, extensive experiments show that RASO consistently outperforms baselines without retrieval-augmented skill initialization and updating.

1 INTRODUCTION

Agent skills are reusable natural-language instructions for completing tasks under a particular execution harness. Although public skill collections hold procedural knowledge across tasks and harnesses, skill optimization has largely relied on costly agent rollouts rather than that external knowledge.

1 INTRODUCTION

  • An agent skill is a reusable, actionable natural-language artifact guiding task completion under a given harness.Harnesses define available tools, file access, and scoring procedures.Source: §1 · p. 1
  • Millions of publicly shared skills encode procedural knowledge across diverse tasks, models, and harnesses, yet optimization methods largely rely on costly agent experience.Existing methods refine skills through training-task rollouts, leaving external skill-corpus priors largely unexplored.Source: §1 · p. 1
  • RASO leverages an external skill corpus as prior knowledge and comprises Retrieval-Augmented Skill Initialization and Retrieval-Augmented Skill Update.The supplied passage introduces RASI as leveraging prior corpus knowledge.Source: §1 · p. 1

Retrieval-augmented skill initialization (RASI)‘

RASI uses external skill knowledge to construct initial skills without agent rollouts, while RASU retrieves knowledge guided by execution feedback to refine skills. RASO adapts retrieved procedures to the target domain and harness, and the reported experiments find strong performance across benchmarks and models.

Retrieval-augmented skill initialization (RASI)‘

  • 92.9% of retrieved skills on SpreadsheetBench originate from a different harness; Cross-Harness Adaptation re-expresses procedures using target-supported objects, commands, and units.RASO abstracts away source-domain-specific instruction rather than directly incorporating retrieved content.Source: Figure 1 · p. 2, Retrieval-augmented skill… · p. 2
  • RASU identifies failure modes in agent trajectories and retrieves relevant external knowledge to address them during iterative skill refinement.The contribution summary describes the retrieved knowledge as guided by execution feedback.Source: Retrieval-augmented skill… · p. 2
  • RASI synthesizes initial skills that outperform both retrieved skills and retrieval-free initialization without requiring agent rollouts.The evaluation covers four benchmarks and two LLMs.Source: Retrieval-augmented skill… · p. 2
  • The reported gains are associated with adapting diverse external knowledge rather than relying on particular source documents.Source: Retrieval-augmented skill… · p. 2

2 RELATED WORKS

Prior work makes procedural knowledge available through textual skills and external resources, but approaches either depend mainly on observed execution or use retrieved knowledge only in limited ways. RASO is positioned as combining external knowledge with rollout feedback throughout skill optimization.

2 RELATED WORKS

  • Agent skills are textual procedural guidance that can be inspected, edited, shared, and reused across models, but their benefit depends on matching the target task and harness.Public collections such as GitSkills make this knowledge available at scale.Source: §2 · p. 3
  • Execution-based optimizers refine skills from rollouts, which limits exploration of procedural knowledge beyond what can be inferred from the agent’s own experience.Examples include TextGrad, GEPA, SkillOpt, and WikiSkill.Source: §2 · p. 3
  • External-knowledge methods either require repeated test-time retrieval and adaptation or use external knowledge only at initialization; RASO uses it for initialization and iterative updates.The passage contrasts these approaches with leveraging external knowledge throughout skill optimization.Source: §2 · p. 3

3 PROBLEM FORMULATION

The formulation treats the natural-language skill as the optimization variable for a frozen model operating through a fixed harness. Candidate skills are constructed from training rollouts, selected on validation performance, and evaluated on held-out tasks; optimization includes both initialization and rollout-guided updating.

3 PROBLEM FORMULATION

  • The setup evaluates a frozen language model through a harness, where executing a task with a skill produces a trajectory scored by a benchmark-specific evaluator.The evaluator compares against a reference answer when available, otherwise using the environment’s native success criterion.Source: §3 · p. 3
  • Candidate skills are constructed from Dtrain rollouts and selected using Dval performance, then the selected skill is evaluated on held-out Dtest.Source: §3 · p. 3
  • RASI builds the initial skill without rollouts, while RASU starts from that skill and uses rollout results to retrieve and adapt knowledge for iterative refinement.Figure 2 shows RASI queries from task and harness descriptions and RASU queries from rollout results.Source: Figure 2 · p. 4
  • Skill optimization includes both constructing an initial skill from task and harness descriptions without rollouts and refining it using training-rollout results.With the model and harness fixed, the natural-language skill is the only decision variable.Source: §3 · p. 4

4 RASO: RETRIEVAL-AUGMENTED SKILL OPTIMIZATION

RASO pairs retrieval and adaptation with two stages: RASI builds an initial skill without rollouts, while RASU uses execution feedback to iteratively refine skills. Both use retrieved procedural knowledge tailored to the target task and harness.

4.1 SKILL RETRIEVAL AND CROSS-HARNESS ADAPTATION

  • RASO retrieves heading-delimited skill sections with BM25 and adapts them into concise lessons tailored to each requirement, task, and harness.Source: §4.1 · pp. 4–5

4.2 RETRIEVAL-AUGMENTED SKILL INITIALIZATION (RASI)

  • RASI generates procedure-query pairs from task and harness descriptions, retrieves and adapts relevant sections, then uses grounded lessons to construct an initial skill without rollouts.Source: §4.2 · p. 5

4.3 RETRIEVAL-AUGMENTED SKILL UPDATE (RASU)

  • RASU analyzes failed trajectories to generate textual gradients and targeted retrieval queries, then adapts retrieved knowledge into lessons for skill updates.Source: §4.3 · pp. 5–6
  • RASU integrates textual gradients and trajectory-grounded lessons into a candidate skill, accepting it only when it outperforms the current skill on validation data.Source: §4.3 · p. 6

5 EXPERIMENT

The experiments evaluate retrieval-augmented initialization and updating across four benchmarks and two models, then analyze component contributions, retrieval settings, and adaptation examples.

5.1 MAIN RESULTS

  • Across both models and all four benchmarks, RASI achieves the highest initialization performance before any subsequent agent rollout.RASI also exceeds retrieval-free initialization and direct skill retrieval.Source: §5.1 · p. 6, Table 1 · p. 7
  • Across both backbones, RASO outperforms the strongest competing skill-optimization method on all four benchmarks.Table 1 reports agent performance for GPT-5.6-Luna and Qwen-3.5-9B, separating initialization-only results from skill updates.Source: Table 1 · p. 7, §5.1 · p. 7

5.2 ANALYSIS

  • Ablations show gains from retrieval-augmented initialization and updating, while adaptation improves both RASI and RASU.The combined components reach 49.03 on OfficeQA and 63.33 on SpreadsheetBench; adaptation adds gains in both stages on both benchmarks.Source: §5.2 · pp. 7–8
  • With RASI held fixed, RASU outperforms alternative update methods on OfficeQA and SpreadsheetBench.It also improves over RASI initialization without subsequent updates by 3.29 and 14.16 points, respectively.Source: §5.2 · p. 8, Table 4 · p. 8
  • K=5 produces the highest performance on OfficeQA and SpreadsheetBench; increasing K to 10 slightly degrades both RASI and RASU.The authors use K=5 to balance knowledge coverage against retrieval noise.Source: §5.2 · pp. 8–9, Figure 3 · p. 8
  • Retrieval from just 1% of the skill corpus improves performance over no retrieval, and performance generally rises as corpus size grows.The largest gains from corpus growth occur on SpreadsheetBench.Source: Figure 4 · p. 8, §5.2 · p. 9
  • In a SpreadsheetBench example, adaptation transfers an empty-group procedure from a mismatched skill so department B remains blank rather than receiving C’s mean.Without the retrieved section, code groups only filtered rows, shifting C’s mean into B’s cell.Source: §5.2 · p. 9, Figure 5 · p. 9

6 CONCLUSION

RASO uses external skills as prior knowledge for initialization and updating, adapting retrieved procedures to the target task and harness. Across four benchmarks and two backbones, it outperforms retrieval-free skill-optimization baselines.

6 CONCLUSION

  • RASO uses an external skill corpus as prior knowledge for both skill initialization and updating.Source: §6 · p. 9
  • Cross-Harness Adaptation adapts retrieved procedural knowledge to the target task and execution harness instead of directly reusing retrieved skills.Source: §6 · p. 9
  • Across four agent benchmarks and two model backbones, RASO consistently outperforms baselines without retrieval-augmented initialization and updating.Source: §6 · p. 9

A.1 IMPLEMENTATION DETAILS

The implementation fixes task and harness descriptions across experimental conditions, controls for benchmark-specific skills in the retrieval corpus, and specifies rollout and baseline procedures.

A.1 IMPLEMENTATION DETAILS

  • Optimization runs for two epochs with minibatches of 40 tasks; rollout processing is chunked into groups of eight for query generation.Chunk-level gradient–query pairs are merged into a final set, consolidating shared failure modes and removing redundant queries.Source: §A.1 · p. 13
  • A per-benchmark blocklist removes entire repositories matching benchmark or dataset keywords from the external corpus.Repository-level exclusion is used because companion skills may encode benchmark-specific procedures without naming the benchmark.Source: §A.1 · p. 13
  • For each benchmark, task and harness descriptions are generated once and held fixed across methods, models, and seeds.The generating agent does not see test instances, and descriptions are not revised after test performance is observed.Source: §A.1 · p. 13
  • RFSI matches RASI’s task and harness inputs and skill-generation procedure but omits grounded lessons, isolating retrieval and adaptation.All update baselines start from the same RFSI skill and use the same task and harness descriptions.Source: §A.1 · p. 13

A.2 BASELINE DETAILS

The appendix describes initialization and update baselines, noting that the optimizer model is shared across methods unless specified and that baselines do not access the external corpus during optimization.

A.2 BASELINE DETAILS

  • Initialization comparisons include no skill, SkillRouter retrieval from the corpus, and RFSI generation from task and harness descriptions without corpus access.Source: §A.2 · p. 13
  • Update baselines are TextGrad, GEPA, SkillOpt, and WikiSkill, with distinct feedback, mutation, edit, and persistent-wiki procedures.Source: §A.2 · p. 13
  • Unless specified otherwise, methods use the same target and optimizer model, and baselines cannot access the external corpus during optimization.Table 5 lists the number of tasks in each split.Source: §A.2 · p. 14, Table 5 · p. 14

A.3 BENCHMARK DETAILS

The evaluation spans document reasoning, spreadsheet manipulation, embodied household tasks, and web navigation, using shared train/validation/test splits where specified. Under fixed initialization, RASU leads the compared update methods across model-initialization combinations; RASO also uses fewer rollouts than some methods and has lower API cost on most benchmarks.

A.3 BENCHMARK DETAILS

  • OfficeQA tests grounded reasoning over Treasury bulletins, while SpreadsheetBench tests real-world spreadsheet edits against multiple file test cases.OfficeQA uses 246 questions; SpreadsheetBench uses 400 tasks, each split into training, validation, and test sets.Source: §A.3 · p. 14
  • ALFWorld evaluates textual interaction in simulated household tasks, and WebShop evaluates product search and selection from natural-language instructions.ALFWorld uses 134 out-of-distribution evaluation games; WebShop evaluates 500 test instructions.Source: §A.3 · p. 14

A.4 SKILL UPDATE METHODS UNDER FIXED INITIALIZATION

  • RASU achieves the highest performance across all four initialization-model combinations, outperforming retrieval-free updates regardless of the initial skill and target model.The comparisons cover Qwen-3.5-9B and GPT-5.6-Luna under RASI and RFSI initialization.Source: §A.4 · p. 15, Table 6 · p. 15, Table 7 · p. 15
  • Under RFSI, RASU beat the strongest baseline by +4.05 (61.07 vs. 57.02) on SpreadsheetBench with GPT-5.6-Luna and by +0.84 (30.60 vs. 29.76) with Qwen-3.5-9B.With GPT-5.6-Luna, it also beat SkillOpt by +2.02 (47.56 vs. 45.54) on OfficeQA; with Qwen-3.5-9B, it beat WikiSkill by +1.08 (38.48 vs. 37.40) on OfficeQA.Source: §A.4 · p. 15, Table 7 · p. 15, Table 8 · p. 15
  • Under RASI with Qwen-3.5-9B, RASU improves the initial skill by +1.75 on OfficeQA and +1.07 on SpreadsheetBench, reaching 42.25 and 31.55.Retrieval-free methods largely fail to improve the RASI initialization in these comparisons.Source: §A.4 · p. 15, Table 6 · p. 15
  • On OfficeQA with Qwen-3.5-9B, RASI alone scores 40.50, above RFSI followed by RASU at 38.48, while combining RASI and RASU reaches 42.25.The passage describes the two stages as contributing gains that neither achieves alone.Source: §A.4 · p. 15

A.5 OPTIMIZATION COST

  • RASO uses exactly as many rollouts as TextGrad and WikiSkill, and fewer than GEPA and SkillOpt.All methods are optimized for the same two epochs; the rollout-count comparison excludes test-set evaluation.Source: §A.5 · p. 16, Table 10 · p. 16
  • At matched budgets, RASO has the lowest cost on three of four benchmarks and is within $0.06 of the cheapest method on SpreadsheetBench.Compared with TextGrad, RASO reduces cost by 42–75%; on WebShop, the costs are $41.03 vs. $10.28.Source: §A.5 · p. 16, Table 9 · p. 16
Loading 2609.38024v1…