PaperlayerView exact revision on arXiv

Source-linked AI summary

SkillRAE: Agent Skill-Based Context Compilation for Retrieval-Augmented Execution

Xiangcheng Meng, Shu Wang, Yixiang Fang

arXiv:2605.10114v1cs.CL

TL;DR

RAE systems retrieve external skills but have paid less attention to compiling their evidence into compact, grounded, immediately usable contexts. SkillRAE builds a multi-level skill graph and performs rescue-aware online compilation, improving benchmark performance, including a 11.7% improvement over the SOTA method on SkillsBench. Its scope is strongest for repositories with explicit procedural text and constraints, not opaque tools or undocumented runtime dependencies.

  • Problem

    Existing RAE work focuses mainly on skill retrieval and task execution, leaving selected skill evidence insufficiently organized for downstream execution.

  • Method

    SkillRAE builds a multi-level graph offline and performs online skill retrieval, subunit rescue, and compact compilation into task-specific guidance.

  • Results

    11.7% improvement over the SOTA method on SkillsBench, with ablations showing context compilation is crucial rather than a mere prompt addition.

  • Takeaways & Limitations

    SkillRAE supports the conclusion that RAE requires compact, grounded, task-specific context compilation beyond skill selection.

  • Takeaways & Limitations

    SkillRAE is most applicable to repositories with explicit procedural text, file/output conventions, and constraints, and does not guarantee downstream context use or recovery from execution-time failures.

Abstract

from arXiv · show

Large Language Model (LLM)-based agents (e.g., OpenClaw) increasingly rely on reusable skill libraries to solve artifact-rich tasks such as document-centric workflows and data-intensive analysis. As these libraries grow, a few works have attempted to study the Retrieval-Augmented Execution (RAE), which often first retrieves some external skills and other knowledge, then compiles the context using retrieved skills, and finally executes the task. Existing works mainly focus on optimizing skill retrieval and task execution, and they pay little attention to how to effectively organize the selected skill evidence in a form that is compact, grounded, and immediately usable for the downstream executors to complete tasks. To fill this gap, we propose SkillRAE, a two-stage RAE approach focusing on skill-based context compilation, which consists of the offline and online stages. Specifically, in the offline indexing stage, it builds a multi-level skill graph over skill communities, skills, and reusable subunits, for capturing their relationships. In the online retrieval stage, it first performs skill-ranked retrieval with selected-subunit evidence export in the graph, and then applies rescue-aware compact compilation to recover the key evidence. Together, these components compile a coarse-ranked skill set into a task-specific context that is compact, grounded, and immediately usable. Experiments on two public benchmarks show that SkillRAE achieves a significant improvement over baselines for RAE. For example, on SkillsBench, it achieves an improvement of 11.7% over the SOTA method. Ablation studies further show that our context compilation is crucial, instead of a mere prompt addition.

1 Introduction

SkillRAE addresses a gap in Retrieval-Augmented Execution by compiling retrieved skills and reusable subunits into compact, grounded, task-specific contexts. Its two-stage graph-based approach improves benchmark performance, with context compilation contributing beyond simple prompt addition.

  • 1 Introduction: A skill is illustrated as a reusable bundle containing a natural-language description and optional linked assets, references, or scripts.This bundle structure motivates treating procedural subunits as reusable evidence rather than only selecting whole skills.
  • 1 Introduction: Existing RAE research emphasizes skill retrieval and execution, while giving less attention to organizing selected evidence into an immediately usable context artifact.Skill routing, repository-aware retrieval, and execution planning address adjacent stages but do not provide the same compilation mechanism.
  • 1 Introduction: Retrieved skills can remain under-resolved because relevant subunits may be hidden within other skills and isolated subunits may depend on skill-specific constraints.The challenge is therefore to compile related skills and subunits rather than merely rank them.
  • 1 Introduction: SKILLRAE builds a multi-level skill graph offline, then retrieves skills, rescues relevant subunits, and compiles them into task-specific guidance online.The resulting context is intended to be compact, grounded, and immediately reusable by downstream agents.
  • 1 Introduction: 11.7% relative improvement over pure curated skills on SkillsBench, with 29.26% mean verifier reward across 87 tasks.SKILLRAE also improves 32.8% relative over the strongest automated baseline.
  • 1 Introduction: Ablations show that context compilation is crucial for forming an executable skill context rather than functioning as a simple prompt addition.The paper presents this finding as evidence that compilation is a substantive component of the method.

2 Related Work

Related work spans agent skills, tool and API selection, retrieval-augmented generation, and Retrieval-Augmented Execution. These areas provide reusable capabilities and evidence access, but RAE specifically concerns retrieving procedural artifacts for task execution.

  • 2 Related Work: Agent skills are reusable procedural artifacts that externalize task-focused know-how, including when to act, how to execute, and how to judge completion.They differ from APIs or tool-use interfaces, which mainly specify callable operations.
  • 2 Related Work: Tool and API selection methods rank callable operations using names, descriptions, schemas, documentation, and demonstrations before downstream invocation.These methods extend agents with external operations for information access, computation, and environment interaction.
  • 2 Related Work: RAG methods ground language models by retrieving external evidence, using vector databases, graphs, or iterative agentic retrieval.Post-processing may filter, summarize, compress, or rewrite retrieved passages before generation.
  • 2 Related Work: RAE extends retrieval augmentation from knowledge-grounded generation to agent execution by retrieving procedural artifacts such as skills, tools, and APIs.Its related methods include skill routing and repository-aware retrieval for selecting or structuring reusable execution resources.

3 Our RAE Approach: SKILLRAE

SKILLRAE builds a multi-level skill graph offline and uses it online to retrieve skills, expose subunit evidence, and compile compact, task-specific context for execution.

  • 3 Our RAE Approach: SKILLRAE: SKILLRAE uses offline graph construction and online retrieval-plus-compilation to produce compact, grounded, immediately usable execution context.The offline stage indexes skill communities, skills, and reusable subunits; the online stage retrieves evidence and compiles it with task-specific guidance.
  • 3.2 Offline Stage: The offline skill graph represents communities, skills, and normalized subunits, preserving executor-compatible skills while exposing fine-grained evidence.Skills are linked to subunits through extraction edges, and skill communities group related skills for later retrieval.
  • 3.3.1 Skill Retrieval: Online retrieval combines top-down community matching with bottom-up subunit evidence to select skills and export task-relevant highlights.The method also retains non-selected rescue candidates for later compilation, while skills remain the executor-facing units.
  • 3.3.2 Context Compilation: Context compilation filters and attaches subunits, generates task-specific guidance, and organizes selected evidence around the task, output contract, and context budget.Rescued subunits become local cues attached to selected skills before the final context is delivered as advisory context to the executor.
  • 3.3.2 Context Compilation: The compiled context changes what the agent sees before execution rather than changing runtime orchestration.Compilation adds local evidence and subunit cues while leaving the source skills, planner, and executor unchanged.

4 Experiments

Experiments evaluate SKILLRAE through end-to-end execution on two complementary benchmarks, comparing it with retrieval, routing, orchestration, and curated-skill baselines. Results show that graph-derived evidence and context compilation improve downstream performance, including across agent backbones and compatible retrieval interfaces.

  • 4.1 Setup: The evaluation uses end-to-end agent execution, with selected skills and compiled guidance sent to the same executor and outputs scored against benchmark-specific requirements.SkillsBench reports mean reward over 87 tasks, while AgentSkillOS reports an aggregate score.
  • 4.2 Overall Performance: 3.06 percentage points on SkillsBench and 1.09 percentage points on AgentSkillOS over their native curated or native baselines, respectively, with SKILLRAE achieving the best overall performance on both benchmarks.It also improves over SkillRouter by 7.22 percentage points on SkillsBench and 2.29 percentage points on AgentSkillOS.
  • 4.3 Detailed Analysis of SKILLRAE: Removing Top-down Retrieval lowers reward from 29.26% to 16.61%, the largest ablation drop, while removing Bottom-up Retrieval or Context Compilation also reduces performance.Top-down retrieval supports coarse matching through skill communities; bottom-up retrieval supplies fine-grained matching, and compilation produces task-specified guidance.
  • 4.3 Detailed Analysis of SKILLRAE: 29.26% and 28.85% rewards under Codex CLI with GPT-5.2 and Gemini CLI with Gemini 3 Flash show similar full-method performance across backbones, with all ablations reducing performance.Top-down Retrieval remains the largest-degradation component under both backbones, while some Gemini-side drops, especially for Context Compilation, are smaller.
  • 4.4 Context Compilation on Compatible External Backbones: Context compilation improves vanilla and LLM-based retrieval when selected skills are exposed through a compact task/output-aware layer, while SKILLRAE’s graph-native compilation raises reward from 22.59% to 29.26%.The comparison separates general context-exposure benefits from additional gains due to graph-derived subunit export and affiliate-aware rescue.
  • 4.5 Summary of findings: Overall, the experiments support combining skill retrieval with compilation of retrieved skills, subunits, and constraints into a compact, grounded, task-specific execution context.This conclusion is supported by the main comparison, ablations, cross-backbone evaluation, and compatible-backbone compilation tests.

5 Limitations

SKILLRAE is most applicable when skill repositories expose procedural text, file/output conventions, and constraints. Its scope is limited when dependencies are hidden in opaque tools, undocumented code, or runtime state, and it does not guarantee downstream context use or failure recovery.

  • SKILLRAE is most applicable to repositories with explicit procedural text, file/output conventions, and constraints.
  • Hidden dependencies in opaque tools, undocumented code, or runtime state may reduce SKILLRAE's effectiveness.
  • As an advisory compiler rather than a planner or controller, SKILLRAE does not guarantee context use or recovery from execution-time failures.

6 Conclusion

The paper studies skill-based context compilation for RAE and proposes SKILLRAE, which constructs a multi-level skill graph before retrieving and compiling relevant skills and subunits. Experiments on SkillsBench and AgentSkillOS show improved downstream execution performance over baseline methods.

  • SKILLRAE constructs a multi-level skill graph offline, then retrieves relevant skills and subunits and compiles them into compact, grounded, task-specific context online.
  • SKILLRAE improves downstream execution performance over baseline methods on SkillsBench and AgentSkillOS.
  • Future work will evaluate SKILLRAE on more skill benchmarks and under real-world constraints such as limited token budgets.

A.1 Implementation Details

The implementation deterministically extracts reusable evidence from skills, organizes communities with fixed clustering settings, and applies bounded rescue and retrieval procedures under a fixed context budget.

  • A.1 Implementation Details: The deterministic extractor collects procedural lines, file and command patterns, library names, and requirement-matched constraints from SKILL.md files.It normalizes text, removes duplicates, filters by token length, and creates skill–subunit extraction edges.
  • A.1 Implementation Details: L2 community construction uses KMeans with random_state=42, n_init=10, and k = ⌊|S|⌋ clusters.
  • A.1 Implementation Details: Rescue selection applies parent and subunit score thresholds of 0.35 and 0.12, with global and per-parent caps of 3 and 1.
  • A.1 Implementation Details: The rescue candidate pool is bounded by min(|ranked|, max(10, 4k)).
  • A.1 Implementation Details: The compiled context budget is fixed at 384 tokens, while retrieval baselines select top-k = 5 skills.The LLM-based retrieval baseline first forms a 32-skill embedding-retrieved candidate pool before LLM selection.

A.2 Implementation Details of Affiliate Attachment

Affiliate attachment deterministically links each rescued subunit to a selected skill using a weighted score over relevance, skill fit, retrieval confidence, and graph relationships. Attachments receive exclusivity and capability-group adjustments before threshold and compact-packet gating.

  • A.2 Implementation Details of Affiliate Attachment: Affiliate attachment scores a rescued subunit against each selected skill using five weighted features rather than changing the selected skill set.The layer does not perform random walk or graph diffusion or introduce new executable skills.
  • A.2 Implementation Details of Affiliate Attachment: The score weights token relevance at 0.15, skill fit at 0.45, retrieval confidence at 0.10, same-group membership at 0.15, and active-group membership at 0.15.
  • A.2 Implementation Details of Affiliate Attachment: Each rescued subunit attaches to the highest-scoring selected skill, with an exclusivity bonus, an outside-group penalty, and later affiliation and compact-packet gates.

A.3 Reproducibility Details

The experiments use fixed benchmark task definitions, controlled skill pools and execution protocols, and documented retrieval, compilation, model, verification, and logging procedures.

  • Experiments use fixed benchmark task lists and native evaluation protocols, with SkillsBench covering 87 tasks and AgentSkillOS covering 30 tasks.SkillsBench uses deterministic verifier rewards, while AgentSkillOS uses its aggregate benchmark score.
  • Benchmark-native methods use curated reference conditions, while retrieval methods draw from benchmark skill repositories containing SKILL.md files and optional resources.Comparable methods share the same task environment, skill-pool exposure, runtime budget, downstream executor, and aggregation rule.
  • The offline graph pipeline extracts, deduplicates, embeds, and connects procedural, element-like, and constraint-like subunits to source skills and communities.The resulting artifacts include skill and subunit nodes, edges, embeddings, canonical representations, and community assignments.
  • Unless stated otherwise, retrieval selects five skills, considers 30 matched subunits, retains two communities, and exports up to three highlighted subunits per skill.Rescue-aware compilation applies parent and subunit score thresholds, global and per-parent caps, and token-Jaccard redundancy filtering.
  • Vanilla retrieval ranks skills from frontmatter similarity, whereas all methods pass selected skills or compiled guidance to the same downstream executor for artifact evaluation.The main comparisons use Codex CLI with GPT-5.2, while component ablations also use Gemini CLI with Gemini 3 Flash.
  • Each run records task summaries, selected-skill metadata, context artifacts, executor logs, verifier outputs, and aggregation files sufficient to recompute reported benchmark scores.SkillsBench means can be recomputed from fixed-task verifier rewards, while AgentSkillOS uses its native aggregate score.

A.4 Compute Resources

Compute resources include a large CPU-and-memory server, while local retrieval and graph operations use CPUs and hosted inference uses external CLI backends. Reported compute is recoverable rather than exhaustive because exploratory and failed runs were not fully audited.

  • The worker server has 56 physical CPU cores, 112 logical threads, 1.0 TiB of RAM, eight RTX A5000 GPUs, and containerized execution software.It runs Ubuntu 24.04 LTS, Linux kernel 6.8.0, Docker 29.3.0, Docker Compose v2.29.7, and Python 3.12.9.
  • The local worker manages graph preparation, retrieval, compilation, task mirroring, containerized execution, verification, logging, and aggregation, while hosted inference uses Codex and Gemini CLI backends.The audited retrieval backend uses one worker by default, although some archived wrappers use up to four.
  • At least 303 planned Harbor invocations and 33.66 recorded wall-clock hours were observed in sampled run directories, representing recoverable compute rather than total project cost.Preliminary, failed, and debugging runs were not exhaustively audited.
Loading 2605.10114v1…