PaperlayerView exact revision on arXiv

Source-linked AI summary

RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources

Yijia Fan, Zonglin Di, Zimo Wen, Yifan Yang, Mingxi Cheng, Qi Dai, Bei Liu, Kai Qiu, Yue Dong, Ji Li, Chong Luo

arXiv:2606.29538v4cs.SEcs.AI

TL;DR

Existing software-agent skill libraries underuse tutorial videos and other multimodal human resources that can convey procedural knowledge. RESOURCE2SKILL distills these resources into a hierarchical multimodal Skill Wiki, improving artifact quality by +11.9 points over no-skill agents and beating agentic-harness baselines in 26 of 28 main-aggregate cells.

  • Problem

    Existing skill libraries largely rely on expert writing, agent traces, or text/code, leaving tutorial videos and other multimodal human expertise underused.

  • Method

    RESOURCE2SKILL distills videos, repositories, articles, documentation, and artifacts into executable skills organized in a hierarchical multimodal Skill Wiki.

  • Results

    +11.9 points in artifact quality over no-skill agents, with RESOURCE2SKILL beating two agentic-harness baselines in 26 of 28 main-aggregate cells.

  • Takeaways & Limitations

    Distilling human-created multimodal resources provides software agents reusable procedural knowledge that improves performance across diverse authoring domains.

  • Takeaways & Limitations

    Online acquisition adds search, distillation, and validation latency and is evaluated separately on a fixed pool rather than as unbounded test-time recall.

Abstract

from arXiv · show

Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.

1 Introduction

RESOURCE2SKILL distills tutorial videos and other human-created multimodal resources into executable skills organized in a hierarchical multimodal Skill Wiki for software agents. It unifies offline library construction with online skill acquisition and improves performance across seven software-authoring domains.

  • Motivation: RESOURCE2SKILL addresses skill-library limitations by distilling reusable executable skills from multimodal human-created resources, especially tutorial videos.Existing libraries are mainly hand-written, derived from agent traces, or mined from text and code, leaving multimodal human resources underused.
  • Method: The framework organizes skills in a hierarchical Wiki combining structured text, executable or adaptable code, visual examples, metadata, and provenance.Text explains applicability and mechanism, code provides tool-grounded execution patterns, and visual examples preserve layout, style, motion, and other perceptual information.
  • Method: A unified resource-to-skill operator builds offline domain libraries and acquires new skills online when user requirements reveal capability gaps.At inference, agents navigate the hierarchical index, form candidate pools, and compose relevant multimodal entries during execution.
  • Evaluation: Across seven software-authoring domains, skill access yields consistent gains over no-skill agents and strong harness baselines.The domains include slide design, web pages, spreadsheets, Blender, CAD, UE5 scenes, and music production.
  • Evaluation: The empirical study isolates the contributions of source diversity, multimodal format, hierarchical organization, selection strategy, and online acquisition.The evaluation uses seven authoring benchmark suites and controlled ablations.

2 Related Work

Prior work makes procedural knowledge explicit for tool-using agents and draws supervision from human-created resources, but commonly treats resources as context or builds narrow text/code libraries. RESOURCE2SKILL instead distills multimodal resources into a validated, hierarchical Skill Wiki for agent execution.

  • LM agents and reusable memory: LM agents combine reasoning, tool invocation, executable actions, and execution feedback, while retrieval, memory, and skill-library methods make procedural knowledge explicit.These agents are increasingly evaluated on realistic web and software-engineering tasks, but procedural knowledge often remains implicit in weights, prompts, or interaction history.
  • Procedural knowledge from human-created resources: Instructional videos, web pages, and documented code provide rich procedural supervision, but raw resources are poorly matched to execution and usually serve as pretraining or retrieval context.Resource2Skill distills these sources into a validated multimodal wiki rather than leaving them in raw form.
  • Comparison with skill-library frameworks: Existing skill-library frameworks mainly grow text- or code-only libraries from single-domain agent traces or failures, use hand-authored bundles, or mine offline skills into domain-specific trees.The comparison includes Voyager [31], AWM [32], ASI [33], SkillFlow [38], Anthropic Agent Skills, and SkillFoundry [27].

3 Method

RESOURCE2SKILL builds executable skills through four stages—construction, wiki organization, selection, and execution—using a shared MCP-mediated browse-select-execute interface. Skills combine structured text, visual evidence, executable code, taxonomy paths, and provenance metadata, while the same construction operator supports controlled online acquisition.

  • Overview: The method comprises construction, wiki organization, selection, and execution, with all stages sharing one MCP-mediated browse-select-execute interface over domain-specific backends.When offline coverage is insufficient, construction is reused online rather than adding a separate pipeline.
  • Skill representation: Each skill combines a taxonomy path, metadata, and complementary text, visual, and code views that describe applicability, evidence, and executable procedures.Visual and code fields may be empty for reference-only entries, while metadata supports filtering, auditing, and provenance.
  • Construction: Construction distills tutorial videos, source repositories, articles, and reference artifacts into normalized wiki entries validated for completeness, provenance, deduplication, modality consistency, and executability.A vision-capable language model extracts modality-specific evidence before a domain-specific predicate applies the five validation checks.
  • Selection: MetaBrowse first uses taxonomy-aware lexical scoring to narrow candidates, then a language model selects a composable subset and may choose zero skills when none fit.The candidate evidence can expose metadata and any available text, visual, and code views.
  • Online acquisition: Online acquisition targets capability gaps with the same construction and validation operator, keeping temporary candidates separate from offline skills as a controlled gap-filler.Targeted queries use the same resource families, and online candidates are validated before exposure to the current task or evaluation split.

4 Experiments

Across seven authoring domains and four agent backends, Resource2Skill consistently improves performance over no-skill agents and off-the-shelf harnesses. Controlled ablations attribute gains to library scale, hierarchical multimodal representation, diverse sources, and online acquisition for novel tasks.

  • Main comparison: +11.9 points: w Skills averages 56.8% versus 45.0% for w/o Skills and wins all 28 model–domain cells; it also beats the stronger harness in 26 of 28 cells.Codex-H and ClaudeCode-H average 50.5% and 50.4%, respectively, while the two exceptions are within one point.
  • Main comparison: Gains are largest in convention-heavy domains, reaching +30 to +40 pp on UE5, while Reaper shows the smallest improvement.The free-form agent often fails to assemble minimum-viable UE5 scenes, whereas Reaper has a stronger no-skill prior.
  • Skill-pool scaling: Performance rises monotonically with library size and saturates near 200 skills; the initial 0→200 slice gains +3.1 to +14.2 pp, while 400→Full adds at most +0.8 pp per domain.Early entries cover common operations and recovery routines; later entries fill domain-specific gaps.
  • Online acquisition: Online acquisition adds only +0.7 pp on standard tasks but raises novel-task performance from 41.2% to 62.8%, a +21.6 pp lift.The offline pool covers common requests, so online search functions primarily as a gap-filler for novel tasks.
  • Wiki and representation ablations: The full wiki is best in every domain, beating flat text by 2.5 to 8.2 pp; multimodal content further raises performance from 65.0% for Text to 68.9% for Full.Visuals add +1.9 pp and code +2.0 pp, while hierarchy, metadata, visual previews, and executable code improve retrieval and grounding.
  • Source and selection ablations: Video is non-substitutable: removing it drops the average from 68.9% to 59.4%, while MetaBrowse reaches 68.9% versus 66.0% for BM25 and wins every domain.Video-only skills outperform the three-source no-video library by 7.4 points; MetaBrowse’s largest margins over retrieval-only selection are Excel (+5.0 pp), PPT (+3.8), and Blender (+2.3).

5 Conclusion

RESOURCE2SKILL distills multimodal human references into a structured, executable Skill Wiki that supports offline construction and controlled online gap filling. Across seven authoring domains and four backends, skill access improves artifact quality and outperforms agentic-harness baselines.

  • Conclusion: +11.9 points over no-skill agents across seven authoring domains and four backends, with skill access beating two agentic-harness baselines in 26 of 28 main-aggregate cells.These results support reusable procedural knowledge distilled from human-created resources as a mechanism for improving software-agent artifact quality.

A Per-Domain Details

The paper implements RESOURCE2SKILL across seven authoring domains with domain-specific tools, rendering paths, and skill libraries. Main comparisons cover all seven domains, while per-condition ablations use a five-domain core and hold out CAD and UE5 to bound compute.

  • Domain coverage: The implemented backends span Web, Excel, PPT, Blender, Reaper, CAD, and UE5, with CAD and UE5 excluded from per-condition ablations.Main results include all seven domains, whereas ablations use Web, Excel, Reaper, PPT, and Blender.
  • Slide design: PPT uses SVG-to-PowerPoint rendering on a fixed 16:9 canvas, with 996 active wiki entries and video or contact-sheet frame scoring.Its categories include layout, typography, palette, charts, icons, photos, shells, and motion.
  • Domain-specific pipelines: Each domain pairs specialized authoring tools with an evaluation render path, including Playwright screenshots, LibreOffice sheets, Blender hero frames, Reaper audio previews, and CAD or UE5 viewport composites.The implementations use vanilla HTML/CSS/JavaScript, openpyxl, Blender bpy, ReaScript over MCP, ezdxf with FreeCAD fallback, and a UE5-MCP bridge.
  • Skill representation: Each skill bundles agent-facing metadata, structured prose describing mechanisms and conditions, executable code, and optional visual thumbnails resolved only when requested.This design keeps visual assets out of the default text context while retaining multimodal access.
  • Inference-time context: BM25 selects K=20 candidates and the language-model selector exposes n=5 full entries, yielding approximately 26–32k tokens of skill-related context per task.The selection step uses approximately 4–10k tokens, followed by approximately 22k tokens for five expanded entries.

B Benchmark Construction and Evaluation Protocol … F Judge Reliability

The evaluation uses wiki-blind, taxonomy-driven benchmark construction, standardized artifact rendering and blinded judging, and reproducible matched comparisons. Skills are distilled through one vision-language-model call plus deterministic validation, while retrieval, composition, and judge reliability are assessed from logged runs and human-rater agreement.

  • B Benchmark Construction and Evaluation Protocol: Benchmark briefs specify domains, requests, output paths, and artifact types without naming skills, sources, or solution steps.Briefs are manually screened for clarity, feasibility, and non-overlap with construction resources.
  • B Benchmark Construction and Evaluation Protocol: Rendered artifacts are scored by blinded domain-appropriate judges using five paper-facing axes, with matched comparisons aggregated over identical brief IDs.Each score records the rendered artifact, rubric breakdown, overall score, and judge metadata for later re-aggregation.
  • B Benchmark Construction and Evaluation Protocol: Resource collection is taxonomy-driven and benchmark briefs are wiki-blind, preventing benchmark identifiers, wiki metadata, or scores from influencing library construction.Acceptance thresholds are frozen before scoring, and benchmark outcomes are not fed back into the library or benchmark generator.
  • C Harness Baseline Configurations: ClaudeCode-H and Codex-H serve as off-the-shelf harness baselines using the same briefs, domain backends, output contract, and matched evaluation protocol.The harnesses retain their default planning and tool-use loops rather than using custom-built or stripped variants.
  • D Construction Operator and Acceptance Predicate: Resource2Skill distills each resource with one vision-capable language-model call, then applies deterministic post-processing and five rule-based acceptance gates.The gates check completeness, provenance, deduplication, modality consistency, and structural executability; failed executable entries remain reference-only and are filtered in verified-only mode.
  • E Retrieval and Composition Quality: Run logs expose BM25 top-20 candidates, five selected skills, selection rationales, execution traces, tool outcomes, and artifact-level failures for retrieval and composition analysis.Composition applies the selected skills once in order, without multi-pass replanning or conflict arbitration.
  • E Retrieval and Composition Quality: Matched-budget selection ablations isolate ranking and presentation, so irrelevant retrieval can cause selector refusal, failed application, or lower artifact scores.The library, agent, judge, decoding seed, and candidate budget remain fixed across strategies.
  • F Judge Reliability: Judge reliability on 17 task–artifact pairs reached ρ = 0.71 and ICC = 0.66 against the median of three human raters on the overall axis.Design and detail aligned most closely with human medians, while utility showed the highest disagreement.

G Per-Cell Paired Outcome Counts and Wilcoxon Tests

Across matched N=80 briefs, Table 6 reports per-cell paired outcome counts and Wilcoxon signed-rank tests comparing w Skills with w/o Skills. The w Skills delta is significant in every reported cell, with 8 of 9 cells reaching p < 10−8.

  • G Per-Cell Paired Outcome Counts and Wilcoxon Tests: The analysis covers matched N=80 briefs for each reported cell.The table summarizes paired comparisons between w Skills and w/o Skills.
  • G Per-Cell Paired Outcome Counts and Wilcoxon Tests: Table 6 partitions each matched cell into strict w Skills wins, ties, and strict w/o Skills wins.It reports paired outcome counts, the matched delta, and the paired Wilcoxon signed-rank p-value.
  • G Per-Cell Paired Outcome Counts and Wilcoxon Tests: p < 10−3 in every reported cell, with 8 of 9 cells at p < 10−8 for the w Skills delta over w/o Skills.These results use paired Wilcoxon signed-rank tests on matched N=80 briefs per cell.

H Per-Axis Main-Comparison Details · I Online Acquisition Details

The paper provides per-axis GPT-5.4 main-comparison details and specifies a bounded, validated, and isolated protocol for online skill acquisition. Per-domain axis means are reported as percentages and average to the overall scores in Table 1.

  • H Per-Axis Main-Comparison Details: Each domain’s five axis values average to its corresponding GPT-5.4 overall score in Table 1.
  • H Per-Axis Main-Comparison Details: The w Skills column reports each domain’s mean and its matched percentage-point difference relative to w/o Skills.Positive differences are marked in green.
  • H Per-Axis Main-Comparison Details: Table 7 reports five-axis GPT-5.4 scores for each domain, with percentages computed by multiplying raw axis means by 10.Each domain uses its own five-axis rubric.
  • I Online Acquisition Details: Online acquisition activates only when the offline wiki fails to return an adequate candidate set for the requested capability.
  • I Online Acquisition Details: The online arm may add at most 100 newly searched and distilled skills, with this budget fixed across task sets to measure coverage gain rather than unbounded resource access.
  • I Online Acquisition Details: Online candidates undergo the same construction predicate as offline candidates, and executable code must pass a domain smoke check or be marked reference-only.Rejected candidates are not exposed to the agent.
  • I Online Acquisition Details: Online entries remain in a separate pool throughout evaluation and are not folded into the offline wiki used by the main comparison or ablations.This prevents online search from silently changing the default skill library.

J Case Studies

Case studies across five authoring domains show that distilled skills can substantially improve artifact quality, while also exposing failures from incomplete grounding and overly conservative composition. These manually authored boundary probes complement, rather than contradict, the matched benchmark aggregates.

  • Case-study design: The study pairs one skill-assisted success and one failure in Web, PPT, Excel, Blender, and Reaper on GPT-5.4, using side-by-side qualitative comparisons.Failures are diagnostic boundary probes, not pure missing-output failures, and are excluded from matched benchmark aggregates.
  • Success cases: In Web and PPT successes, skills produce a richer restaurant page and a denser, more varied all-hands deck than the thinner no-skill outputs.The Web skill arm adds complete content and a styled testimonials block, while the PPT arm adds shell variety, dense slide content, and consistent theming.
  • Failure cases: Failures show that skill use can underperform when artifacts retain unresolved bindings, including sparse Web or PPT outputs, Excel #NAME? errors, and an unrecognizable Blender render.The Reaper failure similarly stays in one section, while the no-skill arm achieves stronger variation or polish; these are quality failures despite successful execution where stated.
  • Success cases: In Excel and Blender successes, skills yield a structured KPI workbook and a recognizable, materially and dramatically lit jewelry render instead of sparse or primitive outputs.The Excel result includes summary-first organization, dense tables, and coherent charts; the Blender result includes rim/key lighting and PBR materials.
  • Success cases: The Reaper success adds grounded sidechain pumping, bass, harmonic layering, and clearer arrangement dynamics, whereas the no-skill track remains competent but flatter.The comparison uses spectrogram and waveform visualizations of the skill and no-skill outputs.
  • Failure analysis: Across failures, partial grounding leaves borrowed surface patterns without resolved parameters, while conservative composition sacrifices variation by adhering too closely to a single pattern.These patterns align with weaker Random-FullPool and Embed retrieval baselines in the selection ablation, emphasizing that skill value depends on successful parameter binding.

K User Study · L Skill Library Schema and Storage

A blinded human A/B study externally validates the automatic judge’s preference direction, while the skill library stores each skill as a provenance-aware multimodal bundle with structured metadata and executable content. Across 200 ratings, w Skills substantially outperforms w/o Skills, and the schema implements the paper’s abstract multimodal skill tuple.

  • K User Study: The blinded A/B study samples balanced artifact pairs across seven domains, using five raters per anonymized side-by-side comparison to choose the better artifact or declare a tie.It serves as external validation of the automatic judge’s preference direction, not as a training or selection signal.
  • K User Study: w Skills wins 136 of 200 human ratings (68.0%), versus 23 wins (11.5%) for w/o Skills, with 41 ties (20.5%); excluding ties, its win rate is 85.5%.The study uses five raters per pair and reports a micro-average over 200 individual votes.
  • L Skill Library Schema and Storage: Each skill is materialized as a modality bundle stored under skills_wiki/<domain>/<skill_id>/.The bundle separates source provenance, textual descriptions, visual examples, executable or adaptable code, and metadata.
  • L Skill Library Schema and Storage: The library organizes source, text, visual, code, and meta.json components so skills retain provenance, explanatory content, visual references, procedures, and validation information.The source directory contains provenance and resource references; text covers overview, mechanism, applicability, and inputs; visual stores thumbnails, screenshots, renders, or diagrams.
  • L Skill Library Schema and Storage: Metadata records skill identity, domain, category path, tags, applicability, source type, provenance fields, and validation status.This supports structured retrieval and validation of stored skills.
  • L Skill Library Schema and Storage: The prose body uses semantic sections for mechanism, use conditions, inputs, and expected effects, matching the organization used in the main text.These sections provide structured descriptions of how and when a skill should be applied.
  • L Skill Library Schema and Storage: The storage format implements the abstract multimodal skill tuple s = (p, xtext, xvisual, xcode, m) introduced in Section 3.1.The tuple corresponds to provenance, text, visual content, code, and metadata.

M Limitations

The evaluation is supported by acceptable judge–human agreement and blinded human A/B preferences, but generalization, online-acquisition costs, and raw-resource retrieval comparisons remain limited.

  • Evaluation limitations: Evaluation relies mainly on GPT-5.4 vision, with acceptable judge–human agreement and independent blinded human A/B corroboration across all seven domains.Reaper uses an audio-capable GPT-4o-series judge; the human study used five raters per pair.
  • Scope limitations: The framework does not claim generalization to domains lacking either a programmatic tool interface or a public stream of procedural content.
  • Online acquisition: Online acquisition adds search, distillation, and validation latency and is evaluated separately on a fixed online pool against Tnovel rather than in the main comparison.Keeping it separate avoids conflating library-scaling effects with uncontrolled context expansion at test time; the study measures coverage gain on capability regions known to be insuff.
  • Retrieval baselines: Retrieval baselines operate over the distilled skill library rather than raw resources under a matched token budget, leaving that comparison for future work.
Loading 2606.29538v4…