PaperlayerView exact revision on arXiv

Source-linked AI summary

Prompt2Skill: Unsupervised Skill Optimization From Natural Language Instructions

Bo Ni, Li Li, Ryan A. Rossi, Franck Dernoncourt, Tyler Derr

arXiv:2609.38593v1cs.CLcs.AI

TL;DR

Automated skill optimization often assumes access to curated, in-distribution task data, which users may not have. Prompt2Skill builds skills from task descriptions by discovering or synthesizing task-matched data and refining skills through reflective editing. Across four benchmarks and five target models, it reports 36.1% average relative improvement over direct prompting.

  • Problem

    Prior automated skill optimization assumes curated, in-distribution labeled task data, although users may have only a task description or a few examples.

    Source: §1 · p. 2, §2 · p. 2

  • Method

    Prompt2Skill derives a task specification, retrieves or synthesizes suitable items, and refines a skill through reflective edits evaluated on fresh held-out batches.

    Source: §1 · p. 2

  • Results

    36.1% average relative improvement over direct prompting across 19 evaluated model–benchmark pairs.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Task descriptions alone can guide automated construction of effective skills for frozen language models.

    Source: §6 · p. 9

    Solver agreement does not establish reference correctness or representativeness of the target distribution.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Skills are external artifacts that Large Language Models (LLMs) consume at inference time to improve their performance on specialized domains by incorporating relevant procedural and domain knowledge. Expert-authored skills are expensive to produce, and the resulting artifacts are not optimized for the specific model that consumes them, whose failure modes can vary with version, scale and training. In addition, emerging tasks may fall outside the scope of existing skill libraries, creating a need to develop new skills before curated training data become available. Recent works have explored automated skill optimization through reflection, but they require a curated, in-distribution training set, which users might not always have. To address these limitations, we present Prompt2Skill, a framework that builds skills from natural-language task description alone. From the prompt, the system derives a task specification, discovers or synthesizes datasets, and refines the skill in a closed loop of reflective editing. Across four domains spanning question answering, reading comprehension, spreadsheet manipulation, and mathematical reasoning, Prompt2Skill consistently outperforms the direct prompting baseline, achieving an average improvement of 10.8 across open-source and frontier models.

1 INTRODUCTION

Prompt2Skill addresses the difficulty of producing scalable, model-adapted skills when users lack curated task data. It builds and refines skills from task descriptions, and its experiments report gains over direct prompting across domains and models.

1 INTRODUCTION

  • Expert-authored skills limit scalability (Li et al., 2026; Anthropic, 2025), while SkillOpt requires curated, in-distribution labels (Yang et al., 2026); one-example gains collapse to a data-blind draft (Yang et al., 2026).Source: §1 · pp. 1–2
  • Prompt2Skill derives a task specification and data pool from a task description, then reflectively edits skills using target-model failures without user-supplied data or weight updates.Candidate edits are accepted only after statistically significant paired comparisons on fresh held-out batches.Source: §1 · p. 2
  • 10.8% points over direct prompting on average across four domains and five target models, with no significant regressions.The study also compares against skills authored by a frontier LLM from the task description alone.Source: §1 · p. 2

2 PROBLEM DEFINITION

The problem setting starts from a natural-language task description and, optionally, a few examples, while the target distribution and evaluation metric remain unobserved. It defines skill quality in terms of task performance that can be measured only after optimization.

2 PROBLEM DEFINITION

  • A skill s is an inference-time text artifact; its value is expected metric µ(M_s(x), y) over task distribution D.Source: §2 · p. 2
  • Prompt-to-skill optimization observes a natural-language task description and optional few examples, but neither target distribution D nor metric µ; quality J(s) is testable only after optimization.Source: §2 · p. 2
  • The overview shows the Data Agent producing a specification, seed skill, and validated items before the Skill Agent refines the skill; final selection consults frozen validation once.Source: Figure 2 · p. 3

3 PROMPT2SKILL

PROMPT2SKILL turns a task request into a target-model skill by constructing task-matched proxy data, iteratively refining skills, and selecting an artifact for deployment.

3 PROMPT2SKILL

  • The framework builds a reusable skill for a target model from a task description and optional examples, without updating model parameters.A Data Agent constructs validated items, and a Skill Agent refines an initial skill through rollouts and reflective edits.Source: §3 · p. 3

3.1 TASK SETUP

  • The setup stage derives a task specification and seed skill from the request, using a proxy evaluation criterion to guide data acquisition and validation.The specification captures input structure, task type, language, answer style, and evaluation criterion; the seed skill is generated before inspecting data or model failures.Source: §3.1 · p. 3

3.2 DATA AGENT

  • The Data Agent generates retrieval queries, including search terms derived from a hypothetical suitable dataset, then inspects returned sources and samples for adaptation suitability.Retrieval maps tool-query pairs to candidate sources, and the agent examines schemas when available.Source: §3.2 · p. 4
  • When retrieved records need conversion, the agent plans input-reference fields and transforms usable records; it can instead generate grounded task items from retrieved material or the task specification.If no suitable material is available, generation is guided by the task specification alone.Source: §3.2 · p. 4
  • Validation checks structural integrity, task compatibility, and reference usability against the specification; passing checks indicate practical usability, not reference correctness or target-distribution representativeness.For text sources, the seed-equipped model also screens answerability on a sample, rejecting sources where it fails every sampled item.Source: §3.2 · pp. 4–5

3.3 SKILL AGENT

  • Each round executes the incumbent skill, uses failure feedback to propose edits, and accepts a candidate only when it beats the incumbent on a fresh paired batch under the screening threshold.The specified per-candidate threshold is κ = 1.Source: §3.3 · pp. 5–6

3.4 SELECTION AND EXPORT

  • Accepted skills are saved, then compared on a common frozen selection set whose scores do not feed later refinement; initial skills remain fallback options.Terminal selection can recover an earlier improvement if later edits generalize less well.Source: §3.4 · p. 6

3.5 ALGORITHM

  • Algorithm 1 summarizes the procedure: construct task-matched proxy data, refine an initial skill through execution feedback and paired acceptance, then export the selected artifact for inference.Source: §3.5 · p. 6

4 EXPERIMENT

PROMPT2SKILL is evaluated across four task domains using task descriptions, proxy data, and target models. Its main results outperform direct prompting consistently, while ablations examine initialization, examples, and optimization under matched proxy data.

4 EXPERIMENT

  • The construction procedure retrieves or generates and adapts task items, then uses rollout feedback to propose edits evaluated against the incumbent on a fresh shared batch before selecting and exporting a skill.The algorithm establishes a task specification and proxy evaluator, validates items, and compares incumbent and candidate skills on paired outcomes.Source: §4 · p. 7

4.1 EXPERIMENTAL SETUP

  • The evaluation covers factual question answering, reading comprehension, mathematical reasoning, and spreadsheet manipulation, with task prompts specifying desired behavior and held-out benchmarks assessing the resulting skills.The five target models are Qwen3-8B, Qwen3-32B, Llama-3.2-1B-Instruct, Claude Haiku 4.5, and GPT-5.5; GPT-5.5 also provides reflective feedback.Source: §4.1 · p. 7

4.2 MAIN RESULTS

  • The off-the-shelf skill lowers Llama-3.2-1B’s SQuAD score from 0.207 to 0.048, while PROMPT2SKILL raises it to 0.492.The reported results note that fixed skills can regress and that skill usefulness depends on the target model and task.Source: §4.2 · pp. 7–8
  • 22.0%-29.1% gains for Qwen and commercial models, and 93.5% for Llama-3.2-1B over its three reported benchmarks; PROMPT2SKILL has the highest average relative improvement for every target model.It achieves the highest mean score in 10 model–benchmark pairs, including all five solvers on SQuAD and all four benchmarks for Qwen3-32B.Source: §4.2 · p. 7, Table 1 · p. 8
  • An audit of archived proxy data found no lexical-overlap flags or workbook-hash matches against the target tests.Source details and coverage are reported in Appendix C.Source: §4.2 · p. 8

4.3 ABLATION STUDIES

  • Default and off-the-shelf seed skills yield broadly comparable performance on SearchQA and SQuAD, with neither initialization consistently outperforming the other.The off-the-shelf seed helps Qwen3-8B on SQuAD but reduces performance for GPT-5.5.Source: §4.3 · p. 8
  • Additional demonstrations do not consistently improve performance: direct SQuAD prompting rises from 70.5% with zero examples to 79.0% with four, while PROMPT2SKILL is nearly unchanged on SearchQA from one to four examples.The reported SearchQA scores are 61.1% and 60.9%, respectively; demonstrations are used during skill construction for PROMPT2SKILL but at inference for direct prompting.Source: §4.3 · p. 8
  • With the same proxy data and initial skill, PROMPT2SKILL improves over SkillOpt by 2.93 percentage points on SearchQA EM and 1.20/0.61 points on SQuAD EM/F1.Both methods use Qwen3-8B as solver and GPT-5.5 as editor, with a cap of 7,400 solver evaluations per task.Source: §4.3 · p. 8, Table 2 · p. 8

5 RELATED WORK

Related work covers automatic textual-instruction optimization, reusable skill authoring, and language-model-driven automation of machine-learning workflows.

Skill Optimization

  • APE generates and selects instruction candidates, Pro-TeGi guides prompt edits with natural-language critiques, and GEPA combines execution-feedback reflection with Pareto-based search.These approaches search over textual instructions to improve model behavior.Source: §5 · p. 9
  • Trace2Skill consolidates execution-trajectory lessons into transferable skill directories, while SkillOpt refines skill documents through bounded edits and validation-gated updates.Both extend automated optimization toward reusable agent skills.Source: §5 · p. 9

AutoML

  • Recent AutoML systems automate workflows using dataset and model retrieval, specialized agents, multimodal interpretation, or tree search over executable solutions.Examples discussed include Prompt2Model, AutoML-Agent, MLZero, and AIDE.Source: §5 · p. 9

6 CONCLUSION

The conclusion presents PROMPT2SKILL as a way to build reusable skills for frozen language models from natural-language task descriptions. It reports improvements over direct prompting and off-the-shelf skills across its evaluated benchmarks and models.

6 CONCLUSION

  • PROMPT2SKILL converts natural-language task descriptions into reusable skills for frozen language models without requiring a supplied target-task training set.Source: §6 · p. 9
  • Its Data Agent constructs task-matched proxy data through retrieval, adaptation, and generation, while its Skill Agent refines skills using execution feedback, reflective editing, and statistical screening on fresh batches.Source: §6 · p. 9
  • Across four benchmarks and five target models, PROMPT2SKILL reports average relative improvements of 36.1% over direct prompting and 83.0% over off-the-shelf skills across 19 model–benchmark pairs.Source: §6 · p. 9

7 AI USE DISCLOSURE

The disclosure describes generative-AI use in research and paper preparation, and states that the authors reviewed AI-assisted work and take responsibility for the final content.

7 AI USE DISCLOSURE

  • Generative AI tools were used for brainstorming a research idea, implementing part of the codebase, and formulating a theoretical proof.Source: §7 · p. 9
  • The authors state that generative AI tools were not used for autoresearch, proposing hypotheses, or providing methodology feedback.Source: §7 · p. 9
  • The authors reviewed AI-assisted work, manually surveyed literature for plagiarism risks in generated research ideas, audited LLM-written claims and code, and accept responsibility for the final content.Source: §7 · p. 9

A RELIABLE SELECTION AFTER ADAPTIVE SKILL REFINEMENT

The selection guarantee bounds the target-performance gap between the exported skill and the best retained skill, accounting for proxy mismatch and selection error. Keeping initial and previously accepted skills lets terminal selection preserve an improvement found earlier in refinement, up to the stated error allowance.

A RELIABLE SELECTION AFTER ADAPTIVE SKILL REFINEMENT

  • Theorem 1 bounds the selected skill’s target-performance gap from the best retained skill using proxy mismatch and selection error.The guarantee assumes a finite candidate collection independent of the selection sample and bounded metrics.Source: A RELIABLE SELECTION AFTER ADAPTIVE… · p. 11
  • Adaptive candidate construction is covered when the candidate collection remains independent of the selection sample..Source: A RELIABLE SELECTION AFTER ADAPTIVE… · pp. 11–12
  • Keeping the initial skill as a fallback gives a selection guarantee relative to that skill, subject to the total error allowance.The passage defines ηn as that total error allowance.Source: A RELIABLE SELECTION AFTER ADAPTIVE… · p. 12
  • Terminal selection can preserve an improvement found anywhere along the retained trajectory, up to the stated error allowance.This motivates retaining previously accepted skills rather than exporting only the final incumbent.Source: A RELIABLE SELECTION AFTER ADAPTIVE… · p. 12

B APPLICATION ON NLP AUTOML

The paper explores Prompt2Skill as an alternative to task-specific fine-tuning for NLP AutoML. In an exploratory comparison on two tasks, Prompt2Skill scores are reported as comparable to Prompt2Model.

B APPLICATION ON NLP AUTOML

  • Prompt2Skill creates a reusable skill for a frozen language model, whereas Prompt2Model fine-tunes a task-specific model.Prompt2Model is described as retrieving datasets and models, generating synthetic data, and fine-tuning.Source: B APPLICATION ON NLP AUTOML · p. 12
  • Figure 4 presents the AutoML performance comparison between P2S and P2M.P2S denotes Prompt2Skill and P2M denotes Prompt2Model.Source: Figure 4 · p. 12, B APPLICATION ON NLP AUTOML · p. 12
  • Prompt2Skill scores are reported as comparable to a modified Prompt2Model baseline on Japanese-to-Python generation and temporal expression normalization.The comparison uses Qwen3-32B for Prompt2Skill, a fixed Qwen-32B fine-tuned baseline, and a plain Qwen-32B baseline; the evaluated sets contain 207 and 400 examples, respectively.Source: B APPLICATION ON NLP AUTOML · p. 12

C DATA SOURCES AND TEST-OVERLAP AUDIT

The paper audits retrieved proxy data for possible overlap with evaluation artifacts. It reports no flagged audited inputs or workbook matches, while noting that the findings apply only to the audited artifacts and that recorded sources do not establish item-level provenance.

C DATA SOURCES AND TEST-OVERLAP AUDIT

  • The audit addresses concern that retrieved data could include exact evaluation items; its zero-detection finding is limited to audited artifacts.The table caption explicitly limits zero detected matches to the audited artifacts.Source: C DATA SOURCES AND TEST-OVERLAP AUDIT · p. 12, Table 3 · p. 13
  • Table 3 records configured or discovered proxy sources, but does not verify item-level attribution or retained counts by source.Source cursors indicate attempted fetching, not the number of items retained.Source: C DATA SOURCES AND TEST-OVERLAP AUDIT · p. 13, Table 3 · p. 13
  • No audited text inputs were flagged by the stated exact, containment, and near-text checks.The checks normalize text and apply specified thresholds for containment and similarity.Source: C DATA SOURCES AND TEST-OVERLAP AUDIT · p. 13
  • No instruction flags or workbook-hash matches were found across the spreadsheet audit’s archived pools and held-out task files.The audit covers 945 items in four archived pools and 1,884 proxy workbooks compared with 560 target files from 280 held-out tasks.Source: C DATA SOURCES AND TEST-OVERLAP AUDIT · p. 13

D IMPLEMENTATION DETAILS

Candidate skills are compared with the incumbent on the same fresh batch, using paired correctness outcomes to assess improvement. Acceptance uses a per-candidate screening threshold, while comparisons across candidates and rounds are not adjusted.

D IMPLEMENTATION DETAILS

  • Each candidate and the incumbent are evaluated on the same fresh batch, with candidate-only successes counted as wins and incumbent-only successes as losses.Items where both succeed or both fail contribute neither a win nor a loss.Source: D IMPLEMENTATION DETAILS · p. 13
  • A candidate qualifies only with more wins than losses and a screening statistic of at least κ=1; the qualifying candidate with the largest net gain is accepted.If no candidate qualifies, the incumbent is retained.Source: D IMPLEMENTATION DETAILS · p. 14
  • The κ=1 threshold is not a conventional 5% significance criterion, and comparisons across candidates or rounds are not adjusted.Fresh batches separate acceptance testing from examples used to propose edits.Source: D IMPLEMENTATION DETAILS · p. 14

E.1 BENCHMARK INPUTS AND EVALUATION

The evaluation uses held-out target subsets, distinct from proxy items used for reflection, acceptance, and selection. Each benchmark has its own input and answer-scoring procedure.

E.1 BENCHMARK INPUTS AND EVALUATION

  • The target evaluation subsets are separated from retrieved or generated items used for reflection, acceptance, and final selection.The main experiments construct skills with no supplied benchmark training examples.Source: §E.1 · p. 14
  • SearchQA evaluation uses 1,400 test items with the original search snippets omitted, scoring normalized extracted answers.The input is the question alone; the partition follows prior work (Yang et al., 2026).Source: §E.1 · p. 14
  • SQuAD evaluation uses 1,000 SQuAD v1.1 validation examples and reports exact match against normalized reference spans.Token F1 is recorded but is not the main-table metric.Source: §E.1 · p. 14
  • AIME evaluation reserves 120 problems across two collections and reports the fraction answered correctly, accepting numerically equivalent answers within 10^-6.The items comprise 90 problems from the AI-MO validation collection and 30 from Math-AI’s AIME 2025 collection.Source: §E.1 · p. 14
  • SpreadsheetBench evaluation uses the held-out test partition of a 400-task collection, with 80 training, 40 validation, and 280 test tasks.The partition follows SkillOpt (Yang et al., 2026).Source: §E.1 · p. 14

E.2 MODELS, INFERENCE, AND REFINEMENT BUDGETS

The experiments use five target solvers, with GPT-5.5 providing reflective feedback. The appendix specifies inference budgets and compares Prompt2Skill with direct, fixed off-the-shelf, and LLM-generated skill baselines.

E.2 MODELS, INFERENCE, AND REFINEMENT BUDGETS

  • The target solvers are Qwen3-8B, Qwen3-32B, Llama-3.2-1B-Instruct, Claude Haiku 4.5, and GPT-5.5; GPT-5.5 supplies reflective feedback.The target solver executes candidates and supplies outcomes for acceptance and selection.Source: §E.2 · p. 15
  • Text benchmarks place the skill in the system message and task input in the user message; spreadsheets share tools and a 40-turn interaction budget.Text output limits are 300 tokens for SearchQA, 600 for SQuAD, and 7,000 for AIME.Source: §E.2 · p. 15
  • Standard text loops request 400 initial proxy items, reserve 150 for selection, and compare candidates on fresh acceptance batches of 120; spreadsheet loops use separate smaller task batches.Text reflection batches request 250 initial items and 150 additional items per round; spreadsheet loops request 22 initial tasks, reserve 12, then request eight reflection and ten acceptance tasks per round.Source: §E.2 · p. 15

E.3 BASELINE CONSTRUCTION

  • Direct provides only a minimal task instruction and answer format, with an empty additional skill body for spreadsheets.The application’s input, output, and tool instructions remain in place.Source: §E.3 · p. 15
  • Off-the-shelf baselines use fixed domain-relevant public skills shared across target models, including general-reasoning, math-logic-reasoning, and xlsx-manipulation.The sources are the reasoning-based-skills and Claude Office Skills repositories; the referenced revisions are recorded in the appendix.Source: §E.3 · p. 15
  • The LLM-generated baseline uses one Qwen3-32B-authored skill per domain, created without demonstrations or execution feedback and reused across target models.Each request asks for a self-contained skill of approximately 600–1,000 words, with no benchmark names, off-the-shelf text, or iterative selection.Source: §E.3 · p. 15

E.4 AGGREGATION OF RESULTS

The paper aggregates relative changes using arithmetic means across benchmarks per model and equally weighted model–benchmark pairs overall. The reported aggregates are relative changes, not pooled accuracy gains.

E.4 AGGREGATION OF RESULTS

  • The main table averages relative changes over each target model’s available benchmarks; the reported mean score is indexed by method, model, and benchmark.Llama-3.2-1B contributes three reported text benchmarks rather than four.Source: §E.4 · p. 16
  • The conclusion aggregates 19 available model–benchmark pairs with equal weight, while each model-level average covers its available benchmarks.Llama-3.2-1B has three reported benchmarks; the other models have four.Source: §E.4 · p. 16
  • 36.1% over Direct and 83.0% over Off-the-shelf are averages of relative changes, not percentage-point gains or relative changes in pooled accuracy.Equal weighting allows a pair with a small baseline score to contribute a large relative change.Source: §E.4 · p. 16

F TASK DESCRIPTIONS AND PROMPT TEMPLATES

The appendix records task descriptions, prompt templates, and evaluated skill artifacts. It explains runtime placeholder notation and the setup and baseline prompts represented in those materials.

F TASK DESCRIPTIONS AND PROMPT TEMPLATES

  • The implementation listings use <<...>> for content inserted at runtime rather than literal text sent to the model.Examples include <<skill>> for the current skill and <<score:.0%>> for its score as a whole-number percentage.Source: F TASK DESCRIPTIONS AND PROMPT TEMPLATES · p. 16
  • The prompt-template notation maps setup, retrieval, adaptation, validation, generation, editing, diagnosis, and rephrasing symbols to their implementation roles.θsetup covers automatic text-task setup and spreadsheet specification and consistency checks.Source: F TASK DESCRIPTIONS AND PROMPT TEMPLATES · p. 16

F.1 TASK DESCRIPTIONS

  • Main task descriptions are supplied without input–output demonstrations; the optional-example study appends examples during task setup.Source: §F.1 · p. 16

F.2 AUTOMATIC TEXT PIPELINE

  • The setup request jointly produces a task specification and seed skill, while code implements the selected matching mode.Source: §F.2 · p. 17

F.3 LLM-GENERATED BASELINE

  • The LLM-generated baseline is authored once per domain by Qwen3-32B from a frozen domain brief, without proxy rollouts or target-model feedback.The same factual-QA skill is used for SearchQA and SQuAD, and generated skills are held fixed across target solvers.Source: §F.3 · p. 19

Shared output-format adapter for fixed text skills

F.4 OFF-THE-SHELF SKILLS

  • The off-the-shelf skill listings reproduce the instruction bodies and bundled Markdown references used in evaluation, with packaging metadata omitted.SearchQA and SQuAD share general-reasoning, AIME uses math-logic-reasoning, and SpreadsheetBench uses xlsx-manipulation.Source: §F.4 · p. 20

F.5 FINAL SELECTED P2S SKILLS

  • The appendix lists one final selected skill for each reported model–task pair, including retained initial skills and data-informed spreadsheet drafts.The Llama-3.2-1B–SpreadsheetBench pair is not reported in the main table.Source: §F.5 · p. 39

Related papers

Loading 2609.38593v1…