Source-linked AI summary
SkillCraft: Can LLM Agents Learn to Use Tools Skillfully?
Shiqi Chen, Jingze Gai, Ruochen Zhou, Jinghan Zhang, Tongyao Zhu, Junlong Li, Kangrui Wang, Zihan Wang, Zhengyu Chen, Klara Kaleb, Ning Miao, Siyang Gao, Cong Lu, Manling Li, Junxian He, Yee Whye Teh
TL;DR
Existing benchmarks offer limited evidence about whether agents can acquire and reuse compositional skills across recurring long-horizon tool-use tasks. SkillCraft introduces a benchmark and protocol for automatically composing, verifying, caching, and reusing Skills, finding up to 80% lower token usage and strong links between composition efficiency and task success.
Problem
Existing benchmarks mainly test single-task success with fixed atomic tools, providing limited evidence about agents’ ability to accumulate, abstract, and reuse compositional skills across tasks.
Method
SkillCraft combines recurring, compositional tasks with Skill Mode, which automatically composes successful atomic tool sequences into executable Skills and caches them for reuse.
Results
Skill reuse reduces token usage by up to 80%, and efficiency gains strongly correlate with task success; high-quality skills also transfer across tasks and models with over 95% execution success.
Takeaways & Limitations
Compositional skill acquisition is a crucial capability for robust long-horizon tool use, while shallow, well-tested skill libraries are more reliable than deeply nested hierarchies.
Takeaways & Limitations
Hierarchical compositions are less reliable because nested dependencies accumulate errors and increase debugging overhead.
Abstract
from arXiv · showhide
Real-world tool-using agents operate over long-horizon workflows with recurring structure and diverse demands, where effective behavior requires not only invoking atomic tools but also abstracting, and reusing higher-level tool compositions. However, existing benchmarks mainly measure instance-level success under static tool sets, offering limited insight into agents' ability to acquire such reusable skills. We address this gap by introducing SkillCraft, a benchmark explicitly stress-test agent ability to form and reuse higher-level tool compositions, where we call Skills. SkillCraft features realistic, highly compositional tool-use scenarios with difficulty scaled along both quantitative and structural dimensions, designed to elicit skill abstraction and cross-task reuse. We further propose a lightweight evaluation protocol that enables agents to auto-compose atomic tools into executable Skills, cache and reuse them inside and across tasks, thereby improving efficiency while accumulating a persistent library of reusable skills. Evaluating state-of-the-art agents on SkillCraft, we observe substantial efficiency gains, with token usage reduced by up to 80% by skill saving and reuse. Moreover, success rate strongly correlates with tool composition ability at test time, underscoring compositional skill acquisition as a core capability.
1. Introduction
SkillCraft targets the undermeasured ability to abstract and reuse compositional tool skills across recurring long-horizon tasks. It introduces a benchmark and Skill Mode protocol, finding substantial efficiency gains and evidence that composition quality relates to success.
- Motivation: Real-world tool-using agents need reusable higher-level tool compositions for recurring structures rather than repeatedly executing isolated atomic calls.These workflows include repeated search–analyze–summarize patterns across documents, repositories, and web services.
- Gap: Existing benchmarks provide limited signal about whether agents can accumulate, abstract, and reuse compositional skills across tasks.They typically fix both the toolset and model at deployment and test single-task solving with given tools.
- Contribution: SkillCraft embeds repeated substructures within tasks to evaluate reusable tool compositions within and across tasks under a fixed budget.The benchmark is designed to require agents to identify and reuse compositions multiple times.
- Benchmark Design: SkillCraft scales task difficulty by increasing both the number of entities and the tool calls required per subtask.Its construction combines adapted benchmark tasks, handcrafted web API tasks, and local file and data-processing tasks.
- Protocol: Skill Mode automatically discovers, caches, and invokes successful tool-call sequences as reusable skills during test-time solving.This lets agents expand their action space through discovery and reuse while accumulating capabilities.
- Results: Up to 80% token usage reduction is reported with Skill Mode, while composition-related efficiency gains strongly correlate with task success.The evaluation includes Gemini-2.5-Pro, Claude-Sonnet-4.5, and GPT-5.1.
- Composition Quality: Deeper automatically generated hierarchies are less reliable than shallow, well-tested skill libraries because nesting amplifies error propagation and debugging overhead.High-quality compositions nevertheless transfer across difficulty levels and models with consistently high execution success.
2. SkillCraft
SkillCraft is designed as a long-horizon, compositional benchmark that makes skill discovery and reuse useful and measurable. Its task repository is built from seed tasks and scaled along entity count and subtask complexity.
- Motivation: Current single-episode benchmarks do not adequately measure tool composition because they test isolated task solving with fixed atomic tools.SkillCraft instead uses long-horizon tasks with repetitive structures to encourage higher-level skill reuse.
- Design Principles: SkillCraft tasks require multi-step, multi-tool reasoning and recurrent structure so discovered skills can be reused across contexts.These properties distinguish composing and generalizing skills from executing isolated calls.
- Construction: The benchmark is constructed through exploration of existing tasks, seed-task creation, and systematic difficulty scaling.Seed tasks combine adapted benchmark tasks, handcrafted web API tasks, and local processing tasks.
- Construction: 126 tasks across 6 difficulty levels are produced from 21 seed tasks by increasing entity number and subtask complexity.Stage 1 explores more than 60 tasks to identify task-design principles.
- Skill Abstraction: Skills consolidate frequently co-occurring atomic tool chains into executable units that represent data flow, control logic, and iteration compactly.The benchmark asks whether models can abstract recurring chains into reusable, code-based Skills and improve efficiency and success.
3. SkillCraft Protocol
The SkillCraft protocol measures whether agents can compose multi-step tool chains into reusable code-based Skills and retrieve them during later execution. It validates candidate skills before storing them in a persistent library.
- Protocol Goals: The protocol quantifies composition and reuse by letting models create higher-level code-based Skills and reuse them within and across tasks.Reuse can accumulate efficiency gains over repeated interactions.
- Skill Library: A Skill Library stores verified executable Skills and metadata through a lightweight MCP interface, making reuse observable through API calls.The interface defines the evaluation boundary for reuse attempts, successful execution, and failure handling.
- Verification: Candidate Skills undergo syntax validation, runtime error reporting, and post-execution quality detection before acceptance.The verifier rejects syntactically invalid or low-quality Skills and returns debugging information for failed executions.
- Execution Loop: When a matching Skill is unavailable or fails, the agent explores with atomic tools and records a successful sequence for composition.The sequence is parameterized and intermediate results are passed through code variables.
- Execution Loop: Only Skills that pass controlled execution and verification are saved for reliable future reuse.Validation occurs in a Coding Environment through a unified tool interface.
4. Evaluation
SkillCraft evaluates agents under consistent task, tool, and environment settings, measuring success, skill behavior, and efficiency. Skill Mode generally improves success and reduces token and cost usage, while composition ability correlates with both success and efficiency gains.
- SkillCraft evaluates state-of-the-art models under the same task prompts, tool endpoints, and environment constraints.
- Success uses accuracy with a task counted as successful when its final score reaches at least 90%.Exec Rate measures successful Skill executions, while Reusing Rate measures average invocations per saved Skill.
- Skill Mode consistently improves success and efficiency across models, sharply reducing average token usage and cost while typically decreasing tool calls.Some models use more conversation turns because cached-skill selection and verification add lightweight steps.
- Skill execution rate correlates with task success (r=0.65), while efficiency savings correlate with baseline success (r=0.53 for Turns Saved and success rate).
- Models differ in composition behavior: stronger models invoke and reuse skills selectively, whereas weaker models tend to over-apply composition.
5. What is a good tool composition?
The study examines whether deeper and transferable tool compositions are better. Deep hierarchies often reduce reliability through error propagation, whereas shallow tested libraries and high-quality transferred skills provide stronger practical benefits.
- 5.1. Is Deeper Composition Always Better?: Hierarchical Mode permits recursive skill invocation, unlike standard Skill Mode, which restricts each skill to atomic tool calls.The experiments allow nesting to a maximum depth of 10.
- 5.1. Is Deeper Composition Always Better?: Hierarchical failures can propagate from null low-level outputs to medium-level TypeErrors and complete high-level task failure.
- 5.1. Is Deeper Composition Always Better?: Deeper composition is not consistently beneficial: GPT-5.2 success falls from 90% with flat Skill to 79% with Hierarchy.Hierarchy can also weaken token savings, from 0.26M for flat Skill to 0.60M for Hierarchy in the reported comparison.
- 5.1. Is Deeper Composition Always Better?: High hierarchical Exec rates of 95–99% do not necessarily translate into higher task success.The results favor shallow, well-tested skill libraries as more reliable and cost-effective than deep automatically generated hierarchies.
- 5.2. Cross-Task Skill Generalization: Cross-task transfer reuses a static Skill cache across difficulty levels without modification or accumulation during target-task execution.Claude-4.5-Sonnet raises Easy→Hard success from 95% to 100% and reduces tokens from 1.92M to 1.56M.
- 5.3. Cross-Model Skill Generalization: High-quality skills can transfer across executor models, with Claude-created skills achieving 100% success across all four target models.
- 5.3. Cross-Model Skill Generalization: Skill quality determines efficiency outcomes: Claude-created skills save 54–81% of tokens, whereas Minimax-created skills range from −48% to +18%.Claude achieves 69.2% savings with Gemini’s skills versus Gemini’s 14.8% self-reuse, showing that executor capability does not replace sound skill design.
6. Conclusion
SkillCraft and Skill Mode measure whether agents can acquire, cache, and reuse compositional tool skills rather than only execute isolated calls. The results show large efficiency gains, strong transfer across tasks and models, and reduced reliability for hierarchical compositions.
- SkillCraft contains 126 tasks with recurring substructures, while Skill Mode lets agents auto-compose, cache, and reuse tool sequences.
- Skill reuse reduces token usage by up to 80% while maintaining or improving success rates.
- Efficiency gains strongly correlate with model intelligence, and skills generalize across tasks and models with execution rates above 95%.
- Hierarchical compositions are less reliable because errors accumulate across nested steps.
- Skill Mode improves efficiency by reducing repeated argument passing and amortizing planning through reusable tool sequences.It can also cache minimal task-relevant fields instead of repeatedly injecting verbose raw tool outputs into context.
B.3. Implementation details
The implementation provides executable macro-like Skills with persistent storage, inspection, and reuse operations. Evaluation uses partial-credit success thresholds and efficiency comparisons restricted to tasks where both modes succeed, under fixed execution limits.
- Skill Storage and Execution: Each saved Skill is stored with executable script code, parameters, documentation, and execution statistics in a workspace cache.
- Primitive tools: The implementation supports saving, listing, running, retrieving, and inspecting saved macros for repetitive workflows.Saved macros accept named arguments and can call external tools inside their scripts.
- Execution interface: Macro execution returns either a success status with results or a failed status with an error.
- Execution interface: Batch tasks with repetitive logic can replace sequential tool calls with macro executions.
- Evaluation Protocol: Tasks use partial-credit criteria, with success defined as achieving at least 90% of the maximum score.Typical criteria include file existence, JSON validity, data completeness, and field-level accuracy.
- Evaluation Protocol: Efficiency metrics are computed only on tasks where both baseline and Skill modes succeed, while each task is capped at 150 turns and a 60-minute timeout.
C.1. Task API Sources
SkillCraft draws on publicly available APIs spanning diverse application domains and requiring structured, multi-step interactions. Its task families wrap distinct API endpoints to evaluate skill composition and reuse.
- 21 task families span six application domains, from entertainment and gaming to science and development.
- Each task family implements 5–7 tool functions that wrap distinct API endpoints.
- The APIs are publicly available REST endpoints requiring structured multi-step interactions.
D.1. Results by task difficulty
Skill reuse becomes more valuable as tasks grow harder: reuse frequency, efficiency gains, and success improvements generally increase on challenging tasks, while execution remains mostly robust. Benefits differ by model capacity and baseline behavior.
- 2.3–3.0× average skill reuse on Easy tasks rises to 3.0–4.9× on Hard tasks across models.
- 77–78% token reduction on Hard tasks for Claude 4.5 Sonnet and GPT-5.2 exceeds their reported Easy-task reductions of 62–79%.
- 70% tool-call reduction for Gemini 3 Pro and 68% for GPT-5.2 on Hard tasks exceed their Easy-task reductions of 29% and 38%, respectively.
- +29 percentage points for DeepSeek-V3.2-EXP on Hard tasks, from 42% to 71%, exceeds its +8-point Easy-task improvement.
- Frontier models gain efficiency more than accuracy, with Claude achieving 72% average token reduction and GPT-5.2 achieving 78%.
- 66–100% skill-execution success across difficulty levels indicates generally reliable transfer, although Kimi-K2-Thinking and DeepSeek-R1 record lower rates.
D.2. Direct execution mode
Direct Exec Mode removes parameterization and save–execute overhead by running single-use hardcoded scripts. It can deliver larger efficiency gains than reusable Skills, but the generalization of Skills carries non-trivial overhead when reuse is unlikely.
- Skill Mode creates parameterized scripts and executes them with arguments, enabling reuse across similar tasks but adding abstraction and workflow overhead.
- Direct Exec Mode writes single-use scripts with hardcoded values and discards them after execution.
- For Claude-4.5-Sonnet, Direct Exec cuts tokens from 1.72M to 0.16M while retaining 96% success, whereas Skill uses 0.34M tokens with 90% success.
- For GPT-5.2, Direct Exec cuts tokens from 1.18M to 0.06M, reduces turns from 24.5 to 4.5, and lowers success from 94% to 85%.
- When tasks are isolated and patterns are unlikely to recur, Direct Exec Mode provides a more efficient alternative.
D.3. Trajectory analysis
Trajectory analysis contrasts models that judge when abstraction is worthwhile with models whose skill creation and execution fail. Successful skill use can combine higher efficiency with task completion, while poor compositions incur repair and fallback costs.
- Successful Reuse: Claude’s cocktail trajectory completes five skill executions successfully and writes the menu in 213,222 tokens.
- Behavioral Divergence: Claude skips skill abstraction for a 9-call easy task but creates and reuses a skill for a harder 15-call task.
- Skill Creation and Execution Failures: DeepSeek’s cat-breed skill omits required fields, producing warnings and requiring 8 additional repair operations.
- Skill Creation and Execution Failures: DeepSeek’s cocktail skill attempts fail with syntax errors, forcing fallback to manual API calls and ultimately failing the task.
- Implications: Claude uses 213K tokens versus DeepSeek’s 1.14M on the hard cocktail task, a reported 5.3× token-savings difference.