Source-linked AI summary
SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories
Zhuoyun Yu, Xin Xie, Wuguannan Yao, Chenxi Wang, Lei Liang, Xiang Qi, Shumin Deng
TL;DR
Long-horizon agents need more precise skill adaptation because trajectory-level feedback can misattribute failures. SkillAdaptor localizes actionable fault steps and updates only responsible skills, improving performance across WebShop, PinchBench, and Claw-Eval, including a largest gain of +1.5 on PinchBench Avg Score%.
Problem
Trajectory-level skill adaptation can diffuse failure signals across unrelated steps and skills, causing overly broad or misdirected revisions.
Method
SkillAdaptor performs training-free step-level attribution by localizing the first actionable fault, diagnosing the relevant skill deficiency, and refining only that skill while freezing the base model.
Results
+1.5 points on PinchBench Avg Score% was the largest reported gain among matched skill baselines, while performance improved across WebShop, PinchBench, and Claw-Eval.
Takeaways & Limitations
Step-level failure attribution supports more precise, stable, and lightweight training-free skill adaptation compatible with existing agent frameworks.
Takeaways & Limitations
The method may weaken under sparse or delayed feedback, missing external interfaces, longer-term deployment, or broader distribution shifts.
Abstract
from arXiv · showhide
Large language model (LLM) agents increasingly rely on reusable external skills to solve long-horizon interactive tasks. Existing training-free skill adaptation pipelines usually update skills from full trajectories or session-level feedback, which makes failure attribution coarse and often produces unstable or overly broad revisions. We propose SkillAdaptor, a training-free step-level skill adaptation framework with explicit failure attribution, and it can plug into OpenClaw-class agent harnesses. Given a failed trajectory, SkillAdaptor identifies a first actionable fault step, links responsibility to candidate skills, and applies targeted updates under explicit acceptance checks while keeping the backbone frozen. We evaluate on WebShop, PinchBench, and Claw-Eval with Kimi-K2.5, GLM-5, and GPT-5.2. SkillAdaptor improves over no-skill and skill-adaptation baselines on all three suites, with the largest single-metric improvements of +1.5 points on PinchBench Avg Score%, +1.8 on Claw-Eval Avg Score, and +1.7 on WebShop success rate. These results indicate that step-level attribution supports more stable and auditable training-free skill maintenance\footnote{The code will be released at https://github.com/zjunlp/SkillAdaptor.}.
1 Introduction
Existing training-free methods often revise skills from trajectory-level outcomes, which can diffuse failure signals across correct steps and unrelated skills. SkillAdaptor instead attributes failures at the step level, refines only the responsible skill, and improves results across WebShop, PinchBench, and Claw-Eval.
- Limitations of prior adaptation: Most existing training-free methods adapt skills from completed trajectories or session-level summaries, making trajectory-level outcomes the main unit of revision.These methods include memory distillation, workflow refinement, retrospective feedback, and iterative skill evolution.
- Failure attribution: Outcome-level adaptation can diffuse failure signals across correct intermediate decisions and unrelated skills, producing overly broad or misdirected revisions.The problem is especially severe when an early error invalidates many subsequent steps.
- SkillAdaptor: SkillAdaptor shifts adaptation from trajectory-level reflection to step-level attribution by locating the first actionable fault and diagnosing the relevant skill deficiency.It refines only the deficient skill while leaving irrelevant skills unchanged, and operates as a modular post-execution plug-in.
- Evaluation: SkillAdaptor is evaluated on WebShop, PinchBench, and Claw-Eval, where it consistently improves over training-free adaptation baselines.The evaluation supports attributing failures to the specific steps and skills that caused them rather than expanding skill libraries or reflecting over entire trajectories.
2 Preliminaries
The framework models an LLM agent acting in an environment, with its execution trajectory serving as the sole evidence for fault analysis and skill updates. It keeps the base model frozen and seeks to improve expected task success by revising or appending textual skills alone.
- 2 Preliminaries: The agent solves sampled tasks by acting in an environment and producing an execution trajectory.The trajectory records the agent’s actions and observations across steps.
- 2 Preliminaries: The trajectory is the only evidence available for subsequent fault analysis and skill updates.At each step, the trajectory contains the action a_t and observation o_t.
- 2 Preliminaries: The skill collection contains textual records with a title, principle, and applicability conditions, while the base model remains frozen.Only the skill collection K is revised or appended across iterations.
- 2 Preliminaries: The objective is to improve expected success over Q through K alone, keeping the procedure training-free and attributing improvements to skill updates.Run(q, K) executes the agent conditioned on a retrieved subset of K, with success defined by the benchmark.
3 Method
SkillAdaptor builds an initially empty skill collection from successful trajectories, retrieves skills during execution, and adapts them from failures through attribution, modification, and qualification. Updates are targeted, redundancy-filtered, and accepted only when they do not reduce execution feedback.
- Skill Initialization: The framework starts without external skills, archives successful backbone-only trajectories, and distills them into the initial collection K0.Subsequent executions retrieve skills from K, and failed trajectories enter adaptation.
- Skill Injection: Candidate skills are retrieved by Qwen3-Embedding-8B cosine similarity, truncated to top-kret, and reranked by the backbone LLM.Each skill includes a title, applicability condition, and behavioral principle.
- Skill Adaptation: Attribution identifies the earliest accountable fault step and estimates which retrieved skills are responsible for failure.The Linker assigns responsibility weights and routes failures to REVISE for misleading skills or GENERATE for missing coverage.
- Skill Adaptation: Modification rewrites the highest-weighted skill for REVISE or synthesizes a skill from the localized fault context for GENERATE.Generated skills whose semantic similarity exceeds θdup are discarded before insertion.
- Skill Adaptation: Candidate updates are accepted only when ∆≥ 0 after comparing re-executions under the current and candidate collections.Otherwise, the original collection is retained unchanged; adaptation runs for up to 10 rounds or stops after 3 unchanged rounds, then K∗ is frozen.
4 Experiments
Experiments across three backbones and three benchmarks show that SKILLADAPTOR consistently improves matched baselines, with largest gains reaching +2.3 on WebShop Score. Ablations, case studies, and round-wise analyses support step-level attribution as a stabilizing mechanism, especially when failures are locally identifiable.
- Experimental setup: Experiments use Kimi-K2.5, GLM-5, and GPT-5.2 across PinchBench, Claw-Eval, and WebShop, comparing six baselines with three independent runs per setting.All methods share backbone models and benchmark settings; tables report mean and spread.
- Main results: +2.3 on WebShop Score (GLM-5) is the largest reported gain against the strongest matched skill baseline, alongside +1.7 percentage points on WebShop Succ% (Kimi-K2.5).Other largest gains are +1.5 on PinchBench Avg Score% (GLM-5) and +1.8 on Claw-Eval Avg Score (Kimi-K2.5).
- Component ablation: Removing Localizer and Linker lowers WebShop success to 28.6% and Claw-Eval Avg Score to 74.5, indicating that step-level attribution targets effective revisions.Removing Qualifier increases spread and lowers WebShop success to 26.3% with ±2.6, allowing unstable skills into retrieval.
- Case study: In a PinchBench CSV and Excel Data Summarization task, SKILLADAPTOR localizes failure to interaction step six and links it to the fourth procedure of the injected skill.This converts a broad trajectory failure into a concrete, directly editable skill span grounded in execution feedback.
- Failure localization: SKILLADAPTOR is most effective when failures expose localizable intermediate decisions, while gains are more moderate when success depends on external systems or persistent state.Data and Code tasks provide especially clear execution traces for revising and reusing faulty skill spans.
- Adaptation rounds: 75% of WebShop accepted writes occur in rounds 1–2, compared with 64% for Claw-Eval and 65% for PinchBench, while later rounds provide diminishing refinements.Accepted writes reach zero after round 5 for WebShop and Claw-Eval and after round 6 for PinchBench; later limitations increasingly reflect backbone capability.
5 Conclusion
SKILLADAPTOR is a lightweight, training-free framework for updating external agent skills. Across WebShop, PinchBench, and Claw-Eval, it improves over no-skill and prior skill-adaptation baselines through step-level failure attribution.
- Conclusion: SKILLADAPTOR is a training-free adaptor framework that updates external agent skills.The framework is designed for agent skill maintenance without additional training.
- Conclusion: Across WebShop, PinchBench, and Claw-Eval, SKILLADAPTOR consistently improves over no-skill and prior skill-adaptation baselines.The conclusion reports consistent improvements across all three evaluation suites.
- Conclusion: Step-level failure attribution enables more precise and efficient adaptation than trajectory-level updates alone while remaining lightweight and framework-compatible.The approach is compatible with existing agent frameworks and uses step-level rather than trajectory-level updates.
Limitations · A Related Work
The paper identifies conditions that limit SkillAdaptor’s effectiveness and situates it among work on long-horizon agents, self-adapting skills, orchestration, and skill reuse. It distinguishes its lightweight, step-level failure attribution and fine-grained skill evolution from trajectory-level or infrastructure-focused approaches.
- Limitations: SkillAdaptor is most effective when failures expose observable intermediate signals and required tool dependencies are available.Performance may weaken under sparse or delayed feedback and missing external interfaces.
- Limitations: The current evaluation covers three public benchmarks, while longer-term deployment and broader distribution-shift evaluation remain future directions.
- A Related Work: Long-horizon agent research targets interactive tasks such as WebShop, software terminals, and tool-augmented workflows, integrating reasoning, action, memory, and tool use.
- A Related Work: Recent self-adapting skill methods extract and refine skills from agent experiences through context-aware retrieval, failure analysis, distribution-shift discovery, and iterative generation and verification.
- A Related Work: Most existing adaptation approaches aggregate failures at the trajectory or session level, limiting precise localization of the execution step responsible for downstream errors.
- A Related Work: Some adaptation methods rely on multi-agent coordination or iterative verification, whereas SkillAdaptor uses a lightweight single-agent framework for direct and efficient skill adaptation.
- A Related Work: Harness and skill-reuse studies improve execution robustness, workflow organization, controlled tool interaction, and skill effectiveness across tasks.
- A Related Work: Unlike these studies, SkillAdaptor addresses post-failure skill updates through step-level failure attribution and fine-grained skill evolution during long-horizon interaction.
B Experimental Details · B.1 Implementation Details
SkillAdaptor uses a frozen, consistent backbone for all adaptation stages while representing skills in OpenClaw-style SKILL.md files. Its implementation combines embedding-based retrieval, duplicate filtering, bounded iterative updates, deterministic execution, and trajectory-only feedback under unchanged benchmark infrastructure.
- B.1 Implementation Details: The same backbone LLM performs initialization rollout, failure localization, responsibility attribution, revision, generation, and qualification.This configuration is maintained for the current experiment setting.
- B.1 Implementation Details: Skills use the OpenClaw-style SKILL.md format, combining lightweight structured metadata, natural-language behavioral instructions, and optional execution guidance.
- B.1 Implementation Details: 0.9 temperature is used for skill extraction, generation, and revision to encourage diverse reasoning traces and broader coverage of failure-induced behaviors.
- B.1 Implementation Details: 0.45 cosine similarity filters retrieval candidates, after which the top-10 skills are reranked by the backbone LLM using the current task description.Qwen3-Embedding-8B provides dense retrieval over the skill collection.
- B.1 Implementation Details: θdup = 0.95 is the duplicate threshold; candidate updates exceeding it are discarded before insertion to suppress redundancy accumulation.The same embedding model is used to compare new or revised skills against the existing collection.
- B.1 Implementation Details: 10 rounds bound skill adaptation, with early stopping after the skill collection remains unchanged for 3 consecutive rounds.
- B.1 Implementation Details: 0 execution temperature is used across benchmarks, while benchmark-specific environments, runtimes, task configurations, sandboxes, tools, grading, and aggregation remain unchanged.WebShop uses its official Gym-based environment and evaluator; PinchBench and Claw-Eval use their provided OpenClaw runtime and task configurations.
- B.1 Implementation Details: The adaptor observes only execution trajectories and success or failure signals, not evaluator logic, reward decomposition, or benchmark-specific scoring details.Experiments vary only the backbone LLM or adaptation method while keeping execution and evaluation infrastructure fixed.
B.2 Benchmarks
The paper evaluates SkillAdaptor on three interactive-agent benchmarks spanning web shopping, long-horizon tool use, and open-ended workflows. Each benchmark uses specified task splits and reports standardized evaluation metrics.
- WebShop: WebShop evaluates multi-step online shopping through 1,624 training instances and a 200-instruction test set, reporting task score and success rate.Methods use identical environment settings on the test set.
- PinchBench: PinchBench evaluates long-horizon tool-mediated agents across coding, DevOps, spreadsheets, logs, meetings, email, calendars, and web research, reporting Avg Score%.It is an OpenClaw benchmark covering diverse tool-mediated workflows.
- Claw-Eval: Claw-Eval evaluates open-ended agents on office, operations, finance, ticket triage, and multi-turn advisory tasks with simulated users.The benchmark includes General tasks and Multi-turn tasks under its standard aggregation protocol.
- Claw-Eval: Claw-Eval reports Avg Score and Pass@3 using the Overall Dimension of its official leaderboard.The Overall Dimension follows the benchmark’s standard aggregation protocol.
B.3 Baselines
The baselines share SkillAdaptor’s training-free base LLM and experiment settings, while differing in how they inject skills, memory, workflows, lessons, or iterative refinement. The comparison includes no-skill, memory, workflow, critique, skill-bank, and OpenClaw-class agent systems.
- Shared setup: All baselines use the same training-free base LLM and experiment settings as SkillAdaptor, with external skills or memory applied according to each baseline’s module.This controls the comparison while preserving each baseline’s intended architecture.
- No-skill baseline: The Base model injects no external skills or experience and evaluates tasks with Kimi-K2.5, GLM-5, and GPT-5.2.It provides the no-skill reference condition across the evaluated models.
- Memory and workflow baselines: A-Mem uses agentic memory, letting the model choose what to store, retrieve, and revise in a persistent store.The implementation follows the public recipe, including benchmark prompts and harness assumptions.
- Memory and workflow baselines: AWM distills reusable workflows from past trajectories and retrieves them on new tasks with lightweight string matching.The benchmark wiring preserves identical tool calls, step limits, and scoring.
- Lesson-based baseline: ExpeL extracts qualitative lessons from high- versus low-reward rollouts and injects them as free-text hints at test time.Its artifacts function as retrospective critiques rather than explicit workflow templates.
- Skill-bank and harness baselines: EvoSkill discovers skills iteratively from traces, while OpenSpace provides skill libraries and refinement loops on an OpenClaw-class harness.EvoSkill results are reported for WebShop; OpenSpace runs use the same PinchBench and Claw-Eval channels as the base model.
C Input Tokens and Interaction Steps · D Prompt Templates
SkillAdaptor trades larger per-step prompts for fewer interaction steps, especially on PinchBench and Claw-Eval, while WebShop remains in a lower token range. Its prompt templates operationalize deterministic failure localization, responsibility attribution, targeted revision, new-skill generation, and qualification gating.
- C Input Tokens and Interaction Steps: Retrieving and injecting skills inflate input tokens by adding procedural descriptions to each task prompt.The effect is more pronounced on PinchBench and Claw-Eval because tool usage and multi-step reasoning repeatedly expose skill-related context.
- C Input Tokens and Interaction Steps: SkillAdaptor increases prompt size while reducing mean interaction steps on PinchBench and Claw-Eval relative to Base.WebShop remains in a lower token band with smaller absolute shifts.
- C Input Tokens and Interaction Steps: For Kimi-K2.5, mean input tokens rise from 13.9k to 16.9k on PinchBench, 11.5k to 13.9k on Claw-Eval, and 1.5k to 2.7k on WebShop.The WebShop increase remains within a lower token range.
- C Input Tokens and Interaction Steps: Mean interaction steps decrease from 10.4 to 9.8 on PinchBench, 7.3 to 5.8 on Claw-Eval, and 18.9 to 15.5 on WebShop.The passage attributes this pattern to shifting decision burden from multi-step interaction toward in-context reasoning.
- C Input Tokens and Interaction Steps: Token usage and interaction length should be interpreted jointly because fewer steps do not directly imply lower overall computational cost.The reduction reflects redistribution from environmental interaction to richer per-step context conditioning.
- D Prompt Templates: The prompt-template appendix provides compact, complete skeletons for failure localization, responsibility linking, skill revision, skill generation, and qualification gating.Templates are written in English for reproducibility in the training-free SKILLADAPTOR adaptor pipeline.
- D Prompt Templates: The shared constraint template requires deterministic, evidence-grounded, directly actionable outputs from an expert agent debugger.A shared safety and quality constraint layer applies across all stages.
- D Prompt Templates: The templates localize one primary fault step, assign responsibility weights to active skill candidates, perform minimal targeted edits, or generate a reusable skill when revision is unsafe.The revision policy preserves working parts and prefers additive edits.