PaperlayerView exact revision on arXiv

Source-linked AI summary

Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills

Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, Guanjun Jiang

arXiv:2603.25158v5cs.AI

TL;DR

LLM agents need high-quality domain skills, but human authoring does not scale and parametric drafts often miss operational pitfalls. Trace2Skill jointly analyzes labeled trajectories, proposes patches in parallel, and consolidates them into portable skill directories. Experiments report transfer across models, benchmarks, and domains, while the authors identify patch-selection cost as a remaining limitation.

  • Problem

    Manual skill authoring does not scale, while parametric-only skill drafts often lack concrete failure modes, workarounds, and operational details from execution traces.

  • Method

    Trace2Skill collects labeled trajectories, proposes trajectory-level patches in parallel, and hierarchically consolidates them into one portable skill without updating agent parameters.

  • Results

    Trace2Skill skills improve performance and transfer across model scales, model families, benchmarks, task domains, and out-of-distribution settings.

  • Takeaways & Limitations

    The resulting skills are portable files used directly at inference, with recurring trajectory lessons compressed into reusable procedures rather than requiring test-time retrieval.

  • Takeaways & Limitations

    Patch selection remains costly because reliable validation requires a large representative set and Bayesian optimization materializes and scores every candidate subset.

Abstract

from arXiv · show

Large Language Model (LLM) agents increasingly rely on domain-specific skills, yet manually authoring such skills does not scale, and skills generated purely from parametric knowledge often miss critical operational pitfalls. We introduce Trace2Skill, a framework that consolidates broad execution trajectories in parallel into a unified skill directory through inductive reasoning over agent experience. Trace2Skill supports both deepening existing human-written skills and creating useful skills from weak LLM-generated drafts. Experiments demonstrate the effectiveness of Trace2Skill across diverse domains, including office workflows, math reasoning, and vision QA. Importantly, the evolved skills are not merely memorized artifacts of the trajectories used to create them: they often transfer across model scales, across model families, and to out-of-distribution settings. For example, skills evolved from Qwen3.5-35B trajectories improve a Qwen3.5-122B agent by up to $57.65$ percentage points on WikiTableQuestions. Further analyses show that Trace2Skill outperforms sequential skill editing and ReasoningBank-style retrieval memories, compresses recurring failures and workarounds into standard operating procedures (SoPs), and yields portable skills that can be reused without parameter updates or test-time retrieval.

1. Introduction

Trace2Skill addresses the difficulty of authoring robust domain skills by jointly analyzing execution traces and consolidating recurring lessons into portable skill files. Experiments report transfer across models, benchmarks, and domains, alongside advantages over sequential editing and retrieval memories.

  • Motivation and approach: Human-written skills can help one agent while hurting another, and parametric drafts often omit concrete failure modes, workarounds, and operational details.These limitations motivate grounding skill evolution in actual execution experience.
  • Motivation and approach: Trace2Skill turns many execution traces into one portable skill directory through parallel analysis and hierarchical consolidation.The framework supports both deepening human-written skills and creating skills from weak LLM-generated drafts.
  • Evidence and analysis: Trace2Skill skills improve performance while transferring across model scales, families, benchmarks, and out-of-distribution tasks.Reported applications include spreadsheets, math reasoning, DocVQA, and document workflow skills.
  • Evidence and analysis: Parallel consolidation is faster and generally stronger than order-dependent editing, while one distilled skill outperforms ReasoningBank-style episodic retrieval.Qualitative analyses report recurring failures and workarounds coalescing into reusable standard operating procedures.
  • Contributions: The framework provides automatic skill creation and deepening, with empirical evidence that trajectory-grounded evolution transfers across LLM scales, families, and out-of-distribution tasks.The contribution includes both the framework and broad transfer evidence.

2. Trace2Skill

Trace2Skill formalizes skills as portable human-readable directories and evolves them from labeled execution trajectories without updating agent parameters. Its pipeline proposes patches in parallel and hierarchically merges them into a coherent skill update.

  • 2.2. Trajectory Analysis: Trace2Skill collects trajectories from a fixed agent, labels them as failures or successes, and proposes patches independently from disjoint trajectory batches.Error and success analysts produce separate patch pools from their respective trajectories.
  • 2.1. Skill and Problem Formalization: A skill is a human-readable directory containing a root SKILL.md and auxiliary references, scripts, or assets.The root stores broadly applicable procedural knowledge, while auxiliary files provide tools or lower-frequency details.
  • 2.1. Skill and Problem Formalization: Skill evolution constructs a new skill from evolving-set trajectories while keeping the fixed agent parameters unchanged.The formal objective evaluates the evolved skill on a disjoint test set.
  • 2.2. Trajectory Analysis: The failure analyst can inspect artifacts, compare outputs with ground truth, and validate fixes, while unexplained failures are excluded from the failure patch pool.The success analyst uses a single-pass workflow to identify reusable behavior patterns.
  • 2.4. Stage 3: Patch Consolidation: Patches are hierarchically merged into one coherent update that deduplicates insights, resolves conflicts, and preserves non-overlapping information.The merge uses deterministic guardrails before applying diff-style edits to the initial skill.
  • 2.5. Skill Deepening and Creation: Skill deepening starts from a human-written skill, whereas skill creation starts from an LLM-generated parametric draft; both use trace-induced patches.The two modes cover settings with and without expert-written skills.

3. Experiments

Trace2Skill is evaluated across spreadsheet, math, and document visual-question-answering settings, using held-out and cross-model tests to assess whether trajectory-derived skills generalize. The experiments compare skill deepening and creation against no-skill, human-written, and parametric baselines.

  • Spreadsheet Experiments: Experiments evaluate spreadsheet skills across held-out, full-benchmark, and out-of-distribution table tasks, using three seeds and official benchmark criteria.SpreadsheetBench-Verified is split into 200 evolution and 200 held-out test samples; WikiTableQuestions and HiTab provide OOD transfer tests.
  • Spreadsheet Experiments: Trace2Skill compares No Skill, Human-Written, Parametric, and +Error, +Success, and +Combined variants for skill deepening and creation.Deepening starts from Human-Written, while Creation starts from Parametric; Qwen3.5-122B-A10B and Qwen3.5-35B-A3B serve as authors and users.
  • Spreadsheet Results: Human-Written is not reliably portable across model scales, while Parametric remains close to No Skill, motivating trajectory-grounded deepening and creation.The two regimes test whether manual skills can become more transferable and whether useful skills can be built from weak seeds.
  • Spreadsheet Results: Both Deepening and Creation produce generalizable skills across author models, user models, and task families, with 35B Deepening +Combined achieving the best absolute Avg.The gains include in-distribution spreadsheet performance, other model scales, and OOD table tasks; +Combined usually gives the largest Avg improvement.
  • Model-Family Generalization: Gemma-4-31B-it and GPT-5.5-high both improve from their own traces and benefit from Qwen3.5-authored skills, demonstrating cross-family generalization.The comparison uses each model’s self-deepened xlsx skill alongside Qwen3.5-122B/35B +Combined skills.
  • Math Reasoning: Trace2Skill improves both held-out DAPO and OOD AIME mathematics, while +Error transfers cleanly between 122B and 35B users.Skills are created from 400 DAPO questions and evaluated on 100 disjoint DAPO questions plus AIME 2026 avg@8 over 30 problems.
  • Visual Question Answering: All evolved DocVQA skills improve over No Skill, with +Combined producing the clearest same-model gains and positive cross-model transfer.Evaluation reports ANLS and Accuracy, where Accuracy is the percentage with ANLS ≥0.5.

4. Analysis

Trace2Skill’s core design choices—parallel consolidation, holistic skills, and agentic diagnosis—improve reuse, efficiency, and patch quality. Further analyses show that recurring patches form reusable procedures, while broader applications transfer gains across held-out domains.

  • Parallel Consolidation: Parallel consolidation preserves or improves quality while reducing SpreadsheetBench production time to about 3 minutes, versus 15 minutes for Seq-B=4 and 60 minutes for Seq-B=1.It outperforms sequential baselines on all 122B metrics and 35B Vrf, with Seq-B=1 modestly higher only on 35B Soft and Hard.
  • Holistic Skill vs. Retrieval: Trace2Skill’s consolidated skill is consistently better than ReasoningBank retrieval on same-model Deepening.The comparison uses the same trajectory pool; OOD retrieval evaluations are excluded because the queries are too semantically distant.
  • Agentic Error Analysis: Agentic error analysis achieves higher Avg than single-call log-only patch generation in most settings.The agentic analyst inspects artifacts, queries ground truth, and validates fixes, whereas +Error LLM reads only execution logs.
  • SoPs Learned: Many-to-one merging converts recurring trajectory lessons into standard operating procedures rather than one-off tricks.Dominant SoPs include recalculating and reading back formulas, verifying target cells, and deleting rows in a corruption-safe order; quirks remain in on-demand references.
  • Patch Composition: Greedy patch selection rises quickly but plateaus below full aggregation, while Bayesian optimization can improve selected metrics at substantial validation cost.Greedy selection misses patch complementarity and interference; BO must materialize and validate candidate subsets.
  • Broader Application: Evolution on source traces improves performance on separate held-out targets in PDF, PPTX, and DOCX workflows.Pass rates rise from 76.9% to 85.3% for PDF, 72.5% to 88.8% for PPTX, and 79.7% to 87.5% for DOCX.

5. Related Work

Related work treats skills as reusable procedural artifacts and agent experience as reusable memory, while concurrent methods evolve skills, policies, or agent loops online. Trace2Skill instead studies static skill directories distilled from one model’s traces for transfer across models and tasks.

  • Agent Skills: Agent skills package task procedures, domain knowledge, and operational guardrails, but mismatched or stale skills can distract or hurt agents.The paper frames Trace2Skill’s skills as portable standard operating procedures.
  • Experience Memory for Agent Self-Evolution: Experience-memory methods store reflections, behaviors, or procedural lessons and retrieve them at test time.These approaches share the premise that execution experience contains reusable structure.
  • Skill and Policy Evolution: Skill and policy evolution methods refine skills or co-evolve policies, skills, and agent loops online.Trace2Skill asks whether one model’s traces can instead be compressed into a static directory that benefits other models and tasks.

6. Conclusion

Trace2Skill distills execution traces into portable skills through parallel patch proposal and one-shot consolidation. The resulting skills transfer across model scales, families, and out-of-distribution tasks, with applicability across multiple domains.

  • Conclusion: Trace2Skill uses parallel analyst sub-agents to propose targeted patches from disjoint trajectory batches, then consolidates them into one declarative skill directory.The resulting skill is portable and used directly at inference.
  • Conclusion: Skills distilled from a single model’s traces transfer across model scales, model families, and out-of-distribution tasks.The framework also shows strong applicability across various domains.

Limitations

Trace2Skill applies all consolidated patches by default, while higher-quality subset selection through Bayesian optimization remains costly. More extensive combinatorial patch-selection analysis is left for future work.

  • Limitations: Bayesian optimization can select higher-quality patch subsets, but each candidate requires materializing and scoring a new skill, making computation grow quickly.The selection signal also depends on a large, representative validation set; the reported sampled validation uses only 32 questions.
  • Limitations: Trace2Skill therefore applies all consolidated patches by default and leaves more thorough combinatorial selection to future work.

Ethics Statement

The study uses publicly available datasets and research-permitted artifacts, with human-authored qualitative annotations. The authors report no additional ethical risks and supervised use of AI coding and grammar tools.

  • Publicly available datasets and research-licensed artifacts are used, avoiding data privacy issues.
  • Human annotations in qualitative analyses were conducted by the authors.
  • The authors identify no other ethical risks associated with the study.
  • AI coding and grammar tools were used under careful human supervision, with unit tests intended to prevent hacking or unaligned behavior.

A. Experimental Details

Experiments use held-out evaluation, multiple seeds, specified open models, and substantial GPU resources. The ablations compare parallel consolidation with sequential editing and retrieval memories across spreadsheet, math, and document tasks, while cross-model transfer reaches 68.00% Vrf in one setting.

  • Experimental setup: Three random seeds, 41, 42, and 43, are used unless otherwise noted, with experiments running on 8-GPU A100 nodes.Total compute is roughly 20,000 GPU hours.
  • Evaluation protocol: SpreadsheetBench Verified uses 200 evolution problems and 200 held-out test problems, excluding evolution examples from scoring.Additional OOD evaluations use WikiTableQuestions and HiTab conversions.
  • Models: The main author/user models are Qwen3.5-122B-A10B and Qwen3.5-35B-A3B, served with vLLM in instruct or thinking modes.
  • Efficiency: ≈3 sequential LLM-call rounds enable parallel consolidation, compared with 60 minutes for Seq-B=1 and 15 minutes for the faster sequential baseline.The parallel process takes about 3 minutes and uses W=128 workers with N≈70 lessons.
  • Ablations: Parallel consolidation is best or tied for best on all reported math metrics while also having the lowest skill-generation wall time.
  • Cross-model transfer: 68.00% Vrf is achieved by GPT-5.5-high-authored Deepening +Combined on Qwen3.5-122B, exceeding No Skill at 27.67% and Human-Written at 48.33%.This is a cross-model trace-induction setting using Qwen traces.

C.1. Agentic vs. LLM Error Analysis

Agentic error analysis improves patch grounding by inspecting artifacts, validating fixes, and rejecting log-level false positives. Consolidation distills recurring spreadsheet failures into high-support SoPs, while iterative greedy selection can regress unrelated tasks.

  • Error analysis: Agentic analysis and single-call +Error LLM strongly agree on only 4 of 33 shared cases, while 18 cases show clear disagreement.
  • Error analysis: Artifact access lets the agentic analyst compare outputs with ground truth and validate candidate fixes, unlike a log-only analyzer.
  • Error analysis: 57% versus 14%: +Error LLM attributes parse errors as the primary cause at a much higher rate than the agentic analyst.The agentic loop can reject explanations contradicted by artifact evaluation.
  • Patch quality: Verified failure mechanisms produce domain-general guardrails that transfer more reliably than patches encoding surface log symptoms.Examples include wrong ranges, stale formulas, corrupted structure, type conversion, and missing read-back verification.
  • Selective aggregation: Greedy patch selection plateaus below full aggregation because edits can regress unrelated correct tasks and repeatedly target semantically overlapping failures.
  • Learned SoPs: 55.1% of patches encode formula recalculation and write-back verification, the most common learned spreadsheet procedure.The workflow runs recalc.py after formula writes and reopens with data_only=True.
  • Learned SoPs: 54.8% of patches recommend openpyxl over pandas.to_excel(), while 42.7% require explicit read-back verification.Structural-edit safety appears in 16.4% of patches, and niche quirks are routed to 13 reference files.
  • Comparative analysis: Agentic +Error analysis has the higher average in three of four author–mode blocks and wins most SpreadsheetBench columns.The exception is 122B-authored Creation, where +Error LLM is slightly higher on Avg because of OOD table benchmarks.

G. Head-to-Head Comparison with Concurrent Skill-Evolution Systems

The head-to-head comparison evaluates complete concurrent systems under a shared open-model protocol, while the authors caution that these scores conflate design choices with model, harness, and engineering differences. Trace2Skill records the highest reported Vrf among the compared systems.

  • Scope and interpretation: The head-to-head comparison complements, rather than replaces, apples-to-apples tests because whole-system scores confound design attribution.The systems also target different goals, models, harnesses, and domains, so results are interpreted conservatively.
  • Comparison design: The appendix reproduces XSkill, EvoSkill, and SkillGen pipelines under one shared protocol rather than isolating individual design choices.
  • Shared protocol: All systems use Qwen3.5-122B-A10B, the same evolution slice, the held-out 200:400 test slice, and official instance accuracy.
  • Concurrent systems: 23.0 Vrf is reached by the XSkill reproduction using lexical-overlap retrieval of its distilled action-level experiences.Replacing the lexical retriever with Qwen3-Embedding-0.6B yields 20.0 Vrf.
  • Concurrent systems: 59.5 Vrf is reached by EvoSkill in the React-style local harness, versus 33.5 with the Claude Code-style harness.

I.2.3. Example Trajectory-Level Patch (Error Analyst Output)

An error analyst converts a trajectory showing unsafe row deletion into a structured skill patch. The patch is later consolidated with 52 related patches into a “Structural-edit safety” standard operating procedure.

  • The proposed patch adds a SKILL.md section on row-manipulation safety, including exact range-boundary enforcement before operations.
  • This patch, together with 52 patches addressing related structural-edit failures, is consolidated into the “Structural-edit safety” SoP.
  • A single failure trajectory revealed deletion outside the specified range and confusion between partially empty data rows and empty padding rows.
  • The analyst identified missing guidance on row and column manipulation and enforcement of range boundaries.
  • The patch requires checking both target-column emptiness and explicit range membership before deleting a row.
Loading 2603.25158v5…