PaperlayerView exact revision on arXiv

Source-linked AI summary

A Wrong Turn Does Not Ruin the Journey: Deviation-Guided Skill Self-Evolution for LLM Agents

Yichun Feng, Jiawei Wang, Haozhe Sun

arXiv:2609.29154v1cs.AI

TL;DR

Failed agent trajectories can contain useful work before an erroneous suffix, making whole-trajectory skill updates liable to mix effective guidance with behavior needing correction. SkillPivot locates the deviation, uses a teacher continuation and suffix contrast to make localized updates, and reports improved performance with transferable evolved skills.

  • Problem

    Failed trajectories may combine useful exploratory prefixes with erroneous suffixes, so treating each failure as a whole can mix effective behavior with harmful guidance in skill updates.

    Source: Introduction · p. 1

  • Method

    SkillPivot detects a deviation point, continues from the same prefix with a stronger teacher, contrasts the suffixes, and generates regression-checked skill deltas.

    Source: Introduction · pp. 1–2

  • Results

    Evolved skills improve the original corpus and generalize beyond source groups across ToolQA task groups and transfer benchmarks.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    Skill self-evolution can produce reusable procedural corrections rather than instance-specific patches, with transferable gains across task groups and evolution rounds.

    Source: Introduction · p. 2

    SkillPivot assumes an existing high-quality skill corpus rather than generating skills from scratch.

    Source: Problem Formulation · p. 3

Abstract

from arXiv · show

Large language model agents increasingly rely on natural-language skills to solve complex tool-use tasks. However, such tasks often admit multiple valid solution paths, making it inappropriate to improve skills by forcing failed trajectories to match a fixed successful trajectory. Moreover, failed trajectories are rarely entirely wrong: an agent may first collect useful evidence and make meaningful progress, but later deviate into an erroneous suffix. We therefore argue that skill self-evolution should identify where productive problem solving begins to break down, rather than reflect coarsely over the entire failure. Based on this insight, we propose SkillPivot, a deviation-point-guided framework for skill self-evolution. SkillPivot detects the transition from a useful prefix to an erroneous suffix using execution validity, goal progress, and action diversity. A stronger teacher then continues from the same prefix and produces a successful alternative under the same interaction history. By contrasting the student's failed suffix with the teacher's successful suffix, SkillPivot generates localized skill updates while preserving already effective guidance. Experiments on ToolQA, LogicBench, and WildClawBench show that SkillPivot consistently outperforms competing skill-evolution methods, improves multiple agent models, and produces compact, transferable skill updates.

Introduction

SkillPivot addresses failures that mix useful exploration with later erroneous actions by locating where problem solving breaks down and updating skills from that point. Its evaluations report transferable gains from evolved skills.

Introduction

  • Natural-language skill corpora provide reusable procedural guidance but may omit boundary cases, recovery strategies, or tool-specific constraints.Tool-using agents must also decide when to query, construct arguments, interpret observations, verify results, and recover from unexpected feedback.Source: Introduction · p. 1
  • Failed trajectories can contain useful exploration before an erroneous suffix, so whole-trajectory reflection risks noisy, overly broad, or instance-specific updates.Agents may gather relevant evidence or narrow the solution space before deviating.Source: Introduction · p. 1
  • SkillPivot detects where productive problem solving begins to fail, then contrasts the student's failed suffix with a teacher's successful continuation from the same prefix.It generates minimal, conditional, auditable deltas and uses regression review to preserve useful rules and check generalization.Source: Introduction · pp. 1–2
  • Evolved skills improve the original skill corpus and generalize beyond source groups, with transferable gains across task groups and evolution rounds.The evaluation also reports benefits to stronger models and continued evolution over multiple rounds.Source: Introduction · p. 2

Related Work

Related work covers tool-augmented agents, experience-based self-evolution, and evolving skill libraries. SkillPivot distinguishes itself by preserving productive prefixes and using only erroneous suffixes to guide localized updates.

Tool-Augmented Language Agents

  • Tool-augmented agents can select APIs and construct arguments, yet benchmarks document persistent challenges in long-horizon reasoning, grounded execution, and reliable completion.The surveyed benchmarks include ToolQA, API-Bank, WebArena, Mind2Web, AgentBench, and AgentBoard.Source: Tool-Augmented Language Agents · p. 2

Agent Self-Evolution

  • Agent self-evolution ranges from updating model parameters with interaction data to frozen-model methods that store feedback, refine outputs, or extract workflows.Parameter updates can require expensive training, curated data, and repeated safety evaluation.Source: Agent Self-Evolution · p. 2

Self-Evolving Agent Skills

  • Skill-evolution systems create, refine, or retrieve reusable skills, while SkillPivot localizes updates to the suffix after a trajectory shifts from productive exploration to error.Existing methods may use complete trajectories, paired outcomes, or cross-trajectory patterns, risking disruption of useful exploration.Source: Self-Evolving Agent Skills · p. 2

Method

SkillPivot treats skill self-evolution as constrained maintenance: it seeks to repair procedural gaps while preserving stable behavior and useful knowledge. It locates a productive prefix’s deviation point, contrasts student and teacher suffixes, and regression-checks localized updates before deployment.

Problem Formulation

  • Skill self-evolution updates an existing corpus with minimal, conditional, auditable deltas that repair procedural gaps while preserving previously stable behavior.The objective also includes transfer to unseen tasks and bounded regression on tasks already solvable with the current corpus.Source: Problem Formulation · p. 3

Framework Overview

  • SkillPivot locates where useful exploration turns erroneous, then uses a teacher continuation, suffix contrast, and regression gate to produce auditable updates.Preserving the student’s useful prefix focuses updates on the divergent suffix and reduces unnecessary rewriting.Source: Framework Overview · p. 3

Student Rollout

  • The student executes tasks with the current corpus, and SkillPivot retains full interaction traces while an evaluator separates successful from failed trajectories.Successes test whether updates damage working behavior; failures help diagnose missing or underspecified procedural knowledge.Source: Student Rollout · p. 3

Deviation Point Detection

  • Deviation detection scores execution validity, new target-relevant progress, and action diversity, then confirms a low-scoring trough only when later scores remain low.Progress counts only positive increases beyond the best earlier target relevance; low action diversity signals similarity to a prior action.Source: Deviation Point Detection · pp. 3–4

Teacher Rollout

  • A stronger teacher continues from the detected prefix, and its continuation is accepted only if it passes the task evaluator.If the teacher fails, the case is discarded or retried from the same deviation point up to a preset attempt limit.Source: Teacher Rollout · p. 4

Paired Suffix Contrast and Skill Generation

  • For verified paired suffixes, a summarizer diagnoses the student’s mistake and missing guidance before a generator proposes a skill edit, description change, or skip.The generator can choose improve_skill, optimize_description, or skip when evidence is insufficient.Source: Paired Suffix Contrast and Skill… · p. 4

Regression-Gated Skill Update

  • Before deployment, a verifier checks that an update preserves useful rules and adds generalizable guidance, accepting only when both criterion scores exceed θ = 0.5.Rejected updates leave the original skill unchanged; accepted updates enter the corpus for the next evolution round.Source: Regression-Gated Skill Update · pp. 4–5

Experiments

Experiments on ToolQA, LogicBench, and WildClawBench test SkillPivot’s performance, transfer, deviation localization, paired suffix contrast, and review. Results show gains across groups and models, while ablations examine update precision and long-term stability.

Experimental Setup

  • Experiments use Qwen3-32B as the default student and Qwen3.5-397B-A17B for SkillPivot’s LLM-based modules, with task metrics following each benchmark’s official protocol.Students may execute up to 20 steps; teacher continuation is limited to 10 steps and may be retried up to three times.Source: Experimental Setup · p. 5

RQ1: Do Evolved Skills Improve Performance and Generalize Beyond Their Source Groups?

  • SkillPivot-g5 raises mean success rate from 53.18% for Non-evo to 59.36%, consistently outperforming Non-evo across all test groups, with gains extending beyond groups used during evolution.Source: RQ1: Do Evolved Skills Improve… · p. 5

RQ2: Does SkillPivot Outperform Other Skill Evolution Methods?

  • SkillPivot achieves the highest accuracy with relatively compact modified skills, while competing methods’ broader rewrites or local edits provide more limited improvement.Its updates turn observed failures into recovery protocols such as alternative date formats, query reformulation, and evidence verification.Source: RQ2: Does SkillPivot Outperform Other… · pp. 5–6, Figure 3 · p. 6
  • Evolved skills improve performance across all five evaluated models, including cases where weaker models become competitive with stronger models using the original corpus.The experiment evaluates models on the same six task groups.Source: Figure 4 · p. 6, RQ3: Do Evolved Skills Also Improve… · p. 6

RQ4: Are Detected Deviation Points Reliable and Driven by Meaningful Signals?

  • Random intervention slightly improves over From Scratch, but low point agreement and more steps and invalid calls indicate that random positions often miss the useful-to-erroneous transition.LLM-selected points can have high agreement yet lower continuation success than SkillPivot; a weaker self-teacher also reduces continuation success.Source: Table 2 · p. 6, RQ4: Are Detected Deviation Points… · p. 6

RQ5: Does Paired Suffix Contrast Produce Better Skill?

  • Paired suffix contrast achieves the best downstream accuracy and shortest updates, whereas whole-trajectory reflection performs worse than the original skill corpus.Failure-only reflection and teacher-only imitation improve accuracy, but their gains are limited.Source: RQ5: Does Paired Suffix Contrast… · pp. 6–7, Figure 3 · p. 6

RQ6: Does Review Improve Long-Term Stability of Skill Evolution?

  • No-review sometimes yields higher short-term accuracy, while review is framed as a long-term constraint against unreliable deltas, overfitting, and negative transfer.The authors caution that review may discard potentially useful deltas early in self-evolution.Source: RQ6: Does Review Improve Long-Term… · p. 7

RQ7: Does SkillPivot Transfer to Other Benchmarks?

Conclusion

SkillPivot addresses skill self-evolution by locating where useful problem solving turns into error, then using teacher continuation and suffix contrast to update skills. The reported experiments find transferable improvements across task groups, evolution rounds, agent backbones, and task environments.

Conclusion

  • Failed trajectories often contain useful exploratory prefixes before deviating into localized erroneous suffixes.The paper frames this as a reason to improve skills without overwriting previously useful procedural knowledge.Source: Conclusion · p. 7
  • SkillPivot detects where failures begin, uses teacher continuation to construct successful suffixes, and generates minimal, conditional, regression-checked skill deltas through suffix contrast.Source: Conclusion · p. 7
  • Experiments report reusable procedural knowledge and transferable improvements across task groups, evolution rounds, agent backbones, and task environments.The paper concludes that maintaining and evolving external skill corpora is a promising path toward more reliable and continuously improving agents.Source: Conclusion · p. 7
Loading 2609.29154v1…