Source-linked AI summary
SkillCommit: Evolving Agent Skills through Behaviorally Validated Scope Expansion
Yu He, Weikai Yang
TL;DR
Existing skill-evolution methods can retain narrow patches or merge related experiences without behavioral validation, limiting compact reusable skill libraries. SkillCommit consolidates experiences through replay-based behavioral compatibility checks and source-preserving abstraction, and consistently improves performance across three benchmarks while transferring across model scales and families.
Problem
Existing skill-evolution methods retain narrow instance-specific patches, causing growing memory and retrieval costs as experience accumulates.
Method
SkillCommit uses replay-based behavioral compatibility checks and source-preserving consolidation to build a compact hierarchical skill library while retaining validated behavior.
Results
SkillCommit consistently outperforms no-skill settings and strong baselines across RuleArena, OpenExempt, and KOR-Bench, with skills transferring across model scales and families.
Takeaways & Limitations
The results support behaviorally validated scope expansion as a way to evolve compact reusable skills while preserving previously validated behavior.
Takeaways & Limitations
Skill induction assumes privileged teacher feedback that is never exposed during execution and discards candidates containing final answers or instance-specific solutions.
Abstract
from arXiv · showhide
Large language model (LLM) agents can continually improve without parameter updates by converting historical experience into reusable procedural knowledge. However, existing methods often consolidate experience based on semantic similarity or LLM judgments, which may merge superficially related but behaviorally incompatible strategies and thereby degrade performance. To address the issue, we propose SkillCommit, an online skill evolution framework that continuously transforms experience into a hierarchical library of reusable skills. Each new experience is initially preserved as an instance-specific patch, retaining the behavior validated in its local context. As related skills accumulate, SkillCommit abstracts those sharing a common behavioral mechanism into higher-level skills. Specifically, for each incoming skill, embedding-based retrieval first identifies candidate related skills. Cross-instance replay and an LLM-based mechanism check determine whether these skills transfer across cases and share a common underlying mechanism. Candidates that pass both checks are abstracted into a higher-level skill and committed only if it preserves the validated behavior of all constituent skills. Experiments on RuleArena, OpenExempt and KOR-Bench demonstrate that SkillCommit consistently improves agent performance across diverse domains. Moreover, the learned skills transfer across model scales and families, enabling cross-model experience transfer.
Introduction
SkillCommit addresses the limits of surface-similarity-based skill evolution by expanding skills only when their behavior transfers and their shared mechanism is validated. It builds an online hierarchical skill library that preserves validated local behavior while improving performance across benchmarks and model settings.
- Motivation: LLM agents need reusable domain knowledge because base-model pretraining omits specialized procedural conventions, operational constraints, and rare boundary cases.Manually eliciting and formalizing this knowledge is costly, motivating automatic skill evolution from trajectories and feedback.
- Problem: Skill evolution must evaluate behavioral compatibility and shared relational structure rather than relying only on surface similarity.Semantic retrieval can select behaviorally incompatible patches or miss patches with a common behavioral mechanism.
- Method: SkillCommit derives locally validated instance-specific patches, groups those with transferable behavior and a common mechanism, then commits higher-level skills only when constituent behavior is preserved.Its three stages are Feedback-Guided Patch Induction, Behavioral Compatibility Grouping, and Source-Preserving Consolidation.
- Evaluation: SkillCommit consistently outperforms no-skill settings and strong baselines across evaluated subsets of RuleArena, OpenExempt, and KOR-Bench.The framework is evaluated as an online approach for building compact skill libraries while retaining previously validated behavior.
- Transferability: Induced skills transfer effectively across model scales and retain utility when transferred across model families.These experiments examine whether the learned procedural knowledge generalizes beyond the model and setting in which it was induced.
Related Work
Prior work frames agent skills as reusable procedural artifacts and studies how external knowledge is organized, associated, and transformed from experience. More recent systems extend this direction toward continually evolving playbooks rather than retaining only individual memories.
- LLM Agents and Skill Systems: Agent skills extend LLM agents with task instructions, operational knowledge, and executable workflows beyond one-off prompting and atomic tool use.Prior work also examines external-knowledge organization through dynamically linked memory notes and recombined knowledge.
- Skill Acquisition and Self-Evolution: Experience-based systems transform execution feedback into reusable textual knowledge, including reflections, transferable insights, and strategies distilled from successful and failed experience.Examples include Reflexion, ExpeL, and ReasoningBank.
- Skill Acquisition and Self-Evolution: More recent systems continually refine an evolving playbook instead of retaining only individual memories.This marks a shift from isolated experience retention toward ongoing skill self-evolution.
SkillCommit
SkillCommit evolves a hierarchical skill library through behaviorally validated scope expansion. It induces source-validated patches, groups them using replay and mechanism assessment, and commits abstractions only when full-scope replay preserves validated behavior.
- Feedback-Guided Patch Induction: SkillCommit first converts failed executions into procedural patches and admits them only after the frozen executor successfully replays the source instance.Teacher feedback contrasts failed and successful behavior to generate guidance, while solution-leaking candidates are discarded before validation.
- Behavioral Compatibility Grouping: Semantic similarity retrieves bounded candidate sets, but behavioral replay and shared-mechanism assessment determine whether patches can support a common higher-level skill.This guards against semantically similar patches with incompatible strategies and differently expressed patches sharing the same procedure.
- Behavioral Compatibility Grouping: For existing skills, SkillCommit retains the top R candidates, with R = 4, then checks cross-instance transfer before proposing scope expansion.A candidate must pass the compatibility threshold and mechanism assessment before the expanded skill proceeds to source replay.
- Behavioral Compatibility Grouping: If no existing skill is compatible, SkillCommit forms a new skill from residual source-validated patches organized into behaviorally compatible groups.Eligible groups include the incoming patch, contain at least m members, and use an anchor procedure that transfers across the group by default.
- Source-Preserving Consolidation: A candidate abstraction is committed only when replay succeeds on every proposed source instance; otherwise, SkillCommit narrows or refines the candidate scope.Candidate versions and replay outcomes are retained for auditing, with refinement continuing for at most L attempts.
Experiments
Across three reasoning benchmarks, SkillCommit improves performance over No Skill and competing experience-learning methods, while its skills transfer across model scales and families. Ablations show that source-replayed patch induction, behavioral compatibility grouping, and source-preserving consolidation each contribute to performance.
- Main results: SkillCommit improves over No Skill and achieves the best result on all 18 task configuration pairs, raising the unweighted mean from 50.80% to 78.15%.Table 1 reports strict accuracy for RuleArena and KOR-Bench and macro-F1 for OpenExempt.
- Instance-level effects: Behavioral validation prevents negative transfer: competing methods repair baseline failures but also overturn cases that the baseline solves.ExpeL repairs 24 Airline and 14 NBA failures, while also overturning eight Airline and three NBA baseline cases; ReasoningBank and ACE likewise help and hurt cases.
- In-family cross-scale transfer: Qwen3.6-35B-A3B-authored skills raise Qwen3.5-9B’s average from 48.97 to 75.25 and improve every task, although transfer is not lossless.Matched execution remains higher on Airline, NBA, and Cipher, while the smaller user performs slightly better on Logic.
- Cross-family and cross-scale transfer: GPT-5.4-authored skills raise Qwen3.5-9B’s average from 48.97 to 69.75 and improve every task, demonstrating transfer across model-family and scale boundaries.The passage states that this transfer is weaker than in-family cross-scale transfer.
- Ablation analysis: Removing Feedback-Guided Patch Induction reduces performance from 71.11% to 53.33%, the largest drop among the three leave-one-stage-out conditions.Source-validated instance patches deployed directly reach 64.44%, attributing the gain to validating and refining proposed patches rather than merely generating memory text.
- Ablation analysis: Removing Behavioral Compatibility Grouping or Source-Preserving Consolidation lowers performance from 71.11% to 66.67%, showing that compatibility screening and full-scope replay provide distinct safeguards.Compatibility Grouping alone reaches 60.00% on first-pass patches, while consolidation helps ensure abstractions preserve validated source behavior.
Conclusion
SkillCommit evolves accumulated agent experience into a compact, hierarchical skill library while preserving previously validated behavior. Its behavioral-evidence-based consolidation yields consistent improvements over no-skill baselines on RuleArena, OpenExempt, and KOR-Bench.
- Framework: SkillCommit transforms accumulated agent experience into a compact, hierarchical skill library without sacrificing previously validated behavior.The framework is designed for online skill evolution.
- Behavioral validation: Broader behavioral claims are admitted only after surviving replay, rather than through semantic similarity or model judgment alone.This grounds skill consolidation in behavioral evidence.
- Evaluation: SkillCommit consistently improves over no-skill baselines on RuleArena, OpenExempt, and KOR-Bench.The conclusion reports improvements across all three evaluated benchmarks.