Source-linked AI summary
SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills
Yingtie Lei, Zhongwei Wan, Jiankun Zhang, Samiul Alam, Zixuan Zhong, Peizhou Huang, Xin Wang, Jingxuan Zhang, Donghao Zhou, Yunta Hsieh, Zhihao Dou, Hui Shen, Yan Xu, Dimitrios Dimitriadis, Tuo Zhang, Mi Zhang
TL;DR
Agents can reuse episodic trajectories, but whether they can distill them into durable procedural skills remains unclear. SkillEvolBench tests this transition across related deployment tasks and finds that agents usually adapt locally rather than form reliable reusable skills, with raw trajectories often outperforming distilled skills.
Problem
It remains unclear whether agents can distill noisy episodic experience into compact procedural skills that transfer beyond the original episode.
Method
SkillEvolBench evaluates skill evolution through role-conditioned task families, verifier-backed updates, frozen deployment tasks, replay, and raw-trajectory controls.
Results
Across ten model configurations and three harnesses, agents often adapt locally but rarely form reliable reusable skills, while episodic traces often outperform distilled skills.
Takeaways & Limitations
The central challenge is selective procedural abstraction that preserves invocation, verification, robustness, and composition while filtering episode-specific noise.
Takeaways & Limitations
The evaluation uses observable outcomes, process checks, rewards, and diagnostics rather than hidden model state or private chain-of-thought.
Abstract
from arXiv · showhide
Large language model (LLM) agents accumulate rich episodic trajectories while solving real-world tasks, but it remains unclear whether such experience can be distilled into reusable procedural skills. We introduce SkillEvolBench, a diagnostic benchmark for evaluating this step from experience reuse to skill formation. It contains 180 tasks across six real-world agent environments, organized into role-conditioned task families with shared latent procedures. Agents learn from acquisition tasks, update an external skill library using compacted trajectories and verifier feedback, and then face frozen deployment tasks testing context shift, adversarial shortcuts, and composition. By comparing self-generated and curated-start skill evolution against no-skill and raw-trajectory controls, SkillEvolBench separates procedural abstraction from base capability, curated prior knowledge, and direct reuse of episodic traces. Across ten model configurations and three agent harnesses, we find that current agents often adapt locally but rarely form robust reusable skills. Skill-based conditions can improve acquisition or replay, and individual models sometimes gain on specific deployment axes, but these gains are unstable under frozen deployment. Raw-trajectory reuse frequently outperforms distilled skills, suggesting that current abstraction procedures discard contextual and procedural cues that remain useful for future tasks. Capacity and cost analyses further show that writing more skills or larger Tier-3 resource libraries is not sufficient: additional updates can improve coverage while introducing episode-specific drift and procedural clutter. These findings position SkillEvolBench as a testbed for measuring when one-off experience becomes durable procedural knowledge rather than task-local memory.
1 Introduction
SkillEvolBench tests whether agents can transform verifier-grounded episodic trajectories into reusable procedural skills before facing harder related tasks. Its controls and frozen evaluations distinguish skill formation and deployment transfer from local recovery and direct episodic reuse.
- Benchmark design: SkillEvolBench converts each learning attempt into a host-side abstraction step that uses an episodic trajectory and verifier feedback to update a persistent skill library.The Skill Author may write, refine, or skip a library update, and the resulting library is frozen before evaluation.
- Evaluation design: The benchmark compares Self-Generated and Curated-Start evolution with No-Skill and Raw-Trajectory controls to separate skill formation, curated priors, and direct episode reuse.Original learning tasks are replayed with the final frozen library to distinguish local recovery from deployment transfer.
- Benchmark design: Role-conditioned task arcs test acquisition, replay, context shift, adversarial robustness, and multi-skill composition rather than a single aggregate success rate.Frozen evaluation tasks specifically probe context shift, adversarial shortcuts, and composition after learning tasks expose limits of naive procedures.
- Findings: Across ten model configurations and three agent harnesses, agents show local procedural adaptation but rarely form reliably reusable skills that transfer to frozen deployment tasks.Skill-based agents can improve acquisition or replay, yet those gains do not consistently transfer to deployment.
- Findings: Raw-trajectory reuse often outperforms distilled skills, exposing a lossy abstraction bottleneck in which proceduralization discards useful contextual and procedural cues.The comparison indicates that episodic traces can be used more effectively than the skills derived from them.
2 Related Work
Related work spans realistic interactive-agent benchmarks, experience reuse without parameter updates, and explicit procedural skill artifacts. Prior studies show that trajectories, reflections, and curated skills can aid task solving, while self-generated skills provide limited average gains.
- From static tasks to realistic agent work: Interactive benchmarks now evaluate agents on multi-step web navigation, software engineering, computer use, tool policies, and workplace workflows.Examples include Mind2Web, MindWeb, WebArena, SWE-bench, OSWorld, τ-bench, and TheAgentCompany.
- Reusing agent experience: Experience-reuse methods improve agents without parameter updates by storing feedback, extracting lessons, retrieving trajectories, or inducing reusable workflows.Reflexion, ExpeL, Synapse, and Agent Workflow Memory represent distinct strategies for reusing prior solving evidence.
- Agent Skills and skill evolution: Agent Skills package procedural knowledge into loadable artifacts, while curated skills improve performance and cold-start self-generated skills provide limited average gains.Related work also studies LLM-generated tools, executable code-skill libraries, skill discovery, memory skills, self-evolution, and trajectory-derived skill libraries.
3 SkillEvolBench
SkillEvolBench tests whether agents transform repeated experience into reusable procedural skills across controlled task families and deployment conditions. Its design combines role-conditioned progression, frozen skill-library evaluation, curated incomplete skills, and layered verification.
- Benchmark structure: SkillEvolBench contains 180 tasks across six real-world agent environments, organized into five procedural families per environment and six role-conditioned tasks per family.Families share underlying procedures while varying failure modes, surface forms, and deployment conditions.
- Role-conditioned progression: Each family progresses from canonical, enriched, and variant acquisition tasks to context-shift, adversarial, and composition deployment tasks.The progression tests whether skills transfer, resist shortcut solutions, and combine with other skills.
- Curated skills: Each family includes a gap-exposed curated skill that supports the canonical task while leaving enriched, variant, adversarial, and compositional cases unresolved.The curated skill is manually refined for controlled granularity and is neither an oracle solution nor copied from a task instance.
- Verification: Verification combines public basic-contract tests, hidden edge-case and distribution-shift tests, and process checks for brittle strategies.Process verifiers inspect traces and artifacts for hard-coded constants, swallowed exceptions, skipped validation, and incomplete repairs.
- Evaluation protocol: Agents update an environment-specific skill library from acquisition trajectories and verifier feedback, then face deployment tasks after the library is frozen.This protocol makes deployment success depend on prior skill formation rather than test-time repair.
4 Skill Evolution Protocol
SkillEvolBench evolves environment-scoped, family-conditioned skill libraries from acquisition evidence and verifier feedback, then freezes them for deployment evaluation. The protocol separates local replay and recovery from transfer to context-shift, adversarial, and composition tasks.
- Episode and library lifecycle: Each episode starts with a fresh environment-scoped library, runs acquisition tasks, compacts execution artifacts with verifier feedback, and freezes the resulting library for deployment.Libraries remain environment-scoped: skills can be visible across later families within an environment but never transfer across environments.
- Starting skill conditions: Three initialization conditions compare experience-based self-generation, fixed zero-shot metadata skills, and gap-exposed curated skills that cover base procedures while leaving missing sub-capabilities.Experience-based induction begins without a family skill and requires execution evidence plus verifier feedback; zero-shot skills are generated before execution and never revised.
- Acquisition and authoring: Acquisition proceeds through canonical, enriched, and variant roles, with family-local authoring based only on same-family skills and acquisition history.The Skill Author emits structured library edits only after eligible acquisition attempts, while the task-solving agent may read the broader environment-level library.
- Frozen deployment: During frozen deployment, agents may read and apply accumulated skills but cannot create, revise, retire, or otherwise modify the library.Deployment uses context-shift, adversarial, and composition roles, testing transfer without adaptation on the evaluation instance.
- Evaluation metrics: The verifier reports outcome, process, overall, and binary success measures, while ESR evaluates frozen deployment and decomposes into CSSR, ARSR, and CompSR.LSR measures acquisition success with updates allowed, and RSR measures replay success on the original acquisition tasks after freezing, capturing local recovery rather than transfer.
5 Experimental Setup and Results
SkillEvolBench evaluates reusable skill formation through acquisition, replay, and frozen deployment, revealing local adaptation but unreliable transfer, abstraction loss, and environment-dependent performance. Additional resources and memory mechanisms can expand coverage or improve selected metrics, but greater capacity and cost do not reliably improve deployment success.
- Overall findings: Reusable skill formation requires gains on acquisition or replay plus frozen deployment, yet current agents show local procedural adaptation without reliable transfer.Deployment tests assess invocation, robustness, and composition rather than replay alone.
- Overall findings: Under Self-Generated Experience, Claude Opus 4.6 improves LSR by 5.5 pp and RSR by 10.0 pp, but decreases ESR, CSSR, and CompSR.This demonstrates that local gains can coexist with deployment failures.
- Capacity and resource bundling: Forced Tier-3 authoring generally increases persisted files, but larger libraries do not reliably improve frozen deployment and can introduce procedural clutter.Claude Opus 4.6 improves from 37.8% to 40.0% ESR under SelfGen-Always+Tier3, while Gemini 3 Flash drops from 35.6% to 27.8%.
- Environment variation: Environment structure strongly affects outcomes: the easiest-to-hardest baseline gaps are 67.3 percentage points for LSR and 42.1 points for ESR.E2 Tool/API has mean CSSR of 84.7%, whereas E3 Data has mean ARSR of 69.8% but mean CompSR of 4.5%.
- Abstraction versus replay: Raw-Trajectory is the strongest baseline by mean RSR, ESR, ARSR, and CompSR, averaging 48.2%, 37.6%, 44.7%, and 25.7%, respectively.The comparison indicates that distilled skills often discard contextual and procedural cues retained in episodic traces.
- Cost and efficiency: Most memory variants increase cost per attempted task relative to No-Skill, while only a smaller subset produces corresponding ESR gains.The cost–success frontier varies substantially by model and variant.
6 Conclusion
SkillEvolBench evaluates whether agents transform episodic task experience into reusable procedural skills rather than merely reuse experience at inference time. Across broad experiments, current agents adapt locally but rarely form reliable skills, revealing selective procedural abstraction as the central challenge.
- Benchmark design: SkillEvolBench tests the transition from experience reuse to skill formation using role-conditioned families, verifier feedback, frozen deployment, replay, and Raw-Trajectory controls.These design elements help distinguish local task recovery from transferable procedural reuse.
- Empirical findings: Across ten model configurations and three agent harnesses, current agentic LLMs often adapt locally but rarely form reliable reusable skills.Skill-based conditions can improve acquisition or replay, but gains remain unstable under context shift, adversarial shortcuts, and composition.
- Empirical findings: Raw-Trajectory comparisons reveal a lossy abstraction bottleneck: agents often use episodic traces more effectively than the distilled skills derived from them.This indicates that current abstraction procedures discard useful cues that remain available in raw experience.
- Implications: The central challenge is selective procedural abstraction: preserving details needed for future invocation, verification, robustness, and composition while filtering local repairs and episode-specific noise.The benchmark measures progress from one-off task experience toward durable procedural knowledge.
A Complete Family Catalog
The appendix catalogs environment-level procedural skill families, with each family representing one skill and containing six role-instantiated tasks. The listed families cover API orchestration, structured data processing, document transformation, and research information synthesis.
- E2: Multi Step Tool Api Orchestration: Multi Step Tool Api Orchestration includes parameter validation, retries, pagination, dependent API chaining, and response validation with fallback.These skills address request preparation, transient failures, complete retrieval, two-step workflows, and response trustworthiness.
- E3: Data Processing Structured Query: Data Processing Structured Query includes schema inspection, type normalization, join-key alignment, null-safe filtering and aggregation, and result sanity checks.The methods emphasize inspecting tabular structure before queries, standardizing values before ordering, aligning keys before merges, handling NaNs, and validating outputs.
- E4: Document Parsing Extraction Transformation: Document Parsing Extraction Transformation includes structured-field extraction, cross-format migration, template filling from context, document-diff comparison, and multi-source merge reconciliation.The listed methods cover extracting fields to JSON, preserving content across flat-format conversions, completing templates, comparing document versions, and merging records by exact keys.
- E5: Research Information Synthesis: Research Information Synthesis includes multi-source search and filtering, evidence-grounded comparison, citation verification, and constrained summarization.The catalog specifies relevance-based selection from 10-20 sources, comparison grounded in evidence, citation checking, and constrained synthesis.
B Task Design Catalog
The task catalog spans source contradiction finding, email triage and drafting, overdue follow-up detection, meeting scheduling, and action-item extraction. These tasks emphasize explicit comparison, classification, context use, temporal checks, time-zone conversion, and structured extraction.
- Information comparison: Contradiction finding compares claims about the same facts across documents, groups them by topic, and reports differing values with their sources.The targeted contradictions include different numbers, dates, names, or statistics.
- Email workflows: Email triage assigns fixed categories, detects explicit urgency markers, and sorts messages from High to Medium to Low priority.The method focuses on keyword-based classification using subject lines or opening sentences.
- Email workflows: Reply drafting reads 3-5-message business threads, answers the latest request, references useful prior context, and matches the original sender’s tone.The intended output is a concise professional reply.
- Email workflows: Overdue follow-up detection checks the last message’s direction, questions or requests, and elapsed business days before sorting waiting items by age.The task identifies threads awaiting the user’s reply.
- Scheduling: Meeting scheduling converts 2-3 participants’ business-hour availability across 2 time zones into UTC, finds overlap, and proposes a local-time meeting.The task assumes fixed UTC offsets and standard business hours.
- Email workflows: Action-item extraction identifies direct requests, named assignments, and explicit commitments, then records each task’s description, assignee, deadline, and source email.The task operates on formal professional email threads.
B.1 E1: Code Debugging & Modification
E1 covers code debugging and modification through LS5 learning tasks and evaluation tasks spanning context shifts, adversarial conditions, and composition. The catalog includes canonical, enriched, and variant learning scenarios with merge-resolution tasks.
- Task settings: E1-LS5 defines canonical, enriched, and variant learning settings for code-debugging tasks.The listed learning scenarios include canonical, enriched, and variant forms.
- Learning tasks: Learning tasks center on resolving or completing merges without breaking utilities, YAML structure, or release workflows.Examples include utility rollout, YAML structure, and v2.1 release-merge scenarios.
- Evaluation tasks: Evaluation tasks test E1-LS5 under context-shift, adversarial, and composition conditions.The catalog explicitly labels these three evaluation settings.
- Evaluation tasks: Evaluation examples include database and end-of-sprint feature merges.These tasks are listed under the context-shift or adversarial evaluation portion of the catalog.
B.2 E2: Tool & API Orchestration
E2 evaluates tool and API orchestration through E2-LS5 learning variants and evaluation conditions, including context shift, adversarial testing, and composition. Its tasks include product synchronization during an API rollout and primary/backup product merging with validation and fallback requirements.
- E2-LS5 includes canonical, enriched, and variant learning conditions.
- Product sync during API rollout targets gap1-nested-wrapper-structural-change.
- E2-LS5 evaluates context-shift, adversarial, and composition conditions.
- Primary/backup product merge uses composition-validation-fallback-merge and requires pre-call parameter validation plus response-validation fallback.
B.3 E3: Data Processing & Structured Query
E3 organizes data-processing and structured-query tasks into learning variants and evaluation settings spanning context shift, adversarial conditions, and composition. Its composition tasks require coordinated procedures such as key alignment, null-safe aggregation, schema inspection, type normalization, and sanity reconciliation.
- E3-LS4: E3-LS4 includes canonical, enriched, and variant learning tasks plus context-shift, adversarial, and composition evaluations.
- Composition tasks: Unified Product Inventory is a cross-family composition task requiring key alignment before merging and null-safe filtering and aggregation.
- E3-LS5: E3-LS5 contains canonical, enriched, and variant learning tasks alongside context-shift, adversarial, and composition evaluations.
- Composition tasks: Operations Quality Report is a cross-family, multi-layer sanity composition requiring schema inspection, type normalization before sorting, and result sanity reconciliation.
C Implementation-Specific Experiment Settings · C.1 Model and API Configuration
The appendix specifies reproducible implementation settings for routing models and providers, compacting trajectories, and configuring Skill Author prompts. Model presets keep benchmark variants provider-independent while routing both Harbor task-solving and host-side LLM calls through explicit configurations without storing API secrets.
- C Implementation-Specific Experiment Settings: The appendix documents model/provider routing, trajectory compaction, and Skill Author prompts for zero-shot generation, experience-based induction, and revision.
- C.1 Model and API Configuration: Baseline and strategy files define memory, retrieval, and authoring behavior, while configs/models/*.yaml supplies model-specific execution settings without changing benchmark logic.Table 7 summarizes the routing parameters.
- C.1 Model and API Configuration: Two LLM call surfaces are used: the Harbor task-solving agent and host-side components such as the Skill Author and LLM-based retriever.A supplied model preset routes both surfaces through the same provider configuration, while secrets remain environment-variable names.
- C.1 Model and API Configuration: OpenAI-family runs use Azure OpenAI with the Codex adapter, OpenAI-compatible environment variables, and responses-wire configuration.The preset uses provider: azure_openai, harbor_agent_name: codex, and agent_model_name: openai/<azure-deployment-name>.
- C.1 Model and API Configuration: The openai/... identifier is used by Codex and LiteLLM, while the actual endpoint is the Azure resource named by AZURE_OPENAI_ENDPOINT.The agent_model_name must match the available Azure deployment name.
- C.1 Model and API Configuration: Claude-family runs use Amazon Bedrock with the Claude Code adapter, an AWS cross-region inference-profile agent model name, and a bedrock/... host-side LiteLLM model.The configuration uses provider: bedrock and harbor_agent_name: claude-code.
- C.1 Model and API Configuration: Gemini presets use gemini-cli with google/<model> and GEMINI_API_KEY, while Kimi presets use a Bedrock-Mantle OpenAI-compatible endpoint with kimi-cli.The Kimi configuration uses provider: bedrock_mantle; the passage truncates its agent_model_name specification.
- C.1 Model and API Configuration: At launch, presets resolve agent_env, configure Harbor, publish SEVB_HOST_LITELLM_* settings, and let runtime fallbacks supply run-level or strategy-level defaults.Runtime reads the published sentinels before falling back to api_base, api_key_env_var, and author_- model settings.
C.2 Trajectory Compaction · C.3 Skill Author Prompts · C.4 Tier-3 Resource-Bundling Prompt and Parser Constraint
The pipeline deterministically compacts trajectories into rich evidence for skill authoring and rough evidence for the raw-trajectory baseline. Skill Author prompts enforce reusable, procedural, future-oriented skills organized across metadata, instructions, and on-demand resources, while revisions should encode repeatable patterns rather than trial-specific noise.
- C.2 Trajectory Compaction: Trajectory compaction is deterministic: it parses recorded trials, selects informative events, and renders bounded markdown summaries in rich and rough modes.Rich compaction supports Skill Author evidence, whereas rough compaction supports the raw-trajectory control.
- C.2 Trajectory Compaction: Rich compaction preserves messages, tool calls, observations, verifier summaries, and available reasoning_content for induction and revision.Revision uses cumulative same-family learning histories, while induction uses the canonical-task compacted trajectory.
- C.2 Trajectory Compaction: Rough compaction drops reasoning_content and retrieves only three same-family learning-role trajectories chronologically for Raw-Trajectory.The design exposes episodic execution evidence rather than the agent’s private abstraction.
- C.3 Skill Author Prompts: The Skill Author is a host-side LLM call whose stable system contract is paired with trial-specific inputs including feedback, trajectory summaries, existing skills, and condition instructions.Revision adds mode-specific diagnosis and cumulative same-family trajectories; induction supplies canonical-task feedback and trajectory evidence.
- C.3 Skill Author Prompts: A skill must generalize across a task class as a reusable capability package, not become a one-off note, trajectory log, or task-specific answer.The contract emphasizes transferable operations, failure patterns, workflows, decision rules, adversarial guardrails, and knowledge for future agents.
- C.4 Tier-3 Resource-Bundling Prompt and Parser Constraint: The three-tier design places always-contextual metadata in Tier 1, activated workflow instructions in Tier 2, and on-demand scripts, references, or assets in Tier 3.Tier-3 files must be explicitly cited from SKILL.md with clear triggers; uncited resources are rejected as dead weight.
- C.3 Skill Author Prompts: Skill authoring prioritizes comprehensive and adversarial-aware coverage, concise decision rules, and instructions written for future agents rather than human readers.The prompt specifically favors validation checks and explicit “if X, do Y” rules over generic advice.
- C.3 Skill Author Prompts: Revisions should update skills only for repeatable family-level failure patterns that can be encoded as reusable workflows, decision rules, or Tier-3 artifacts.One-off quirks should produce a NoOp, and task-specific answers must never be written into the skill library.
D Full Tier-3 Capacity Ablation Results
Tier-3 capacity ablations show that enlarging skill libraries through forced resource bundling does not reliably improve frozen deployment success. Local gains in LSR or RSR often fail to transfer to ESR, CSSR, ARSR, or CompSR.
- Capacity effects: Forced Tier-3 resource bundling increases library size without reliably improving frozen deployment success.The full ablation covers always-update conditions and compares acquisition, replay, frozen deployment, and three deployment-role metrics.
- Capacity effects: Tier-3 bundling can improve local metrics, especially LSR or RSR, but these gains do not consistently carry over to frozen deployment metrics.The comparison includes Curated-Always and SelfGen-Always Tier-3 variants against corresponding No-Skill baselines.
- Model variability: Some SelfGen-Always+Tier3 or Curated-Always+Tier3 configurations gain locally, while others lose frozen deployment performance despite larger libraries.The table reports success rates and percentage-point deltas relative to corresponding No-Skill results.