Source-linked AI summary
Write, Execute, Refine: From Skill Followers to Skill Optimizers via Reinforcement Learning from Execution Feedback
Kang Peng, Zhiwei Zhang, Yichen Zhang, Zezhong Wang, Yiming Du, Geng Tu, Baojun Wang, Bin Liang, Ruifeng Xu, Kam-Fai Wong
TL;DR
Agents can benefit from expert-written procedural skills but struggle to author skills that reliably improve tool use. WER trains a separate Skill Optimizer from execution trajectories and programmatic verification, improving Pass@1 over the no-skill baseline by 7.80 points on BFCL v4 and 3.85 points on τ2-bench.
Problem
Agents benefit from expert-written procedural skills, but agent-authored skills perform 8–11 points below using no skill, leaving skill authorship underdeveloped.
Method
WER trains a separate Skill Optimizer through phase-wise refinement states built from matched successful and failed executions scored by a programmatic verifier.
Results
7.80 and 3.85 percentage points: WER raises domain-averaged Pass@1 over the No Skill baseline on BFCL v4 and τ2-bench, respectively.
Takeaways & Limitations
These results support treating skill refinement as a distinct capability for turning execution feedback into reusable procedural guidance.
Takeaways & Limitations
WER has been evaluated only on BFCL v4 multi-turn and τ2-bench, both with programmatic verifiers, so transfer to open-ended evaluation remains unestablished.
Abstract
from arXiv · showhide
Expert-written natural language skills can improve tool-using agents, yet agent-authored skills perform 8-11 points worse than using no skill. This gap suggests that following procedural guidance and improving it from execution evidence are distinct capabilities. Inference time loops can repair skills but do not improve the model that writes the next one. We study how to organize execution experience from intermediate skills into training states for an optimizer. We introduce WER (Write, Execute, and Refine), a multi-phase framework that trains a Skill Optimizer outside a frozen executor. The optimizer proposes skills, a frozen agent executes each repeatedly, and a programmatic verifier scores the outcomes. The scores provide relative credit and select mixed-outcome records. Matched successful and failed trajectories from these records form the next phase's refinement states, so the optimizer learns from the consequences of its earlier outputs. On BFCL v4 multi-turn and tau2-bench, WER improves average Pass@1 over the no-skill baseline by 7.80 and 3.85 points, respectively. Under an identical refinement workflow, it outperforms the same backbone without optimizer training by 9.35 and 10.29 points. The trained 4B optimizer reaches 76.63 percent on BFCL v4, outperforming all evaluated off-the-shelf general-purpose models used as skill optimizers on average.
1 Introduction
WER addresses the gap between benefiting from procedural skills and reliably authoring them by training a dedicated Skill Optimizer on execution feedback. It uses frozen-agent executions, verifier rewards, and matched successful and failed trajectories to construct refinement states for iterative skill improvement.
- WER framework: WER keeps the Skill Optimizer outside the sandbox, has a frozen skill-conditioned agent execute multiple candidates repeatedly, and uses a deterministic verifier for group-relative rewards.Candidates with mixed outcomes yield controlled local comparisons through matched successful and failed trajectories under fixed task, skill, and executor conditions.
- Motivation: Expert-curated skills raise average pass rate from 33.9% to 50.5%, whereas agent-authored skills fall 8–11 points below using no skill.This exposes a gap between following procedural guidance and writing reliable skills.
- Motivation: Inference-time drafting-and-revision loops can improve the current skill but typically leave the skill writer unchanged.Each new task still depends on the base model to diagnose execution logs and translate failures into general procedural corrections.
- Motivation: 10% plausible but wrong experience reduces τ 2-bench Pass@1 from 82.5 to 77.2, while self-verification recovers almost none of the loss (83.3→83.2).Repairing one skill at inference time therefore does not teach the writer to repair the next.
- Contributions: WER treats refinement-state construction as central and uses phase-wise self-bootstrapping to turn matched execution outcomes into diagnostic experience for later optimization.The framework couples candidate-level relative optimization with cross-phase construction of refinement states.
2 Related Work
Prior work improves external skill artifacts, trains task agents with skill mechanisms, or learns skill-management decisions. WER instead trains the Skill Optimizer from multi-turn, programmatically verified execution consequences of its own revisions while keeping the external text interface.
- Skill construction and inference-time refinement: External-skill methods create, retrieve, revise, or optimize natural-language artifacts without changing the acting model.Examples include Voyager, Trace2Skill, OPRO, PromptAgent, EvoPrompt, DSPy, and TextGrad.
- Skill construction and inference-time refinement: WER retains the external text interface but trains the Skill Optimizer from multi-turn trajectories and programmatically verified outcomes in stateful tool environments.Unlike prior artifact-optimization methods, WER trains the optimizer itself rather than only optimizing the current artifact.
- Skills in agentic reinforcement learning: Most skill-augmented reinforcement-learning methods update the task agent itself, including systems that integrate skill libraries, jointly learn skill decisions, or evolve skill banks alongside the policy.The cited examples are SAGE, Skill1, SkillRL, and ReSkill.
- Learned skill management and refinement: WER differs from learned skill-management methods by pairing mixed-outcome executions of an intermediate skill into refinement states for training.Alternative revisions can be compared within a group, and the optimizer learns from selected execution consequences of its previous revisions.
3 Method
WER trains a text-only Skill Optimizer to revise procedural skill documents from execution evidence while a frozen agent performs the tool interactions. Programmatic verification supplies relative outcome signals, and matched successful and failed trajectories become refinement states for later training phases.
- Skill representation and optimization: The Skill Optimizer edits short procedural markdown skills while a downstream agent executes them unchanged in a stateful tool environment.Each skill contains a name, task-family description, numbered workflow, and notes; its body is prepended to the downstream agent’s system prompt.
- Execution: A frozen agent injects each candidate skill into an instrumented environment and produces full tool-call trajectories, including arguments, responses, errors, and the terminal state.The executor parameters are never updated during training, and the trajectory is passed verbatim so the optimizer can identify where to edit.
- Execution: Deterministic environment verifiers judge success by comparing terminal states with reference solutions rather than using model-generated scores.BFCL multi-turn compares environment objects, while τ2-bench compares the terminal database and required actions.
- Reward and update: Candidates receive relative credit under matched refinement states, combining parseability, verifier outcomes across rollouts, and reasoning-length control before a clipped GRPO update.The task reward remains deliberately coarse because finer behavioral credit would require putting a model in the scoring loop.
- Experience and refinement: Middle-outcome records retain matched successful and failed runs of the same skill on the same task, whose trajectories supply the next phase’s refinement states.Across phases, training shifts toward cases where the current skill is nearly sufficient, making targeted edits more likely to change outcomes.
4 Experiments
WER improves tool-use performance over no-skill and untrained-optimizer baselines, with gains increasing across training phases and concentrating in the first two inference-time revisions. Its trained optimizer also surpasses larger general-purpose models, while revisions progressively correct distinct execution failures.
- WER Improves Agentic Performance of Base LLMs: 7.80 and 3.85 percentage points: WER raises average Pass@1 from 68.83% to 76.63% on BFCL v4 and from 46.87% to 50.72% on τ 2-bench.WER improves three of four BFCL v4 domains and all three τ 2-bench domains.
- WER Learns to Refine Skills from Execution Feedback: 9.35 points on BFCL v4 and 10.29 points on τ 2-bench: WER training improves the same backbone under an identical refinement protocol.The untrained optimizer leaves BFCL v4 at 67.28% and lowers τ 2-bench from 47.70% to 40.43%.
- Phase-wise Self-Bootstrapping: 69.35% to 71.29% to 76.63%: average Pass@1 increases monotonically across Phases 1, 2, and 3.The comparison evaluates checkpoints from each phase under the same protocol.
- Inference-time Refinement Depth: 76.63%: the first two inference-time revisions raise average Pass@1 from 67.28%, while a third revision lowers it slightly to 75.33%.The results show diminishing returns as revisions accumulate.
- Skill-optimizer Backbones: 76.63%: the WER-trained Qwen3-4B optimizer exceeds GPT-5.5 by 1.88 points and surpasses general-purpose optimizers scoring 69.91% to 74.75%.WER leads in File System and Travel and ties GPT-5.5 in Vehicle.
- Revision Analysis: Two revisions correct different failures in a representative BFCL v4 task: the first fixes file creation, and the second fixes numerical aggregation, yielding success in both rollouts.The seed skill fails in both file handling and numerical aggregation, while neither initial rollout succeeds.
5 Conclusion
WER trains a separate Skill Optimizer to refine reusable agent skills from execution trajectories and programmatic verification while leaving the downstream executor unchanged. Its phase-wise self-bootstrapping improves revisions over an untrained backbone and supports skill refinement as a distinct capability.
- 5 Conclusion: WER trains a separate Skill Optimizer to revise natural-language skills from execution trajectories and programmatic verification signals while keeping the downstream executor unchanged.The framework targets reusable agent skills refined from execution feedback.
- 5 Conclusion: Each training phase constructs refinement states from earlier revisions’ consequences, enabling the optimizer to learn from its own outputs through phase-wise self-bootstrapping.This organizes execution feedback into successive training states.
- 5 Conclusion: Experiments on BFCL v4 and τ 2-bench show that WER produces more effective revisions than the same backbone without optimizer training.The trained 4B optimizer also outperforms all evaluated general-purpose models used in the same role on BFCL v4.
Limitations
WER’s evaluation is limited to verifier-supported benchmarks, so transfer to open-ended settings, unseen tools, or different executors remains unestablished. Its refinement-state design may also face context-scalability bottlenecks on longer, multimodal, or repository-scale tasks.
- Evaluation scope: WER has been evaluated only on BFCL v4 multi-turn and τ2-bench, both with programmatic verifiers.These results do not establish transfer to open-ended evaluation, unseen tool interfaces, or different executor models.
- Context scalability: WER preserves matched successful and failed trajectories verbatim when constructing subsequent refinement states.This retains diagnostically useful execution details, but state size grows with longer and more complex agent interactions.
- Context scalability: Context scalability has not been evaluated on substantially longer-horizon tasks, multimodal trajectories, or large skill repositories.These settings may make the growing refinement states a bottleneck.
A Additional Experimental Details · A.1 Benchmark Details · A.2 Baseline Details
The appendix details the BFCL v4 and τ2-bench evaluation settings, then describes Skill-R1 and Trace2Skill as baseline skill-optimization approaches. The benchmarks emphasize stateful, multi-turn tool use and the baselines derive or refine skills from execution evidence.
- A.1 Benchmark Details: BFCL v4 contains 200 multi-turn-base tasks evenly distributed across File System, Vehicle Control, Trading Bot, and Travel Booking.Tasks may require multiple tool calls across conversational turns, intermediate tool outputs, and coherent state-dependent actions.
- A.1 Benchmark Details: BFCL v4 requires agents to maintain interaction context while using domain-specific APIs and cross-functional utilities.The supplied passage identifies context maintenance, intermediate outputs, and state-dependent action sequences as core task demands.
- A.1 Benchmark Details: τ2-bench evaluates conversational tool agents in Airline, Retail, and Telecom domains using policies, task-specific tools, stateful environments, and an LLM-based user simulator.Airline and Retail center on policy-constrained customer-service workflows.
- A.1 Benchmark Details: τ2-bench’s Telecom domain uses dual control, requiring the agent and user to operate distinct tools over a shared environment.This setting adds coordination demands beyond the policy-constrained workflows described for Airline and Retail.
- A.2 Baseline Details: Skill-R1 formulates skill optimization as recurrent reinforcement learning while keeping the task model frozen.A lightweight generator uses task context, previous rollouts, and verified outcomes to produce successive skill revisions.
- A.2 Baseline Details: Skill-R1 uses a bi-level group-relative objective combining intra-generation and inter-generation advantages.The former compares rollouts under the same skill, while the latter rewards improvements across successive revisions.
- A.2 Baseline Details: Trace2Skill builds transferable skills from agent execution trajectories without updating model parameters.Parallel analyst agents extract trajectory-local lessons and propose patches, which are hierarchically consolidated into a conflict-free skill directory.
- A.2 Baseline Details: Trace2Skill supports refining existing skills or constructing skills from an initial weak draft through hierarchical trajectory-based consolidation.The design uses multiple analyst agents to process trajectories in parallel before consolidating their proposed patches.
A.3 Training Details
Optimizer training uses GRPO with four rollouts per prompt on eight Ascend 910B NPUs, with a 1 × 10−6 cosine-schedule learning rate and specified sampling and length limits.
- Training setup: Eight Ascend 910B NPUs on a single node run optimizer training with the verl framework.Training uses GRPO with a batch size of 6.
- Optimization: 1 × 10−6 is the learning rate, using a cosine schedule with warmup.
- Sampling and lengths: 4 rollouts per prompt are sampled at temperature 0.95 with top-k sampling using k = 50.Maximum prompt and response lengths are 19,000 and 4,096 tokens, respectively.
A.4 Per-Domain Results for Skill-Optimizer Backbones
Table 4 provides the full per-domain breakdown for the skill-optimizer backbone comparison summarized in Figure 4. Results are reported using BFCL v4 Pass@1 percentages, with the best results bolded.
- Per-Domain Results: Table 4 reports the full per-domain breakdown for the skill-optimizer backbone comparison.The comparison is summarized in Figure 4.
- Per-Domain Results: BFCL v4 performance is measured with Pass@1, in percent.The table compares different skill optimizers.
- Per-Domain Results: The table bolds the best results.This identifies the strongest result in each reported comparison.
B Extended Case Study Analysis · C Prompt Templates
WER’s case study shows skills improving across successive execution-feedback rounds, while the prompt templates define reusable formats for skill revision and merging. The refinement process progressively addresses concrete execution failures in a representative BFCL v4 task.
- B Extended Case Study Analysis: Three refinement stages trace how WER improves a skill on a BFCL v4 multi-turn task requiring report analysis, aggregation, and file writing.Each stage evaluates the current skill with two independent agent rollouts.
- B Extended Case Study Analysis: The initial seed skill leaves file creation and numerical aggregation underspecified, causing both rollouts to fail.The task requires writing the rounded result to a newly created statistics.txt file.
- B Extended Case Study Analysis: The first revision instructs the agent to create a missing file before writing, eliminating the file-operation error while leaving one rollout’s aggregation incorrect.The revision responds directly to the observed failure.
- B Extended Case Study Analysis: The second revision clarifies that all relevant values must be collected before aggregation and that rounding should be applied.These instructions target the remaining numerical error described in the case study.
- B Extended Case Study Analysis: The task prompt asks for the mean of revenue, expense, and profit, rounded to the nearest integer and written alone to a new statistics.txt file.It specifies that the file should contain only the integer.
- C.1 Skill Revision Prompt: The skill revision prompt frames a skill as a reusable prompt for handling a class of similar tasks and supplies a format placeholder, {skill_instance}.The example content is explicitly unrelated to the current task; only its format should be followed.
- C.2 Skill Merging Prompt: The skill merging prompt uses the same reusable-skill definition and format placeholder when consolidating related skills.Its associated figure describes consolidation into a single reusable skill.