Source-linked AI summary
SkillGate: Training In-Policy Skill Selection in Long-Horizon Agents
Qingyao Li, Wenxiang Jiao, Shuai Shao, Kangning Zhang, Yuan Lu, Yi Guo, Weiwen Liu, Weinan Zhang, Yong Yu
TL;DR
In-policy skill selection remains under-addressed, and outcome-only RL fails to train it reliably in long-horizon tasks. SkillGate separates selection and execution credit, achieving 53.2% trial success versus 47.0% for identical-budget outcome-only RL across five agentic benchmarks.
Problem
In-policy skill selection remains under-addressed despite determining which instructions condition the agent’s subsequent actions.
Method
SkillGate partitions token-level credit into disjoint channels, assigning outcome advantage to execution tokens and action-local advantage to tokens naming the selected skill.
Results
53.2% trial success versus 47.0% for identical-budget outcome reward alone across five agentic benchmarks.
Takeaways & Limitations
Selection credit can be trained as a distinct decision, improving task performance while reducing exposure to misleading skills and skill reads.
Takeaways & Limitations
The setup assumes a terminal verifier supplies the task score as the system’s only reward.
Abstract
from arXiv · showhide
Agent frameworks increasingly package procedural knowledge as skills: instruction files an agent reads on demand, while public libraries now hold thousands of them. Which skill to read has thus become a decision the policy itself makes in the middle of an episode, yet no existing signal trains it. We show that the default remedy, outcome-rewarded RL over the candidate slate, cannot teach it, for a structural reason we identify and name selector credit starvation: under a broadcast, sequence-level advantage, the few tokens that name the chosen skill carry a vanishing share of the loss, and the credit they inherit is increasingly wrong-signed as trajectories lengthen. A correct choice is punished whenever the execution after it fails, even though the choice itself is among the most valuable decisions in the trajectory. Auditing a completed run's own training artifacts confirms all three properties, each worsening monotonically with horizon. SkillGate removes the failure by construction: it partitions the token support into two disjoint credit channels, outcome credit reaching only execution tokens, and a separate action-local advantage reaching exactly the skill-naming tokens, positive only when a trajectory's single read is the correct one. On five agentic benchmarks under a 16-candidate slate, SkillGate lifts a 9B policy from 40.8% to 53.2% trial success, well ahead of the identical budget spent on outcome reward alone, while cutting exposure to misleading candidates by two thirds and reading fewer skills.
1 Introduction
In-policy skill selection remains difficult because the policy’s read choice is made mid-episode and conditions subsequent actions, while outcome-only RL buries its signal in trajectory-wide execution credit. SkillGate separates selection and execution credit, improving trial success while reducing misleading-skill exposure and skill reads.
- Motivation: In-policy skill selection is the policy’s mid-episode choice of which skill to read, and the opened skill conditions every subsequent action.Candidates are written to look alike, with only a few tokens distinguishing the correct name from a plausible wrong one.
- Method: SkillGate partitions one policy-gradient update into disjoint execution and selection channels, preventing task outcomes from revising the skill choice.The task channel removes the entire skill-read tool call, while the selector channel applies an action-local advantage only to tokens naming a single correct read.
- Results: SkillGate cuts misleading-skill exposure by two thirds while reading fewer skills, indicating a behavioural gain rather than a capability effect.Its selection weight is made length-invariant, so a decision’s weight no longer depends on the trajectory length containing it.
- Problem: Selector credit starvation causes outcome-only RL to dilute and corrupt the selection signal as trajectories lengthen, despite stable task value.On 12,800 training trajectories, the choice’s loss share dilutes 7× with length, credit becomes increasingly wrong-signed, and matched prompt groups show a +11.2 pp success gap for the correct read.
- Results: 53.2% trial success versus 47.0% for the identical budget spent on outcome reward alone demonstrates SkillGate’s effectiveness across five agentic benchmarks.The method also outperforms supervised selection, preference learning, external routers, and reference models with roughly forty times the parameters.
2 Problem Setup
The setup defines episodes as multi-turn tool-use trajectories in which the agent selects from a slate of skill candidates and may read their bodies. Training uses only assistant-generated tokens, while a terminal verifier supplies the sole task reward.
- Skill slate: Each task presents a slate of K skill candidates, including one task-specific oracle skill verified to solve the task.A skill consists of a name, one-line description, and SKILL.md body; only the name and description are initially visible.
- Episodes: An episode is a multi-turn tool-use loop that ends when the agent stops or exhausts fixed turn and wall-clock budgets.The trajectory concatenates the prompt, assistant messages, and environment observations.
- Episodes: The terminal verifier returns R(τ) ∈[0, 1] from benchmark tests or a grader, and it is the system’s only reward.Only assistant-generated tokens are trained; observations are masked out.
- Selection and execution: Read actions are tool calls that open skill bodies, attributed only from assistant-generated text and only for files under skill directories.Each attributed read carries a skill identity and slate category, whether performed through the read tool or a shell command.
3 SkillGate
SkillGate trains skill selection and task execution through disjoint credit channels: outcome advantages reach execution tokens, while action-local selection advantages reach only the tokens naming the chosen skill. Its selector rewards exactly one clean oracle read, while preserving outcome-based execution learning and preventing credit from reflecting whether the selected skill’s content ultimately worked.
- Credit channels: SkillGate assigns task advantages only to execution tokens and selection advantages only to identity spans, keeping the two credit channels disjoint by construction.The task mask removes complete read calls, while the selector trains only credited skill-name spans.
- Selection credit: A clean single read of the oracle receives positive selection credit, whereas misleading reads, repeated reads, or reading additional candidates receive zero utility.The single-read requirement prevents the policy from gaining credit by reading everything.
- Selector safeguards: The action-centred selector advantage has zero weighted sum within each rollout group and becomes silent when all actions tie, creating no general pressure to read more or fewer skills.A group with no clean oracle read or only clean oracle reads yields Asel ≡ 0.
- Token weighting: Token-weight rescaling preserves the task channel’s total learning mass after deleting read calls and gives every credited action equal total selector weight.When M > 0, selector token weights sum to the original batch mass N; when M = 0, the selector channel is silent.
- Credit semantics: Selection credit depends only on the chosen read action, so an oracle read is reinforced whether or not the subsequent task succeeds.Execution tokens independently receive outcome-based reward or penalty, regardless of which skill was read.
4 Experiments
Across five agentic benchmarks, SkillGate improves 9B task success by learning selective, single-oracle skill access rather than merely increasing reads. Ablations and diagnostics show that action-local credit on the trajectory’s only oracle read drives this improvement while reducing exposure and inference cost.
- Evaluation setup: Five benchmarks use K = 16 mixed slates, 30-turn and 850-second trial budgets, and a 385-trial outcome protocol with disjoint training and evaluation oracle identities.The slate contains one oracle, five misleading hard negatives, and ten library skills sampled from 2,045 community skills; training uses 491 tasks excluding Claw-Eval.
- Task performance: Outcome-only RL increases both oracle and misleading exposure, whereas SkillGate produces the strongest joint shift toward reliable skill access and task success.An oracle-only intervention improves a frozen SFT executor by roughly eleven points, while Selection BC improves choice without matching the downstream result.
- Credit ablation: Only credit on the trajectory’s single oracle read closes the gap between action selection and clean behavior, converting selector signal into task success.Group-level and trajectory-level credit fail to identify the chosen skill or encourage correctness; action credit improves behavior but does not penalize extra reads.
- Credit diagnostics and efficiency: 0.14% is the median loss weight carried by skill-naming tokens, diluting roughly 7× across trajectory lengths while nearly two in five oracle-reading trajectories receive negative advantage.Despite this starvation, oracle reading is worth +11.2 pp of success within matched prompt groups, and SkillGate uses fewer distinct reads, turns, and cumulative input tokens than outcome-only RL.
5 Related Work
Related work studies skill selection under progressive disclosure, where agents choose which library skill to open, and develops more localized training signals for multi-turn trajectories. Other work applies hierarchical or skill-oriented reinforcement learning to decompose trajectories, learn skill policies, and train skill internalisation.
- Selecting a skill from a large library: Progressive-disclosure frameworks make skill routing a decision: agents see a skill name and one-line description before deciding what to open.Audits report that many public-library skills lack usable routing descriptions, while benchmarks measure the resulting confusion at scale.
- Where the training signal lands in a multi-turn trajectory: Outcome-rewarded policy gradients attach one advantage to every rollout token, motivating research on more localized credit assignment for agents.Related approaches include formation-gain step scores, paired advantage channels for tool contribution, conditional efficiency channels for tool invocation, and empirically calibrated per-turn rewards.
- Where the training signal lands in a multi-turn trajectory: Agent Lightning decomposes agent trajectories into transitions within a hierarchical reinforcement-learning framework.This work is presented alongside methods assigning credit at selected decision points and other structured reward approaches.
- Skill-oriented reinforcement learning: Skill-oriented reinforcement learning co-evolves reusable skills and task policies, internalises skills from paired skill and no-skill rollouts, or learns whether to invoke a skill.The passage lists these as distinct directions within skill-oriented RL.
6 Conclusion
SkillGate trains mid-episode skill selection by separating outcome credit for execution from action-local credit for skill-naming tokens, addressing the selector’s vanishing and increasingly wrong-signed training signal.
- SkillGate: SkillGate partitions each trajectory’s token support into two disjoint credit channels: outcome credit reaches execution tokens, while action-local advantage reaches only tokens naming the chosen skill.The design trains the agent’s mid-episode choice of which skill to read as a decision.
- Motivation: The design is motivated by measurements showing that skill selection receives a vanishing and increasingly wrong-signed share of training credit.These measurements use a finished run’s own training artifacts.
A The 385-trial protocol
The evaluation comprises 385 trials across two task sets, using a fixed 16-candidate protocol and consistent prompts, interfaces, and grading controls. Results are aggregated across separate batches without outcome-based trajectory filtering, with a complete rerun recovering unavailable Skill-free RL records.
- Composition: 385 trials combine 224 non-Claw trials from 56 tasks evaluated four times and 161 Claw-Eval trials evaluated once per task.Claw-Eval includes 14 tasks from the repeated evaluation set and 147 disjoint held-out tasks.
- Evaluation controls: All methods use fixed task lists, 16-candidate mixed slates, slate orders, prompts, decoding seeds, context windows, and benchmark-specific graders.The protocol uses a 65,536-token context window, disabled hidden thinking, and each model’s specified tool interface.
- Aggregation: 385 trials are concatenated from separate task-set batches without rescoring, selecting, or discarding trajectories based on model outcome.Unavailable records for 14 repeated Claw-Eval tasks in Skill-free RL were replaced by a complete rerun under identical settings, preserving the task set and denominator.
B Protocol for the frontier reference rows
The frontier reference rows use the same tasks, slates, prompts, budgets, and graders as trained rows, with interface adaptations to measure task performance rather than formatting artifacts. They use native function calling, adjusted loop detection, and include only models that sustain tool-enabled multi-turn interaction, so they are absolute references rather than an exhaustive leaderboard.
- Reference design: The frontier block matches trained rows on tasks, skill slates, prompt content, turn budget, and graders, but serves as a capability reference rather than a controlled baseline.Two interface adaptations prevent formatting quirks from determining measured task performance.
- Interface adaptations: Each frontier model uses its provider’s native function-calling interface while keeping tools, observations, and success criteria unchanged.The manual schema remains for models trained with it because imposing it on unrelated API models can reject valid calls.
- Interface adaptations: Frontier rollouts stop repeated tool calls, using a longer duplicate-call signature to avoid mistaking legitimate shared-prefix calls for loops.The stopping threshold and other rollout limits are unchanged, and the same rule applies to every frontier model.
- Reference scope: Only models that successfully dispatch tool calls and sustain multi-turn interaction are included, making the frontier block an absolute reference rather than an exhaustive leaderboard.Models with systematic interface failures or immediate termination are omitted rather than treated as weak task capability.
C Per-method results on the 147 held-out Claw tasks
On 147 corrected, held-out Claw tasks evaluated once per model, SkillGate uniquely improves task performance while reducing both oracle and misleading skill exposure. The evaluation uses repaired oracle skills and a standard mixed slate, with grader mean reported as partial credit.
- Evaluation setup: 147 corrected Claw tasks form a clean generalisation test, with no task overlap with training or the repeated 14-task protocol.Each task is evaluated once per model, supporting task-level pass@1 directly.
- Evaluation setup: Oracle skills were verified and repaired because unverified targets would make selection metrics measure the wrong thing.Repairs addressed defects including wrong endpoints or response fields, fixture assumptions, dates, and unsafe operation ordering.
- Method comparison: SkillGate is the only principal method that leads task performance while improving both oracle and misleading exposure.Oracle and misleading reads are overlapping events, so their percentages can sum past 100%.
- Method comparison: Task-mask only increases oracle reading and misleading reading, showing that removing harmful outcome credit does not make the policy selective.Action credit without the single-read constraint performs strongly but incurs more misleading reads.
D Full read-behaviour table
The full read-behaviour analysis distinguishes protocols and clarifies how to interpret exposure and read-rate metrics. Table 6 uses repeated trials, where longer interactions raise absolute read rates but preserve method ordering.
- Protocol: Table 6 reports full read behaviour on the 280-trial repeated-trial protocol used by Tables 2 and 3 and Figure 4.Absolute read rates are higher on this subset because repeated tasks permit longer interactions.
- Metric definitions: Oracle and misleading exposure are multi-label events that can sum past 100%, while reads/trial counts distinct skill names.These definitions apply to interpreting the behaviour breakdown in Table 6.
- Interpretation: Untrained rows’ low misleading exposure reflects rare reading and abstention rather than selective skill choice.This distinction is important when comparing exposure across methods.
E Uncertainty on the controlled comparison
The controlled comparison resamples tasks rather than trials and supports a positive pooled trial-level improvement for SkillGate over outcome-only SkillRL, but not a statistically conclusive pooled task pass@4 improvement. Accordingly, the paper reports the trial-level gain without claiming significance.
- Controlled comparison: +7.9 pp pooled trial success favored SkillGate over SkillRL (outcome only), with a 95% interval of [+2.1, +14.3] excluding zero.Tasks were resampled with replacement 50,000 times after averaging each task’s repeated trials.
- Controlled comparison: +7.1 pp pooled task pass@4 favored SkillGate, but its 95% interval [−1.4, +15.7] did not exclude zero.The resampling used the same task indices for both methods.
- Controlled comparison: Figure 6 reports SkillGate minus SkillRL (outcome only) per benchmark, using observed differences and 95% task-clustered bootstrap intervals.“All” denotes the pooled result, not a sixth benchmark.
F Agent prompt and skill interface
The skill-selection interface exposes a manual read call whose identity span is the selected skill name, while presenting 16 candidate names, descriptions, and paths before action. One candidate is the oracle among five misleading hard negatives and other bystanders, so the policy must choose from short descriptions and use the exact listed path.
- Read-call surface: The read interface copies the selected slate entry’s path into a manual function call targeting /skills/<skill-name>/SKILL.md.The call uses a read function with a path parameter.
- Read-call surface: The selected candidate is directly observable in assistant-generated tokens through the <skill-name> identity span, excluding surrounding function markup.The identity span covers only the skill name, while the broader call span includes the function name and markup.
- Selection instruction: Before acting, the model sees all 16 candidates’ names, one-line descriptions, and paths, with the same selection instruction during training and evaluation.The instruction permits reading at most one skill up front and requires the exact listed path.
- Candidate representation: The fixed-order slate hides category labels and contains one oracle, five misleading hard negatives, and remaining relevant or irrelevant bystanders.Similar names and descriptions force selection from the task and short descriptions before opening a skill body.