UCOB: Learning to Utilize and Evolve Agentic Skills via Credit-Aware On-Policy Bidirectional Self-Distillation

1 INTRODUCTION

• Retrieved skills can improve exploration and sample efficiency, yet stale, generic, or state-mismatched guidance can help in one state and mislead in another.This makes the assumption that skill-conditioned behavior is always a better teacher fragile.Source: §1 · pp. 1–2
• UCOB treats skill-conditioned and no-skill prompts as on-policy views of the same model, letting the higher-return view within a task-anchor group teach the other.This locally internalizes useful skill behavior and corrects misleading skill-induced behavior.Source: §1 · p. 2
• UCOB evolves skills through dual-granularity memory, utility-aware retrieval, and reflection self-training, using credit evidence to update memory and the skill writer.The framework links skill use during policy learning to subsequent skill evolution.Source: §1 · p. 2
• UCOB beats SOTA baselines such as SDAR and Skill1 by up to 23.5 and 18.0 absolute success-rate points on ALFWORLD and WEBSHOP.The contribution summary reports gains across model scales.Source: §1 · p. 2

2 OBSERVATIONS

• The diagnostics compare fixed-direction skill/no-skill protocols and use anchor-state returns to determine which view is the local teacher.The return comparison is between skill-conditioned and no-skill views within the same task and anchor state.Source: §2 · pp. 3–4
• Skill-conditioned evaluation does not consistently dominate no-skill evaluation, and remains weaker throughout WEBSHOP training in the fixed-teacher diagnostic.The result challenges treating the skill-conditioned view as a privileged teacher.Source: §2 · p. 3, Figure 3 · p. 3
• Dual rollouts narrow the gap between views but do not make skill-conditioned behavior consistently better; on WEBSHOP, skill-conditioned rollouts have lower average reward.Both views receive on-policy RL updates, while distillation direction remains fixed.Source: §2 · pp. 3–4, Figure 4 · p. 3
• Same-task, same-anchor return gaps have mixed signs, and the better branch can switch across decision steps within a trajectory.Trajectory-level summaries can therefore provide the wrong local update direction.Source: §2 · p. 4, Figure 5 · p. 4
• UCOB turns matched skill/no-skill rollouts into credited pairs, using the higher-value branch for cross-context distillation and the evidence to update task- and state-level skills.The approach contrasts with trajectory-based state-skill summaries such as D2Skill (Tu et al., 2026).Source: §2 · p. 4

• The agent observes a task state and interaction history, generates a textual response from which an executable action is parsed, and receives rewards across turns.Source: §3 · p. 4
• Table 1 compares representative skill-conditioned methods by state-level skill use and supervision, noting that SDAR gates tokens but does not redirect harmful teachers.Source: Table 1 · p. 5
• Skill-conditioned and no-skill prompts induce two context views of the same policy, with retrieved skills included only in the former.Source: §3 · p. 5
• Skill-conditioned OPSD uses student-sampled rollouts and per-token divergence, while common variants fix the skill-conditioned view as teacher without verifying its local advantage.Source: §3 · p. 5

4 METHOD: UCOB

• UCOB compares skill-conditioned and no-skill views at the same task-anchor state, distills from the higher-return view, and feeds credit into policy and skill updates.Source: §4 · p. 6

4.1 STEP 1: DUAL-GRANULARITY SKILL RETRIEVAL

• Dual-granularity retrieval maintains task- and state-level skill pools, ranking candidate skills with utility-aware UCB scores before adding retrieved skills to the conditioned prompt.Source: §4.1 · pp. 6–7

4.2 STEP 2: MIXED ROLLOUTS AND ANCHOR-STATE GROUPING

• UCOB samples skill-conditioned and no-skill trajectories from the current policy, then groups repeated anchor states within each task for matched comparisons.Source: §4.2 · p. 7

4.3 STEP 3: CREDIT-AWARE BIDIRECTIONAL SELF-DISTILLATION

• For each same-task, same-anchor group, CBSD selects the higher-return record as reference and distills it into the opposite context, reversing direction when skills are locally harmful.Source: §4.3 · p. 7

4.4 STEP 4: SKILL MEMORY EVOLUTION AND JOINT TRAINING

• UCOB writes task skills from rollout groups and state skills from local return contrasts, updates skill utilities from baseline-relative success, and trains the reflection writer with utility-derived advantages.Source: §4.4 · p. 8

• At a fixed task-anchor context, the skill-minus-no-skill return gap equals the difference in expected local advantages, because both views share a state-value baseline.Source: §5 · p. 9
• With correctly estimated branch ordering and a small update, lower-to-higher distillation improves the old-policy local advantage surrogate and adds a positive first-order term to the return lower bound.Source: §5 · p. 9
• For mixed-sign gaps, fixed skill-to-no-skill distillation can be locally harmful when the gap is negative; CBSD instead flips direction to keep the signed local update positive.Source: §5 · p. 9

6.1 EXPERIMENTAL SETUP

• Evaluation covers ALFWORLD success, WEBSHOP task score and success, and SEARCH-QA accuracy across seven datasets; three Qwen backbones are compared with skill-free RL, skill-memory, and self-distillation baselines.The Table 2 caption specifies that ALFWORLD Succ is overall success and SEARCH-QA Avg is the mean across seven datasets.Source: §6.1 · pp. 9–10, Table 2 · p. 10

6.2 MAIN RESULTS

• UCOB achieves the best ALFWORLD overall success rate for all three backbones and the best WEBSHOP success rate for every backbone, with Qwen3-1.7B gains over Skill1 of +23.5 and +18.0 points, respectively.SEARCH-QA results remain within 0.8, 0.9, and 0.2 points of the best average accuracy across the three backbones.Source: Table 2 · p. 10, §6.2 · pp. 10–11

6.3 ABLATION STUDY

• Removing anchor-state grouping, state-level skills, or reflection self-training moderately degrades performance, while removing paired rollouts or CBSD causes a larger drop.Without explicit cross-context distillation, memory-augmented rollouts leave skill/no-skill comparisons to sparse RL returns rather than converting local return gaps into dense supervision.Source: Figure 7 · p. 11, §6.3 · p. 11
• One-way or fixed-direction CBSD variants are less robust than local return-based teacher selection, and top-K distributional support stabilizes its target; default λCBSD=0.1 performs best on both environments.Fixed SkillRL-initialized memory lags evolving memories, and external GPT-5.4 remains below self-trained reflection overall.Source: Figure 7 · p. 11, §6.3 · p. 12, Table 3 · p. 12

6.2 MAIN RESULTS

• The gains are strongest in multiturn interaction tasks, where state tracking and recovery leave smaller models more headroom, while SEARCH-QA depends more on search quality, evidence reading, and synthesis.The paper reports that the SEARCH-QA average remains competitive rather than uniformly dominant.Source: §6.2 · p. 11

6.4 ANALYSIS

• Both task-level and state-level skill pools grow during training, while retrieved utility tends to remain positive, the useful-skill ratio stays non-trivial, and reflection signals remain active.The analysis defines Retrieved Utility as average utility of retrieved skills and Useful-Skill Ratio as the fraction of skills whose utility exceeds its initial value.Source: Figure 8 · p. 12, §6.4 · p. 13
• Teacher routing favors no-skill-to-skill updates more often, and UCOB improves both evaluation views; case studies show it selects the skill view when useful and the no-skill view when retrieved guidance misleads.The resulting policy remains effective whether skills are provided at evaluation or not.Source: §6.4 · pp. 12–13, Figure 9 · p. 12, Figure 12 · p. 14
• Under a matched 8-GPU Qwen3-1.7B training protocol, UCOB reaches higher validation success within comparable wall-clock budgets and shows modest, localized extra overhead; in sequential training its final three-environment average is 68.4%, versus 57.0% for SFT and 51.8% for SDAR.The stream uses 150 updates per environment; UCOB retains 82.8% on ALFWORLD after subsequent stages and preserves WEBSHOP performance after SEARCH-QA training (75.8% →77.3%).Source: Figure 10 · p. 13, §6.4 · p. 13, Figure 11 · p. 14

Language agents with memory and skills

• Skill-memory methods vary in whether skills serve as retrieved context, evolving external memory, or behavior to internalize; UCOB addresses when imperfect retrieved behavior should be internalized or corrected.The cited approaches include systems for building, curating, selecting, and internalizing skills.Source: §7 · p. 14

Agentic RL and credit assignment

• Agentic RL work seeks finer-grained credit assignment for sparse-reward multiturn tasks, while UCOB uses same-task, same-anchor-state returns to select which context view teaches the other.This makes credit assignment determine not only which action tokens to reinforce, but also which context view serves as local teacher.Source: §7 · p. 15

On-policy self-distillation with privileged context

• Skill-SD and SDAR use retrieved skills as fixed privileged teachers, whereas UCOB selects direction from same-anchor-state returns to internalize useful skills and correct misleading context.The related-work passage distinguishes this credit-aware teacher selection from prior OPSD-style filtering.Source: §7 · p. 15

8 CONCLUSION

• Skill-conditioned prompts are state-dependent teachers rather than universally reliable privileged views, so on-policy skill rollouts still require local credit assignment.CBSD compares skill-conditioned and no-skill views under the same task and anchor state, selecting the higher-return view as local teacher.Source: §8 · p. 15
• UCOB combines bidirectional self-distillation with dual-granularity memory evolution, utility-aware retrieval, and reflection self-training to internalize useful skill behavior and correct misleading context.Source: §8 · p. 15
• Across ALFWORLD, WEBSHOP, and SEARCH-QA, UCOB achieves consistent gains, with analyses reporting useful evolving memories, sequential-environment adaptation, and localized training overhead.Source: §8 · p. 15

Fixed teacher without skill rollouts

• In the fixed-teacher protocol, only the no-skill prompt acts in the environment; the skill-conditioned prompt supplies an OPSD target on the same student-generated prefixes.The skill-conditioned view receives no rollout-based RL updates under its own context.Source: A DETAILED PROTOCOLS FOR THE… · p. 19

Dual rollouts with fixed-direction OPSD

• Dual-rollout OPSD executes both prompt views, but keeps skill-to-no-skill distillation fixed, testing whether on-policy skill rollouts alone resolve teacher ambiguity.The passage reports that dual rollouts reduce exposure mismatch but do not remove ambiguity when the fixed teacher is locally weaker.Source: A DETAILED PROTOCOLS FOR THE… · p. 19

Anchor-state value-gap diagnostic

• The diagnostic groups records sharing a task and anchor state, averages return-to-go by prompt branch, and uses their gap to identify the more reliable local view.Positive gaps favor the skill-conditioned view; negative gaps favor no-skill, and observed gaps are mixed-sign and can switch across steps.Source: A DETAILED PROTOCOLS FOR THE… · p. 19

B FULL TRAINING ALGORITHM

• UCOB groups skill-conditioned and no-skill on-policy decisions by task and canonical anchor state, then selects the highest-return local record for comparison.The loop records each decision’s branch, prompts, response, and return-to-go before forming the groups.Source: B FULL TRAINING ALGORITHM · p. 20
• For each anchor-state group, UCOB distills from the higher-return context into the opposite view only when the return gap exceeds a margin.A winning skill-conditioned view teaches the no-skill view; a winning no-skill view corrects the skill-conditioned context.Source: B FULL TRAINING ALGORITHM · p. 20
• The loop retrieves task- and state-level skills to build skill-conditioned prompts, alongside no-skill prompts without retrieved skills.It uses both pools at encountered states before generating mixed on-policy rollouts.Source: B FULL TRAINING ALGORITHM · p. 20
• UCOB writes task-level skills from rollout reflections and state-level skills from anchor-state contrasts, then updates utilities, prunes memories, trains the reflection writer, and optimizes the policy.These memory and reflection updates occur in the final stage of the training loop.Source: B FULL TRAINING ALGORITHM · p. 20

Optimization and rollout

• ALFWorld and WebShop use group size 8, batch size 16, discount factor γ = 0.95, and 150 training steps on 8 GPUs.Their maximum horizons are 50 and 15, respectively; the actor learning rate is 10−6.Source: C IMPLEMENTATION DETAILS · p. 20

CBSD

• CBSD uses turn-level, response-scope top-K OPD with Ktok = 32, return-gap margin ϵCBSD = 0.05, and at most two accepted pairs per anchor-state group.Its confidence gate is enabled, and supervision is applied to the action span.Source: C IMPLEMENTATION DETAILS · p. 21

Dynamic memory and reflection

• Both skill pools retrieve top-3 items after top-10 sentence-transformer candidate filtering, with utility updates using EMA rate βU = 0.2.Reflection writes task-level items per rollout group and state-level items per qualifying anchor-state contrast; self-training starts after a 30-step warmup in Qwen3-1.7B runs.Source: C IMPLEMENTATION DETAILS · p. 21

SEARCH-QA runs

• Search-QA uses group size 8, training batch size 128, validation batch size 512, and a four-step maximum search horizon.Runs train for 150 steps on 8 GPUs and validate at the final step.Source: C IMPLEMENTATION DETAILS · p. 21

D LOCAL POLICY-IMPROVEMENT VIEW OF CBSD

• Within a shared task-anchor context, the skill-minus-no-skill return gap estimates a local advantage difference, not a raw trajectory-level preference.The shared state-value baseline cancels in the comparison.Source: D LOCAL POLICY-IMPROVEMENT VIEW OF CBSD · pp. 21–22
• Under a trust-region-controlled state-distribution shift, the idealized update has a first-order local improvement with only a second-order error in the bound.The analysis is local to a fixed policy update and task-anchor group, not a global convergence guarantee for the evolving system.Source: D LOCAL POLICY-IMPROVEMENT VIEW OF CBSD · pp. 21–22
• CBSD selects the teacher direction from the higher-return branch and distills its behavior into the opposite context view.Its implemented token-level objective is described as a KL-proximal stochastic approximation using a higher-return reference and a nonnegative gap-dependent weight.Source: D LOCAL POLICY-IMPROVEMENT VIEW OF CBSD · pp. 21–22
• The token-level CBSD loss applies positive-weight imitation toward the locally higher-return branch, with weight increasing in the accepted return gap.The reference distribution is stop-gradient from the higher-return branch, while the target-branch distribution receives the update.Source: D LOCAL POLICY-IMPROVEMENT VIEW OF CBSD · p. 22
• When the skill-conditioned view has lower local value, fixed skill-to-no-skill distillation can decrease the local surrogate, whereas CBSD reverses direction.When the skill-conditioned view has higher local value, it is instead the improving teacher.Source: D LOCAL POLICY-IMPROVEMENT VIEW OF CBSD · pp. 22–23
• Return gaps alone do not establish that the higher-return branch is closer to the local optimal response distribution.The stated closeness result is conditional on the higher-return branch also being the better local reference.Source: D LOCAL POLICY-IMPROVEMENT VIEW OF CBSD · p. 23