PaperlayerView exact revision on arXiv

Source-linked AI summary

Before the Next Rollout: Diagnosing Skill Utility Shifts from Policy Updates

Yifan Wang, Zhenyi Gai, Yiwen Zhao, Mingxuan Jiang, Keyang Chen, Zeping Li, Hongfeng Chai, Yixin Cao

arXiv:2610.39358v1cs.AI

TL;DR

Policy updates can change the utility of unchanged skills, while earlier trajectories and failure feedback may not isolate their current contributions. This paper uses fixed-context policy-response shifts and training advantages to diagnose utility changes; the readouts identify declining utility and guide revisions that improve downstream performance while reducing token use.

  • Problem

    Policy updates can alter the marginal utility of unchanged skills, so earlier trajectory evidence may not reflect their contribution under the updated policy.

    Source: §1 · pp. 1–2

  • Method

    The method combines fixed-context skill–control policy shifts with training advantages to form reward-directed readouts using existing trajectories, without additional post-update diagnostic rollouts.

    Source: §1 · p. 2, §3.2 · p. 4, §3.3 · p. 5

  • Results

    Readouts are positively associated with measured utility declines across three benchmarks, and readout-guided revision improves downstream performance while reducing policy and editor token consumption.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Policy-update signals provide information for identifying declining-utility skills and prioritizing more targeted skill revision.

    Source: §1 · p. 2, §5 · pp. 8–9

    The readouts depend on training advantages and access to both policy checkpoints, and scores prioritize skills rather than provide calibrated utility-change estimates.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Skill-evolution methods typically analyze execution trajectories to guide agent skill refinement. However, policy updates can change the behavioral effects and marginal utility of an unchanged skill. Attribution based on earlier trajectories may therefore no longer reflect the skill's contribution under the updated policy. Moreover, trajectory failures can arise from multiple factors, making individual skill contributions difficult to isolate. We evaluate whether policy-update signals contain predictive information about changes in skill marginal utility. At fixed contexts from pre-update training rollouts, we measure how policy updates change the contrast in token preferences between skill and control conditions. Combining these interaction shifts with advantage estimates yields reward-directed readouts for diagnosing utility changes. The readouts reuse existing training rollouts and require only forward passes for single-step action scoring, without additional post-update rollout evaluation. Our experiments show that policy updates can alter skill marginal utility and that policy-update readouts carry diagnostic information for identifying skills with declining utility. Integrating readout-guided prioritization into skill evolution improves downstream performance while reducing policy and editor token consumption.

1 Introduction

Policy updates can change the marginal utility of unchanged skills, making earlier trajectory evidence less reliable. The paper introduces reward-directed policy-update readouts for diagnosing utility shifts and prioritizing skill revision, reporting diagnostic value and downstream benefits.

1 Introduction

  • Because policy updates can change the effects of unchanged skills, earlier trajectories may no longer accurately reflect their utility under the updated policy.The paper defines marginal utility as the difference in expected task success between using a skill and a matched control without its task-specific guidance.Source: §1 · p. 1
  • Reward-directed readouts are positively associated with measured utility declines across three benchmarks.The readouts are intended to identify skills with declining utility.Source: §1 · p. 2
  • The authors frame policy-update signals as predictive information about utility shifts before additional post-update rollout evaluation.They propose prioritizing potentially declining skills for revision.Source: §1 · p. 2
  • Readouts combine policy-response changes with advantage estimates from existing training batches, avoiding additional post-update rollouts for diagnosis.The policy-response changes compare skill and control conditions.Source: §1 · p. 2
  • Readout-guided revision improves downstream performance and reduces policy and editor token consumption versus full-failure revision under a shared editor and matched budgets.The reported comparison uses the same editor and matched revision budgets.Source: §1 · p. 2

2 Related Work

Related work studies experience-driven skill evolution, skill utility measurement, and methods for predicting which skills merit assessment. This paper focuses on policy-update signals as a low-overhead source of information about utility shifts in unchanged skills.

Experience-driven skill evolution

  • Experience-driven methods distill interaction feedback into persistent skills and revise them iteratively, with some approaches jointly adapting skill libraries and policies.Historical feedback may become less reliable for maintenance as the policy evolves.Source: §2 · pp. 2–3

Skill utility evaluation and prediction

  • Skill-evaluation work measures incremental guidance benefits, while lifecycle methods use contribution estimates for retention or distillation.Predictive approaches also estimate transferability from descriptions or allocate budgets using reward predictors under frozen models.Source: §2 · p. 3
  • This work examines policy-update signals for utility shifts in unchanged skills and reuses them for diagnosis without a separate utility predictor or new post-update rollouts.The passage contrasts this focus with existing measurement and prediction approaches.Source: §2 · p. 3

3 Methodology

The method measures skill-utility changes across policy updates, derives reward-directed readouts from fixed-context policy responses, and uses them to prioritize skill revision. It reuses pre-update contexts and training-batch information rather than requiring additional diagnostic rollouts.

3.1 Measuring Skill Utility Shifts

  • Skill marginal utility is the expected task-success difference between a skill and its matched control, and its change measures how that difference shifts across a policy update.Both conditions continue from shared reference anchors with equal remaining interaction budgets; a negative change indicates less marginal benefit after the update.Source: §3.1 · p. 3

3.2 Fixed-State Policy Readouts

  • The method compares policy checkpoints at the same recorded training contexts, forward-scoring skill and control conditions to measure changes in token-level action preferences.The skill library remains fixed within each update window.Source: §3.1 · p. 3, §3.2 · p. 4

3.3 Reward-Directed Skill Diagnosis

  • The readout combines skill–control interaction shifts with batch advantages; negative projections oppose the batch-derived reward direction and increase reassessment priority.Signed aggregation preserves projection direction and magnitude, with positive projections offsetting negative ones.Source: §3.1 · p. 3, §3.3 · pp. 4–5

3.4 Readout-Guided Skill Revision

  • At each update window, the method ranks skills and selects up to k highly ranked candidates invoked in failed batch trajectories for editor review.The editor may revise the library under a shared mutation budget; proposals are checked on paired validation tasks and decoding seeds, and rejected proposals leave the current library unchanged.Source: §3.4 · p. 5

4 Experiment

Across six benchmarks, the experiments assess whether policy-update readouts identify skills with declining utility and whether using them to guide revision improves performance and efficiency. Analyses also examine candidate selection and WebShop editing dynamics.

4.1 Experimental Settings

  • The evaluation covers six benchmarks and compares policy-learning, fixed-skill, and skill-evolution baselines using diagnostic, downstream-performance, and efficiency metrics.Closed-loop methods use three independent 150-iteration runs, with revision opportunities every five iterations and a shared editor.Source: §4.1 · pp. 5–6

4.2 Main Results

  • Reward-directed readouts improve average AP from 0.590 to 0.746, AUROC from 0.525 to 0.802, and directional Spearman correlation from 0.032 to 0.531 over magnitude-only readouts.P@5 rises from 0.367 to 0.667; magnitude-only readouts correlate more strongly with absolute utility change, while advantages add directional information.Source: §4.2 · pp. 6–7
  • Readout-guided revision outperforms the strongest skill-evolution baseline by 6.07 and 5.59 percentage points on ALFWorld Seen and Unseen, 3.76 on WebShop, and 3.82 on LogicBench; it remains competitive on NQ and SpreadsheetBench.It achieves the highest performance in five of seven evaluation settings; Table 2 reports performance across six benchmarks.Source: Table 2 · p. 6, §4.2 · p. 7
  • On ALFWorld, editor input tokens fall by 76.7% (14.940M to 3.475M), and readout-guided revision also reduces average interaction steps on ALFWorld and WebShop.Despite less editing context, candidate acceptance rates are higher across all six benchmarks; Table 3 compares efficiency with full-failure attribution.Source: §4.2 · p. 7, Table 3 · p. 7

4.3 Analysis and Discussion

  • Under the same revision budget, reward-directed candidate selection beats the strongest alternative by 5.00 and 4.47 percentage points on ALFWorld Seen and Unseen, and 3.20 points on WebShop.Failure-frequency and magnitude-only selection also improve over random selection but remain below reward-directed prioritization; Table 4 compares selectors under a shared top-5 budget and revision protocol.Source: §4.3 · pp. 7–8, Table 4 · p. 7
  • On WebShop, the skill library grows from 54 to 72 over 150 RL iterations, with category-specific skills accounting for most growth and accepted edits shifting from additions to modifications.Category-specific skills increase from 39 to 55, while general skills rise from 15 to 17; Figure 4 depicts library composition and editing operations per 25 iterations.Source: §4.3 · p. 8, Figure 4 · p. 8
  • In a WebShop case study, readout-guided selection prioritizes ele_001 and its failure evidence, leading to broader initial searches while retaining explicit constraint checks before purchase.The comparison contrasts targeted evidence with full-failure attribution that presents all 54 skills and failed trajectories to the editor; Figure 5 illustrates the case.Source: §4.3 · p. 8, Figure 5 · p. 8

5 Conclusion

Policy updates can change the marginal utility of unchanged skills. The paper introduces reward-directed readouts of these shifts and reports that their use in skill evolution improves downstream performance while reducing inference and revision costs.

5 Conclusion

  • Policy updates can change the marginal utility of unchanged skills, and reward-directed readouts combine fixed-context skill–control policy shifts with training advantages to diagnose those changes.The readouts require forward scoring on existing trajectories; experiments report improved identification of declining-utility skills and better downstream performance with lower inference and revision costs.Source: §5 · pp. 8–9

Limitations

The approach has stated dependencies and a limited evaluation scope.

Limitations

  • Readouts depend on training advantages and both policy checkpoints, may be less informative with noisy rewards or sparse skill observations, and were evaluated on text-based skills with a fixed router.Fixed-context comparisons do not capture state-visitation or routing shifts; scores prioritize skills rather than calibrate utility changes, and evaluation and candidate validation still incur interaction costs.Source: Limitations · p. 9

A.1 Dataset Details and Splits

The appendix describes the benchmark task settings and the initial skill collections used in the experiments.

A.1 Dataset Details and Splits

  • The six benchmarks span embodied household interaction, web shopping, retrieval-based question answering, spreadsheet manipulation, and logical reasoning.ALFWorld uses text commands, WebShop uses search and click actions, HotpotQA uses retrieval and reasoning, SpreadsheetBench involves executable code, and LogicBench evaluates logical inference.Source: §A.1 · p. 11
  • Natural Questions evaluates short-answer question answering with retrieval, while LogicBench uses an evaluation subset in a single-response setting.Table 5 summarizes benchmark task counts and interaction modes.Source: §A.1 · p. 11, Table 5 · p. 11

A.2 Initial Skill Libraries

  • Initial libraries reuse SkillRL collections for ALFWorld, WebShop, HotpotQA, and NQ, and the 19 released SRA-Bench logical-reasoning skills for LogicBench.HotpotQA and NQ share a search-oriented collection; all methods use the same initial collection within each benchmark, with general and task-specific skills treated as routable entries.Source: §A.2 · p. 11, Table 6 · p. 11
  • SpreadsheetBench starts with 32 skills distilled from successful and failed training-task traces, covering procedures such as workbook inspection, data transformation, formula handling, and output verification.Table 6 summarizes the initial libraries, and Table 7 summarizes additional hyperparameters and computational resources.Source: §A.2 · p. 11, Table 6 · p. 11, Table 7 · p. 12

B.1 Training and Evaluation Configuration

The appendix documents the controlled training and evaluation comparisons, statistical testing, and cost accounting. It also specifies how readouts are aggregated and checked for validity and coverage.

B.1 Training and Evaluation Configuration

  • Table 7 summarizes default settings, while controlled comparisons share initialization and router inputs and keep diagnostic skill banks fixed.Closed-loop revisions occur only at window boundaries, and readouts reuse the initial pre-update batch.Source: §B.1 · p. 12
  • Cost accounting averages interaction steps and policy tokens across evaluation episodes, including failures, and counts editor usage across revision requests and retries.Router computation is excluded from policy-token counts.Source: §B.1 · p. 12
  • Paired bootstrap tests use identical task resamples across methods, average decoding repetitions within tasks, and apply Holm correction within each benchmark.Significance is defined as adjusted p < 0.05.Source: §B.1 · p. 12

B.2 Readout Validity and Aggregation

  • Hierarchical aggregation weights tasks equally after averaging tokens within decisions and trajectories, or within answers and questions for single-response tasks.Compared readouts use identical scored positions and weights, independent of projection values and utility outcomes.Source: §B.2 · p. 12
  • The magnitude-only baseline measures centered interaction magnitude without advantages; centering removes the vocabulary-wide mean while leaving the reward-directed projection unchanged.Every skill with scored positions in the batch is included, with zero-advantage positions contributing zero while retaining their weights.Source: §B.2 · p. 13

B.3 Paired Utility Evaluation

The appendix measures paired utility changes using shared reference anchors and reports cross-seed uncertainty. It also specifies readout-based candidate selection, editing constraints, and paired validation.

B.3 Paired Utility Evaluation

  • Utility comparisons reuse first-invocation anchors from pre-update reference rollouts, comparing the original skill with a control that removes its task-specific guidance.Both policies continue from shared anchors, with matched decoding seeds and freely evolving continuations.Source: §B.3 · p. 13
  • Task-equal averaging yields utility estimates under the reference distribution, reported in percentage points as the post-update minus pre-update marginal utility.The evaluation uses six independent 150-iteration runs per benchmark and 36 run–window comparisons, with paired hierarchical bootstrap confidence intervals.Source: §B.3 · p. 13

B.4 Editor Prompt and Revision Settings

  • Every five RL updates, readout arms select up to five scored skills that appeared in failed trajectories, ranking candidates by score and breaking ties by skill ID.The failure-driven baseline instead receives the full skill library and all failed trajectories from the same batch.Source: §B.4 · p. 13

B.3 Paired Utility Evaluation

  • 36 of 47 skill estimates supported by at least two tasks have intervals excluding zero: 17 indicate declines and 19 increases.Figure 6 shows pointwise 95% paired-bootstrap confidence intervals; intervals entirely below or above zero support decline or increase labels.Source: §B.3 · p. 13, Figure 6 · p. 14

B.4 Editor Prompt and Revision Settings

  • Each editing opportunity allows at most three mutation units and one valid candidate library; Figure 7 specifies editor inputs, revision instructions, and JSON output.Editing is skipped without eligible candidates or failure evidence, and NOOP retains the current library without validation.Source: §B.4 · p. 15, Figure 7 · p. 16
  • Proposed libraries are compared with the incumbent under the updated policy using paired validation tasks and decoding seeds, and are accepted only under the predefined rule.The gate subset is shared across editing arms, with an acceptance tolerance of η = 0.05.Source: §B.4 · p. 15

Related papers

Loading 2610.39358v1…