PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds

Yuxuan Liu, Zhaochen Su, Yuhao Zhang, Jiahe Guo, Zhongwei Xie, Huihao Jing, Lingyun Xie, Qing Zong, Yauwai Yim, Zhixiong Zhang, Haoran Li, Yangqiu Song

arXiv:2608.02636v1cs.SEcs.AI

TL;DR

The paper addresses whether persistent skill evolution benefits from successful trajectories, failed trajectories, or both, and whether test-time computation can recover comparable gains. It introduces a controlled multi-round evaluation across models and benchmarks with fixed procedures and varied feedback views. The results show sparse, model- and benchmark-dependent gains, with validation selecting evolved skills in 11 of 14 settings and all selected evolutions including failed trajectories.

  • Problem

    Existing evidence does not isolate how successful and failed trajectories affect persistent skill evolution or whether extra inference can replace persistent skill updates.

    Source: Introduction · p. 1, Self-Evolving Agent Skills · p. 2

  • Method

    The study compares Normal, Fail-only, and Success-only feedback under fixed execution, revision, validation, and round-budget protocols across 42 runs and 14 model–benchmark settings.

    Source: Introduction · p. 2

  • Results

    Persistent skill evolution is sparse and feedback-dependent: validation selects evolved skills in 11 of 14 settings, nine improve released-test performance, and all selected evolutions include failed trajectories.

    Source: Introduction · p. 2, Main Results · p. 4, Conclusion · p. 7

  • Takeaways & Limitations

    Persistent self-evolution is better characterized as validation-filtered search with model- and benchmark-dependent returns than as steady improvement from additional rounds.

    Source: Conclusion · p. 7

    The study does not evaluate dedicated skill benchmarks such as SkillsBench and SkillLearnBench, so generalization beyond its task suite remains untested.

    Source: Limitations · p. 8

Abstract

from arXiv · show

Self-evolving skill systems promise to improve agents by turning execution feedback into persistent skill updates without changing the underlying model. Yet it remains unclear when further evolution helps, how successful and failed trajectories shape revision, and whether extra test-time computation can recover the same gains. To address these questions, we present a controlled evaluation framework across five benchmarks and three models. Our primary study contains 42 feedback runs across 14 supported model-benchmark settings. Within each setting, we hold the executor and optimizer configuration, revision procedure, validation rule, and round budget fixed, while varying only the feedback shown to the optimizer: successes and failures (Normal), failures only, or successes only. Evolution is sparse: only 55 of 388 candidates establish byte-distinct validation bests. Validation-based selection chooses an evolved skill in 11 of 14 settings, nine of which improve released-test performance. All 11 selections come from feedback conditions that include failed trajectories, although the relative ranking of Normal and Fail-only varies across settings. Validation and downstream evaluations on test, robustness, and transfer sometimes favor different feedback views. A broader SearchQA analysis covering eight models shows similarly sparse, feedback-dependent dynamics. In the GPT-5.5 test-time-scaling controls, oracle Parallel Sampling comes within 0.43 points of the evolved SearchQA skill but remains 30.96 points behind on SpreadsheetBench; Sequential Refinement recovers neither gain. Overall, persistent skill self-evolution is better understood as sparse, validation-filtered search with model- and benchmark-dependent returns, rather than steady improvement from additional rounds. The implementation is available at https://github.com/HKUST-KnowComp/rethinkskill.

Introduction

The paper isolates how successful and failed trajectories shape persistent skill evolution and tests whether additional rounds or inference can reproduce its gains. Across controlled studies, evolution is sparse, feedback-dependent, and often improves downstream performance.

Motivation

  • Existing comparisons do not isolate feedback composition because revision, selection, and evaluation procedures vary across systems.Before-and-after scores also conceal accepted revisions, rollbacks, stopping decisions, and whether gains reflect persistent skills.Source: Introduction · p. 1

Controlled Evaluation

  • The primary study holds the executor, optimizer, revision procedure, validation rule, and round budget fixed while varying Normal, Fail-only, and Success-only feedback.It uses 42 feedback runs across 14 model–benchmark settings, with common parent skills within each setting.Source: Introduction · p. 1

Main Findings

  • Only 55 of 388 candidates establish byte-distinct validation bests, showing that skill evolution is sparse rather than steadily improving.The analysis records candidate identities, acceptance, rollback, and best-so-far updates.Source: Introduction · pp. 1–2
  • Validation selects an evolved skill in 11 of 14 settings, and nine of those selections improve released-test performance.Nine also improve robustness and transfer, while seven improve both.Source: Introduction · p. 2

Feedback Comparison

  • All 11 evolved selections come from Normal or Fail-only feedback, whereas Success-only is never selected in the primary study.Normal is selected in nine settings and Fail-only in two, so every selected evolution includes failed trajectories.Source: Introduction · p. 2

Related Work

Prior self-evolving systems differ in whether they preserve successful behavior, diagnose failures, or combine both. Existing benchmarks and systems do not isolate these feedback sources under one fixed multi-round evolution procedure.

Self-Evolving Agent Skills

  • Success-gated, diagnosis-oriented, and mixed-trajectory systems respectively use successful discoveries, failures, or both as revision evidence.These approaches encode different intuitions about preservation, defect exposure, and broader but potentially conflicting feedback.Source: Self-Evolving Agent Skills · p. 2
  • Table 1 groups representative self-evolving systems by the execution feedback they use.The grouping organizes the related systems around successful, failed, or mixed evidence.Source: Table 1 · p. 2
  • Prior systems evaluate their native feedback policies as wholes rather than comparing Normal, Fail-only, and Success-only under fixed revision and validation procedures.Thus, the comparative roles of successful and failed evidence in persistent-skill search remain unclear.Source: Self-Evolving Agent Skills · p. 2

Agent Skill Evaluation and Benchmarks

  • Agent-skill benchmarks cover utility, continual learning, process quality, retrieval, organization, and cross-task evolution but do not isolate alternative feedback sources in fixed multi-round procedures.Examples include SkillsBench, SkillLearnBench, OpenClawBench, SkillsWild, SkillJuror, and SEA-Eval.Source: Self-Evolving Agent Skills · p. 2

Framework Overview

The framework repeatedly executes a current skill, exposes a selected feedback view to a revision operator, validates the resulting candidate, and freezes the best skill for controlled evaluation. It separates next-round acceptance from strict best-checkpoint updates.

Self-Evolution Process

  • The revision operator proposes a candidate from the current skill and condition-specific evidence.The candidate is distinct from the automatically carried-forward next-round skill until validation determines acceptance.Source: Framework Overview · p. 3

Validation-Gated Skill Revision

  • Artifact-level bests require both strict validation improvement and a byte-distinct candidate, excluding byte-identical score changes as execution variability.Each feedback view returns its validation-best skill, and validation alone selects among the three views before downstream evaluation.Source: Framework Overview · p. 3

Execution and Feedback Construction

  • Each round executes the current skill and constructs Normal, Fail-only, or Success-only feedback for revision.Normal includes both successful and failed trajectories; the other views contain only the named trajectory type.Source: Framework Overview · p. 3

Validation-Gated Skill Revision

  • Validation advances a candidate when its score does not decrease and retains the current skill after regression.A separate checkpoint updates only after strict validation improvement.Source: Framework Overview · p. 3

Controlled Evaluation

  • After evolution, frozen skills are evaluated through a common execution-and-verification path under controlled conditions.Changing the skill, execution conditions, task, or verifier isolates parent–evolved comparison, scaling, task variation, or verifier replacement.Source: Framework Overview · p. 3

Experiments

Across 14 matched model–benchmark settings, evolution is sparse but often beneficial: validation selects evolved skills in 11 settings, with downstream gains varying across benchmarks and feedback views. The experiments also test robustness, transfer, and test-time scaling under controlled protocols.

Experimental Setup

  • The protocol evaluates three models across five benchmarks using matched parent skills, fixed ten-round evolution, and feedback-only differences among Normal, Fail-only, and Success-only.Each round uses 40 training trajectories, or 36 for LiveMath, followed by full-split validation and stopping rules.Source: Experimental Setup · pp. 3–4
  • Test-time controls compare one-call evolved-skill deployment with oracle Parallel Sampling and Sequential Refinement under shared execution, verifier, and call budgets.SearchQA and SpreadsheetBench are the primary contrasts.Source: Experimental Setup · p. 4

Main Results

  • Validation selects evolved skills in 11 of 14 settings, and nine selected skills improve released-test performance.Gains range from 0.5 to 37.7 points; SpreadsheetBench has the largest reported gains, while LiveMath and SearchQA are more variable.Source: Main Results · p. 4, Table 2 · p. 4
  • All evolved selections include failed trajectories: Normal accounts for nine selections, Fail-only for two, and Success-only is never selected.The parent is retained in three settings, and selected rounds span both early and late evolution.Source: Main Results · p. 4
  • Across the feedback trajectories, 55 establish byte-distinct validation bests; Figure 2 marks these separately from other candidates, best rounds within feedback views, and benchmark-level selections.Source: Main Results · p. 5, Figure 2 · p. 5
  • Robustness and transfer each improve in nine of the 11 evolved selections, while seven improve both.SpreadsheetBench is positive across test, robustness, and transfer for all three models, but other benchmarks show mixed outcomes.Source: Main Results · p. 5

Analysis

Across models and benchmarks, evolved skills show sparse, feedback-dependent search: generalization varies by setting, later rounds are non-monotonic, and persistent changes can outperform extra test-time sampling on workflow-heavy tasks.

Cross-Model Evidence on SearchQA

  • Seven of eight SearchQA models improve released-test performance with validation-selected skills, but selected views, rounds, and validation-to-test rankings differ by model.Six of seven complete-result models improve by 0.86–15.07 points, while Gemini decreases by 0.36 points.Source: Cross-Model Evidence on SearchQA · p. 5

Generalization of Evolved Skills

  • Generalization depends on the benchmark, model, and retained skill: SpreadsheetBench improves across all three models, while LiveMath includes model-dependent reversals.Gemini–LiveMath gains 22.6 released-test points, 18.5 robustness points, and 16.7 transfer points; GPT-5.5 improves robustness but loses released-test and transfer performance.Source: Generalization of Evolved Skills · pp. 5–6

Effects of Feedback Composition

  • No feedback view has a fixed advantage: Normal is selected most often, Fail-only produces more validation improvements, and Success-only improves fewer settings on every reported metric.Normal is selected in nine settings versus two for Fail-only and none for Success-only; Fail-only produces validation improvements in 11 of 14 settings.Source: Effects of Feedback Composition · p. 6

Effects of Additional Evolution Rounds

  • Thirty-eight of 55 byte-distinct validation bests occur in rounds 1–4, yet six of 11 final selections first appear in rounds 6–9, making four rounds insufficient for most final selections.Late gains follow rejected candidates or saturation rather than steady improvement; DocVQA never exceeds its 96.2 parent across 23 candidates.Source: Effects of Additional Evolution Rounds · p. 6

Self-Evolution versus Test-Time Scaling

  • Oracle Parallel Sampling comes within 0.43 points of the evolved SearchQA skill but remains 30.96 points behind it on SpreadsheetBench.Sequential Refinement remains near the parent on SearchQA and declines on SpreadsheetBench, whereas SpreadsheetBench evolution encodes a multi-step workflow that sampling rarely reproduces.Source: Self-Evolution versus Test-Time Scaling · p. 7, Figure 3 · p. 7

Conclusion

The conclusion characterizes persistent skill self-evolution as sparse, validation-filtered search whose returns depend on feedback, models, and benchmarks rather than increasing steadily with rounds.

Conclusion

  • Only 55 of 388 candidates establish byte-distinct validation bests, while validation selects evolved skills in 11 of 14 settings.Selected skills improve released-test performance, robustness, and transfer in nine settings each, with seven improving both robustness and transfer.Source: Conclusion · p. 7
  • All 11 primary evolved selections include failed trajectories, but the leading failure-containing view varies across models and benchmarks.Normal accounts for nine selections and Fail-only for two; Success-only is never selected in the primary study.Source: Conclusion · p. 7
  • Test-time scaling can approximate modest answer-form gains but not the persistent workflow changes needed for SpreadsheetBench.Parallel Sampling comes within 0.43 points on SearchQA yet remains 30.96 points behind on SpreadsheetBench; Sequential Refinement reproduces neither gain.Source: Conclusion · p. 7

Limitations

The study’s evidence spans five heterogeneous benchmarks and 14 model–benchmark settings, but does not cover dedicated skill benchmarks such as SkillsBench and SkillLearnBench.

Limitations

  • The evaluation does not include dedicated skill benchmarks such as SkillsBench and SkillLearnBench, limiting evidence about broader skill-dependent and continual skill-generation tasks.The authors propose applying their matched feedback-isolation and validation-selection protocol to those benchmarks in future work.Source: Limitations · p. 8

Experimental Protocol and Evaluation Design

The evaluation combines fixed benchmark pools and metrics with a matched multi-round evolution protocol, then applies post-selection diagnostic and test-time-compute evaluations.

Benchmarks, Metrics, and Data Splits

  • Table A1 defines benchmark tasks, sources, fixed evaluation pools, and metrics, while treating ALFWorld as a separate analysis.Reported hard scores use the primary metric, with benchmark-specific scoring conventions for OfficeQA and DocVQA.Source: Table A1 · p. 10, Benchmarks, Metrics, and Data Splits · p. 10

Evolution Protocol and Feedback Isolation

  • The common protocol compares feedback views under fixed evolution controls and records complete multi-round trajectories and artifact identities.The protocol uses shared execution, feedback-isolation, and artifact-identity controls across matched comparisons.Source: Table A2 · p. 10, Table A3 · p. 11

Post-Selection Evaluation Design

  • Post-selection evaluation uses fixed panels for robustness and transfer and allocates GPT-5.5 budgets for diagnostic and test-time-scaling analyses.The design specifies robustness and transfer tiers, benchmark-specific probes, and maximum additional target-model calls per control.Source: Table A4 · p. 11, Table A5 · p. 11, Table A6 · p. 12

Complete Evolution and Endpoint Results

Across the primary runs, evolution produced sparse, feedback-dependent search trajectories, with validation selecting evolved skills in most settings but not always the same feedback view. The broader SearchQA analysis shows the same model-dependent pattern.

Validation and Released-Test Results

  • The ALFWorld runs produced three validation new bests among 30 candidates, while Success-only supplied the validation-selected skill and Normal was stronger on robustness and transfer.Repeated executions also showed that unchanged artifact identity can leave a branch at the parent despite evaluation variation.Source: Validation and Released-Test Results · p. 12
  • Among the selected skills, nine improved released-test performance, while the primary validation and released-test results were reported across five benchmarks for GPT-5.5 and nine settings for Gemini and DeepSeek.The supplied tables cover the parent and three feedback views, with parent retention indicated by r = 0 or byte-identical markers.Source: Table A7 · p. 13, Validation and Released-Test Results · pp. 13–14, Table A8 · p. 14

Evolution Dynamics

  • 55 of 388 candidates established byte-distinct validation new bests, and validation selected evolved skills in 11 of 14 primary settings.The evolution summary covers 42 primary runs; the corresponding tables track candidate yield, selection, and early stopping.Source: Table A9 · p. 15, Table A11 · p. 17
  • Failure-containing feedback produced new bests in 44/267 candidates versus 11/121 for Success-only, and all 11 primary evolved selections came from failure-containing runs.Failure-containing runs produced at least one new best in 21/28 cases, compared with 6/14 Success-only runs.Source: Evolution Dynamics · p. 17, Table A11 · p. 17
  • 38/55 new bests appeared in rounds 1–4, yet six of 11 selected evolved skills first appeared in rounds 6–9, showing that late discoveries determined most final selections.The round-wise yield table records candidate success by round, while early stopping reduced the number of attempted candidates.Source: Table A9 · p. 15, Evolution Dynamics · p. 17, Table A10 · p. 17

Cross-Model SearchQA Results

  • In the broader SearchQA analysis, validation-selected skills improved released-test performance for seven of eight models, with Normal or Fail-only selected for six models.Selected rounds ranged from 1 to 9, extending the sparse, feedback-dependent pattern across model families.Source: Cross-Model SearchQA Results · p. 17, Table A12 · p. 17

Post-Selection Generalization

Post-selection evaluations tested whether evolved skills generalized beyond released-test performance and whether validation-based selection matched downstream outcomes. Generalization was broad but not universal, and selection gaps remained possible.

Deployment Variability

  • Robustness probes changed surface form, added distractors, or altered the output contract, while transfer probes moved across tasks, sources, or distribution shifts.Each probe score averaged three deployments on a fixed panel.Source: Deployment Variability · p. 17, Table A14 · p. 18
  • Three-repeat evaluations used fixed benchmark-specific panels and paired parent-versus-evolved differences, while the SpreadsheetBench row used the paired Normal diagnostic.DocVQA compared independent parent executions.Source: Deployment Variability · p. 17, Table A13 · p. 18

Robustness and Transfer

  • Nine of 11 selected evolved skills improved released-test performance, robustness, and transfer separately, while seven improved all three measures.Every selected skill improved at least one of robustness or transfer.Source: Table A15 · p. 18, Robustness and Transfer · p. 19
  • SpreadsheetBench showed the strongest cross-model case: all three models selected an evolved skill, with released-test gains ranging from 28.8 to 37.7 points.Robustness and transfer were positive for every SpreadsheetBench model.Source: Robustness and Transfer · p. 19

Validation-to-Test Selection Gaps

  • Validation-based cross-view selection can leave performance on the table when another parent-or-feedback-view artifact has a higher released-test score.The selection-gap table defines this difference as the best released-test score minus the validation-selected score.Source: Table A16 · p. 19

Artifact-Level Evidence

Artifact-level evidence examines how sparse, validation-filtered evolution produces persistent, task-specific skill revisions. It combines evidence on candidate retention and distinct skill artifacts with excerpts detailing concrete spreadsheet-operation procedures and verification practices.

Selected Skills and Provenance

Cross-Model Convergence in Selected Skills

  • Selected skills converge on answer normalization for SearchQA, executable workbook postconditions for SpreadsheetBench, and theorem-level option auditing for LiveMath, with model-specific clauses.SearchQA emphasizes canonical entity answers; SpreadsheetBench verifies saved values; LiveMath audits hypotheses, quantifiers, meta-options, and answer contracts.Source: Cross-Model Convergence in Selected… · p. 21

Comparative Case Studies of Feedback Views

  • Normal and Fail-only views retained different revisions for unsupported answers and formula-like spreadsheet edits, while Success-only helped repeated task-wide specifications but failed on sparse incidental evidence.Both SpreadsheetBench candidates reached 82.1 validation, and the reported result used the Fail-only skill; the Success-only comparison includes selected SearchQA skills and a rejected LiveMath revision.Source: Comparative Case Studies of Feedback… · pp. 21–22, Table A20 · p. 22, Table A21 · p. 22

End-to-End Evolution Traces

  • 82.0 validation emerged only in GPT-5.5–SearchQA round 9 after earlier alternatives failed, while Gemini SpreadsheetBench rose from 38.5 to 82.1 through cumulative procedural repairs.The SearchQA gain came from a narrower property-value answer rule; SpreadsheetBench added literal values, loading safeguards, and execution-defect fixes before later proposals tied or regressed.Source: End-to-End Evolution Traces · p. 22, Table A22 · p. 23, Table A23 · p. 23
  • DeepSeek LiveMath improved from 40.0 to 54.3 with Normal feedback, whereas Success-only reduced validation to 11.4 and retained the parent.The selected Normal revision combined label formatting with guarded meta-option selection after seven unsuccessful revisions; the Success-only branch inferred a strength/equivalence rule from eight successes.Source: End-to-End Evolution Traces · p. 23, Table A24 · p. 24

Representative Selected-Skill Excerpts

  • Representative artifacts operationalize minimal canonical answers, theorem-precise option selection, and executable spreadsheet manipulation using literal Python-computed values and verifier-safe workbook handling.The excerpts specify exact answer contracts, hypothesis and quantifier checks, formula avoidance, dual loading, and preservation of workbook structure.Source: Representative Selected-Skill Excerpts · pp. 24–26
  • The excerpt safely deletes rows by collecting matching indices, sorting them in descending order, and removing them one by one.Descending deletion prevents index shifts from skipping rows or targeting the wrong rows.Source: Representative Selected-Skill Excerpts · p. 26
  • For contiguous data blocks separated by blank rows, the skill keeps each block’s header at the top rather than sorting it as data.The excerpt explicitly warns against mistakenly including headers in block sorting.Source: Representative Selected-Skill Excerpts · p. 26
  • The operational workflow explores the workbook, writes and executes solution.py, verifies the output file, and checks target cells for literal values.The workflow specifies INPUT_PATH and OUTPUT_PATH and uses a Dual-Load pattern.Source: Representative Selected-Skill Excerpts · p. 26

Test-Time Scaling and Verifier Diagnostics

The paper compares test-time scaling controls with evolved skills and separately measures how verifier choice changes observed gains.

Test-Time Scaling Calculation

  • Parallel Sampling uses oracle any-success and Sequential Refinement uses only the last response on the common support, with outcome-independent nested allocations for SearchQA.The oracle Parallel estimate is an upper bound rather than a deployable pass@1 estimator.Source: Test-Time Scaling Calculation · p. 27, Table A26 · p. 27
  • The selected SearchQA Normal evolution run costs 2,750 target-model calls, amortized as 1 + 2,750/n calls per deployment across n deployments.The metric includes the selected evolution run and subsequent deployments.Source: Test-Time Scaling Calculation · p. 27, Figure A2 · p. 28
  • GPT-5.5 test-time-scaling allocations differ by benchmark: SearchQA uses item-specific budgets, while the other benchmarks evaluate every item through their maximum budget.SearchQA assigns 25 items two attempts, 460 three, 229 four, 138 five, and 548 six attempts.Source: Table A26 · p. 27
  • At maximum budget, Table A25 reports percentage-point changes from the parent for Evolved, Parallel, and Sequential, defining Gap as Evolved minus Parallel.Source: Table A25 · p. 27

Output-Locked Verifier Sensitivity

  • The verifier disagreement rate is 4.8%, but rescoring leaves gains unchanged on SpreadsheetBench, DocVQA, and LiveMath, changes OfficeQA by 1.0 point, and changes SearchQA from −3.0 to 0.0.The comparison rescored fixed parent and evolved outputs with the original and one separately fixed alternative verifier per benchmark.Source: Output-Locked Verifier Sensitivity · p. 29, Table A28 · p. 30
Loading 2608.02636v1…