Source-linked AI summary
The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements
Yuanhao Zeng, Shaocheng Lan, Ao Lu, Kuang Yan, Heng Lin, Xinang Xu, Yuan Luo, Yueting Wang, Yexin Li, Ruo Shan, Kan Ren
TL;DR
Agent evolution can produce conflicting method rankings and often stalls, raising the question of whether it fails to propose improvements or cannot identify them reliably. The paper tests the Resolution Hypothesis across methods and candidate artifacts, finding that noisy evaluation makes both selection and method comparisons unreliable. It concludes that evaluation resolution, as well as proposal quality, matters for improvements to accumulate.
Problem
Agent evolution's effectiveness and method rankings vary across settings, while it remains uncertain whether proposals or evaluation's ability to identify improvements are the limiting factor.
Source: §1 · pp. 1–2Method
The paper tests the Resolution Hypothesis using variance decomposition, repeated method runs, and controlled candidate pools to examine method rankings and selection reliability.
Source: §3.1 · p. 4, §5 · p. 6Results
Helpful artifacts exist, but selection identifies the better candidate about half the time, and common practice identifies the truly best of four text-artifact methods on Terminal-Bench 33% of the time.
Source: §1 · p. 2Takeaways & Limitations
Evaluation resolution determines which improvements can be retained and accumulated, so progress may require making smaller improvements reliably detectable.
Source: §7 · pp. 9–10The experiments cover current text-artifact and harness evolution methods, leaving it open whether resolution similarly constrains program evolution.
Source: Limitations · p. 10
Abstract
from arXiv · showhide
Agent evolution follows a simple loop: a proposer writes candidate artifacts (prompts, skills, memories) and selection accepts those that score higher. This loop has shown promise, yet practitioners face a puzzle: methods trade wins across papers, and evolution often fails to bootstrap on certain tasks. We trace this puzzle to the Resolution Hypothesis: agent evolution stalls not because methods fail to propose better artifacts, but because noisy evaluations cannot reliably identify them. Across seven evolution methods and six models (over 10M LLM calls), we find support for the hypothesis at both levels where evaluation scores are used. At the candidate level (choosing artifacts), helpful candidate artifacts exist, but selection is no better than random at telling them from harmful ones, so errors accumulate across rounds. At the method level (ranking methods), typical margins of a few percentage points are smaller than the difference between two independent runs of the same method, so published rankings often rest on luck. Together, these findings indicate that agent evolution is often bottlenecked by reliably identifying improvements. This calls for a shift in how we develop and evaluate self-improving agents, from designing ever more candidate improvement systems to making true improvements detectable and accumulable.
1 Introduction
The paper argues that agent evolution is bottlenecked by noisy evaluation that cannot reliably identify improvements, despite useful artifacts being available. It tests this at the candidate and method levels and finds that selection errors and run-to-run variation undermine reliable gains and rankings.
1 Introduction
- 33% of the time, current common practice identifies the truly best of four text-artifact methods on Terminal-Bench; candidate selection is also about half-right.The paper reports that the latter is no better than a coin flip and that score increases need not correspond to better artifacts.Source: §1 · p. 2
- The Resolution Hypothesis says evolution is bottlenecked by identifying improvements, not proposing them; below evaluation resolution, selection degrades to chance.Proposal determines the size of improvements on offer, while resolution determines how small an improvement can still be reliably identified.Source: §1 · p. 2
- The study tests seven methods across six models and four agent benchmarks, with nine additional benchmarks used to re-examine previously reported gains.The authors ran each method eight times per model and benchmark and evaluated resulting artifacts up to 16 times, totaling 1,030 runs and over 2.1 million test trajectories.Source: §1 · pp. 2–3
- Typical method margins are smaller than the difference between independent runs, while the largest reported gains exploit evaluation shortcuts and stall after evaluation repair.The paper reports that shortcuts lift nearly every task at once, producing gains large enough to exceed evaluation resolution.Source: §1 · p. 3
- Helpful candidate artifacts exist, but common score-based and LLM-judgment selectors perform near chance at identifying the better candidate.The paper reports that selection errors compound across rounds and produce large differences between independent runs of the same method.Source: §1 · p. 3
- The authors suggest enlarging proposed improvements or making smaller ones reliably detectable, including through lower-variance checks and separate proposer and selector evaluations.They argue higher-resolution feedback may be as important as more powerful evolution algorithms.Source: §1 · p. 3
2 Related Work
Prior work studies several forms of agent evolution and documents fragility, while related concepts address noisy selection and evaluation. This paper isolates evaluation noise from run-to-run variation and tests proposal quality separately from selection reliability.
2 Related Work
- Agent evolution methods vary in whether they evolve prompts [2], skills [24] [37], memories [27] [43], or executable code [18] [42].Prior analyses report run-to-run and task-order sensitivity, sparse feedback-dependent gains [19], and skill-induced failures or contamination from uncontrolled accumulation [7] [17].Source: §2 · p. 3
- Noisy selection connects this work to best-arm identification, uncertainty-aware RL evaluation [1], and PACE's false-commit analysis [30].The paper distinguishes evaluation noise from run-to-run variance and uses controlled pools to test proposal quality and selection reliability independently.Source: §2 · p. 3
- Prior findings point to noisy selection as a shared failure mode across agent-evolution methods.The paper frames its contribution as isolating that mechanism rather than treating each reported fragility separately.Source: §2 · p. 3
3 Variance Decomposition and Experimental Setup
The paper separates evaluation noise within an artifact from true-quality variation across runs, because each affects which improvements can be identified. It applies this framework across seven methods, multiple models and benchmarks, and experiments on variance, reported gains, candidate pools, and repeated trajectories.
3 Variance Decomposition and Experimental Setup
- The section frames the experiments around separating evolution-algorithm stochasticity from evaluation-instrument noise.It introduces variance decomposition as the basis for determining whether benchmark scores can identify improvements.Source: §3 · p. 3
3.1 Two Sources of Score Variance
- Observed-score variance combines evaluation noise within an artifact with variation in true artifact quality across independent runs.Identifying a fixed artifact's gain over a reference depends on evaluation noise alone; comparing methods depends on both variance sources.Source: §3.1 · pp. 3–4
3.2 Experiments
- The study covers seven text-artifact and harness methods across six models, with variance measurement on four benchmarks and gain audits on twelve.The gain audit re-examines reported gains under released and repaired evaluations; the variance study repeatedly evolves and evaluates each method to estimate both variance sources.Source: §3.2 · p. 4
- Controlled skill pools test proposal gains on disjoint training and test tasks, while selection rules are benchmarked on candidates from Terminal-Bench and DeepSWE trajectories.A separate experiment re-evaluates intermediate artifacts from eight SkillOpt-Lite and eight GEPA runs; the experiments consumed over 10M LLM calls and 400K CPU core hours.Source: §3.2 · p. 4
4 Resolution at the Method Level
Typical method margins are often smaller than single-run evaluation noise, while the largest reported gains are reproduced by one rule and largely vanish after evaluation repair. These findings show how evaluation resolution limits both ranking methods and retaining genuine improvements.
4.1 Single Reports Cannot Rank Methods
- For over 60% of same-model, same-benchmark configuration pairs, the gap between all-run means was below one single-run standard deviation.Across methods, 53% to 79% of comparisons fell below that threshold; the typical score standard deviation was 1.7 to 4.1 pp.Source: §4.1 · p. 5, Figure 2 · p. 6
- The median reported method margin was 3.4 pp, and identifying it with 80% power required about 5 to 23 runs per method across four benchmarks.At the lower-quartile margin, Terminal-Bench required 181 runs; a single run picked the best of four text-artifact methods only 33% of the time there, versus 25% by random guess.Source: §4.1 · p. 5, Figure 2 · p. 6
4.2 Large Reported Gains Come from Evaluation Shortcuts
- Across 27 model–benchmark pairs with reported gains above 3 pp, one added rule recovered a median 90% of the gain.The rule typically adapted to a grader's answer-format preference or to a benchmark version that differed from the official evaluation.Source: §4.2 · p. 5
- The median gain across audited shortcuts fell from 17.6 to 0.8 pp after evaluation repair, and rerunning methods did not restore it.A single rule recovered a median 90% of reported gains above 3 pp across 27 model–benchmark pairs; rules typically exploited evaluation shortcuts.Source: §4.2 · pp. 5–6
- The controlled contrast supports the prediction that evolution stalls when only small, task-specific improvements remain below evaluation resolution.The audited shortcuts created large detectable gains; repairing the evaluation removed those gains.Source: §4.2 · p. 6
5 Resolution at the Candidate Level
Candidate pools contain genuinely helpful skills, but practical selectors identify better artifacts at about chance and fail to capture the pool's available gains. Across rounds, noisy selection inflates in-loop scores and increases run-to-run variance.
5.1 Helpful Artifacts Exist
- At K=100, Quality@K exceeded its equal-quality null by +3.3 to +3.4 pp, showing that larger candidate pools contain genuinely stronger skills.The study evaluated 100 proposed skills on 10 disjoint Terminal-Bench v2.1 test tasks; under the 2-hour budget, Quality@K rose from 49.2% at K=1 to 61.8% at K=100.Source: §5.1 · pp. 6–7, Figure 4 · p. 7
- The strongest shared skill reached 57.5% versus the 49.2% pool mean under the 2-hour budget, an 8.3 pp lead across all 10 test tasks.Under original timeouts, the strongest shared skill scored 48.1% versus 39.7%, an 8.4 pp lead.Source: §5.1 · p. 7
5.2 Selection Cannot Identify Them
- Execution-based selection chose the better artifact in 49.7% of head-to-head comparisons, while native SkillOpt-Lite and GEPA selection reached 48.6%.Regressions were 30.5% of the pool but 62.1% of accepted candidates; practical selectors remained near random as K increased to 100, while an oracle gained with K.Source: Figure 4 · p. 7, §5.2 · p. 8
- Model-based judges also lacked reliable pairwise accuracy; reading full trajectories added noise rather than restoring identification.Their reading noise was independent of evaluation noise, so the two noise sources accumulated rather than canceled.Source: §5.2 · p. 8
5.3 Noise Accumulates Across Rounds
- After eight evolution runs, in-loop gains reached 6.9 to 9.0 pp, while independent re-evaluation found gains between +0.4 and −3.2 pp.Noisy accept/reject decisions repeatedly select lucky high-scoring candidates, so validation scores rise without corresponding capability gains.Source: §5.3 · p. 8, Figure 5 · p. 9
- Corrected run-to-run variance reached 4.2 to 24.5 squared pp by the final round, as per-round selection errors compounded across runs.The resulting differences between independent runs could be as large as method margins reported in the literature.Source: §5.3 · p. 8, Figure 6 · p. 9
6 Towards Better Evaluation Resolution
The paper proposes practical ways to improve evaluation resolution at both the candidate and method levels. These approaches offer diagnostics, but do not replace end-to-end evaluation or guarantee that behavioral changes improve task success.
Candidate level: finer-grained feedback
- Programmatic checks of intermediate behavior can expose shifts missed by terminal pass/fail scores, with unmasked-diagnostic rates showing 13×–57× lower evaluation noise.Such checks are cheap diagnostics, not a fix for selection: triggering more intermediate checks does not necessarily mean solving more tasks.Source: §6 · p. 9
Method level: component-wise evaluation
- Component-wise proposer and selector curves can reveal where stagnation lies, while reporting evaluation noise and run-to-run variance helps readers judge whether margins exceed resolution.These diagnostics split a below-resolution end-to-end comparison into component comparisons that exceed resolution; they do not replace end-to-end evaluation.Source: §6 · p. 9
7 Conclusion
The conclusion identifies evaluation resolution, rather than proposal alone, as the factor governing which improvements can be retained and accumulated. It recommends resolution-aware evaluation and reporting so claimed gains can be judged against noise.
7 Conclusion
- Evaluation resolution determines which proposed improvements can be retained and accumulated; insufficient resolution loses true improvements in noise.Large changes that exploit evaluation shortcuts can dominate because they remain detectable despite noisy evaluation.Source: §7 · p. 9
- The paper recommends resolution-aware evaluation through less noisy intermediate metrics, separate proposer and selector diagnostics, and reporting evaluation noise alongside run-to-run variance.These measures let readers judge whether claimed margins exceed evaluation resolution.Source: §7 · p. 10
Limitations
The study’s conclusions are bounded by its focus on frozen-weight agent evolution and a finite set of methods, models, and benchmarks. It also leaves open whether its findings extend to program evolution, larger candidate pools, or future stronger models.
Frozen weights only
- The experiments cover frozen-weight methods that evolve external state; weight-updating reinforcement-learning methods may face different resolution constraints.The external artifacts include prompts, memories, skills, and harnesses.Source: Limitations · p. 10
Candidate pool scope
- Candidate-level evidence is limited to Terminal-Bench v2.1, leaving larger pools, other generation strategies, and other benchmark domains open.The study evaluates pools of 100 skills and re-evaluates trajectories for two methods and two models.Source: Limitations · p. 10
Model strength and resolution
- Future, substantially stronger models could have lower evaluation noise, but the study cannot rule this out.In the tested grid, noise shrinks mainly near the ALFWorld success-rate ceiling, where little headroom remains; Terminal-Bench noise stays at 3.5 to 4.0 pp across three models.Source: Limitations · p. 10
Grid coverage
- Seven methods, six models, and four benchmarks provide broad but non-exhaustive coverage; different artifact representations or task structures may have different variance profiles.Source: Limitations · p. 10
Beyond agent artifacts
- The experiments cover text-artifact and harness evolution, not program evolution, so whether resolution constrains program evolution remains open.Program evolution scores deterministic code and may structurally reduce evaluation noise through cheap parallel re-evaluation and accessible diagnostics.Source: Limitations · p. 10
Use of AI Assistants
This material describes the study’s use of AI assistants, related work, variance analysis, and experimental setup. It specifies how the two sources of score variation are estimated and summarizes the models and evolution protocols.
Use of AI Assistants
- The authors used AI coding assistants for experiment and analysis code, reviewed all outputs, and manually checked candidate instructions flagged in the gain audit.Source: Use of AI Assistants · p. 10
A Extended Related Work
- Prior work spans varied propose-evaluate-select methods and documents fragility, while this study separates evaluation noise from run variation and examines proposal quality apart from selection reliability.Related analyses address run variance and task order, feedback-dependent gains [19], and skill-induced failures or accumulation [7] [17]; noisy selection connects to best-arm identification, uncertainty-aware RL evaluation [1], and false-commit analysis [30].Source: A Extended Related Work · p. 14
B.1 Derivation of the Variance Decomposition
- An evolution experiment samples an artifact through method stochasticity, then evaluates it with execution randomness; true quality is the expected observed score given that artifact.The variance decomposition is exact: score fluctuations arise from evaluation noise or from drawing a different artifact.Source: §B.1 · p. 14
B.2 Estimators for the Two Variance Components
- The estimators use repeated evaluations per artifact to estimate evaluation noise and remove its contribution from variation among artifact means; both variance estimators are unbiased.A negative run-to-run variance estimate is reported as zero; this occurred in 9 of 117 method configurations.Source: §B.2 · p. 15
C Experimental Setup Shared Across Experiments
- The shared experimental-setup appendix identifies which models, methods, and benchmarks each experiment group uses and where its protocol is described.Source: C Experimental Setup Shared Across… · p. 15
C.1 Models and Inference
- Six models span the capability spectrum, and each method role uses the same model so no method benefits from a stronger artifact-writing model.Inference settings differ by benchmark: three use medium reasoning effort and a 16,384-token cap, while Terminal-Bench uses provider defaults.Source: §C.1 · p. 15
C.2 Methods and Evolution Budgets
- Seven methods evolve text artifacts or harnesses under matched per-model budgets, but differ in proposal and acceptance protocols.Validation scores use one rollout per task; Trace2Skill and ReasoningBank keep proposals without score-based selection on three benchmarks, while harness methods use differing training- or validation-score rules.Source: §C.2 · pp. 15–16, Table 2 · p. 16
C.3 Benchmarks
The benchmarks combine task-specific scoring and agents with separate training, validation, and held-out test splits. Experiment coverage varies with cost, with the most expensive settings run on subsets of models.
C.3 Benchmarks
- Benchmarks use varied environments and scoring, including ALFWorld household-goal success, OfficeQA normalized exact match, and binary per-task scores.SpreadsheetBench uses the Verified-400 subset and scores 281 test tasks; its released grader does not recalculate formulas.Source: §C.3 · pp. 16–17, Table 3 · p. 17
- Terminal-Bench v2.1 uses 18 training, 18 validation, and 53 held-out test tasks, with task-specific tests in isolated sandboxes.The split is stratified by task difficulty.Source: §C.3 · p. 17
- Candidate-skill experiments use separate pools of 100 skills to measure out-of-sample proposal gains and selection reliability.The proposal pool evaluates on 10 test tasks disjoint from the 10 training tasks; the selection pool spans 11 Terminal-Bench tasks.Source: §C.3 · pp. 17–18
C.4 Coverage of Each Experiment Group
- Model coverage follows cost: the cheapest text-artifact settings use all six models, while the more expensive harness and Terminal-Bench settings use subsets.A harness evolution run consumes a median of 26M tokens; a Terminal-Bench run consumes about 43M tokens, plus 424 containerized test episodes per resulting artifact.Source: §C.4 · p. 18, Table 5 · p. 18
D.1 Construction of Figure 2 and Full Results
The analysis separates run-to-run artifact variation from evaluation noise, then compares that uncertainty with method differences and reported literature margins. Across the surveyed results, typical margins are small relative to the uncertainty of a single run.
D.1 Construction of Figure 2 and Full Results
- 217 of 341 paired configuration comparisons fall below the diagonal, meaning their mean-score difference is smaller than the standard deviation of a single-run comparison.The share is similar for method pairs and method–baseline pairs.Source: §D.1 · p. 18
- Single-run score distributions overlap for most method, model, and benchmark combinations, and many are multimodal because independently evolved artifacts differ in quality.Figure 7 shows distributions resampled from observed artifacts and repeated evaluations.Source: §D.1 · p. 19, Figure 7 · p. 20
D.2 Repetitions Required for Method-Level Comparisons
- At the median literature margin of 3.4 pp, identifying a method difference requires about 5, 10, 19, and 23 independent runs on SpreadsheetBench, OfficeQA, ALFWorld, and Terminal-Bench.With many evaluations per artifact, the corresponding requirements fall to about 1, 5, 9, and 3 runs.Source: §D.2 · p. 19, Table 7 · p. 19, Figure 2b
- A single run picks the top text-artifact method correctly 67%, 51%, 73%, and 33% of the time on SpreadsheetBench, OfficeQA, ALFWorld, and Terminal-Bench.A random guess gets the top method right 25% of the time.Source: §D.2 · pp. 19–20
D.3 Margins Reported in Agent-Evolution Papers
- Across 171 literature columns with an optimization competitor, the median proposed-method margin over its strongest competitor is 3.4 pp, while the median gain over the unoptimized baseline is 9.3 pp.The survey includes main results tables from nine papers and excludes analyses, ablations, and appendix tables.Source: §D.3 · pp. 20–21
- None of the 178 surveyed results columns reports an error bar or variance estimate, and only six Trace2Skill columns average over multiple evolution runs.In four papers, the strongest competitor scores at or below the unoptimized baseline in 15 columns.Source: §D.3 · p. 21
E.1 Audit Design
- SkillOpt-Lite reports 36–45 pp SpreadsheetBench gains, and the study reproduces gains of 29–38 pp with its own evolved artifacts.On LiveMathematicianBench, released SkillOpt-Lite skills yield reproduced gains of 27–37 pp.Source: §E.1 · p. 22
- The audit compares released gains, a baseline with one artifact-derived rule, and repaired evaluations; repairs restore official grading or use semantic grading.The one-rule test establishes that a sentence can reproduce a gain, not that the artifacts contain nothing else of value.Source: §E.1 · pp. 21–22
E.2 Per-Benchmark Results
The audit finds that some large reported gains track evaluation shortcuts, while repaired evaluations leave smaller, benchmark-specific gains and substantial unresolved headroom. Re-evolution and benchmark-level analyses distinguish shortcut effects, genuine improvements, and limits on what the audit can establish.
E.2 Per-Benchmark Results
- Where an evaluation shortcut is present, a one-rule baseline matches reported gains and repaired scores shrink near zero; on several benchmarks reported gains exceed repaired headroom.The audit's one-rule test does not reproduce every gain.Source: §E.2 · p. 23, Table 10 · p. 23, Figure 8 · p. 24
- HotpotQA retains a 5–8 pp gain after repair, while HoVer gains arise from retrieval strategies and a single sentence recovers different shares across models.For HoVer, the sentence recovers 43% and 41% of gains for GPT-5.4 nano and GPT-5.6 Luna, and 91% for DeepSeek-V4-Flash.Source: §E.2 · p. 23
- On LiveMathematicianBench, the evolved skill more often gets the stronger-result option followed than a single rule, but repaired gains remain near zero.The rule recovers 0% to 51% of the gain, and the skill differs in its reasoning, default choice, wording coverage, and repeated instruction.Source: §E.2 · p. 23
E.3 Re-evolving Under Repaired Evaluations
- With repaired feedback, text-artifact gains on OfficeQA and SpreadsheetBench are at most 1.9 pp, except SkillOpt-Lite's 5.8 pp on SpreadsheetBench with DeepSeek-V4-Flash.HotpotQA is an exception: GEPA improves retrieval for all three models under either evaluation.Source: §E.3 · pp. 24–25
- On repaired evaluations, some harness gains are larger but trace to specific changes: restoring ALFWorld history or writing spreadsheet values instead of formulas.Meta-Harness and AHE improve two models by 16–32 pp on ALFWorld; AHE gains 11–13 pp on spreadsheet tasks requesting formulas, versus 3–4 pp on other tasks.Source: §E.3 · p. 26
E.4 Headroom After Repair
- SearchQA scores can be earned without retrieval, while LiveMathematicianBench's stronger-result option is correct in all 85 released-split questions where it appears.The option appears in 46 of 106 test questions, allowing a model that always selects it to answer those 46 correctly without reading the mathematics.Source: §E.4 · pp. 26–27
- Terminal-Bench leaves 28 pp and 46 pp of headroom for two models, yet the best method gains only 1.0 and 3.6 pp, below single-run standard deviation.HotpotQA, SpreadsheetBench, and HoVer retain gains after repair, with retrieval or adaptation to the recalculation engine identified as their sources.Source: §E.4 · p. 27
E.5 Scope of the Audit
- The one-rule audit shows that a rule can reproduce a gain, not that the gain requires that rule; it can also miss gains spread across multiple strategies.The repaired LiveMathematicianBench test balances the leaked option only to half-correct, so capability is scored on questions without that option.Source: §E.5 · pp. 27–28
F.1 Candidate Pools, Timeouts, and Evaluation Protocols
The appendix details how the candidate pools, artifact-quality estimates, and selectors are constructed, and tests whether averaging validation rollouts changes evolution outcomes. It also describes per-task selection curves and independent trajectory re-evaluation.
F.1 Candidate Pools, Timeouts, and Evaluation Protocols
- The Proposal Pool tests 100 skills generated from 10 training tasks on 10 disjoint test tasks, while the Selection Pool tests 100 skills across 11 difficulty-stratified tasks.Proposal skills received 16 rollouts per artifact–task pair; Selection Pool artifacts received 8, extended to 15 for the ten highest-scoring per task, with 12 baseline rollouts per task.Source: §F.1 · p. 28
- At K=100, Proposal Pool excess over the matched null stayed positive across all five timeout cutoffs, from +3.22 to +3.41 pp.The pooled K=1 success rate rose from 39.69% under original timeouts to 49.23% at two hours; extended budgets particularly benefited tasks such as path-tracing-reverse and rstan-to-pystan.Source: §F.1 · p. 28, Figure 9 · p. 29, Figure 10 · p. 30
F.1.1 Proposer Research Protocol
- The proposer protocol directs researchers to investigate failure mechanisms, verify fixes with probes, and turn evidence into actionable skills stripped of training-instance literals.It emphasizes checking actual verifier assertions and workflow decision points rather than relying on plausible narratives or comments.Source: §F.1.1 · pp. 28–30
F.2 Posterior Estimation of Quality@K
- An empirical-Bayes hierarchical model regularizes noisy artifact success counts, estimating task-specific beta-prior parameters and posterior success probabilities.The primary estimator caps prior concentration at κmax=100; Quality@K estimates the best quality among K uniformly drawn candidates, relative to an equal-quality null.Source: §F.2 · p. 31
- With κmax=100, the two-hour Proposal Pool curve reached 61.77% versus a 58.36% null at K=100 (+3.41 pp), separating cleanly for K≥4.A cap of 30 inflated the matched null and left the +2.20 pp excess within overlapping bands; caps of 300 and infinity yielded larger excesses, while infinity had a point-mass degeneracy on near-homogeneous tasks.Source: §F.2 · p. 31, Figure 11 · p. 32
F.3 Selection Operators and Evaluation Metrics
- Selectors are evaluated against held-out artifact-quality rollouts that they do not observe, and native SkillOpt-Lite and GEPA selection logic is also assessed.The appendix defines pairwise accuracy, false acceptance, and improvement and regression rates; it also defines equivalent budget as the oracle candidate count needed to match a selector’s K=100 quality.Source: §F.3 · p. 32, §F.4 · p. 33
- Averaging 16 validation rollouts raised SkillOpt-Lite’s mean final test score from 61.4% to 64.4% and its best artifact’s score from 64.8% to 72.7%.Single-rollout validation sometimes gave ordinary candidates inflated scores that stalled evolution; averaging was costly and could make greedy acceptance more conservative.Source: §F.3 · p. 33
F.4 Per-Task Selection Curves
- Per-task Selection Pool curves show selected-artifact success probability against candidate budget K using held-out quality references; trajectory re-evaluation targets eight rollouts per artifact–task pair.The re-evaluation uses 119 artifact versions from independent SkillOpt-Lite and GEPA runs on the same 18 validation tasks; its variance correction retains negative estimates to avoid upward bias in small early-round estimates.Source: §F.4 · p. 33, Figure 12 · p. 34, G Trajectory Re-evaluation · p. 35
H Low-Variance Process Diagnostics
Process metrics can reveal behavioral changes at lower evaluation noise than terminal pass/fail, but they are diagnostics rather than selectors. The appendix also presents three candidate skills and examples of their concrete guidance.
H Low-Variance Process Diagnostics
- For the baseline and all seven methods, process-metric evaluation noise was 13 to 57 times lower than terminal-reward noise.The process metric measures the fraction of check commands whose failure diagnostics are not masked; it is computed per trajectory for comparison with terminal pass/fail.Source: H Low-Variance Process Diagnostics · p. 35, Table 13 · p. 36
I.1 Shareable Artifacts in the Candidate Pool
- Seeds 027 and 099 tied for the best average gain across 11 tasks at +11.4 pp, while seed 058 scored perfectly on mteb-retrieve but added little when shared globally.All three artifacts present a software-engineering skill focused on preserving behavior, implementing the requested API, and verifying with tests.Source: §I.1 · p. 36, §I.1.1 · p. 36, §I.1.2 · p. 37
I.1.1 Artifact seed-027 (best average gain, +11.4 pp)
- Seed 027 warns against substituting idiomatic names for exact required public identifiers and against hand-rolling tokenizer or embedding calls instead of following the named package’s conventions.Its guidance calls for extracting exact symbols and messages and verifying role-specific encoding behavior.Source: §I.1.1 · p. 36
I.1.2 Artifact seed-099 (tied best, +11.4 pp)
- Seed 099 cautions that arbitrary text separators and shared query/document prompt settings can change exact token counts or retrieval rankings.It also advises preserving the literal public identifiers and signatures expected by hidden checkers.Source: §I.1.2 · p. 37
I.1.3 Artifact seed-058 (best on mteb-retrieve)
- Seed 058 stresses matching documented tokenizer text shape, special-token settings, and distinct query versus passage prompts for retrieval encoders.It also recommends keeping the exact public identifier and signature reachable to the checker rather than renaming or re-wrapping it.Source: §I.1.3 · pp. 37–38
Related papers
This paper cites
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering2018
- GLM-5: from Vibe Coding to Agentic Engineering2026
- DocVQA: A Dataset for VQA on Document Images2020
- Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces2026
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills2026
- Rethinking the Evaluation of Harness Evolution for Agents2026