PaperlayerView exact revision on arXiv

Source-linked AI summary

Verifier-Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Tasks

Tianyi Xu, Yaolun Zhang, Yiran Wu, Joe Nguyen, Shengyu Dai, Saba Moeinizade, Zhenwen Shao, Qingyun Wu, Huazheng Wang

arXiv:2610.17609v1cs.AI

TL;DR

Open-ended tasks lack answer keys, so the verifier's judgment can affect both the response served and experience retained. UCoE co-evolves a contrast-tested executable verifier with harness state while model parameters stay fixed. It outperforms matched direct generation across six open-ended benchmarks and raises the mean on three verifiable benchmarks.

  • Problem

    Open-ended tasks lack ground truth, so feedback judgments determine both the response served and the experience retained for later tasks.

    Source: §1 · p. 1

  • Method

    UCoE evolves an executable verifier alongside harness state with model parameters fixed, selecting candidates on declared-preference response contrasts.

    Source: §1 · p. 2

  • Results

    Across six open-ended benchmarks, UCoE leads matched direct generation at both actor sizes (59.00% preference with Qwen3-8B); on three verifiable benchmarks, mean rises from 36.06% to 38.48%.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Driving capacity uses weakest-family discrimination to bound selection gain and provide a criterion for verifier updates.

    Source: §1 · p. 2

    Transfer from constructed contrasts to natural revisions depends on how well contrast families represent the errors those revisions contain.

    Source: §3.2 · p. 6

Abstract

from arXiv · show

Training-free self-improvement lets an agent use feedback to revise the response it is about to return and to update the harness state that shapes how it handles later tasks. On open-ended tasks that feedback rarely comes from reference answers or executable tests, so the agent has to rely on a model-based verifier. When the verifier is wrong, the current response can get worse, and misleading experience can persist across the task stream. A verifier that never changes also grows stale as the agent's responses improve and task demands shift. We introduce Unified Co-Evolution (UCoE), a training-free method that evolves an executable verifier program together with the agent's harness state while all model parameters stay fixed. UCoE synthesizes candidate verifier programs and tests them on response contrasts, pairs of responses that differ in one declared property and have a known preferred side. It selects the candidate whose discrimination on its weakest contrast family is highest, a quantity we call driving capacity. The selected verifier diagnoses one weakness in each draft, guides a single repair, decides which response is served, and determines whether the accepted repair may enter the harness state. As tasks complete, their diagnostics seed new candidate verifiers, which must beat the current verifier on the same contrasts before adoption. Across six open-ended benchmarks, UCoE averages 59.00% preference against matched direct generation with Qwen3-8B and 57.19% with Qwen3.8-27B, margins of 9.00 and 7.19 percentage points. On three verifiable benchmarks, the same method raises the mean from 36.06% for direct generation to 38.48%. Code is available at https://github.com/tianyi0216/Unified-Co-Evolution.

1 INTRODUCTION

Open-ended self-improvement ties response selection to persistent harness updates, yet lacks ground-truth feedback. UCoE co-evolves a contrast-tested verifier with harness state and outperforms matched direct generation across six open-ended benchmarks.

1 INTRODUCTION

  • Open-ended tasks lack ground truth, so the agent must judge correctness, coverage, relevance, and style; a mistaken judgment can harm the served response and persist in harness state.Source: §1 · p. 1
  • Holistic judges leave quality dimensions implicit (Gunjal et al., 2026; Shen et al., 2026), while fixed training-free verifiers may grow stale as responses and task demands shift (Anonymous, 2026b; Li et al., 2026).Source: §1 · pp. 1–2
  • UCoE evolves an executable verifier with harness state while parameters stay fixed, selecting contrast-tested candidates and using one diagnosis-guided repair to choose the served response and retained experience.Diagnostics from completed tasks seed candidates, which must meet update criteria on the same contrasts before replacing the verifier.Source: §1 · p. 2
  • UCoE achieves the highest preference against matched direct generation on every open-ended benchmark at both actor sizes, averaging 59.00% with Qwen3-8B and 57.19% with Qwen3.8-27B.On three verifiable benchmarks, it raises the mean from 36.06% to 38.48% without a loss on any benchmark.Source: §1 · p. 2
  • Driving capacity measures discrimination on the weakest contrast family, preventing strength on easy properties from masking a blind spot and providing a criterion for verifier updates.The paper also states that driving capacity bounds achievable selection gain.Source: §1 · p. 2
  • UCoE reports state-of-the-art training-free self-improvement on nine benchmarks, with gains also reported for GPT-5.4 actors and a Codex agent harness on SpreadsheetBench-2.The main experiments also report improved native metrics on every open-ended benchmark.Source: §1 · p. 2

2 RELATED WORK

Prior work adapts responses, experience, and harnesses, or develops model-based feedback for open-ended tasks. UCoE also adapts the feedback process, testing whole verifier programs on declared-preference contrasts while keeping adaptation outside model parameters.

2 RELATED WORK

  • Prior agents refine responses or adapt retained experience and prompt-level harnesses, but take feedback or selection as given (Madaan et al., 2023; Shinn et al., 2023; Lee et al., 2026a).Source: §2 · p. 3
  • Open-ended verifiers use rubrics, pairwise rewards, supplied signals, or executable code; UCoE instead tests whole programs on contrasts with declared preferences (Gunjal et al., 2026; Jia et al., 2025; Ma et al., 2024).Source: §2 · p. 3
  • Prior co-evolution spans rubric systems and executable reward DAGs, often relying on training, rollout traces, or labeled anchors; UCoE keeps adaptation outside model parameters and evaluates changes on declared-preference contrasts (Anonymous, 2026b; Li et al., 2026).Source: §2 · p. 3

3 METHOD

UCoE keeps the actor fixed while adapting harness state and an executable verifier over an ordered task stream. It selects verifiers using controlled response contrasts, then uses the deployed verifier to guide one repair and govern later harness and verifier updates.

3.1 PROBLEM SETUP

  • The fixed actor drafts from harness state, returns a response before task-derived state is updated, and never receives benchmark answers or final-evaluator feedback.The evaluation is prequential: each response uses state inherited from earlier tasks, before the current task contributes to the next state.Source: §3.1 · pp. 3–4

3.2 VERIFIER SYNTHESIS AND CONTRAST-BASED SELECTION

  • UCoE builds controlled response contrasts by breaking one property at a time, including general failure families and recurring benchmark-specific relations.Relations are retained only when they recur across at least three development prompts and can be expressed as a single binary criterion and targeted edit.Source: §3.2 · pp. 4–5
  • Driving capacity ranks verifiers by discrimination on their weakest contrast family, maximizing worst-case preferred-response selection across mixtures of tested families.The empirical candidate winner is evaluated on a selection split; later replacements must also beat the current verifier under held-out and regression criteria.Source: §3.2 · p. 5

3.3 RESPONSE REVISION AND CO-EVOLUTION

  • The verifier diagnoses one weakness, guides a local repair, and serves the revision only when its reward lead exceeds a margin; otherwise the draft is served.Only accepted repairs may add task-derived content to harness state; the bounded procedural-memory implementation adds at most one generalized trigger–action procedure per accepted revision.Source: §3.3 · p. 6
  • At scheduled checkpoints, UCoE uses completed-task diagnostics to propose verifier candidates, adopting a winner only after contrast-based improvement and regression checks.Any adopted verifier takes effect on the next task; otherwise the current verifier is retained.Source: §3.3 · p. 6

4 EXPERIMENTAL SETUP

Experiments compare UCoE with matched feedback baselines on open-ended and verifiable benchmarks, then test transfer in a Codex workbook-editing harness.

Benchmarks and models

  • The open-ended suite covers assistance, writing, multi-turn instruction following, and refinement; actors are Qwen3-8B and Qwen3.8-27B, alongside three objective-endpoint benchmarks.The open-ended benchmarks are Arena-Hard, WildBench, WritingPrompts, WritingBench, MultiChallenge, and RefineBench’s Free-Form split.Source: §4 · pp. 6–7

Baselines and evaluation

  • Four feedback baselines receive the same fixed-model, one-revision protocol, while open-ended outcomes use blinded, tie-adjusted pairwise preference against matched Direct Generation responses.Pairwise judgments are made in both presentation orders; ties receive half credit, so values above 50% favor the evaluated method.Source: §4 · p. 7

Results

  • 59.00% and 57.19% preference against matched Direct Generation, gains of 9.00 and 7.19 points, make UCoE best on all six benchmarks at both actor sizes.These are tie-adjusted preferences against matched Direct Generation responses.Source: Table 1 · p. 7, §4 · p. 8
  • 38.48% versus 36.06% is UCoE’s three-benchmark mean; it improves IFBench and Hard-MATH while matching Direct Generation on BigCodeBench-Hard.The table reports accuracy for IFBench and Hard-MATH, and Pass@1 for BigCodeBench-Hard.Source: §4 · p. 8, Table 2 · p. 8

Ablations

  • On Arena-Hard, removing current-task repair drops preference from 58.30% to 50.80%; removing the other components lowers it to 56.60–56.70%.The ablations remove one component at a time; Table 3 reports tie-adjusted preference against Direct Generation.Source: §4 · p. 8, Table 3 · p. 8

Harness transfer

  • In a 75-task SpreadsheetBench-2 Codex-harness stream, UCoE reaches 22.7% exact-match accuracy versus 17.3% for Direct Generation and LLM as a Judge.Gains come from financial-model and debugging tasks, while LLM as a Judge leads on template tasks.Source: §4 · pp. 8–9, Figure 3 · p. 8

Mechanism diagnostics

  • Repair targets vary by task: correctness leads on Arena-Hard at 78.9%, while emotion leads on WritingPrompts at 53.0%; selected verifier updates also raise frozen-pair selection gain.The two reported WritingPrompts updates raise selection gain by 10.16 and 4.69 points.Source: §4 · p. 9, Figure 4 · p. 9
  • External judgments favor 73 of the 94 proposals selected by the updated verifier; 76 of 639 procedure retrievals reuse a procedure after at least ten intervening tasks.Among the 94 proposals, seven judgments favor the draft and 14 are ties.Source: §4 · p. 9, Figure 4 · p. 9

6 CONCLUSION

UCoE co-evolves an executable verifier and harness state around a fixed actor, extending adaptation to the feedback process itself. Across open-ended and verifiable benchmarks, it improves on matched direct generation and raises the verifiable-task mean while preserving BigCodeBench-Hard Pass@1.

6 CONCLUSION

  • Across six open-ended benchmarks, UCoE improves on matched direct generation at both Qwen actor sizes; across three verifiable benchmarks, it raises the mean while preserving BigCodeBench-Hard Pass@1.Repairing the current task contributes most of the gain; verifier updates change selection behavior, and accepted-repair procedures are reused many tasks later.Source: §6 · p. 10
  • UCoE evolves an executable verifier program and harness state together while keeping the actor fixed.Response contrasts choose the verifier, which shapes responses and retained experience.Source: §6 · p. 10

AI USE STATEMENT

  • The authors report that generative AI tools assisted with writing and polishing the manuscript, and that they take responsibility for its content.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

  • The work identifies risks of reinforcing verifier errors or biases, retaining sensitive information without filtering, and amplifying harmful content.It recommends independent safety evaluation, human oversight, retention and deletion rules, and rollback mechanisms.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

  • The reproducibility statement directs readers to appendices covering proofs, benchmark and implementation details, costs, transfer, sensitivity, intervals, and qualitative examples.It also points to the lifecycle in Algorithm 1.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS

The limitations concern how well contrast-tested verifiers generalize, how verifier errors can affect both served responses and retained procedures, and the scope of theoretical claims. The lifecycle uses completed-task diagnostics for later candidate synthesis, while the analysis relates discrimination on targeted contrasts to selection behavior under stated assumptions.

A LIMITATIONS

  • Contrast performance does not guarantee reliable discrimination on natural revisions, interacting errors, or task properties absent from the declared families.A mistaken verifier judgment can also accept a harmful repair and store a misleading procedure.Source: A LIMITATIONS · p. 15

B ALGORITHM

  • At initialization and later checkpoints, UCoE synthesizes verifier candidates from completed-task diagnostics; failed or undefined update comparisons retain the current verifier.The harness state is H, diagnostic history is E, and the fixed relation inventory is R.Source: B ALGORITHM · p. 15

C THEORETICAL ANALYSIS

  • The theoretical analysis studies selectors on fixed response-pair distributions and states that population bounds require the given sampling assumptions.Source: C THEORETICAL ANALYSIS · p. 15

C.1 PAIRWISE DISCRIMINATION AND SELECTION GAIN

  • On valid targeted contrasts, centered accuracy measures selection of the preferred response, but these contrasts measure discrimination on the tested property rather than acceptance of natural revisions.The identity relating selector changes to adequacy applies to binary adequacy on a fixed pair distribution; carrying it to a changing deployed stream needs further assumptions.Source: §C.1 · pp. 15, 17, 18

C.2 WORST-FAMILY SELECTION OVER A FINITE VERIFIER POOL

The section connects worst-family verifier selection to performance across mixtures of tested contrast families and gives a finite-pool sampling guarantee. Its conclusions apply to declared, fixed family distributions and depend on the least-sampled family.

C.2 WORST-FAMILY SELECTION OVER A FINITE VERIFIER POOL

  • Maximizing signed driving capacity over a finite verifier pool also maximizes the worst-case success probability over mixtures of the tested families.For any selector, the worst mixture puts all its weight on the family with the lowest success probability.Source: §C.2 · p. 18
  • The finite-pool guarantee assumes fixed selector and family sample sizes, one comparison per source, and predeclared outcomes for missing or invalid scores.Candidate-dependent removal of comparisons is excluded.Source: §C.2 · pp. 18–19
  • The selected verifier’s capacity is at most ρδ/2 + τ/4 below the best pool candidate under the finite-pool guarantee.With four-decimal rounding, τ ≤ 10^-4; the sampling term is controlled by the least-sampled family.Source: §C.2 · pp. 19–20
  • A candidate’s measured gain over the current verifier must exceed 2ρδ to certify a population increase in signed capacity on the uniform event.This is a sufficient guarantee threshold; the update rule instead adopts based on empirical selection gain and held-out comparisons.Source: §C.2 · p. 20

C.3 EVOLUTION UNDER THE EMPIRICAL VERIFIER-UPDATE RULE

The update rule requires a selection-split improvement and held-out comparisons that favor the candidate without excessive family regression. The section distinguishes these exact empirical guarantees from population bounds, which require fresh independent validation sources and limit regression without certifying a gain.

C.3 EVOLUTION UNDER THE EMPIRICAL VERIFIER-UPDATE RULE

  • The update rule requires a strict selection-split worst-family gain, more candidate-only than current-only held-out successes, and no family regression beyond κ.A missing required family check retains the current verifier; initialization adopts a maximin winner without the paired test.Source: §C.3 · p. 21
  • Adoption strictly raises signed capacity on the selection split and pooled success on held-out comparisons, while held-out capacity may regress within the allowed tolerance.For κ = 0.10, each family’s held-out success rate may decrease by at most five percentage points; these are exact statements on the observed comparisons.Source: §C.3 · p. 22
  • With fresh validation sources independent of the pre-validation information, every population family change is at least its observed change minus its sampling allowance with conditional probability at least 1−δ.The bound is for one fixed selector pair and ranges over families, not the candidate pool.Source: §C.3 · pp. 22–23
  • The population bound limits regression on that event but does not certify a gain, and it applies to one checkpoint conditional on pre-validation information.Source: §C.3 · p. 23

D.1 BENCHMARKS AND DATASET SPLITS

The appendix specifies benchmark coverage and splits, then defines the bounded procedural memory used as harness state in the main experiments.

D.1 BENCHMARKS AND DATASET SPLITS

  • The six open-ended benchmarks span general assistance, writing, multi-turn instruction following, and response refinement; verifiable tasks add compliance, mathematics, code, and spreadsheets.Table 4 lists each benchmark’s task type, domain, and evaluated-task count.Source: §D.1 · p. 23, Table 4 · p. 24
  • Task order is fixed before deployment, calibration contrasts use disjoint sources, and adaptive methods never see benchmark answers or final-evaluator feedback.Source: §D.1 · p. 23

D.2 HARNESS-STATE INSTANTIATION

  • Harness state is a bounded procedural memory with up to 12 entries globally and two per quality dimension, retrieved as no more than two procedures within 64 words.Accepted repairs can add at most one distilled procedure after the response is returned; rejected revisions update usage statistics only.Source: §D.2 · p. 24

D.3 BASELINE IMPLEMENTATIONS

The appendix standardizes baseline revision behavior and details how UCoE builds, selects, applies, and updates verifiers using controlled response contrasts.

D.3 BASELINE IMPLEMENTATIONS

  • Revision-enabled baselines rescore one repair with their own feedback and serve it only when normalized reward rises by more than 0.01.Methods share a revision prompt and isolated procedural memories under the same rules.Source: §D.3 · p. 24

D.4 DEFAULT HYPERPARAMETERS

  • The Qwen3-8B and Qwen3.8-27B runs share UCoE’s default Arena-Hard hyperparameters, including checkpoint schedule and contrast quotas.Source: §D.4 · p. 24, Table 6 · p. 25

D.5 CONTRAST-SET CONSTRUCTION

  • UCoE tests candidates on contrasts made by changing one declared property of an actor-generated calibration response, with selection and held-out splits kept disjoint.Reference edits include omission, unsupported-claim injection, and irrelevant insertion; additional relation families require recurring, single-property differences.Source: §D.5 · pp. 24–26

D.6 IMPLEMENTATION DETAILS

  • Candidate verifiers are selected by their weakest-family centered accuracy on selection contrasts, then checked against the current verifier on held-out contrasts.The verifier scores semantic criteria while mechanical criteria are computed directly; diagnosis targets the weakest participating criterion and gives one localized repair instruction.Source: §D.5 · p. 26, §D.6 · p. 26
  • At checkpoints, recent task summaries seed one candidate verifier, which is adopted only if the update criteria are met; otherwise the current verifier remains.An adopted candidate first affects the next task.Source: §D.6 · p. 26

D.7 EVALUATION

The evaluation section defines the pairwise preference endpoint and describes a separate spreadsheet-agent study with bounded experience state, exact-match checking, and verifier programs grounded in workbook evidence.

D.7 EVALUATION

  • Open-ended preference is tie-adjusted pairwise preference against the same actor’s matched Direct Generation response, with ties worth half credit and 50% representing parity.A larger model judges both presentation orders without method identities, and the judgments are merged per task.Source: §D.7 · p. 27

D.8 AGENT-HARNESS STUDY ON SPREADSHEETBENCH-2

  • The spreadsheet study uses GPT-5.6-luna as actor and GPT-5.6-sol as verifier in isolated Codex sessions with a fixed task skill and sandbox.UCoE evolves only an experience block of at most 12 trigger-procedure entries, capped at 2,400 characters; tools and skill file remain fixed.Source: §D.8 · pp. 27–28
  • SpreadsheetBench-2 evaluation covers 75 business spreadsheet tasks and counts a task correct only when the recalculated workbook exactly matches the reference.UCoE is compared with Direct Generation and a fixed holistic-judge method on the same tasks and inputs; the agent never sees references, checker, or evaluator feedback.Source: §D.8 · p. 28
  • Spreadsheet verifier programs score requirements, formulas, references, preservation, numerical plausibility, recalculated errors, and whether the workbook changed from input.Preservation and error checks include deterministic tool checks; diagnosis receives frozen scores and returns one localized repair instruction.Source: §D.8 · p. 28

E.1 BENCHMARK-SPECIFIC RESULTS

UCoE has the highest overall mean on the reported native metrics for both actor sizes, while individual benchmark comparisons show that its lead is not uniform across baselines.

E.1 BENCHMARK-SPECIFIC RESULTS

  • UCoE’s largest reported native-metric gains are on Arena-Hard: style-adjusted scores rise from 7.0 to 10.2 at 8B and from 51.6 to 60.9 at 27B.It also leads WildBench at both sizes and RefineBench at 8B, and ties for the MultiChallenge lead at 27B.Source: §E.1 · p. 28
  • Rubric-based baselines score slightly higher on WritingBench Critic, LLM as a Judge leads MultiChallenge at 8B, and SpyRL edges RefineBench at 27B by 0.01.Thus the native metrics confirm preference results without showing a uniform UCoE lead over every baseline.Source: §E.1 · p. 28
  • 44.73 versus 43.42 at 8B and 69.87 versus 68.58 at 27B are the overall native-metric means for UCoE and the next method, respectively.Overall is the unweighted mean after scaling the two 1-to-10 ratings to 0 to 100.Source: Table 8 · p. 28, §E.1 · p. 28
  • Figure 5 plots listed-price cost per served task against tie-adjusted preference and breaks UCoE task costs down by model.Costs exclude final evaluation; the preference comparison uses matched direct generation.Source: Figure 5 · p. 29

E.2 COST ANALYSIS

UCoE averages 59.00% preference at $0.0142 per task, spending more than simpler baselines but outperforming them on mean preference.

E.2 COST ANALYSIS

  • 59.00% preference at $0.0142 per task is UCoE’s six-benchmark average against Qwen3-8B.Figure 5 compares preference with listed-price cost.Source: §E.2 · p. 29
  • At $0.0160 per task, SpyRL costs more than UCoE on five benchmarks and is less preferred on all six.SpyRL is the only baseline in UCoE’s price range.Source: §E.2 · p. 29
  • UCoE costs 2.2 to 2.8 times as much as simpler baselines, with mean preference 10.35 to 13.99 points higher.The verifier model accounts for most UCoE cost on every benchmark, making verifier scoring a future savings target.Source: §E.2 · p. 29

E.3 CROSS-BACKBONE TRANSFER AND ROBUSTNESS

Across other actors, hyperparameter settings, statistical intervals, and actor-generated revisions, the reported evidence supports UCoE’s transfer and robust performance, with MultiChallenge a weaker spot.

E.3 CROSS-BACKBONE TRANSFER AND ROBUSTNESS

  • UCoE leads all three actors on Arena-Hard at 56.70, 59.10, and 59.60, and GPT-5.4-mini on WildBench and RefineBench at 62.26 and 55.22.MultiChallenge is weaker: GPT actors are near parity at 51.17 and 50.98, where LLM as a Judge and DREAM do better.Source: §E.3 · p. 29, Table 9 · p. 29
  • 54.10 preference favored adopting one selected verifier candidate over keeping the current verifier across 244 mathematics tasks.The matched runs shared responses, verifier, and procedural memory through task 256; the comparison had 37 wins, 17 losses, and 190 ties.Source: §E.3 · p. 30

E.4 HYPERPARAMETER SENSITIVITY ON ARENA-HARD

  • 57.00% to 59.40% preference against direct generation across hyperparameter settings, with no default-versus-variant comparison separating from parity.Raising ϵ from 0 to 0.05 reduced accepted revisions from 195 to 107; the tested settings did not materially change served quality.Source: §E.4 · p. 30

E.5 STATISTICAL SUPPORT FOR THE MAIN TABLES

  • UCoE’s 95% interval lies above parity in eleven of twelve settings; the exception is MultiChallenge at 27B, whose interval touches parity at 49.80.The intervals use paired bootstrap resampling; Table 11 repeats the main preference entries with pointwise 95% intervals.Source: §E.5 · p. 30, Table 11 · p. 31

E.6 NATURAL-REVISION AUDIT ON WRITINGBENCH

  • Accepted revisions had higher Critic preference than rejected ones: 70.89% versus 51.46% at 8B and 69.47% versus 51.87% at 27B.Applying the recorded decisions yielded Critic gains of +0.072 and +0.036 per task, though some harmful revisions were accepted and helpful ones rejected.Source: §E.6 · pp. 30–31

E.7 QUALITATIVE EXAMPLES AND VERIFIER PROGRAMS

This section presents qualitative examples of UCoE’s verifier programs and their use of response contrasts to score responses, guide repairs, and update verifier and harness state. The examples show successful and failed repairs, contrast-gated adoption or rejection of verifier candidates, and limits in diagnoses and preference labels.

Complete verifier programs

  • 111 of the first 128 diagnoses identified correctness as the weakest criterion, and V1 was adopted after outperforming V0 on selection and held-out contrasts.V1 raised correctness’s weight from 0.25 to 0.5, added soundness and readability criteria, and multiplied the weighted sum by 0.4+0.6 co; a task-256 candidate lost on the selection split and was not adopted.Source: §E.7 · pp. 31–32

Qualitative examples

  • A diagnosis naming one defect enabled a successful repair that changed three lines while preserving the derivation, arithmetic, and answer.In the failed case, a false diagnosis led the actor to remove a correct derivation; the verifier then accepted the shorter answer after its correctness score flipped.Source: §E.7 · p. 32

Stored procedures

  • The procedure created at task 417 was retrieved twelve times and, three tasks after creation, accompanied a verifier-accepted revision preferred by the final evaluator.The stored rule instructed the actor to remove redundant failed attempts.Source: §E.7 · p. 32

E.7 QUALITATIVE EXAMPLES AND VERIFIER PROGRAMS

  • The deployed verifier uses one program to score drafts, revisions, and response contrasts, with model-scored semantic criteria and locally computed mechanical criteria.Its restricted expression evaluator aggregates criterion scores; the program cannot access embeddings, tools, the network, or benchmark answers.Source: §E.7 · p. 38
  • V1 raises correctness weight from 0.25 to 0.5 and scales every score by correctness, limiting fully incorrect responses to at most 0.4 of their weighted sum.The update retains nine existing criteria, adds soundness and safety plus readability, and reduces weights for instruction adherence, completeness, relevance, clarity, root-cause specificity, precision, and mechanical checks.Source: §E.7 · p. 41

Unsupported-claim insertion

  • The base response was preferred over the version with an inserted claim, with verifier scores of 0.90 versus 0.03, a margin of +0.87, and this contrast counted as a success.Separately, omitting the example input and printed output scored 0.90 versus 0.90 (margin +0.00) and was not counted as a success.Source: §E.7 · p. 42

Contrast examples

  • The verifier gave an irrelevant-content contrast a +0.10 margin (base 0.90 vs degraded 0.80), but did not separate the root-cause-specificity pair (0.35 vs 0.35).The root-cause relation was mined post hoc from blinded coding-prompt pairs, and its preference direction was a construction hypothesis rather than an independently verified global label.Source: §E.7 · pp. 42–43

End-to-end successful repair

  • The Bayes’ theorem revision scored 0.950 versus the draft’s 0.168 and was served; its verifier margin was 0.782, compared with 0.25 under V0.The diagnosis identified an incorrect false-positive-rate definition, and the revision corrected it; correctness and soundness moved from 0 to 1 while other criteria were unchanged.Source: §E.7 · pp. 44–45

A failed repair

  • The diagnosis wrongly rejected the correct result µ2(j) = n − j + 1 for depending on n, and the revision removed the derivation without changing the boxed answer.The supplied limitation states that the claim that the function must not depend on n is wrong for this identity, whose upper limit is n.Source: §E.7 · pp. 45–46

Score change and selection decision

  • One revision was served at aggregate score 0.95 after the verifier’s 0.1673 + 0.01 threshold, although the final evaluator preferred the draft.The verifier judged the bare answer correct and sound; the distilled procedure proposal was absent from final state and not reused. Under V0 the pair gained 0.25, compared with 0.783 under V1.Source: §E.7 · p. 46

Two checkpoint decisions

  • At task 128, the synthesis diagnostics recorded 35 accepted revisions among 128 tasks, with correctness weakest in 111 diagnoses.The residual diagnostics also listed six memory rules.Source: §E.7 · p. 47
  • At task 128, the selected verifier beat the current verifier on worst-family centered accuracy (−0.1667 vs. 0.0) and mean centered accuracy (0.2708 vs. 0.4583).On held-out families, the candidate improved omission and irrelevant-insertion centered accuracy, tied on unsupported claims and root cause, and won 19 paired comparisons only to 2 for the incumbent; no family regressed by more than 0.10. The paired lower bound cleared 0.5, but per-family bounds missed the threshold, so the authors make no population-level claim.Source: §E.7 · p. 47
  • At task 256, the candidate had worst-family centered accuracy −0.6667 versus 0.0 and mean centered accuracy −0.0208 versus 0.375, so the incumbent was retained.The held-out comparison was not run because the candidate showed no strict selection gain; diagnostics covered 27 accepted revisions across tasks 129–256, with correctness weakest in 107 diagnoses and six stored procedures.Source: §E.7 · p. 47

One procedure from creation to reuse

  • A task-417 clarity repair raised its score from 0.8698 to 0.95, and its distilled procedure was stored for later retrieval.The procedure advised removing redundant failed attempts and showing only the successful derivation; its final-state record lists 12 retrievals, 5 accepted and 8 rejected uses. At task 420, two procedures were retrieved and the revision scored 0.1078 to 0.9497, but that outcome cannot be attributed to one procedure.Source: §E.7 · p. 47

Related papers

Loading 2610.17609v1…