PaperlayerView exact revision on arXiv

Source-linked AI summary

Metrics That Write Themselves: Evolving an Evaluator from Its Own Blind Spots

Xing Zhang, Yanwei Cui, Guanghui Wang, Zhihao Lin, Peiyang He

arXiv:2608.18744v1cs.AIcs.CLcs.SE

TL;DR

Open-ended outputs lack reliable automatic metrics, limiting how effectively agents can improve. EvalCEGAR evolves executable defect detectors from counterexample collisions, and six of eight runs produce operators that help on unseen tasks, including exact transfer to a second benchmark.

  • Problem

    Open-ended outputs such as reports lack reference answers and settled criteria for quality, while existing substitutes miss semantics or inherit judge biases.

  • Method

    EvalCEGAR searches for correct–incorrect answer collisions in the current operator pool, targets the shared blind spot, and widens the operator interface when attempts fail.

  • Results

    Six of eight runs produce operators that help on 428 unseen tasks, and one matches a hand-written comparator exactly on HumanEval+ while using 36 versus 103 flags.

  • Takeaways & Limitations

    A small, readable executable metric can be authored automatically and transfer across task pools without per-candidate model calls.

  • Takeaways & Limitations

    The operator-generation process remains behaviorally repetitive, with 183 flagging candidates realizing only 96 distinct flag sets.

Abstract

from arXiv · show

Agents improve quickly against a reliable automatic metric and stall without one, and the applications that need them most, report generation among them, are the ones nobody knows how to score. Can the metric write itself? Saying what makes an answer good is hard; pointing at something wrong with one is easier, so the metric we evolve is a pool of small Python operators that each flag a candidate for one named defect, or abstain, and vote. Asking a model for operators directly does not work: 183 candidates realise only 96 distinct behaviours, from one narrow region of an enormous space. EvalCEGAR instead borrows counterexample-guided abstraction refinement from program verification. It reads the pool as an abstraction and searches for a collision, two answers the operators score identically, one correct and one not. That pair, not a prompt, is the authoring request, and when a collision defeats every attempt the loop widens what an operator may read rather than resampling. On MBPP+ and HumanEval+, a sandbox whose hidden unit tests give exact ground truth, the loop writes a 55-line operator that closes 15.4% of the gap between flagging nothing and a perfect filter on 428 unseen tasks (+0.0065, p=0.0010) at a quarter of our best hand-written operator's flags. On the benchmark it never saw it matches that operator's effect exactly on a third of the flags. Six of eight runs admit such an operator and all six help out of sample; our 15 hand-written operators applied together as one filter lose accuracy. An LLM judge on the same information ties that delta on a nearly disjoint set of candidates, and charges a model call per candidate forever where the operator charges none.

1 Introduction

The paper evolves an automatic metric as a voting pool of defect-specific Python operators, authoring new operators from counterexamples that expose blind spots in the current pool. EvalCEGAR is validated on code because MBPP+ and HumanEval+ provide withheld hidden tests with exact correctness labels.

  • Motivation: A reliable metric enables rapid agent improvement, whereas progress stalls without one and common substitutes miss semantics or introduce judge-specific biases.The introduction contrasts automatic signals with reference overlap and LLM judges.
  • Metric representation: The metric is a voting pool of small Python operators that flag one named defect, pass clean candidates, or abstain.A candidate is rejected when enough operators flag it.
  • Failure of direct generation: 295 operators authored through direct prompting yielded no discovery, illustrating that resampling remains confined to a narrow region of the operator space.The paper identifies this narrow sampling region as one of two obstacles to direct operator generation.
  • EvalCEGAR: EvalCEGAR treats the operator pool as an abstraction and requests refinement from a collision: two answers with identical operator signatures but different ground-truth outcomes.The approach adapts counterexample-guided abstraction refinement from program verification.
  • Empirical validation: 6 of 8 runs produced an operator that helped on 428 unseen tasks; the best used 55 lines of Python, matched the best hand-written operator on a second benchmark at a third of its flags, and tied an LLM judge on a near-disjoint set.The reported out-of-sample results motivate validating the approach on code with exact hidden-test correctness.

2 Related work

Related work improves evaluation models, judge criteria, program search, diversity, and testing, but EvalCEGAR differs by evolving an executable evaluator rather than scoring candidates with a fixed scalar or benchmark. Its search uses counterexample-guided requirements, repartitioned targets, and neither a predefined property nor a second implementation.

  • Automatic evaluation of open-ended output: Learned metrics and judge frameworks still produce a single, inseparable scalar, whereas EvalCEGAR produces individually falsifiable executable checks combined in a pool.The cited passage contrasts overlap replacement and explicit judging criteria with EvalCEGAR’s different output object.
  • LLM-driven program search and CEGAR: FunSearch and automated agent design search programs against fixed evaluators or benchmarks; EvalCEGAR instead searches what an evaluator candidate must prove before admission.With no fixed fitness function, the design problem shifts from mutating candidates to specifying admission requirements.
  • Diversity and test generation: Unlike quality-diversity archives, EvalCEGAR repartitions the target after each admission, and unlike test generation, it requires neither a property to falsify nor a second implementation.The passage distinguishes its diversity pressure from explicit behavioral archives and its setting from property-based and differential testing.

3 Preliminaries

The evaluation studies Python-programming tasks where visible checks admit both correct and incorrect model solutions, and selection accuracy measures how often a metric’s kept candidate is correct. It compares learned and hand-written operators under hidden-test ground truth, with explicit safeguards against oracle reconstruction.

  • Tasks and data: MBPP+ and HumanEval+ tasks use hidden test suites as exact oracles, while solvers see only task statements and visible example checks.Visible checks are not held out; the hidden suite alone supplies ground truth.
  • Selection accuracy: Selection accuracy is the probability that a uniformly sampled candidate retained from V(t) is correct after the metric removes flagged candidates.If every candidate is flagged, sampling falls back to V(t); V(t) contains solutions passing visible checks, including duplicates.
  • Decidability and headroom: 58 of 428 held-out tasks are decidable, yielding selection accuracy from 0.7170 with nothing flagged to 0.7593 for a perfect operator.Headroom reports each absolute delta as a fraction of this reachable range.
  • What we compare against: 15 hand-written operators form the human baseline, while the strongest single comparator flags candidates disagreeing with the task-solution plurality and an LLM judge answers once per candidate.The hand-written pool contains ten syntax-tree checks and five execution-based checks, applied together as one filter.
  • Keeping the loop honest: Four screens reject operators that could reconstruct the oracle through syntax, generator constants, prompt keywords, or text-matched flags.None of these screens fires on the 15 hand-written operators.

4 Method

EvalCEGAR treats the operator pool as an abstraction, targets its largest correct–incorrect collision, and asks an authoring model for an operator that resolves that blind spot. It escalates the operator interface when needed, admits operators only when they improve the deployed decision, and composes admitted operators by voting.

  • Core loop: The loop finds two candidates with identical pool scores but different correctness, requests an operator for that fault pattern, widens its interface when necessary, and admits it if deployment improves.The pool is treated as an abstraction, and admission is based on the deployed decision rather than fault-class recall.
  • Collision targeting: The largest signature class containing both correct and incorrect candidates becomes the next blind-spot target.The specification expands beyond the colliding pair to cover every incorrect training sample failing in the same way, together with every correct one.
  • Interface escalation: After rmax=3 failed level-1 attempts, the target escalates from op(task, code) to op(task, code, ctx).The ctx interface exposes candidate peers and allows running them on chosen inputs under a hard budget of 600 observations per operator call over at most 16 peers.
  • Operator admission: Admission requires ∆> 0, at least 3 tasks helped, and helped > hurt on the training split.The method explicitly avoids a fault-class recall floor because sparse, precise operators can be useful despite low recall.
  • Operator composition: A candidate is flagged when at least m=2 of k operators in a selected subset S flag it, with k = |S| ≤ Kmax=6.The subset objective uses labelled training samples alone, with no development split, tuned threshold, tie-break, or learned weights.

5 Results and analysis

EvalCEGAR produces transferable operators only when both of its mechanisms are present, while composition succeeds after replacing a biased objective with balanced accuracy. The resulting operator matches the comparator on an unseen benchmark, and every non-vacuous admitted composition is safe on the held-out endpoint.

  • Unseen-task performance: The authored 55-line operator reaches 94.2% of the comparator’s effect with one quarter as many flags and improves by +0.0065 overall.On the 58 decidable tasks alone, its improvement rises to +0.0481, with Jaccard overlap 0.245 against the closest hand-written operator.
  • Unseen-task performance: 6 of 8 runs admit an operator, and all six improve out of sample, with median +0.0029 and range +0.0009 to +0.0065.Four of six are individually significant, all six are level-2 operators, and all four fitting screens are clean.
  • Transfer: On HumanEval+, the authored operator matches the comparator’s same 10 helped tasks, one hurt task, and +0.0125 improvement, while outperforming it on parsimony.Across all three pools, it is the only one of 13 measured rules positive and clearing its own per-pool null on every pool.
  • Mechanism ablations: Removing either mechanism stops admission: no level-1 attempt is admitted, and Benjamini–Hochberg finds no discoveries at a 10% false-discovery rate.An independent search likewise admits no significant level-1 candidates but admits 6 of 16 level-2 candidates for +0.0054; the gate ledger attributes the barrier to needing a third distinct helped task.
  • Composition: Balanced accuracy is best in all four composition cells, closing 39.8–90.2% of the gap to the oracle argmax, whereas the frozen criterion ranks only from the 4.9th to 85.7th percentile.The defect is objective bias: balanced accuracy penalizes false alarms 5.33× less than the frozen 1:1 objective, and zero of 2,007,292 non-vacuous subsets harms the endpoint.

6 Negative results and limitations

EvalCEGAR’s main limitations are behavioural redundancy, wasted admissions, and fragile comparisons. Its evaluations also rely on a fixed peer electorate and benchmarks from one distribution family, while a judge reaches the same delta with complementary errors.

  • Behavioural duplication: 183 candidates realise only 96 distinct flag sets despite no source-text pair among 187 candidates reaching Jaccard 0.8.Targeting reduced diversity, while novelty recovered a genuinely new operator in only 2 of 11 blocked candidates.
  • Waste, not exhaustion: 9 of 14 admissions rediscovered an already selected operator, and the pool’s +0.0076 composition headline failed after screening.Rarefaction still found 2.00 new classes at the sixth seed, indicating waste rather than exhaustion.
  • What the numbers do not show: Every ∆ is measured against flagging nothing, while no two rules separated in 9 paired tests and verdicts changed for all 35 operators under a threefold larger electorate.The differences occupied only 2–9 tasks, so level-2 flags require a pinned peer electorate.
  • Evaluation limits: Both benchmarks come from one distribution family, requiring a third oracle port for an out-of-sample task list; a judge ties the operator’s delta.The judge therefore provides a complementary error profile rather than a higher ceiling.

7 Conclusion · A Request scope, screens and nulls

EvalCEGAR shows that an evaluation metric can be authored automatically as a readable 55-line Python operator, while unresolved counterexamples expose when the metric’s vocabulary has run out. Its authoring request covers a whole fault kind, excludes oracle-like shortcuts, and constrains operators to prompt-visible execution evidence.

  • 7 Conclusion: 55 lines of Python close 15.4% of the distance to a perfect operator on 428 unseen tasks.The resulting artefact has no weights and is small enough to read.
  • 7 Conclusion: Admitting on the decision rather than coverage moves admissions from none to one per seed.When nothing resolves a counterexample, widening the interface changes 336 narrow-interface attempts admitting none into a wider interface admitting six.
  • 7 Conclusion: TP −(nTP/nFP)FP reaches the top decile of a space whose precision-flavoured objective sits in the bottom 2%.The classes are weighted using the training labels on a selection endpoint.
  • 7 Conclusion: Where unit tests exist, running them is the appropriate route; unresolved counterexamples mark the limit of the metric’s vocabulary otherwise.The latter setting is the one that motivated the method.
  • A Request scope, screens and nulls: The authoring request covers a whole fault kind, not two colliding samples, because a two-sample request is satisfiable by a lookup table.The fault kind is whichever category contains the colliding sample: visible-check failures or hidden-suite-only failures.
  • A Request scope, screens and nulls: The prompt-keyword screen rejects operators that answer from task identity rather than the candidate, treating them as oracle reconstruction by dispatch table.This is the subtlest of the four screens in Section 3.
  • A Request scope, screens and nulls: Level 2 exposes peers, a runner, prompt-visible asserts, and an input generator, while hiding the hidden suite.Inputs perturb prompt-visible assert arguments; operators exceeding the observation cap abstain, and the shuffle null uses b=1000 seeded resamples for exact reruns.

B The transfer budget · C Targeting versus accepting

EvalCEGAR’s transfer budget is about 10 decidable examples for authoring plus roughly 60 labelled candidates for selecting voting operators. Separating collision-based targeting from acceptance increases metric yield without reducing peak quality.

  • B The transfer budget: 10 training tasks were decidable, yielding 80 correct–wrong pairs and four distinguishable levels of the training objective.An example informs the loop only when it contains both a good and bad candidate surviving visible checks.
  • B The transfer budget: 20–30 decidable tasks mark the aggregation-tuning crossover: below this, selection collapses to the union in 9 of 9 directions.Above the crossover, rotating a held-back fold recovers +0.0077 (p ≤0.0004), identical at m=2, 3, 4.
  • B The transfer budget: +0.0077 is 57% of the oracle’s gain once aggregation tuning exceeds the 20–30-task crossover.The recovery is identical at m=2, 3, 4.
  • B The transfer budget: ∼10 decidable examples author operators for a new domain, while ∼60 labelled candidates determine which admitted operators vote.The second quantity does not require one example to contain both a correct and incorrect candidate.
  • C Targeting versus accepting: 4 admissions to 0 favored collision-based admissions in an early synthetic-bank ablation, but the result failed to replicate on the natural bank and was discarded.The ablation rejected 31 of 36 undirected candidates as untargeted.
  • C Targeting versus accepting: 1.67 distinct metrics per seed versus 0.38 resulted when collision remained the target but was removed as the acceptance test.This is a 4.4× increase in distinct-metric yield, with peak quality unchanged at +0.0063 versus +0.0065.

D The mechanism ablations in full · E The gate ledger in full

The ablations compare recall-gated and narrow-only operator-authoring arms under matched labels and loop-generated candidates, while the gate ledger exposes how escalating interfaces shaped admissions. Admission required candidates to help on at least three training tasks, and only the wider interface reached that threshold.

  • D The mechanism ablations in full: Both ablation arms ran inside the loop on their own candidates and read the same training-split labels, avoiding cross-arm operator rescoring.Neither gate call hit its abort guard.
  • D The mechanism ablations in full: 30% recall across the 140 still-missed fault classes and at most 2% false alarms defined the recall-gated arm’s admission floor.The floor applied to samples that must pass.
  • D The mechanism ablations in full: 48 level-1 attempts per target gave the narrow-only arm twice the level-1 budget per target.
  • E The gate ledger in full: 124 of 336 level-1 attempts survived earlier screens and reached the gate shown in Figure 3.Targets were attempted at the narrow interface first, so escalating runs contain mostly narrow consultations.
  • E The gate ledger in full: Each of the six operators admitted across eight runs had a recorded level-1 rejection immediately beforehand.The ledger also served as sensitivity analysis for the admission floor.
  • E The gate ledger in full: 55 of 124 consultations would be admitted at the narrow interface under a ≥2 threshold, but the transfer of those admissions was unmeasured.This limitation means the interface-level admission count does not establish downstream benefit.
  • E The gate ledger in full: Admission required candidates to help on three training tasks, and only the wider interface reached that threshold.Figure 3 includes every admission-gate consultation, including those that did not end in admission.

F The selection criterion: registration, robustness, and the class weight

The section evaluates pre-registered selection criteria, adopts C6 without tuning, and tests its robustness across splits, benchmarks, and class-weight values. C6 remains preferred under registered checks, while the training-split class ratio lies within HumanEval+’s optimal plateau.

  • Registration: 6 criteria were registered with formulas and predicted rankings before scoring the landscape.Table 4 also includes C7, the only criterion added afterward.
  • Registration: C6 is the adopted untuned criterion, while 0.1877 is the training split’s class ratio nTP/nFP.C7 is the induction baseline’s admission signal and the only row not registered in advance.
  • Robustness: 4 of 4 split-half cells re-selected C6, placing it at the 85.2–98.2nd percentile of half B’s landscape versus J’s 9.3–82.4th.The per-benchmark analysis also identifies HumanEval+ as the criterion’s best pool.
  • Class weight: 12 λ values were scored in each of 4 cells; HumanEval+’s optimal plateau contained λ=0.1877 in 4 of 4 cells.λ=1 was strictly outside the plateau every time, while λ=0 was never strictly better.

G Peer-dependent verdicts · H Redundancy and the diversity veto

Peer-dependent operators gain discriminating power from the task’s full peer electorate, while the redundancy veto favors parsimony without improving quality. The five-operator rule benefits from expanded peer context, but the veto remains no better than the plain union.

  • G Peer-dependent verdicts: Every level-2 verdict depends on ctx, which contains every raw task sample, making the operator’s judgment a function of its peer electorate.The two effects—peer dependence and its consequences—were measured separately.
  • G Peer-dependent verdicts: λ=nTP/nFP=0.1877 lies within HumanEval+’s optimal held-out plateau, whereas J’s λ=1 is excluded.The value was read off the MBPP+ training split.
  • G Peer-dependent verdicts: 0 of 35 operators change any verdict against a frozen electorate, establishing exact referential stability across held-out and training data.This identity is checked rather than assumed.
  • G Peer-dependent verdicts: 1996 flags are added and 440 removed when the electorate grows from 3424 to 12744 samples, with one operator more than doubling its flags.The wider interface increases discriminating power but reduces stability, even when candidate code is unchanged.
  • G Peer-dependent verdicts: +0.0009 to +0.0036 is gained by every frozen selection under the grown electorate, with no material reordering.This supports peer context as a source of signal rather than conformity alone.
  • G Peer-dependent verdicts: +0.0220 is scored by the fixed five-operator rule on 9320 candidates, with z=8.34, p=0.0010, and +79/−12 tasks.The rule was selected on the training split and never re-selected on the same draw.
  • H Redundancy and the diversity veto: +0.0065/+0.0057/+0.0047 at m=2, 3, 4 is achieved by the veto, at or below the plain union, while harming 9/11/11 tasks versus 8.Development folds disable the veto in 5 of 9 directions.
  • H Redundancy and the diversity veto: +0.0040 is retained by the frozen rule on 5 novel operators in the rediscovery-heavy pool.The veto buys parsimony—7 operators for 26 operators’ effect—rather than quality.

I The judge and induction baselines

The judge can complement the loop-authored operator, but its limited recall and per-candidate cost constrain its usefulness. Verifier induction works only at level 2, while alternative finite-pool selection criteria recover more held-out performance than the published ranking.

  • The judge and induction baselines: The author model is more precise than the cheap judge (0.677 vs. 0.403) but less useful because it flags fewer bad candidates.Reasoning before answering improves the cheap model’s delta while lowering its flag count; both author-model configurations reach roughly half that delta.
  • The judge and induction baselines: The judge and loop-authored operator overlap on only 27 of 257 flags, yielding a Jaccard of 0.105.Their disagreement makes Boolean unions potentially informative rather than redundant.
  • The judge and induction baselines: 47 calls produced the operator, whereas judging one endpoint pass requires 1762 calls, or 37.5× the production run.Future candidates continue to incur a judge call, while scored candidates cost nothing for the operator.
  • The judge and induction baselines: Level-1 verifier induction produced no separation from its null, but level 2’s best single check reached +0.0074.The verifier commissions candidates from balanced labelled samples and admits them using true-positive and precision thresholds.
  • The judge and induction baselines: C7 beats the published criterion in three cells, ties it in the fourth, and remains effective as the pool grows from 26 to 35 operators.C6 recovers more of the gap in all four cells, while C7 selects subsets with Jaccard 0.22–0.67 against the published choice.
Loading 2608.18744v1…