PaperlayerView exact revision on arXiv

Source-linked AI summary

EvalResearchBench: Can AI Agents Design Their Own Evaluations?

Yaolun Zhang, Tianyi Xu, Yujie Zhao, Jishen Zhao, Qingyun Wu, Huazheng Wang

arXiv:2610.04184v1cs.AI

TL;DR

Repeated complex-benchmark runs make evaluation costly and slow iteration, while experts currently reduce that burden by designing compact suites. ERB tests whether agents can design and revise frozen evaluators under fixed budgets; the best pairwise agreement is about 75%, below the 91% ceiling from target disagreements.

  • Problem

    Repeatedly running complex benchmarks incurs API and compute costs and delays experiments, motivating the question of whether agents can design affordable, informative evaluations.

    Source: §1 · p. 2

  • Method

    ERB gives researcher agents target materials, development references, candidate APIs, and fixed budgets to build and revise executable evaluators before independent assessment.

    Source: §1 · p. 2

  • Results

    About 75% pairwise agreement is the best result, below the 91% ceiling set by disagreements among targets, and no researcher leads on every metric.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ERB provides an executable benchmark for assessing agents’ ability to construct evaluation feedback instruments through autonomous, budgeted experimentation.

    Source: §6 · p. 9

    Many Astra spreadsheet responses exhaust their answer allowance, so observed separation reflects those tasks under the chosen execution conditions.

    Source: §4.3 · p. 7

Abstract

from arXiv · show

Recursive self-improvement (RSI) relies on evaluation feedback to assess progress and guide further research, yet repeatedly running complex benchmarks is costly and slows iteration. Human experts reduce this cost by selecting benchmark subsets or designing compact suites. We ask whether AI agents can automate this design process and introduce EvalResearchBench (ERB), a benchmark for autonomous evaluation research. Given target materials, development references, candidate APIs, and fixed time and API budgets, an agent called the researcher selects or synthesizes tasks, implements graders, and revises them in pilot tests before freezing an executable evaluator for coding, co-work, and reasoning. We study 9 researchers and 13 candidate models and compare each frozen evaluator with 14 target benchmarks on score concordance and pairwise agreement. The best evaluators order about 75\% of candidate pairs as the targets do, below the 91\% ceiling set by disagreements among the targets. No researcher leads on every metric, and the best evaluator on development targets is not the best on sealed targets hidden from the researcher. A human-designed sample of public tasks remains a strong baseline, and the evaluator with the lowest recorded execution cost attains the highest pairwise agreement. Agents repair tasks and graders through pilot feedback, yet their evaluators can still truncate answers, exhaust the evaluation budget, or let a few questions dominate a domain score.

1 INTRODUCTION

ERB asks whether agents can design affordable evaluators that reproduce complex benchmarks, then tests frozen suites against target benchmarks. Results show imperfect ranking agreement, limited transfer from development to sealed targets, and persistent execution and scoring issues.

1 INTRODUCTION

  • Repeated complex-benchmark runs cost API and compute resources and delay experiments, motivating compact human-designed suites and autonomous evaluation design.Earlier improvement systems generally rely on human-supplied evaluation protocols, leaving evaluation design as a remaining autonomy challenge.Source: §1 · p. 2
  • ERB has agents design, pilot, revise, and freeze executable evaluators, which are independently assessed against development and sealed targets.Researchers receive target materials, development references, and candidate APIs; sealed candidates and targets remain hidden during research.Source: §1 · p. 2, Figure 2 · p. 3
  • The study evaluates 9 researcher configurations on 13 candidates and 14 targets across coding, co-work, and reasoning, measuring score concordance and pairwise ordering.Score measures concordance, while PA measures the fraction of candidate pairs ordered as the target does; construction and recurring execution costs are separated.Source: §1 · p. 2
  • About 75% pairwise agreement is the best result, below the 91% ceiling from target disagreements; Astra leads overall Score, while Grok 4.7 leads PA.No evaluator leads in every domain: Grok 4.7 leads coding, Astra co-work, and Opus 5 reasoning.Source: §1 · p. 2
  • On identical candidates, seven of eight agent-designed evaluators exceed Random120 in Score, but only four in PA; Random120 remains a strong baseline, and fitting visible references does not ensure transfer.Source: §1 · p. 3
  • The evaluator with the lowest recorded execution cost achieves the highest PA, yet final suites can truncate answers, exhaust budgets, or overweight a few questions.Agents do repair tasks and graders through pilot feedback, but these failure modes remain.Source: §1 · p. 3

2 RELATED WORK

Related work covers evaluation design and efficient measurement, as well as autonomous research and self-improvement. ERB focuses on agents constructing the feedback instrument through budgeted experimentation.

2.1 EVALUATION DESIGN AND EFFICIENT MEASUREMENT

  • ERB extends work on coverage, task construction, grading, benchmark design, and efficient measurement by evaluating agents’ autonomous selection and development of measurement strategies.Related approaches include AutoBencher, BenchAgents, BenchBench, tinyBenchmarks, and adaptive-testing work.Source: §2.1 · p. 3

2.2 AUTONOMOUS RESEARCH AND SELF-IMPROVEMENT

  • Unlike broader autonomous-research and self-improvement systems, ERB evaluates construction of the feedback instrument itself and assesses the submitted evaluation independently of its creation process.Related work spans iterative experimentation, resource-bounded research, replication, post-training, and systems such as The AI Scientist and Agent Laboratory.Source: §2.2 · pp. 3–4

3 EVALRESEARCHBENCH

ERB asks a researcher agent to build a reusable evaluator across three domains under bounded resources. It freezes the design before independent assessment and measures both score agreement and candidate ordering against target benchmarks.

3.1 RESEARCH TASK

  • The research task equips an LLM agent to measure candidate model–execution configurations across coding, co-work, and reasoning using supplied target materials, tasks, traces, references, and APIs.The agent may select, synthesize, or simplify tasks, repair graders, adjust weights and answer allowances, and inspect pilot responses within fixed budgets and a deadline.Source: §3.1 · p. 4

3.2 INDEPENDENT ASSESSMENT

  • Independent assessment runs the frozen evaluator on development and withheld candidates against visible and withheld target references, with separate assessment caps and incomplete-item rules.Assessment records per-item rewards, termination reasons, and usage; a domain is missing if it has no completed positive-weight item.Source: §3.2 · pp. 4–5

3.3 SCORE AGREEMENT AND ORDERING

  • Score is 100 times concordance correlation coefficient, while PA evaluates ordering on non-tied reference pairs and awards full, half, or zero credit for correct, tied, or reversed measured order.CCC penalizes mean and scale mismatch; eligible targets require at least six matched candidates and nonconstant references.Source: §3.3 · p. 5

4 EXPERIMENTS

The experiments compare agent-designed evaluators with target benchmarks and a public-task baseline, then examine how task selection, suite size, weighting, and costs shape measurement quality. Overall leaders differ by metric and target split, while design choices affect both agreement and recurring execution cost.

4.1 SETUP

  • Random120, a source-stratified public-task baseline, draws 40 coding, 40 co-work, and 40 reasoning tasks from sixteen sources, then assesses twelve complete candidates against the same fourteen targets.Source: §4.1 · p. 6

4.2 MAIN RESULTS: AGREEMENT WITH THE REFERENCE PANEL

  • Astra leads overall Score while Grok leads PA; domain strengths vary, with no researcher dominating both metrics across the three domains: coding, co-work, and reasoning.Source: Table 2 · p. 6, §4.2 · p. 7
  • Fable 5.1 leads development Score, but Grok leads sealed Score, so development-target fit does not ensure transfer.The comparison uses frozen evaluations on development and withheld targets.Source: §4.2 · p. 7
  • Random120 remains a strong baseline: its public sources, domain quotas, and graders encode task-design knowledge, especially in coding.Seven of eight agent-designed evaluators exceed Random120 in Score on identical candidates, but only four exceed it in PA.Source: §4.2 · p. 7

4.3 WHAT SHAPES EVALUATION QUALITY?

  • Task groups vary in measurement value: Astra’s spreadsheet tasks contribute strongly to both co-work metrics, while K3’s hard coding additions improve Score but slightly reduce PA.Removing K3’s hard coding group or Astra’s repository traces has only modest effects relative to same-domain, same-size random deletions.Source: §4.3 · p. 7
  • Larger domain-stratified question subsets raise mean Score and reduce variation in both Score and PA across question draws.Small subsets can omit useful task families or overrepresent a few responses, even when domain proportions are fixed.Source: §4.3 · p. 8, Figure 4 · p. 8
  • Equal item weights substantially improve Grok’s coding and co-work Score, while overall PA barely changes on the same responses.Six of Grok’s 20 co-work questions carry almost all scoring weight; Fable 5.1 distributes influence more evenly.Source: §4.3 · p. 8, Figure 5 · p. 9

4.4 WHAT MAKES AN EVALUATION COST-EFFICIENT?

  • Grok combines low assessment cost with the highest overall PA and a competitive Score; candidate answer demands and grading methods are distinct cost drivers.Grok favors short code, output predictions, and numeric checks with local graders, while Fable 5.1 uses longer deliverables and a costly rubric judge.Source: §4.4 · pp. 8–9

5 CASE STUDY: HOW RESEARCHERS REVISE EVALUATIONS

Researchers use development feedback to broaden evaluation coverage, repair tasks and graders, and adjust execution settings.

5 CASE STUDY: HOW RESEARCHERS REVISE EVALUATIONS

  • Astra develops repository-tracing tasks through source inspection and adjusts execution after pilot responses are truncated.Pilot feedback also resolves mismatches between prompts and rubrics.Source: §5 · p. 9
  • GLM corrects invalid examples and numerical answers during evaluator revision.Source: §5 · p. 9
  • K3 adds harder coding tasks after pilot scores approach the ceiling.Source: §5 · p. 9

6 CONCLUSION

ERB measures autonomous evaluation research under fixed budgets, separating score reproduction, ordering, and execution cost. Its target-level assessment reports comparable metrics and fixed score-scale conventions.

6 CONCLUSION

  • ERB uses independent assessment, pilot audits, and component and subset analyses to examine autonomous evaluation design under fixed budgets.Source: §6 · p. 9

A TARGET-LEVEL ASSESSMENT AND SCORE SCALES

A.1 REPRESENTATIVE TARGET SCATTERPLOTS

Target references can disagree in score level and candidate ordering, limiting how well one evaluator can reproduce them. Target composition also changes which evaluator leads on raw-score concordance.

A.1 REPRESENTATIVE TARGET SCATTERPLOTS

  • Astra scores 55.78 on GDPval and 42.23 on BrowseComp; the three evaluators score 55.33–63.94 on HLE and 3.62–7.83 on AA-LCR.Figure 7 plots measured scores against target references for one development and one sealed target per domain.Source: §A.1 · p. 17, Figure 7 · p. 18
  • On six shared coding candidates, the ProgramBench and SWE-bench Multilingual reference means are 0.0075 and 0.876, creating a raw-score error floor for their pair.CCC also depends on covariance and dispersion, while PA reports ordering.Source: §A.1 · p. 18
  • Fable 5.1 leads Score on the eight-target compatible-reference subset, while Astra leads on the primary fourteen-target panel.The subset is selected by shared-candidate overlap and reference-score RMSE, without using researcher scores.Source: §A.1 · pp. 18–19
  • 91.20% is the reference-imposed overall PA ceiling, versus 74.87% for the best observed artifact.The ceiling reflects cross-target ordering conflicts; each column optimizes its own target set.Source: §A.1 · p. 19
  • Fable 5 produced 20 of 39 candidate-domain reports before the budget cap stopped assessment; the main ranking excludes it.Table 2 separately reports a scenario assigning its 19 unobserved scores a value of one.Source: §A.1 · p. 19
  • Assigning every candidate a score of 0.5 yields overall Score 0 and PA 50.00% across the fourteen target panels.All measured pairs tie in this no-model-call sanity check.Source: §A.1 · p. 19

A.2 COST ACCOUNTING

  • Cost accounting separates researcher inference, shared development and helper testing, and frozen assessment, using standardized API-equivalent estimates.Source: §A.2 · p. 19

A.3 TASK DESIGN, EXECUTION COST, AND SCORE CONCENTRATION

The appendix examines how suite construction, execution costs, answer limits, and question weights relate to agreement and score variability. It also documents the source-stratified Random120 baseline and the fixed-panel sensitivity setup.

A.3 TASK DESIGN, EXECUTION COST, AND SCORE CONCENTRATION

  • Fable 5 candidate-call costs are 16.73 and 7.12 times Grok's for GPT-5.5 and Astra, while completed-item counts differ by less than 10%.Source: §A.3 · p. 20
  • Grok's question counts of 62/20/25 correspond to weight-equivalent counts of 21.84/6.78/19.07; its six highest-weight co-work items carry 93.87% of that score.The weight-equivalent count equals the question count under uniform weights and decreases as weights concentrate.Source: §A.3 · p. 20
  • GPT-5.5, Astra, and Sol score 100% on Grok co-work, while Fable 5.1 scores 79.22%, 81.73%, and 70.52% for those candidates.The contrast accompanies Grok's 74.87% overall PA and 22.59 overall Score, illustrating that cost, spread, and target agreement are distinct measures.Source: §A.3 · p. 21
  • Declared answer allowances describe a full-suite protocol setting, not observed token consumption or a billing bound.Assessment expenditure is split by candidate and grader handles; recorded run totals can include partial executions.Source: Table 4 · p. 20, §A.3 · p. 21
  • The passage reports a final increase for that family from 8,000 to 16,000 tokens; Opus’s development diagnostic recorded 36 capped responses out of 102, including 32 empty replies.Opus’s final code sets reasoning, coding, and co-work caps at 24,000, 16,000, and 8,000 tokens, respectively.Source: §A.3 · p. 22
  • Fable 5's rubric helper initially fails to return a parsable verdict, delaying response-matrix delivery and leaving its predefined family weights in place.The researcher increases the helper allowance, adds bounded retry samples, and changes error handling to preserve other item results.Source: §A.3 · p. 22

B FROZEN-SUITE SENSITIVITY AND SOURCE-STRATIFIED BASELINE

B.1 SOURCE-STRATIFIED RANDOM120 BASELINE

  • Random120 samples 120 public tasks across three domains from sixteen sources using human-set source quotas and fixed local graders.The source quotas cover multiple task types; sampling is independent of target reference scores.Source: §B.1 · p. 22

B.2 MATCHED-PANEL COMPARISON AND SCORE DECOMPOSITION

The matched-panel results change with the aggregation unit: Grok leads under target-equal weighting, while Astra leads under domain-equal weighting. The section also reports how score concordance is decomposed and how panel sensitivity is assessed.

B.2 MATCHED-PANEL COMPARISON AND SCORE DECOMPOSITION

  • The comparison recomputes all eight complete-domain artifacts on the same twelve candidates and fourteen target-reference intersections used for Random120.Target scales and eligibility remain unchanged; Table 9 also reports a domain-equal aggregate.Source: §B.2 · p. 22, Table 9 · p. 23
  • Candidate-panel sensitivity uses 10,000 resamples; its percentile ranges reflect paired differences from Random120, not new research episodes or target families.A common mask retains 9,916 draws with defined metrics for every method and target.Source: §B.2 · p. 23
  • Table 10 reports per-target CCC as rCb, with Cb penalizing mean and dispersion mismatch; its factor means are not macro-CCC.Overall Score is reported with either equal weight for all fourteen targets or equal weight for each domain after averaging its targets.Source: Table 10 · p. 24
  • Target-equal weighting gives Grok Score 22.18 and Astra 21.35; domain-equal weighting gives Astra 25.20 and Grok 21.14.The main leaderboard retains target-equal weighting.Source: §B.2 · p. 24

B.3 QUESTION-COUNT PROTOCOL

The question-count analysis tests how nested subsets of three frozen suites compare with their full versions. At 90 questions, the share of draws within one Score point of the full-suite value varies substantially across suites.

B.3 QUESTION-COUNT PROTOCOL

  • Astra, GLM-5.3, and K3 suites contain 112, 100, and 107 questions, respectively; the analysis draws 1,000 domain-stratified subsets at four sizes.Subsets contain 10, 30, 60, or 90 questions, and retained weights are renormalized within each domain.Source: §B.3 · p. 24
  • The same question subset is applied to every candidate, and saved grades—including wrong and empty answers—are reused without new model calls.Metrics are recomputed per target before target-equal averaging; percentile bands describe question-draw variation on the fixed panel.Source: §B.3 · p. 24
  • At 90 questions, 34.4% of Astra, 70.5% of GLM, and 63.3% of K3 draws fall within one Score point of their full-suite values.These percentages describe subsets within each artifact, not rankings across researchers.Source: §B.3 · p. 24, Figure 9 · p. 25

C RESEARCH DECISIONS AND NATIVE-COMPONENT ANALYSIS

The section examines component-level effects, Grok’s concentrated weighting, and DeepSeek’s grading-oriented task choices. It also specifies the constraints on researcher submissions and their development and assessment budgets.

C RESEARCH DECISIONS AND NATIVE-COMPONENT ANALYSIS

  • Removing Astra’s eight spreadsheet tasks lowers co-work Score by 31.94 points and PA by 12.57 percentage points; removing ArXiv-math components raises reasoning Score and PA for all three suites.Astra’s spreadsheet component has 46/96 responses without final answers at the output cap.Source: C RESEARCH DECISIONS AND… · p. 25, Figure 10 · p. 26, Figure 11 · p. 27

C.1 GROK: FROM PILOT RESPONSE PATTERNS TO FROZEN WEIGHTS

  • Six Grok co-work items receive 93.87% of total weight after its program searches response patterns on three development candidates.The reported weight-concentration quantity is not the number of statistically independent questions.Source: §C.1 · pp. 25–26, Figure 12 · p. 28

C.2 DEEPSEEK V4 PRO: GRADING CONVENIENCE AND MEASUREMENT COVERAGE

  • DeepSeek’s reasoning Score exceeds Random120’s, while coding accounts for its largest deficit; the authors interpret simplified task forms as preserving checkability while compressing capability differences.Its co-work checks ask for specified text, numbers, or formats, measuring compliance more directly than independent professional problem solving; short answers simplify grading, not necessarily reasoning.Source: §C.2 · pp. 26, 28

D RESEARCH INSTRUCTIONS AND INPUTS

  • Researchers submit at least 100 scored questions across coding, co-work, and reasoning, with fixed aggregation rules and frozen graders, under bounded development and separate assessment budgets.The research period and wallets constrain development, while assessment uses a separate wallet and per-candidate cap; assessment results are not returned to the researcher.Source: D RESEARCH INSTRUCTIONS AND INPUTS · p. 28

E CASE STUDY: HOW RESEARCHERS REVISE EVALUATIONS

Astra’s case study follows source inspection into task construction, then uses pilot feedback and audits to revise questions and grading criteria. The examples show feedback prompting changes to task validity, coverage, and execution settings.

E CASE STUDY: HOW RESEARCHERS REVISE EVALUATIONS

  • Astra identifies a coding-coverage gap and builds eight multi-file repository traces, including a task about cached outputs after in-place mutation versus dependency replacement.The supplied fixture generates expected outputs for grading; Figure 13 describes a 112-question suite frozen for independent assessment.Source: E CASE STUDY: HOW RESEARCHERS REVISE… · p. 28, Figure 13 · p. 29
  • An 11/12 pilot exposed an unstated requirement for mean close time, so Astra added it to the question; a later audit relaxed an overly restrictive cloud-migration rubric.The audit retained the brief’s security and delivery requirements, and the saved pilot score preceded the repairs.Source: E CASE STUDY: HOW RESEARCHERS REVISE… · p. 29, Figure 13 · p. 29
  • GLM repairs invalid examples and numerical answers, K3 adds five hard coding tasks after near-ceiling pilots, and Astra changes answer allowances and grouped execution after truncation.The examples illustrate revisions to task validity, coverage, and execution settings.Source: E CASE STUDY: HOW RESEARCHERS REVISE… · p. 29

Related papers

Loading 2610.04184v1…