PaperlayerView exact revision on arXiv

Source-linked AI summary

MetaBench-Harness: Unlocking End-to-End Optimization of Benchmark Harnesses

Xuanjun Chen, Hua-Hsuan Chen, Wei-Chung Lu, Yinghao Ma, Jyh-Shing Roger Jang, Hung-yi Lee

arXiv:2609.33411v1cs.AIcs.CLcs.SE

TL;DR

Static benchmarks are struggling to keep pace with LLM capabilities, while existing evolution methods rely on rigid task-level rules. MetaBench-Harness searches over the benchmark-generation workflow with linked harness and benchmark loops. On CodeContests and AIME-2024, it produces challenging, discriminative benchmarks and steadily improves several measures of benchmark quality.

  • Problem

    Static benchmarks are increasingly inadequate as LLM capabilities evolve, and existing automated evolution methods remain constrained by rigid rules.

    Source: §1 · pp. 1–2

  • Method

    MetaBench-Harness uses an inner loop to generate benchmarks and an outer loop to refine harnesses using accumulated trajectory memory.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    On CodeContests and AIME-2024, MetaBench-Harness produces challenging, discriminative benchmarks and steadily improves evolution reasonableness, benchmark competency, and evaluator robustness.

    Source: §1 · p. 2, §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that the framework provides a solution to benchmark saturation through multi-dimensional evolution and diverse difficulty levers.

    Source: §1 · p. 2, §5 · p. 9

Abstract

from arXiv · show

Rapid progress in Large Language Models (LLMs) is saturating static benchmarks faster than they can be designed. While existing automated evolution frameworks attempt to generate harder questions by perturbing individual tasks, they remain constrained by rigid, hard-coded generation rules. Moving beyond the evolution of isolated tasks, we propose to optimize the benchmark generation workflow itself end to end with MetaBench-Harness, a dual-loop search framework. Specifically, the inner loop utilizes a benchmark harness to generate a new benchmark in each round, while the outer meta-harness orchestration layer iteratively refines and searches over harness implementations based on historical evolution trajectories. By applying MetaBench-Harness to the competitive programming CodeContests and Olympiad mathematics AIME-2024 datasets, we demonstrate that the evolved benchmarks are challenging and discriminative for frontier models. Trajectory and quality analyses verify that MetaBench-Harness enables multi-dimensional evolution, steadily improving evolution reasonableness, benchmark competency, and evaluator robustness across successive rounds. Furthermore, case studies reveal its effective utilization of diverse difficulty levers to reframe problems and elevate required capabilities. Ultimately, this work provides a solution to the pressing challenge of benchmark saturation.

1 INTRODUCTION

Static benchmarks are increasingly inadequate as LLM capabilities evolve, while existing automated approaches remain constrained by rigid rules, limited generalization, or narrow domains. MetaBench-Harness searches over benchmark-generation harnesses and produces challenging, discriminative benchmarks with improving quality across successive rounds.

1 INTRODUCTION

  • Static datasets are increasingly inadequate amid rapid model evolution and contamination, and continuous, robust benchmark evolution remains unresolved.Source: §1 · p. 1
  • Existing approaches use predefined task operations, narrow domains, fixed schedules, or restricted environments, limiting comprehensive responses to benchmark saturation.Source: §1 · pp. 1–2
  • The framework treats evolution, discriminability, and diversity as core requirements, including capability-focused tasks and varied difficulty levers.Source: §1 · p. 2
  • MetaBench-Harness searches across evolving harnesses and benchmarks, using accumulated trajectory memory and controlled access to prior code and execution traces.Source: §1 · p. 2
  • On CodeContests and AIME-2024, the framework generates challenging, discriminative tasks and steadily improves benchmark competency and evaluator robustness.Source: §1 · p. 2

2 RELATED WORK

Prior dynamic benchmark methods are limited by fixed generation pathways, narrow domains, or task-specific objectives. MetaBench-Harness instead searches over benchmark-generation pipelines, using historical artifacts to refine the harness itself.

Static vs. Dynamic Benchmark Generation

  • Earlier self-evolving methods use predefined operations, while specialized frameworks target narrow niches and EvoCodeBench refreshes benchmarks on fixed schedules without active model feedback.Source: §2 · p. 2
  • MetaBench-Harness uses outer-loop orchestration with selective access to historical code and execution traces to support a dynamic benchmark-generation pipeline.Source: §2 · p. 3

Adaptive Utilization of External Knowledge

  • The framework applies adaptive context interaction within harness engineering, allowing the proposer to access historical code, scores, and execution traces.Source: §2 · p. 3

Search Over Executable Code

  • Unlike Meta-Harness, which optimizes individual downstream tasks, this work searches and evolves full benchmark-generation harnesses.Source: §2 · p. 3

3 METABENCH-HARNESS: HARNESS FOR BENCHMARK GENERATION

MetaBench-Harness optimizes benchmark-generation workflows through an outer harness-search loop and an inner benchmark-generation loop. Trajectory memory and proxy-model feedback guide harness updates, while generated tasks pass through validation and lifecycle management.

Method Overview

  • A benchmark harness transforms seed benchmarks, trajectory memory records prior optimization, and meta-harness orchestration drives iterative refinement.Source: §3 · p. 3

Objective

  • The optimization objective balances challenge and discriminability by rewarding large score differences between proxy models alongside low average performance.Source: §3 · p. 3

Meta-Harness Orchestration Guided by Trajectory Memory

  • The outer loop proposes harness updates from historical artifacts, ranks candidates, selects one, and applies it to generate the next benchmark checkpoint.Source: §3 · p. 4, Figure 2 · p. 4

Benchmark Generation Governed by Agentic Harness

  • The inner loop discovers scenarios, synthesizes tasks with deterministic evaluators, validates them on proxy models, and audits scenario lifecycles.Source: §3 · p. 4, Figure 2 · p. 4

End-to-End Optimization Procedure and Deliverables

  • Validation feedback and round states are appended to trajectory memory to guide subsequent search, and the procedure returns the optimized benchmark and harness.Source: §3 · p. 5

Module Instantiation of METABENCH-HARNESS

  • Structured Markdown scaffolds specify module functionality, inputs, outputs, and constraints; two proxy models support benchmark validation and audit.Source: §3 · p. 5

4 EXPERIMENTS

Experiments evaluate MetaBench-Harness on CodeContests and AIME-2024, finding that it produces challenging, discriminative benchmarks and that successive evolution improves task and evaluator quality. Analyses also show domain-specific harness adaptations and broader capability demands in evolved tasks.

4 EXPERIMENTS

  • Experiments use CodeContests for 10 rounds and AIME-2024 for 5, with pass rate as the primary metric because each generated task has a programmatic evaluator.Claude-Sonnet-5 serves as Magent; Claude-Sonnet-5 and GPT-5.4 provide proxy difficulty and discrimination signals.Source: §4 · p. 5
  • Held-out model evaluations use zero-temperature decoding, a 32,000-token output cap, and a 900-second timeout per call.The evaluated models include DeepSeek-V3.1, Kimi-K2.5, Gemini-2.5-Flash, and two distilled DeepSeek-R1-Qwen models.Source: §4 · pp. 5–6

4.1 MAIN RESULT: PERFORMANCE OF HELD-OUT MODELS ON EVOLVED BENCHMARKS

  • DeepSeek accuracy falls to 0.322 and Qwen-7B to 0.044 on MetaBench-Harness, while naïve prompting yields a perfect 1.000 for top models.The comparisons use AIME-2024 variants; the authors attribute the naïve-prompting outcome to trivial or flawed tasks without execution feedback.Source: §4.1 · p. 6, Table 1 · p. 6
  • On late-stage MB-AIME, non-reasoning models reach 0.100–0.167 accuracy, while Kimi-K2.5 and DeepSeek-V3.1 score 0.533 and 0.333.Kimi-K2.5 and DeepSeek-V3.1 also generate an additional 14,764 and 5,536 tokens, respectively, versus the base dataset.Source: §4.1 · pp. 6–7, Table 2 · p. 6, Table 3 · p. 7

4.2 BENCHMARK HARNESS EVOLUTION ANALYSIS AND DATA QUALITY ASSESSMENT

  • CodeContests harness edits concentrate on Measurement (39%) and Telemetry (11%), whereas MB-AIME edits focus on Generation (60%).The authors connect coding edits to richer execution data and evaluation needs, and math edits to evolving tasks with deterministic answers.Source: §4.2 · p. 7, Table 4 · p. 7
  • Evolved tasks involve more capability stages; human–AI agreement for labeling difficulty levers and capability dimensions is 97.1% and 96.1%.Figure 3 shows both domains tightening answer criteria and compounding constraints, with math emphasizing structural composition and coding increasing required work.Source: Table 5 · p. 7, §4.2 · p. 8, Figure 3 · p. 8
  • Quality assessment checks reasonableness, competency, and evaluator robustness, with overall success requiring all three dimensions to pass simultaneously.The final-round overall success rate substantially exceeds the cross-round average, and the authors report an upward trend across rounds.Source: §4.2 · p. 8, Table 6 · p. 8
  • Case studies show coding evolution shifting a constructive-math seed into code auditing, while an AIME counting task gains instruction-compliance and distractor-resistance demands.The transformations use meta-level shift and scaffolding removal for coding, and compound conjunction and distractor injection for math.Source: §4.2 · p. 9, Table 7 · p. 9

5 CONCLUSION

MetaBench-Harness optimizes benchmark generation end to end through inner-loop task generation and an outer meta-harness that refines implementations using historical trajectories. On CodeContests and AIME-2024, it produces challenging, discriminative benchmarks and improves benchmark and evaluator quality over successive rounds.

5 CONCLUSION

  • MetaBench-Harness combines inner-loop task generation with outer-loop harness refinement based on historical trajectories.The paper frames this as end-to-end optimization of benchmark-generation workflows.Source: §5 · p. 9
  • Experiments on CodeContests and AIME-2024 show benchmarks that are challenging and discriminative for models.Source: §5 · p. 9
  • Trajectory and quality analyses report steady gains in evolution reasonableness, benchmark competency, and evaluator robustness across rounds.Case studies also report more required capabilities and problem-solving techniques, using diverse difficulty levers to reframe problems.Source: §5 · p. 9

REPRODUCIBILITY STATEMENT

The pipeline generates a matching code-based evaluator for every new task to support consistent, fair future evaluations and avoid instability from subjective LLM-as-a-judge scoring.

REPRODUCIBILITY STATEMENT

THE USE OF LARGE LANGUAGE MODELS

The paper describes LLMs as agents for task generation, evaluator synthesis, and meta-harness orchestration, as well as target models in experiments; manuscript use was limited to language refinement. The implementation runs through configuration-driven Markdown scaffolds, with archived artifacts carrying state between stages and an outer loop revising the inner benchmark-generation workflow.

THE USE OF LARGE LANGUAGE MODELS

  • LLMs serve as framework agents and experimental target models; manuscript use was limited to language refinement, not scientific claims or core ideas.Generation quality was assessed through human validation.Source: THE USE OF LARGE LANGUAGE MODELS · p. 10

A.1 MASTER INITIALIZATION

A THE IMPLEMENTATION DETAILS OF METABENCH-HARNESS

  • Magent runs sequential Markdown-scaffold modules, reading prior artifacts and writing required outputs; file passing carries state between stages.A stage is complete when its required files exist on disk, rather than through persistent context.Source: A THE IMPLEMENTATION DETAILS OF… · p. 13

A.2 OUTER LOOP: META-HARNESS ORCHESTRATION

  • The outer meta-harness reviews archived telemetry and proposes scoped revisions to harness prompts or configurations, preserving seed lineage and deterministic scoring.Proposals include evidence, expected score direction, risks, and rollback plans.Source: §A.2 · p. 13, §A.2.1 · p. 14

A.2.2 STAGE 2: HARNESS EVALUATOR

  • The evaluator scores candidates across ten dimensions and rejects revisions that weaken deterministic verification, seed lineage, or task validity.Weighted judgment prioritizes difficulty pressure, evaluator reliability, task validity, seed fidelity, and reproducibility.Source: §A.2.2 · p. 14

A.2.3 STAGE 3: HARNESS SELECTOR

  • A selector ranks proposals across rounds, may select none, and checks invariants; an applier makes only approved scoped edits and reports modified files.The workflow allows at most one candidate per round unless two are complementary and low-risk.Source: §A.2.3 · p. 14, §A.2.4 · p. 14

A.3 INNER LOOP: BENCHMARK GENERATION

  • Each inner-loop round follows scenario discovery, task and evaluator synthesis, model validation, and lifecycle audit.Source: §A.3 · p. 15

A.3.1 STAGE 1: SCENARIO DISCOVERY

  • Scenario discovery over-generates seed-derived candidates, validates lineage and novelty, refills an active budget of 10, and enforces transformation diversity.For CodeContests, acceptable transformation axes include reasoning burden, edge cases, testing strategy, output contract, and solution debugging.Source: §A.3.1 · p. 15

A.3.2 STAGE 2: TASK AND EVALUATOR SYNTHESIS

The workflow synthesizes tasks and evaluators, validates model performance, and audits scenario lifecycles to guide subsequent evolution. Its implementation specifies agent settings, while the appendix introduces supplementary experimental details.

A.3.2 STAGE 2: TASK AND EVALUATOR SYNTHESIS

  • High-performing or universally solved task families cannot receive same-difficulty siblings; escalation uses validity-preserving levers, and evaluators must pass three probes with reliability grades of at least 4.Acceptance also requires valid lineage, low copy risk, and empirical-difficulty checks; AIME-2024 tasks additionally need a deterministic answer checker that reproduces the expected answer.Source: §A.3.2 · pp. 15–16

A.3.3 STAGE 3: MODEL VALIDATION

  • The model-validation stage generates per-round runners, handles token budgets and truncation explicitly, and scores outputs programmatically, assigning zero with error types to missing, failed, or truncated outputs.Per-task means and cross-model score spreads flag tasks as too easy or non-discriminative for audit.Source: §A.3.3 · p. 16

A.3.4 STAGE 4: LIFECYCLE AUDIT

  • Lifecycle audits distinguish confounded scores from capability signals, assign each scenario a keep, stabilize, or reseed decision, and stop only for specified novelty, evaluator, or production failures.Round summaries audit executable components for reproducibility and distill improvement signals for the outer loop.Source: §A.3.4 · p. 16

A.4 IMPLEMENTATION DETAILS OF METABENCH-HARNESS

  • Claude-Sonnet-5 drives the workflow end to end, with proxy-model requests routed through OpenRouter and generation limited to 16,000 tokens and 600 seconds per call.Errors such as empty responses, timeouts, and budget overflows are handled, with returned token counts and scores recorded.Source: §A.4 · p. 17

B ADDITIONAL EXPERIMENTAL DETAILS AND RESULTS

B.1 EXPERIMENTAL SETUP FOR MAIN RESULT OF HELD-OUT MODEL EVALUATION

The evaluation setup compares unevolved seeds, selected early and late rounds, and alternative evolution methods, while separating proxy models from held-out evaluators. The appendix also details how harness edits are audited.

B.1 EXPERIMENTAL SETUP FOR MAIN RESULT OF HELD-OUT MODEL EVALUATION

  • For the main benchmark comparison, original CodeContests and AIME-2024 serve as baselines, with rounds 2 and 10 selected for MB-CodeContests and rounds 1 and 5 for MB-AIME.The selected rounds represent early and late evolution stages.Source: §B.1 · p. 17
  • Evolution-method comparisons use AIME-2024-derived datasets from AutoEvoEval, TRACE, naïve prompting, and a fixed harness to assess alternative methods and harness-level evolution.Naïve prompting removes the outer loop and viability checks; the fixed-harness condition also removes the outer loop but retains seed items and evaluation logic.Source: §B.1 · p. 17
  • Claude-Sonnet-5 and GPT-5.4 serve as proxy models, while held-out models from standard-instruct and test-time-scaling classes evaluate the resulting benchmarks.The two Qwen reasoning models use a 96,000-token reasoning phase followed by a 32,000-token answer-extraction phase.Source: §B.1 · pp. 17–18

B.2 EXPERIMENTAL SETUP FOR EVOLUTIONARY DYNAMICS AND QUALITY ANALYSIS

  • The evolutionary-dynamics and quality analyses cover outer-loop adaptations, task generation, benchmark reasonableness, competency, robustness, and seed-to-evolved-task case studies.Source: §B.2 · p. 18

B.2.1 HARNESS-LEVEL EVOLUTIONARY DYNAMICS ANALYSIS

  • The harness-edit audit classifies changes into measurement, generation, execution, and telemetry, finding a steady minority of edits maintains the instrument through reliability fixes and score filtering.The authors describe this maintenance as reinforcing the evaluation environment while the inner loop explores more complex problems.Source: §B.2.1 · p. 18

B.2.2 QUALITATIVE ANALYSIS ON TASK EVOLUTION

The qualitative trajectories show the system correcting measurement problems before escalating difficulty, while responding to model behavior and filtering confounds. The trajectories differ between coding and mathematical tasks.

Coding Problems

  • In CodeContests Round 1, formatting failures affected 121 of 180 model-task pairs; correcting the evaluator in Round 2 raised scores to 0.948 and 0.591 without changing task difficulty.Subsequent escalation drove scores down to 0.785 and 0.396, with a three-scenario capability split.Source: §B.2.2 · p. 19
  • By Round 10, the CodeContests model-score gap reaches 0.546 (0.839 vs. 0.293), compared with 0.042 in Round 1, after refinement and difficulty escalation.Rounds 5–7 stalled; the system retired scenarios that failed to grow harder after two attempts and intensified the remaining tasks in Rounds 8–9.Source: §B.2.2 · p. 19

Mathematical Problems

  • The AIME-2024 trajectory begins with model separation at 0.750 vs. 0.125, with GPT-5.4 failures attributed to incorrect numeric derivations rather than format rejection.Source: §B.2.2 · p. 19
  • In AIME rounds 2–5, validation distinguishes shared reasoning errors, divergent model performance, execution timeouts, and instruction drift, pausing escalation for confounded scenarios and restoring a dropped reasoning constraint.The authors say this filtering stabilizes the evaluation environment so task complexity can be escalated without treating superficial confounds as capability limits.Source: §B.2.2 · p. 19

B.2.3 QUANTITATIVE ANALYSIS OF EVOLVED BENCHMARK CHARACTERISTICS

The quantitative analysis tracks difficulty levers, evaluated capabilities, and solving techniques across CodeContests and AIME-2024. It reports domain-specific shifts that enrich task composition rather than merely lengthening reasoning.

Distribution of Difficulty Levers

  • AIME-2024 uses Output Contract Tightening (70%) and Compound Conjunction (70%), while CodeContests uses Output Contract Tightening (91%) and Compound Conjunction (53%) as primary structural difficulty levers.AIME-2024 also uses Distractor Injection (40%); CodeContests uses Scale Up (33%), Meta Level Shift (30%), Multi-case Batching (24%), and Adversarial Input Seeding (22%).Source: §B.2.3 · p. 20

Shift in Evaluated Capabilities

  • CodeContests shifts toward Construction Search (20% to 90%) and Output Contract (95%), while AIME-2024 shifts toward Multi-case Bookkeeping (30% to 100%) and Output Contract (70%).CodeContests' original Domain Reasoning (96.4%) declines; AIME-2024 retains Domain Reasoning (90%) while Spec Parsing drops from 83.3% to 30%.Source: §B.2.3 · p. 20

Evolution of Solving Techniques

  • Evolved CodeContests tasks demand Delimiter Protocol Compliance (96%) and Magnitude Bound Control (50%), while AIME-2024 emphasizes Combinatorial Enumeration (90%), Maximality Saturation Reasoning (80%), and JSON Schema Compliance (70%).Closed Form Derivation (63.3%) disappears from AIME-2024, and Exact Integer Arithmetic falls from 76.7% to 10%.Source: §B.2.3 · p. 20

Overall

  • The authors characterize CodeContests evolution as constraint-based and AIME-2024 evolution as structural, with both adapting mechanisms to domain demands and enriching task composition.Source: §B.2.3 · p. 20

B.2.4 QUALITY ASSESSMENT OF THE EVOLVED BENCHMARKS

Across evolutionary rounds, benchmark quality is exploratory and non-monotonic: despite temporary reliability drops, trajectories recover to strong final robustness and acceptance. Case studies track how seed tasks deepen through expanded capabilities, solving techniques, and difficulty levers.

B.2.4 QUALITY ASSESSMENT OF THE EVOLVED BENCHMARKS

  • After sharp evaluator-robustness drops, both trajectories recover to final metrics above 90% Evaluator Robustness and a 70% overall “All Three Yes” acceptance rate.Evolutionary Reasonableness remains consistently high; the assessment covers each round and uses an isolated Claude-Opus-5 judge.Source: §B.2.4 · p. 23
  • Figure 9 traces quality across rounds, showing an exploratory, non-monotonic trajectory with temporary reliability crashes followed by recovery.The cited examples are CodeContests Round 8 and AIME Round 4, where Evaluator Robustness collapses temporarily.Source: §B.2.4 · p. 23, Figure 9 · p. 23
  • Figure 8 presents a simplified version of the system prompt used for LLM-as-a-judge quality evaluation.The prompt outlines essential evaluation criteria while omitting operational overhead such as I/O formatting.Source: §B.2.4 · p. 23, Figure 8 · p. 23

B.2.5 DETAILED CASE STUDIES: FROM SEED TASKS TO EVOLVED TASKS

  • The round-by-round case studies track task formulation from seed to deeper stages, alongside expanding capabilities, solving techniques, and applied difficulty levers.Table 8 presents this breakdown for MB-CodeContests and MB-AIME.Source: §B.2.5 · pp. 23–24, Table 8 · p. 24

Related papers

Loading 2609.33411v1…