Automating Benchmark Design

1 Introduction

• Static benchmarks quickly become obsolete as models improve, making it difficult to distinguish state-of-the-art model performance.Dynamic benchmarks can avoid saturation through recalibration or harder data, but still rely heavily on manual authoring and updates.Source: §1 · p. 1
• Parameterized environment design offers a dynamic-benchmarking route, but non-trivial parameter spaces are intractable to search by naive random sampling.Many randomly sampled configurations may be trivial or unsolvable.Source: §1 · p. 1
• BeTaL iteratively proposes benchmark parameters, evaluates a generated sample with the target model, and feeds results back to the designer for refinement.The closed-loop process can target properties such as difficulty or realism; this paper focuses on target difficulty.Source: §1 · pp. 1–2
• BeTaL treats benchmark design as optimization over parameters that determine properties such as difficulty, then uses LLM reasoning to search that space.The framework aims to reduce the burden of designing and continually updating benchmarks, and can repurpose outdated static benchmarks.Source: §1 · p. 2
• BeTaL produces benchmarks with 5.3%–13.2% deviation from target difficulty, a 2–4× improvement over baselines across the evaluated tasks.The evaluation covers modified existing benchmarks and new mathematical and spatial reasoning benchmarks.Source: §1 · p. 2

2 Methodology

• The methodology frames BeTaL as an LLM-in-the-loop framework that iteratively designs benchmarks to meet user-specified goals.Its system begins with a loosely defined environment template that can be refined and instantiated.Source: §2 · p. 2, §2.1 · p. 3

2.1 Preliminaries

• An environment is defined by controllable parameters, a simulator generates problems with ground-truth solutions, and the user specifies a target performance level and distance measure.The paper uses target difficulty as its performance measure and assumes verifiable or procedurally generated solutions.Source: §2.1 · p. 3

2.2 BeTaL: Benchmark Tuning with LLM-in-the-loop

• BeTaL prompts a designer LLM for parameter values, instantiates a benchmark, evaluates the target model, and returns performance feedback for the next iteration.It retains the configuration with the smallest observed gap from the target and returns it after the allotted iterations.Source: §2.2 · p. 4

3.1 Benchmarking Tasks

• The tasks cover arithmetic sequences, spatial reasoning on a grid, and simulated airline customer service with tool use and database updates.Their complexity can be varied through task-specific parameters such as sequence length, board size, action types, and airline-task variables.Source: §3.1 · pp. 4–5

3.2 Baselines

• The baselines are RS+PPR, which replays favorable parameter configurations, and two Best-of-N variants that select configurations using a predictive model or target-model rollouts.RS+PPR combines uniform random sampling with noisy variants drawn from a buffer.Source: §3.2 · p. 5

3.3 Designer and Target Models

• GPT-5, Claude Opus 4.1, and Grok 4 serve as designer models, while o4-mini is the target model and three models evaluate the generated benchmarks.The evaluation models are o4-mini, Gemini 2.5 Flash, and Claude 3.7 Sonnet.Source: §3.3 · p. 5

3.4 Metrics

• Performance gap is the absolute difference between the target model’s observed performance and the specified target performance.The paper assumes the task performance measure is inversely proportional to difficulty.Source: §3.4 · p. 5

3.5 Experiment Protocol

• The protocol uses four target levels—Hard 0.25, Medium 0.50, Easy 0.75, and Trivial 0.90—and runs iterative methods for 10 iterations or samples 10 configurations for non-iterative methods.The best parameters are used to generate a larger evaluation dataset; results are repeated with three random seeds and reported with 95% confidence intervals.Source: §3.5 · p. 5
• Table 1 reports that BeTaL outperforms iterative and Best-of-N baselines in parameter search and evaluation across all three tasks and designer models.The table reports average performance gap in percent, using o4-mini as the target model.Source: Table 1 · p. 6
• Table 2 reports that chain-of-thought prompting does not consistently produce strong designer-model performance or outperform random sampling.Claude Opus-4.1 is competitive on arithmetic sequences and τ-Bench, while reported values use average performance gap with o4-mini as target.Source: Table 2 · p. 6

• BeTaL outperforms RS+PPR and Best-of-N variations across benchmarks and designer models, with its authors attributing the advantage to feedback-driven parameter refinement.The reported performance gap shrinks more strongly than RS+PPR over 10 iterations, by more than 20% on both τ-Bench and Spatial Reasoning.Source: §4 · p. 6
• BeTaL consistently outperforms baselines at each target difficulty level across Arithmetic Sequences, Spatial Reasoning, and τ-Bench.The evaluation uses o4-mini as the target model and GPT-5 as designer; similar performance is noted with Claude Opus 4.1 and Grok-4 as designers.Source: §4 · p. 7, Figure 2 · p. 7
• Difficulty gaps vary by benchmark: τ-Bench and Spatial Reasoning show their largest gaps at Trivial difficulty, while Arithmetic Sequence’s largest gap occurs at Hard.The paper attributes the Arithmetic Sequence pattern to several degenerate solutions.Source: §4 · p. 7
• BeTaL’s performance gap shrinks faster than RS+PPR’s within 10 iterations on Spatial Reasoning and τ-Bench, averaged across difficulty levels and designer models.The comparison indicates more efficient search for favorable environment parameters.Source: Figure 3 · p. 7
• Designer performance depends on task domain: Grok-4 and GPT-5 do well on mathematical and logical tasks, while Claude-Opus-4.1 excels on τ-Bench Airline.On τ-Bench Airline, the performance gap is 7.7 ± 5.2% for Claude-Opus-4.1, compared with 13.2 ± 10.3% for GPT-5 and 10.3 ± 12.4% for Grok-4.Source: §4 · p. 7
• BeTaL-designed benchmarks transfer across evaluation models; on τ-Bench, o4-mini-generated benchmarks yield comparable performance for Claude 3.7 Sonnet and Gemini 2.5 Flash, with BeTaL outperforming baselines.The paper also reports that stronger or weaker evaluation models should follow the target difficulty trends with larger or smaller performance-gap magnitudes, respectively.Source: §4 · pp. 7–8, Figure 4 · p. 8
• Chain-of-Thought prompting alone remains unreliable for benchmark design: Claude Opus-4.1 beats the random baseline only on Arithmetic Sequence and τ-Bench, while failing on Spatial Reasoning.GPT-5 and Grok-4 underperform even further in these results.Source: §4 · p. 8

C4. Can LLMs also generate better parameter spaces?

• 24.4% and 23.3% performance gaps for GPT-5 and Opus 4.1 on trivial and easy benchmarks exceed the human-space gaps of 15.6% and 13.3%, respectively.The authors attribute these gaps to limited flexibility and controllability, particularly for smooth difficulty scaling across target performances.Source: Figure 6 · p. 9, C4. Can LLMs also generate better… · p. 9
• 1.1% and 1.7% are the reported performance gaps for AI-generated parameter spaces on medium and hard benchmarks, respectively.The comparison is between parameter spaces generated by Opus 4.1 and humans.Source: Figure 6 · p. 9, C4. Can LLMs also generate better… · p. 9
• The experiment prompts Claude Opus 4.1 to generate a complete τ-bench parameter space, then manually implements feasible parameters in the task generator.Opus 4.1 adds controls for cooperation level and whether preferences are explicit or implicit.Source: C4. Can LLMs also generate better… · p. 9
• The authors conclude that current LLMs show partial autonomy in environment design, while full self-sufficiency in parameter-space generation remains an open challenge.They report that AI-generated spaces can capture structural patterns for challenging, calibrated benchmarks, but have limited flexibility and controllability across target performances.Source: C4. Can LLMs also generate better… · p. 9

Automating benchmark design

• Task-generation approaches include BENCHMAKER [16] and CHASE [17], while other work evolves tasks through perturbation, probing, alternation, or multi-agent coordination [19] [20].The paper distinguishes its approach by parameterizing benchmark design and using target-model feedback to tune it across domains.Source: §5 · p. 10

Environment design for curriculum learning

• UED methods adapt environments through adversarial generation [21], replay-based curation [11], or evolutionary mutation [22]; LLM-driven variants include EnvGen [23], LLM-POET [24], and R-Zero [25].BeTaL differs by avoiding a training loop and supporting adaptive benchmark generation with open and closed models.Source: §5 · p. 10

Scaling environments and datasets

• AgentScaler [26], APIGen [27], ToolACE [28], and Gaia2 [29] scale agentic environments or datasets, whereas this work focuses on automating evaluation.Source: §5 · p. 10

LLMs as optimizers

• LLM-based optimizers such as OPRO and evolutionary variants LEO and target mathematical tasks or prompt optimization, while this work applies optimization to benchmark design.Source: §5 · p. 10

6 Conclusion, Limitations and Future Work

• Across arithmetic, spatial reasoning, and agentic domains, BeTaL reduces performance gaps by 2-4× compared with LLM and non-LLM baselines.The conclusion frames this as potential support for evaluation systems that evolve alongside advancing models.Source: §6 · p. 10
• BeTaL assumes access to parameterized, verifiable task generators, depends on designer-model reasoning strength and careful prompts, and was evaluated on few domains.Multimodal and more subjective tasks remain unexplored.Source: §6 · p. 10
• Future work could optimize objectives such as realism and diversity, explore multi-agent or co-evolutionary loops, and add human oversight.The paper presents these as possible ways to enhance adaptability and reliability.Source: §6 · p. 10

Arithmetic sequences task

• In the arithmetic task, an agent returns operations that recursively transform input x into output y, and its predicted sequence is executed and checked against the ground truth.Source: §A.1 · p. 13
• Arithmetic task difficulty depends on operator choice, sequence length, input range, and whether x is integer or floating-point.The benchmark uses addition, subtraction, multiplication, division, square root, and power-of-two operations.Source: §A.1 · p. 13

Spatial Reasoning

• Spatial-reasoning queries ask for coordinates, tile number, orientation, or relative location, with structured answers checked against programmatically computed ground truth.Source: §A.1 · p. 13
• Spatial controls specify board size, permitted board and particle movements or rotations, action counts, and whether boundary overflow wraps around.Source: §A.1 · pp. 13–14

Human Designed τ-bench Airline

• Human-designed τ-bench airline tasks use discrete itinerary controls, including action, passenger, and baggage counts and booking or routing choices, to vary complexity.The passage says that more actions, passengers, bags, multiple strategies, connecting flights, or round trips add complexity or reasoning constraints.Source: §A.1 · p. 14

Opus 4.1 Designed τ-bench Airline

• Opus 4.1’s τ-bench space adds behavioral and informational controls, including information revelation, cooperation, and preference clarity, to steer the target model’s failure rate.The designer receives a target failure rate and returns structured parameters with a rationale.Source: §A.1 · p. 14

A.2 Detailed Baselines

• For RS+PPR, p = 0.5 sets the probability of sampling from the good-parameter buffer, and Δ = 0.1 sets the gap threshold for good parameters.Both settings were used across all settings.Source: §A.2 · p. 14
• BoN-ML experiments explored over 800 parameter configurations and architectures, using 100 samples per domain and 5-fold cross-validation.Features ranged from 13 to 74 per domain and were derived directly from environment parameters.Source: §A.2 · p. 14
• Small neural networks performed best for Arithmetic Sequences, Random Forests for Spatial Reasoning, and gradient boosting for τ-Bench.Models were selected by the highest cross-validation R2 score; Table 3 summarizes the 5-fold cross-validation training results.Source: §A.2 · p. 14, Table 3 · p. 15

A.3 Details of LLM Models

• The latest GPT-5 version as of Sep 25, 2025 was undisclosed; the listed versions also include Opus 4.1, Grok 4, o4-mini, claude3.7, and gemini-2.5-flash.The appendix provides version identifiers for the named models.Source: §A.3 · p. 15
• Designer models use temperature 0.5 and target models 0.0 by default, except two Claude models are only available at temperature 1.The exception applies to claude-opus-4-1-20250805 and claude-3-7-sonnet-20250219.Source: §A.3 · p. 15
• The default reasoning budgets are 4096 tokens for designers and 1024 for target models, but grok-4-0709 has no configurable reasoning budget.Source: §A.3 · p. 15

A.4 Dataset Sizes

• Search rollouts use 10, 30, and 250 samples for Arithmetic Sequence, τ-Bench, and Spatial Reasoning; evaluation uses 75, 50, and 500, respectively.Source: §A.4 · p. 15

A.5 Additional Results

• In all but a few settings, BeTaL iteratively improves parameter estimates and converges toward the desired performance gap after a few iterations, supporting LLMs as optimizers.The convergence study spans settings from trivial to hard difficulty.Source: §A.5 · p. 15, Figure 10 · p. 17
• In spatial reasoning, designers adjust board size and action options to tune difficulty, generally using larger boards and more actions for harder settings and reducing board actions to simplify tasks.For hard configurations, models generally prefer more capabilities and actions; for medium and trivial settings, they reduce board actions close to zero while retaining some particle actions.Source: §A.5 · p. 15, Figure 12 · p. 19, Figure 13 · p. 20
• Evaluation on o4-mini shows low performance gaps; across other models, τ-Bench has similar performance, while Spatial Reasoning and Arithmetic Sequences show larger gaps but retain relative difficulty levels.The benchmarks were designed using o4-mini as the target model.Source: §A.5 · p. 15, Figure 9 · p. 16
• BeTaL achieves low performance gaps across different designer models; all three show similar results across datasets and target difficulty levels, with Figure 11 results using o4-mini as target model.The cited comparison names Claude Opus-4.1 and Grok-4 as designer models.Source: §A.5 · p. 15, Figure 11 · p. 18

B Prompts

• The prompts direct designers to choose arithmetic-sequence, spatial-environment, and airline-task parameters to target regret, configure a 2D grid world, or approach a target failure rate.The airline prompt defines failure rate as 1 − pass@1; the arithmetic prompt asks for feedback-informed parameter choices.Source: B Prompts · pp. 23, 25
• The prompts tune task parameters such as action limits, information completeness, and user cooperation to steer the student model’s 1 −pass@1 toward a target failure rate.Source: B Prompts · p. 27

C Teacher Model Reasoning Traces

• The appendix presents teacher-model reasoning traces showing how environment parameters are selected to target difficulty levels across tasks.Source: C Teacher Model Reasoning Traces · p. 33

C.1 Arithmetic Sequence

• The arithmetic-sequence teacher targeted 25.0% accuracy after earlier environments reached 100.0%, choosing integer inputs and repeated squaring to make exact computation difficult.The proposed sequence used four pow and four mul operations, producing eight squarings and about 411 digits for an input near 40.Source: §C.1 · p. 33
• For arithmetic sequences, BeTaL selected {mul, add, sqrt} with N = 7 and K = 3, using float inputs capped at 20 to target ∼0.25 accuracy.The rejected {pow, div, sqrt} configuration was judged too ambiguous; the selected operations were intended to make reverse inference difficult but feasible.Source: §C.1 · p. 34
• A BoN-ML arithmetic-sequence trace targeted ∼0.25 accuracy with N = 8, K = 3, and a float-valued sequence combining mul, pow, and sqrt.The trace attributes the challenge to ambiguity from squaring and square-root operations, while describing the task as solvable with careful analysis.Source: §C.1 · p. 35

C.2 Spatial Reasoning

• For 75% spatial-reasoning accuracy, the BeTaL design used a 10x10 board, wrap-around, up to two board rotations, and bounded particle moves and rotations.The configuration was intended to require tracking multiple transformations, producing errors in about 25% of cases.Source: §C.2 · p. 35
• A BoN-TM spatial-reasoning trace targeted 75% accuracy with a 12x12 grid, wrap-around, 90- and 180-degree board rotations, and limited board movement.It also proposed full particle movement, limited rotation, and moderate action counts of 2–3.Source: §C.2 · p. 36
• A BoN-ML spatial-reasoning trace targeted 75% accuracy using an 8x8 grid, two board rotations, and three particle movements across roughly five steps.The trace emphasized coordinate transformations and particle movements relative to orientation as sources of moderate difficulty.Source: §C.2 · p. 36

C.3 τ-bench

• For τ-bench, one trace proposed four actions with two passengers and three baggages as an intermediate configuration intended to move failure rate toward 0.75.The trace compared it with configurations of four actions, two passengers, and two baggages, and five actions, two passengers, and three baggages, each reported at 0.7 failure rate.Source: §C.3 · p. 36
• Another τ-bench design targeted a 0.75 failure rate by increasing action and passenger complexity, adding competing booking criteria and toggles, and limiting cabin variation.A separate trace proposed four actions, two passengers, and two baggages, with varied booking-strategy, directness, round-trip, cabin, and insurance options.Source: §C.3 · p. 37