PaperlayerView exact revision on arXiv

Source-linked AI summary

BenchAgents: Multi-Agent Systems for Structured Benchmark Creation

Natasha Butt, Varun Chandrasekaran, Neel Joshi, Besmira Nushi, Vidhisha Balachandran

arXiv:2410.22584v2cs.LGcs.AIcs.CL

TL;DR

High-quality benchmarks are needed to evaluate new capabilities, but existing datasets can saturate or become contaminated and manual creation is costly and difficult to scale. BENCHAGENTS uses agents to plan, generate, verify, and evaluate benchmarks across language and vision tasks. The resulting evaluations find reasoning-model advantages on scheduling and constrained writing, persistent difficulty with negation, and visual-processing limits on causal reasoning.

  • Problem

    Evaluation datasets can saturate or become contaminated, while manually creating new benchmarks is costly, time-consuming, and difficult to scale.

    Source: §1 · p. 1

  • Method

    BENCHAGENTS uses interacting agents to plan benchmark specifications, generate instances, verify their quality, and implement evaluation metrics, with developer feedback at each stage.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    Reasoning models outperform non-reasoning models on scheduling and constrained text generation, with gains rising with task complexity, while models struggle with negation and visual causal reasoning is limited by image processing.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Reasoning-capable models show promise for productivity applications involving scheduling and writing, although negation constraints remain challenging across models.

    Source: §7 · p. 8

    For the open-ended generative BA-Causal task, evaluation uses LLM-as-judge, with metric–human alignment discussed in Appendix E.2.

    Source: §4.3 · p. 4

Abstract

from arXiv · show

Evaluation insights are limited by the availability of high-quality benchmarks. As models evolve, there is a need to create benchmarks that can measure progress on new and complex generative capabilities. However, manually creating new benchmarks is slow and expensive, restricting comprehensive evaluations for any capability. We introduce BenchAgents, a multi-agent framework that methodically leverages large language models (LLMs) to automate evaluation benchmark creation while inherently ensuring data and (evaluation) metric quality. BenchAgents decomposes the benchmark creation process into planning, generation, verification, and evaluation, each of which is ] orchestrated via LLM agents. These agents interact with each other and utilize feedback from benchmark developers to improve and flexibly control data diversity and quality. We use BenchAgents to create benchmarks to evaluate capabilities related to planning, constraint satisfaction, and causal reasoning spanning both language and vision modalities. We then use these benchmarks to study state-of-the-art models and extract new insights into common failure modes and model differences.

1 Introduction

Existing evaluation datasets can saturate or become contaminated, while manual benchmark creation is costly and hard to scale. BENCHAGENTS automates benchmark creation and evaluation, and its benchmarks reveal differences and shared failure modes across models.

1 Introduction

  • Evaluation datasets can saturate or become contaminated, and manual benchmark creation is costly, time-consuming, and difficult to scale.Without benchmarks, new capabilities are often demonstrated with anecdotal examples or small, noncomprehensive test sets.Source: §1 · p. 1
  • BENCHAGENTS uses planning, generation, verification, and evaluation agents to create diverse benchmarks with developer feedback at each stage.The planning agent specifies data parameters and constraints, verification checks, and evaluation metrics; generation produces code and instances, while verification and evaluation implement checks and metrics.Source: §1 · p. 1
  • Reasoning models outperform non-reasoning models on scheduling and constrained text generation, with larger gains as task complexity rises; all models struggle with negation constraints.Visual causal reasoning performance is limited by visual processing of images.Source: §1 · p. 1

2 Related Work

Prior automated benchmark-creation approaches support fine-grained evaluation but are often specialized to particular tasks or depend on existing seed data.

2 Related Work

3 BENCHAGENTS

BENCHAGENTS turns a developer’s task specification into structured plans, generated instances, verification checks, and executable evaluation metrics. Developers can refine the plans, while generated instances are checked for quality and evaluation can combine programmatic and model-based methods.

3 BENCHAGENTS

  • The Planning Agent takes a natural-language task specification and optional seed examples, then decomposes benchmark creation into plans for the other agents.The framework outputs diverse, verified instances and metrics with executable evaluation code.Source: §3 · p. 2
  • The plan specifies parameters and value ranges, optional constraints, and tools or libraries to control instance diversity and generation.It also defines verification checks and an evaluation metric suite; developers can refine proposed elements to match their measurement goals.Source: §3 · pp. 2–3
  • The Generation Agent samples parameter and constraint values into metadata, then uses templates or prompted LLM generation to produce instances.Tools and libraries can enrich generation, including multimodal content when required by the benchmark specification.Source: §3 · p. 3
  • Verification checks assess instance clarity, completeness, consistency, feasibility, and task-specific complexity.Checks can be implemented as programmatic code or model-based prompts; evaluation can combine programmatic metrics with context-aware model judgments for open-ended outputs.Source: §3 · pp. 3–4
  • Agents’ generated code, verification checks, and evaluation metrics are themselves verified and evaluated for correctness.The paper points to Section 5 for details.Source: §3 · p. 4

4 Benchmark Generation

BENCHAGENTS is used to generate benchmarks for calendar scheduling, long-form constrained text generation, and visual causal reasoning. Each task defines parameters, constraints, quality checks, and evaluation measures suited to its target capability.

4.1 Calendar Scheduling (BA-Calendar)

  • BA-Calendar addresses limited coverage in NATURALPLAN, which covered two participants and one-day events.The paper motivates the benchmark by the importance of scheduling for planning and reasoning capabilities.Source: §4.1 · p. 4
  • BA-Calendar varies scheduling parameters and constraints, checks clarity, completeness, consistency, and feasibility, and scores complexity by constrainedness.Its parameters include participants, availability, and time blocks; constraints include duration, buffers, and blackout periods.Source: §4.1 · p. 4

4.2 Long-form Text Generation (BA-Text)

  • BA-Text targets long-form responses with complex content-based constraints, including positive, negative, positional, sequencing, conditional, and iterative requirements.Verification checks clarity, completeness, consistency, and feasibility, while complexity is measured by the ratio of applied to defined constraints.Source: §4.2 · p. 4

4.3 Visual Causal Reasoning (BA-Causal)

  • BA-Causal uses controlled 2D physics simulations to test whether models predict the next state from four image frames and choose among one true answer and three distractors.Instances are retained only when their complexity is below 0.99, supporting clear answer separability; evaluation uses exact-match accuracy.Source: §4.3 · pp. 4–5

5 Benchmark Quality Assessment

BENCHAGENTS combines verification checks with filtering to produce high-quality instances, and its generated benchmarks show broad parameter coverage and non-trivial difficulty. Developer feedback and human assessments provide additional validation, though a human baseline for BA-Text is costly.

5 Benchmark Quality Assessment

  • Instances failing any programmatic or model-based verification check are filtered, yielding high-quality datasets across the three tasks.BA-Calendar has consistently higher instance quality than the more complex BA-Text; BA-Causal instances are correct by construction and receive a feasibility check.Source: §5 · p. 5
  • GPT-4o performance drops monotonically with increasing complexity in buckets containing more than 100 instances, indicating that added constraints raise difficulty.The authors report this pattern across all tasks.Source: §5 · p. 5
  • Generated instances have broad parameter coverage, and the authors report that they outperform manually curated benchmarks on this measure.The reported coverage analysis is in Appendices C and D.Source: §5 · p. 5
  • Two annotators scored 85% on BA-Calendar and 97.5% on BA-Causal; BA-Text lacks a human baseline because each instance takes annotators about 1–1.5 hours.The authors describe the human assessment as a solvability check.Source: §5 · p. 5
  • Developers refine plans and verify generated code and prompts, with intervention needs varying by task.BA-Calendar needed planning refinement for logical constraint parsing, while BA-Causal interventions focused on implementation correctness and verification criteria.Source: §5 · p. 5

6 Model Analysis

Model performance varies across the benchmarks: reasoning models lead on scheduling and constrained text generation, while visual causal reasoning remains harder from images than from text. The analyses also identify weaknesses on complex constraint combinations, visual distractors, and long-form instruction following.

6 Model Analysis

  • Reasoning models exceed 80% pass-all accuracy on BA-Calendar, while non-reasoning models remain below 50%.Most models satisfy more than 70% of constraints, but non-reasoning models less reliably satisfy all of them.Source: §6 · p. 6, Figure 2 · p. 6
  • On BA-Calendar, non-reasoning models and o3-mini-high lose pass-all accuracy as constrainedness increases; reasoning models improve on buffer-time and priority constraints.o3-mini-high underperforms other reasoning models overall and on simpler constraints, while models more often report no solution for infeasible than feasible instances.Source: §6 · p. 6, Figure 2 · p. 6

Task 2. Long-form Constrained Text Generation Results

  • The best BA-Text model reaches 60% pass-all accuracy, and reasoning models outperform non-reasoning models by a smaller margin than on BA-Calendar.The larger gap between fraction passed and pass-all accuracy indicates that models struggle to satisfy every constraint in a problem.Source: Task 2. Long-form Constrained Text… · pp. 6–7, Figure 4 · p. 7
  • BA-Text performance is stable on positional and negative constraints, but iterative and conditional constraints show greater variation; all models score similarly low on negative constraints.On challenging combinations, Llama 3.1 405B drops from over 60% on conditional-only tasks to 20% when iterative constraints are added; reasoning models lead on these examples.Source: Task 2. Long-form Constrained Text… · pp. 7–8, Figure 4 · p. 7
  • GPT-4o, o1, and Gemini-2-Pro achieve the highest BA-Causal accuracy, while GPT-4o and o1 perform near perfectly with text inputs but drop on image inputs.The text-versus-image comparison suggests that the task is more challenging when models must use visual inputs.Source: Figure 5 · p. 7, Task 2. Long-form Constrained Text… · p. 8
  • On BA-Causal, GPT-4o accuracy falls from above 80% to about 50% as option similarity increases, while greater material or shape diversity lowers accuracy for high-performing models.As object counts rise, the top models improve while lower-performing models decline.Source: Figure 5 · p. 7, Task 2. Long-form Constrained Text… · p. 8

7 Conclusion

BENCHAGENTS automates structured, diverse benchmark creation from developer specifications while minimizing manual effort and retaining human oversight. Its benchmarks expose model-specific strengths and failure modes, including persistent difficulty with negation and a visual-perception bottleneck in causal tasks.

7 Conclusion

  • Reasoning-capable models outperform others on scheduling and writing tasks, while negation remains challenging and visual perception is identified as the key causal-task bottleneck.The conclusion connects the latter finding to a need for stronger multimodal grounding.Source: §7 · p. 8

A More Details on Benchmark Generation (§ 4)

This section details how BenchAgents generates and verifies benchmark instances for calendar scheduling, constrained text, and simulated visual causal reasoning. It also describes the resulting datasets and reports parameter coverage and constraint distributions for BA-Calendar and BA-Text.

Agent Configuration

  • BA-Calendar generation samples scheduling parameters and constraints, a candidate meeting slot, and participant availability before forming each prompt and filtering instances.The instance is a (prompt, parameters, constraints) triple, and verification filtering leaves 2,000 instances. Time slots must satisfy meeting duration and any additional constraints, such as buffer time.Source: Agent Configuration · pp. 11–13

A.2 BA-Text

  • For BA-Text, the data-generation plan includes an example constraint for each user parameter and constraint group; its normalized edit distance is 0.02, indicating minimal changes were made.The procedure prompts the LLM to generate a topic grounded in instance parameters, then sequentially generate constraints with all previous constraints in context.Source: §A.2 · pp. 15–16

A.3 BA-Causal

  • BA-Causal presents four simulated frames and asks vision models to select the correct next state from four options, with distractors drawn from earlier trajectory states or hypothetical variations.Simulations vary gravity, object properties, initial velocities, and interaction dynamics to test causal consistency in physics-based scenes.Source: §A.3 · p. 17

1. Space-Level Parameters

  • The BA-Causal plan varies space-level conditions including gravity, fixed simulation dimensions, and boundary friction and elasticity.The described ranges include gravity magnitude 0–10 m/s², a suggested 800-by-800-pixel space, and friction and elasticity values from 0 to 1.Source: §1 · p. 18

2. Object-Level Parameters

  • Object-level simulation varies shape, dimensions, physical properties, initial positions, and velocities; the generator simulates 20 steps and selects a collision-centered frame sequence with three distractors.The input uses frames t1–t4 and the answer is t5; the answer and distractors are randomly ordered among the multiple-choice options.Source: §2 · pp. 18–19
  • For each BA-Causal instance, G-Agent samples parameters and a trajectory, selects input and option frames, and records an answer ID; V-Agent filtering leaves 500 verified instances.Each instance is represented as (inputs, options, answer_id).Source: §2 · p. 19, Table 14 · p. 19

B V-Agent’s Pass Rates

  • Figures 7a and 7b report pass rates for each verification test across all G-Agent-generated instances for BA-Calendar and BA-Text, respectively.The supplied passage does not state individual pass-rate values.Source: B V-Agent’s Pass Rates · p. 20, Figure 7 · p. 20

C BA-Calendar’s Parameter Coverage & Comparison to NATURALPLAN

D BA-Text’s Parameter Coverage

  • BA-Text has a relatively uniform distribution across constraints, with positive constraints more frequent and positional and sequencing constraints less frequent.The passage attributes the pattern potentially to G-Agent sampling and to positional and sequencing constraints conflicting more often and being filtered by V-Agent.Source: D BA-Text’s Parameter Coverage · p. 20

E More Details on Human Annotation Experiment

The appendix describes human assessments of the verification and evaluation agents, including annotation criteria, sampling, and the metrics reported for their comparisons with human annotations.

E.1 Human Assessment of V-Agent

  • For V-Agent reliability, two annotators assess each model-based check on 50 pre-filtered instances per dataset, and both must mark a check as passed for ground truth.Checks include clarity, completeness, and consistency, plus feasibility for BA-Text.Source: E More Details on Human Annotation… · p. 20, §E.1 · p. 21
  • The V-Agent assessment reports accuracy, precision, and recall against human-annotated ground truths.Figure 11 presents V-Agent scores compared with human annotations.Source: §E.1 · p. 21, Figure 11 · p. 22

E.2 Human Assessment of E-Agent

  • For BA-Text, two annotators assess each evaluation test on 20 V-Agent-filtered instances; the study reports accuracy and precision/recall for E-Agent checks.Ground truth uses the worse of the two annotations, and predicted values come from E-Agent tests run on GPT-4o solutions; Figures 12 and 13 show accuracy and human-versus-agent scores, respectively.Source: §E.2 · p. 21, Figure 12 · p. 22, Figure 13 · p. 22

F Experimental Settings

The experimental settings specify zero-shot model evaluation and a text-only BA-Causal variant that describes simulated images with object attributes and coordinates. The variant retains next-state prediction from a sequence of frames and multiple-choice options.

F Experimental Settings

  • Each test instance is evaluated by zero-shot inference with temperature 0, top_p 0.95, and max_tokens 2000, then assessed using E-Agent criteria.The evaluation produces constraint-satisfaction test results for each instance.Source: F Experimental Settings · p. 22

F.1 BA-Causal Text-Only

  • The BA-Causal text-only variant represents each image using object shape, colour, size, and x,y coordinates, while asking for the next state from four choices.The prompt asks models to describe changes across four successive frames before selecting the next state, with a response format specifying changes and an answer letter.Source: §F.1 · pp. 22–23

G Additional Results on Model Analysis (§ 6)

Additional results report performance across increasing constrainedness, parameter ranges, and availability-schedule complexity.

H Quantifying Developer Feedback

Developer feedback varied by benchmark, while additional figures break down model performance across calendar, text, and causal tasks.

H Quantifying Developer Feedback

  • BA-Calendar required more developer feedback than BA-Text, with edits addressing logical-condition parsing for programmatic tests.The authors suggest this intervention supports better data generation for BA-Calendar.Source: H Quantifying Developer Feedback · p. 25
  • For BA-Causal, normalized edit distances were 0.00 for planning, 0.27 for generation code, and 0.34 for verification-plan refinement.Developer effort shifted toward implementing PyMunk simulation and trajectory sampling, and defining feasibility checks such as answer–distractor separation.Source: H Quantifying Developer Feedback · p. 25, Table 15 · p. 25

I Full Evaluation Plots

  • Figure 16 breaks down BA-Calendar results by fraction passed, pass-all versus constrainedness, no-solution outcomes, and pass rates for applied constraints.Source: Figure 16 · p. 26
  • Figure 17 shows BA-Text fraction-passed and pass-all results, plus pass rates for individual applied constraints.Source: Figure 17 · p. 27
  • Figure 18 reports BA-Causal accuracy overall and by object count, material diversity, complexity, and shape diversity, and compares text-only with image tasks.Source: Figure 18 · p. 27
Loading 2410.22584v2…