PaperlayerView exact revision on arXiv

Source-linked AI summary

BenchMeUp: Building Software Engineering Benchmarks from Natural-Language Specifications

Islem Bouzenia, Michael Pradel

arXiv:2610.37283v1cs.SE

TL;DR

Manually constructing software-engineering benchmarks is labor-intensive, while general-purpose agents can produce inconsistent or hard-to-reproduce instances. BenchMeUp coordinates LLM agents to construct, validate, and package benchmarks from natural-language specifications; across four tasks, 180 of 198 instances met all criteria, compared with 10 of 200 from Codex.

  • Problem

    Manual benchmark construction requires researchers to identify instances and sources, prepare execution environments, establish expected outcomes, and validate correctness; general-purpose agents can also select inconsistently and produce hard-to-reproduce instances.

    Source: §I · p. 1

  • Method

    BenchMeUp turns a natural-language specification into shared search rules and a construction plan, then uses LLM agents to find and construct instances, replay-check them in fresh environments, and package them behind a common interface.

    Source: §I · p. 2, B. Overview · p. 3

  • Results

    180 of 198 constructed instances meet all criteria, compared with 10 of 200 delivered by Codex for the same specifications.

    Source: §I · p. 2, §VI · p. 11

  • Takeaways & Limitations

    189 of 196 packaged instances support end-to-end evaluation, allowing eight existing techniques to be compared on common instances.

    Source: §I · p. 2

    Replay confirms outcomes required by agent-written checks, but does not establish that the checks are correct or that instances faithfully represent the task.

    Source: §IV · p. 10

Abstract

from arXiv · show

Software-engineering research relies on benchmarks that evaluate new techniques on a set of problem instances. Unfortunately, manually constructing benchmarks is labor-intensive, as researchers must find suitable instances, prepare execution environments, establish expected outcomes, and validate correctness. This paper presents BenchMeUp, an approach for automatically creating software-engineering benchmarks from natural-language specifications. Given a specification defining the desired instances, data sources, and evaluation procedures, BenchMeUp generates search rules and a construction plan, invokes parallel LLM agents that identify candidate instances and construct executable instances, validates the results in fresh execution environments, and packages the instances behind a unified command-line interface. We evaluate the approach on four tasks: program repair, issue reproduction, compiler autotuning, and fixing static-analysis warnings. Targeting 50 instances per task (200 in total), BenchMeUp constructs 198 instances, of which 196 (99\%) are successfully packaged. Search and construction take 36 minutes and \$0.79 per instance, on average. Evaluating eight existing techniques on the generated benchmarks, we find that 189 of 196 packaged instances (96\%) support end-to-end evaluation, allowing users to study the effectiveness of the techniques. Checking every constructed instance against its source, the specification, and its expected outcome, we find that 180 of 198 (91\%) meet all criteria, compared with 10 of 200 instances delivered by Codex (general-purpose coding agent). Overall, we show that LLM-based agents can automate benchmark construction across diverse software-engineering tasks and produce executable instances that existing techniques can use, while validity beyond execution still needs to be checked.

I. INTRODUCTION

Manual benchmark construction requires substantial work, and existing automation does not create real-world, executable benchmarks from natural-language specifications. BenchMeUp uses a coordinated LLM-agent pipeline to build and validate benchmarks; across four tasks, most constructed instances meet the stated criteria, unlike instances from a general-purpose agent.

I. INTRODUCTION

  • Manual construction requires researchers to find instances and sources, prepare artifacts and dependencies, establish expected outcomes, and verify requirements.Task-specific scripts, project setup, and manual inspection can add effort; benchmark reuse raises concerns about overfitting and representativeness.Source: §I · p. 1
  • Existing automation targets individual tasks or construction steps, while no known approach creates real-world, executable benchmarks from natural-language specifications.Other natural-language benchmark generators focus on synthetic questions for language-model evaluation.Source: §I · p. 1
  • 180 of 198 constructed instances meet all criteria, compared with 10 of 200 delivered by Codex for the same specifications.The criteria check each instance against its source, specification, and expected outcome.Source: §I · p. 2
  • BenchMeUp translates a researcher-written specification into shared search rules and a construction plan for agents to follow.The specification can describe desired instances, data sources, and evaluation procedures without requiring the researcher to know which repositories contain suitable instances.Source: §I · p. 2
  • The pipeline validates agent-generated instances in fresh execution environments and packages completed instances behind a common interface.The paper also evaluates the approach on four tasks, compares it with a general-purpose agent, and applies eight existing techniques to the benchmarks.Source: §I · p. 2

II. APPROACH

BenchMeUp takes a researcher-written benchmark specification and produces executable instances behind a common command-line interface. Its overview divides construction into four phases, beginning with shared search rules and a construction plan.

A. Problem Statement

  • A program-repair specification defines the required buggy Java program and exposing test, along with the conditions for a repair to count as correct.The specification also identifies the task, instance contents, sources, evaluation conditions, and desired benchmark size.Source: A. Problem Statement · p. 2
  • BenchMeUp’s package provides instances and a common command-line interface for evaluation, including the versions, patches, environment definition, and build and test scripts.The goal is to construct benchmarks for tasks within a given budget without requiring researchers to implement instance collection and execution.Source: A. Problem Statement · p. 2

B. Overview

  • The four phases derive shared search rules and a construction plan, find candidates, construct and replay-check instances in fresh containers, then package and verify them through a common interface.Search and construction agents run in parallel, sharing summaries of exploration or construction problems and attempted solutions.Source: B. Overview · p. 3

C. Phase 1: Generating Search Rules and Construction Plan

1) Generating the Search Rules:

Shared search rules resolve underspecified choices before candidate search, and a reviewed construction plan gives agents common steps, artifacts, and checks for turning candidates into instances.

1) Generating the Search Rules:

  • Unspecified thresholds and ambiguous terms can lead search agents to apply inconsistent candidate-selection policies; shared rules settle these choices before searching begins.The rules identify candidates, permitted sources, retention conditions, and information to record; the program-repair example operationalizes popularity as at least 100 stars and requires relevant development activity and program and test changes.Source: 1) Generating the Search Rules · pp. 3–4
  • A second LLM reviews the proposed search rules for omitted requirements and unresolved ambiguities, but approval checks consistency with the specification rather than objective correctness.If the researcher-set iteration limit is reached without approval, the specification and feedback return to the researcher instead of starting a search with unapproved rules.Source: 1) Generating the Search Rules · p. 4

2) Generating the Construction Plan:

  • The construction plan specifies stages, artifacts, and checks, and a second LLM reviews it against the specification and search rules before construction agents begin.For program repair, stages include setting up the repository, separating program and test patches, building buggy and fixed versions, and checking that exposing tests fail on the buggy version and pass on the fixed version; common commands support consistent evaluation.Source: 2) Generating the Construction Plan · p. 4

D. Phase 2: Finding Candidates

Phase 2 assembles a pool of plausible candidate records using the specification and accepted search rules, while runnable-instance construction waits for Phase 3.

D. Phase 2: Finding Candidates

1) Searching with Agents:

Search agents use specification-derived rules to find and record suitable candidates, while coordination helps parallel agents explore different directions. Researchers set search targets and timing, subject to available resources and service limits.

1) Searching with Agents:

  • Search agents autonomously explore data sources, inspect promising records, and record candidates that satisfy the search rules.They may query sources, run scripts, and investigate candidate histories further.Source: 1) Searching with Agents · pp. 4–5

2) Coordinating Search Agents:

  • Staggered starts and shared summaries of queries, filters, and explored repositories steer parallel agents toward different search directions.Summaries are periodically updated and distributed once all agents are running.Source: 2) Coordinating Search Agents · p. 5
  • Researchers set the candidate target, agent count, delays, synchronization interval, and time budget, constrained by computing resources and service limits.Source: 2) Coordinating Search Agents · p. 5

3) Checking and Deduplicating Candidates:

BenchMeUp verifies and deduplicates candidate records, then turns selected candidates into instances through staged artifact construction and replay checks in fresh containers. Failed candidates can be revised, dropped, or replaced under researcher-set limits.

3) Checking and Deduplicating Candidates:

  • Before entering the shared pool, candidates must pass a fresh-container existence check against their recorded identities and fields; the pool is then deduplicated.Agents provide verification scripts, and candidates are admitted only when checks pass.Source: 3) Checking and Deduplicating Candidates · p. 5

E. Phase 3: Turning Candidates into Instances

1) Constructing and Replaying Instances:

  • Construction proceeds stage by stage only after required checks pass, with earlier stages rerun to catch regressions; persistent failures or unmet criteria can cause a candidate to be dropped.Researchers set attempt, target, and time limits; passing instances are added for packaging.Source: 1) Constructing and Replaying Instances · pp. 5–6

2) Coordinating Construction Agents:

BenchMeUp coordinates construction agents by sharing lessons from replay feedback and refreshing agents in waves. It packages completed instances behind a common command-line interface and verifies the packaged commands in fresh containers.

2) Coordinating Construction Agents:

  • Construction agents work on separate candidates and periodically receive shared lessons summarizing problems, attempted solutions, and replay feedback.Agents are replaced in waves as context grows, with current lessons passed to successors and a grace period for instances near completion.Source: 2) Coordinating Construction Agents · p. 6

F. Phase 4: Packaging the Benchmark

  • Packaging maps each instance’s scripts and files to plan-defined public commands through a manifest, giving users a uniform interface across instances.Replay reports are retained for inspection, and the shared command-line program dispatches requests to instance-specific scripts.Source: F. Phase 4: Packaging the Benchmark · p. 6
  • Packaged commands are rerun in a fresh container, and instances that fail the plan’s required checks are excluded.Source: F. Phase 4: Packaging the Benchmark · p. 6

III. EVALUATION

The evaluation asks whether BenchMeUp can construct benchmarks effectively and efficiently, whether they support end-to-end evaluation, and how their quality compares with a general-purpose agent. It covers tasks with varied languages and construction requirements under specified model and compute budgets.

III. EVALUATION

  • The evaluation poses four questions about construction effectiveness and efficiency, end-to-end support, benchmark characteristics and quality, and comparison with a general-purpose agent.Source: §III · p. 6

1) Tasks:

  • The four selected tasks span different languages and instance types, with requirements including commit inspection, dependency setup, tests, static analysis, and runtime measurements.Tasks were chosen as common software-engineering tasks with existing benchmarks for comparison.Source: 1) Tasks · p. 6

2) Models and Budgets:

  • The experiments use four search agents with six-hour limits and four construction agents with a 48-hour wall-clock budget, with synchronization and staggered starts.Construction uses DeepSeek-V4-Flash by default; Claude Opus 5 handles search, rules, and plan generation.Source: 2) Models and Budgets · p. 7

B. RQ1: Effectiveness and Efficiency of BenchMeUp

BenchMeUp constructs and packages nearly all targeted instances across four tasks, with replay catching environment-dependent failures before acceptance. The packaged benchmarks support evaluation of existing techniques, while construction time and cost remain concentrated in particular phases.

B. RQ1: Effectiveness and Efficiency of BenchMeUp

  • Construction began once 50 candidates were collected and ran concurrently with continued search, stopping at 50 constructed instances per task.This stopping rule applied to the experiments; researchers can let both phases continue.Source: B. RQ1: Effectiveness and Efficiency… · p. 7

1) Effectiveness:

  • 198 instances were constructed and 196 passed packaging checks; construction reached 50 instances in three tasks but only 48 for static-analysis warnings.Construction agents tried 332 of 639 unique candidates and dropped 134, either because candidates failed instance criteria or could not be constructed.Source: 1) Effectiveness · p. 7
  • 108 of 198 constructed instances initially failed replay, most often because builds relied on residual container state or tests failed in replay.BenchMeUp returns failures for correction and accepts instances only after successful replay in a fresh container; replay does not establish that checks distinguish correct from incorrect inputs.Source: 1) Effectiveness · p. 7

2) Time and Cost:

  • $0.79 per constructed instance and 36 minutes were the average LLM cost and time; construction accounted for 80–96% of time.Search accounted for 64% of LLM cost, while construction accounted for 36%.Source: 2) Time and Cost · p. 7

C. RQ2: Existing Techniques on the Constructed Benchmarks

  • 189 of 196 packaged instances (96%) could be used by the evaluated techniques; their solve counts varied across tasks and techniques.RepairAgent produced plausible patches for 37 of 47 usable repair instances with GPT-5-mini and 22 with GPT-5-nano; ReProAgent and mini-swe-agent reproduced 19 and 20 of 46 usable issues.Source: C. RQ2: Existing Techniques on the… · p. 8

D. RQ3: Characteristics and Quality of Benchmarks

The benchmarks extend project and instance coverage, include larger patches and programs than some reference benchmarks, and are mostly valid in a 40-instance inspection. The inspection also identifies unusable packages and cases where the combined correctness check misses special cases.

1) Quantitative Characterization:

  • The program-repair and issue-reproduction benchmarks cover more projects than their references, though those projects generally have fewer GitHub stars.None of the 31 repair projects appears in Defects4J or GitBug-Java; researchers can adjust the minimum-star threshold in search rules.Source: 1) Quantitative Characterization · p. 8
  • Most issue-reproduction instances come from pull requests merged in 2026, which may be less likely to appear in evaluated models’ training data; contamination was not measured.Source: 1) Quantitative Characterization · p. 8
  • The constructed benchmarks contain larger patches than Defects4J and SWT-bench and larger programs than cBench; none of the repair fixes changes only one line.RepairAgent fails more often on instances with larger patches. The static-analysis benchmark also includes developer fixes verified by rerunning SonarQube.Source: 1) Quantitative Characterization · p. 8

2) Qualitative Inspection:

  • The qualitative inspection randomly sampled 40 instances, including five solved and five unsolved by the best-performing technique for each task, and checked them against upstream repositories and specification criteria.The inspection also tested interfaces with recorded and incorrect inputs.Source: 2) Qualitative Inspection · p. 9
  • 37 of 40 inspected instances were valid; three issue-reproduction instances had an incorrect problem description or a reference test unrelated to the reported issue.Minor problems included unrelated formatting changes and feature requests described as issues.Source: 2) Qualitative Inspection · p. 9
  • Seven packaged instances could not be used by evaluated techniques: three for repair, three for issue reproduction, and one for static-analysis warnings.Causes included external dependency changes, such as removed build files or incompatible pip versions.Source: 2) Qualitative Inspection · p. 9
  • In six inspected instances, the combined check command missed special cases, although basic commands handled incorrect inputs correctly in all 40.A patch that skips a test can leave no failing test, allowing the combined check to accept it even when the test command reports that no test ran.Source: 2) Qualitative Inspection · p. 9

E. RQ4: Comparison with a General-Purpose Agent

BenchMeUp produces more instances meeting the full set of source, specification, reproducibility, and evaluation criteria than Codex. Its evaluation also highlights specification interpretation and instance validity as remaining concerns.

E. RQ4: Comparison with a General-Purpose Agent

  • Codex sessions received each task specification and were asked for five instances apiece, with an eight-hour and $2.50 limit per session.Larger requests made Codex stop early or exhaust its context window; no session reached the time or cost limit.Source: E. RQ4: Comparison with a… · p. 9
  • The seven criteria cover source reality, admissibility and uniqueness; completeness and buildability; and functional commands with expected outcomes.The authors check all criteria with task-adapted scripts.Source: E. RQ4: Comparison with a… · p. 9
  • 180 of 198 BenchMeUp instances meet all criteria, compared with 10 of 200 Codex instances.Codex delivers 200 instances in 48 hours, averaging 14 minutes and $0.07 per instance; only 30 of its 91 complete instances build.Source: E. RQ4: Comparison with a… · p. 9
  • BenchMeUp failures cluster around specification filters and expected outcomes, especially issue-reproduction projects below the required 20 merged pull requests.Some issue-reproduction tests also fail to expose the issue by failing before the fix and passing afterward.Source: E. RQ4: Comparison with a… · p. 9

IV. DISCUSSION AND THREATS TO VALIDITY

  • In 108 of 198 instances, a stage that worked in the agent’s container failed in a fresh one, often because builds relied on agent-installed or leftover files.Container snapshots can preserve execution environments, though base images, packages, and repositories may change.Source: §IV · p. 9
  • The four manually written specifications may have been easier for BenchMeUp than specifications from other researchers, and the evaluation covers only GitHub data.It does not establish effectiveness for tasks requiring sources such as non-GitHub issue trackers or app stores.Source: §IV · p. 10
  • With Opus 5, BenchMeUp delivered 50 instances per task at about $10 per instance and about 9 minutes per instance, versus 25 minutes with deepseek-v4-flash.Opus 5 also constructed instances from 40 candidates that deepseek-v4-flash dropped; static-analysis warnings were excluded for budget reasons.Source: §IV · p. 10
  • Replay verifies outcomes required by agent-written checks, but does not establish that those checks are correct or faithfully represent the task.The authors recommend inspecting sample instances before using a benchmark.Source: §IV · p. 10

V. RELATED WORK

Prior work collects benchmarks and automates parts of instance construction, often for particular tasks or sources. BenchMeUp instead derives construction procedures from researcher-written specifications, and the paper reports that its instances substantially more often meet all criteria than those from a general-purpose agent.

V. RELATED WORK

  • Existing benchmarks span code generation, program repair, issue resolution, and other tasks, while many are curated for one task and ecosystem.The paper positions BenchMeUp as automating construction for arbitrary software-engineering tasks rather than proposing one new benchmark.Source: §V · p. 10
  • Repository-collection tools such as PyDriller, BugSwarm, and GitBug-Actions are tied to particular sources and instance types; BenchMeUp takes these from the specification.Its search agents use the specified source and selection criteria.Source: §V · p. 10
  • Related task-construction approaches often use manually designed, task-specific instance definitions, agent roles, and correctness oracles; BenchMeUp derives procedures from researcher-written specifications.The paper also distinguishes its broader software-engineering task coverage from approaches focused on issue resolution or vulnerabilities.Source: §V · p. 10
  • Construction-time correctness conditions and replay do not settle whether an oracle is adequate; the authors leave that judgment to researchers.Prior studies have found solution leakage and weak tests in SWE-bench, and some patches counted as correct fail developers’ tests.Source: §V · p. 10

VI. CONCLUSION

  • 180 of 198 BenchMeUp instances meet all criteria, compared with 10 of 200 from a general-purpose agent.The conclusion presents BenchMeUp as a way to build executable benchmarks from specifications.Source: §VI · p. 11
  • BenchMeUp packages instances behind a common interface, aiming to help researchers evaluate techniques on new tasks, languages, or software versions without manual construction overhead.The paper’s motivation notes that benchmarks support comparisons, reproduction, and measuring research progress.Source: §VI · p. 11

Related papers

Loading 2610.37283v1…