PaperlayerView exact revision on arXiv

Source-linked AI summary

TestSuiteBench: Coding Agents Struggle to Generate Strong Test Suites from Specifications

Kazuki Egashira, Niels Mündler-Sasahara, Mark Niklas Müller, Martin Vechev

arXiv:2610.13065v1cs.SE

TL;DR

Complete test-suite generation from specifications alone remains largely unmeasured, despite its relevance when new software lacks existing tests and reference implementations. TestSuiteBench evaluates this task across 165 applications using requirement-targeted mutants. Even the strongest agent kills only 55% of mutants, despite high test validity; providing the reference executable does not improve mutation scores.

  • Problem

    Existing test-generation benchmarks focus on individual issues, few files, or single scenarios and provide the implementation, leaving complete-suite generation from specifications alone unmeasured.

    Source: §1 · p. 1

  • Method

    TestSuiteBench evaluates specification-only test-suite generation on 165 applications using detailed specifications, reference artifacts, and mutants designed to violate individual requirements.

    Source: §1 · pp. 1–2, §2.2 · p. 4

  • Results

    The strongest agent’s tests kill 55% of mutants, four of six agents remain below 40%, and mutation scores remain unchanged when agents receive the reference executable.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Agents can build software they cannot verify, and generated suites often miss input combinations needed to expose violated requirements.

    Source: §6 · p. 9

    The benchmark’s detailed specifications may be more favorable than those encountered in practice, so its evaluation may overestimate agents’ test-generation performance.

    Source: §5 · p. 9

Abstract

from arXiv · show

Coding agents are increasingly used to create large software projects. While code-generation benchmarks have scaled accordingly to full applications, test-generation benchmarks remain limited to individual functions or bug fixes. In particular, generating complete test suites for complex software from scratch remains unexplored. To address this gap, we introduce TestSuiteBench, a benchmark of 165 open-source applications, each with a structured and detailed specification of intended behavior, a reference implementation, a reference test suite, and on average 74 mutants. Each mutant is generated from a single requirement of the specification and refined to isolate that behavior: 73\% are killed by exactly one reference test and, on average, by only 0.2\% of the remaining tests. We find that even frontier coding agents struggle to generate strong test suites given our specifications, achieving mutation scores below 60\%, despite their suites being largely valid and 88\% of their tests passing on the reference implementation. We observe that generated test suites often lack relevant input combinations and are smaller than reference test suites. Meanwhile, implementations generated for the same specifications pass 82\% of the reference tests. These results show agents can build software they cannot verify, a gap that directly limits fully autonomous software development.

1 INTRODUCTION

TestSuiteBench addresses the lack of benchmarks for generating complete test suites from specifications alone, a capability needed when new software has no existing tests or reference implementation. It evaluates this task on 165 applications and finds that agents’ tests are often valid but weak at detecting requirement violations.

Motivation

  • Test-generation benchmarks remain limited to smaller tasks and provide the implementation, leaving complete-suite generation from specifications unmeasured.The gap is difficult to address because natural-language specifications can be ambiguous, inconsistent, or incomplete, and generic mutations may not violate specific requirements.Source: §1 · p. 1

Contribution

  • TestSuiteBench covers 165 open-source applications and asks agents to generate a complete test suite using only a natural-language specification.Instances include reference implementations, reference test suites, and mutants targeting specification requirements.Source: §1 · pp. 1–2

Benchmark design

  • Mutants target single requirements and are refined to isolate behavior; 73% are killed by exactly one reference test.The benchmark reports validity as the fraction of tests passing on the reference implementation and mutation score as the fraction of mutants killed by valid tests.Source: §1 · p. 2

Evaluation

  • 55% of mutants are killed by the strongest agent’s suite, while four of six agents kill fewer than 40%.Generated tests are 88% valid on average, and providing the reference executable leaves mutation scores unchanged, pointing to test design as the bottleneck.Source: §1 · p. 2

Implications

  • Generated suites are often too small and miss input combinations needed to expose violated requirements.The agents’ ability to implement from the same specifications contrasts with their difficulty generating strong tests.Source: §1 · p. 2

2 CONSTRUCTING TESTSUITEBENCH

TestSuiteBench is built from repository artifacts, with detailed specifications, requirement-targeted mutants, iterative refinement, and manual checks. Its 165 instances support evaluation of generated tests and implementations using validity, mutation, and reference-pass metrics.

2 CONSTRUCTING TESTSUITEBENCH

  • The benchmark construction proceeds from a draft specification to requirement-targeted mutants, then iteratively resolves specification ambiguities and contradictions.The resulting instance aligns test-admissible implementations with those consistent with the specification.Source: §2 · p. 3, Figure 2 · p. 3

2.1 GENERATING A DRAFT SPECIFICATION

  • Specifications are drafted to describe every tested behavior and refined to remove ambiguity, contradiction, and underspecification.The initial test suites are filtered for internet access, excessive resource use, and flaky behavior; this leaves 165 initial instances.Source: §2.1 · pp. 3–4

2.2 MUTANT CONSTRUCTION

  • Mutants are designed to violate specific requirements while remaining otherwise indistinguishable, with only a subset of test functions selected to cover the specification.Candidate mutants are associated with targeted test functions and refined to reduce unrelated kills.Source: §2.2 · p. 4
  • 73% of refined mutants are killed by exactly the targeted reference test, up from 32.4% before refinement.Mutants killed by more than 20% of unrelated test functions are discarded.Source: §2.2 · p. 4

2.3 SPECIFICATION REFINEMENT

  • Specification refinement iterates test and implementation generation, failure attribution, and correction until fewer than 5% of generated test failures are attributed to the specification.The process increases implementation reference-pass rate and test validity, while mutation score remains low.Source: §2.3 · pp. 4–5, Figure 3 · p. 4

2.4 MANUAL QUALITY CONTROL

  • The resulting specifications, mutants, and reference suites are manually checked, with intervention required for 34 instances.Source: §2.4 · p. 5

2.5 TESTSUITEBENCH

  • The benchmark contains 165 instances across Rust, Go, and C/C++, with a median 5k-word specification, 679 reference test functions, and 67 mutants.Reference implementations range from 538 to 1.6M lines of code, with a median of 7k.Source: §2.5 · p. 5

2.6 EVALUATION METRICS

  • Mutation score measures the fraction of mutants killed by generated tests, while validity measures generated tests that do not kill the reference executable.Implementation correctness is measured by reference pass rate, the fraction of reference tests that do not kill the generated implementation.Source: §2.6 · p. 5

3 EXPERIMENTAL EVALUATION

Frontier agents produce largely valid tests and implementations, but their test suites kill few requirement-targeted mutants. Suites tend to be small, and performance also weakens as specifications grow.

3.1 MAIN RESULTS

  • 55.5% is Claude Opus 5’s mutation score M, ahead of GPT 6 Astra at 50.1%; all other agents score below 40%.Opus costs $10.56 per instance and takes 37 min on average; Astra costs $2.18 and takes 12.4 min.Source: Table 2 · p. 6, §3.1 · p. 6
  • Most models achieve validity scores V of 88%-96% and reference-test pass rates R of 86%-93%, while mutation scores M remain 38%-56%.DeepSeek V4 Flash and GPT 5.6 Luna are exceptions to the general pattern of valid tests and implementations.Source: §3.1 · p. 6
  • Cost and mutation score M have a logarithmic relationship; higher reasoning effort raises both, from 31.3% at $0.45 on low to 46.5% at $2.4 on xhigh.All evaluated models except Gemini 3.8 Flash lie on the Pareto frontier.Source: §3.1 · p. 6, Figure 5 · p. 6

3.2 ADDITIONAL ANALYSIS

  • Generated test suites are generally smaller than reference suites; Opus’s gap is 8.9%, while Sol’s is 79.9%, and larger suites tend to accompany higher M.Instances with fewer reference tests also tend to have higher M.Source: §3.2 · p. 6, Figure 6 · p. 7
  • Mutation scores M negatively correlate with specification size across models: Opus ρ = −0.16 and Sol ρ = −0.42.The analysis relates size to the number of specification sections.Source: §3.2 · p. 7, Figure 6 · p. 7
  • Generated tests often cover core functionality but omit input combinations; one Sol test used only one file and missed a mutant that Opus killed by checking two.Opus attempted executable and mutant creation most often, but created mutants in only 33 of 165 instances; such attempts were not necessary for Sol to outperform DeepSeek.Source: §3.2 · pp. 7–8, Figure 7 · p. 7
  • Giving GPT 5.6 Sol the reference executable raises validity V from 94.4% to 98.4%, while mutation score M changes only from 37.7% to 38.1%.The authors conclude that specification ambiguity does not explain the low mutation scores in this analysis.Source: §3.2 · p. 8, Table 4 · p. 8
  • Replacing the benchmark specifications with developer documentation on 30 instances lowers Sol’s mutation score from 25.5% to 4.0%.Sol writes 40% fewer tests, with validity falling from 86.7% to 75.4%; the documentation induces three times as many underspecification failures per test.Source: §3.2 · p. 8

4 RELATED WORK

Prior coding benchmarks focus on implementing projects, while test-generation benchmarks generally limit tasks to specific issues, files, or scenarios and provide an implementation. TestSuiteBench instead evaluates repository-scale suite generation from specifications.

Repository-Level Coding Benchmarks

  • Repository-level coding benchmarks task agents with implementing projects, whereas TestSuiteBench measures test-suite generation from specifications.NL2Repo-Bench supplies detailed specifications and ProgramBench supplies a reference executable, but both ask agents to (re)implement programs.Source: §4 · p. 8

Test Generation Benchmarks

  • Existing test-generation benchmarks target bugs, issues, individual files, or specified scenarios, and generally supply an implementation or executable.The reviewed benchmarks include SWT-Bench, SWE-Mutation, TestGenEval, MultiFileTest, and RESTestBench.Source: §4 · pp. 8–9
  • Prior benchmarks’ golden reference suites per task are orders of magnitude smaller than TestSuiteBench’s, alongside their narrower task scopes and implementation access.Table 6 compares inputs, use of mutation score and validity, and whether evaluations target coding agents or LLMs.Source: §4 · p. 9

5 LIMITATIONS AND FUTURE WORK

The benchmark’s findings are bounded by its sampled mutants, unusually detailed specifications, reference implementation as ground truth, and application scope.

Mutants as a Proxy for Test Strength

  • Mutants sample requirements uniformly, so gaps in untargeted requirements go unpenalized and individual mutants need not reflect real-world fault distributions.The authors audit single-test-killed mutants to reduce the risk that they reward tests tied to exact inputs.Source: §5 · p. 9

Specification Realism

  • Specifications are more detailed than typical developer documentation, a favorable condition that may cause the evaluation to overestimate test-generation performance in practice.On the sampled instances, mutation scores are 25.5% with the benchmark specifications versus 4.0% with developer documentation.Source: §5 · p. 9

Reference Behavior as Ground Truth

  • Specifications, reference tests, and mutants inherit bugs and incidental behavior from the reference implementation, though obvious cases were manually removed.Subtler inherited cases may remain; grounding artifacts in an existing implementation enables construction to scale.Source: §5 · p. 9

Scope

  • TestSuiteBench covers command-line applications in compiled languages; interpreted languages are outside its current scope because the reference executable would expose source code.The authors say the pipeline could be extended to interpreted languages such as Python and TypeScript.Source: §5 · p. 9

6 CONCLUSION

TestSuiteBench measures whether agents can turn specifications into strong test suites across 165 applications. Frontier agents often produce valid tests and implementations that pass reference tests, yet their tests kill only just over half the mutants.

6 CONCLUSION

  • TestSuiteBench covers 165 applications and uses mutants that each violate a single specification requirement.It measures whether agents can turn natural-language specifications into strong test suites.Source: §6 · p. 9
  • Even the strongest agent’s suites kill just over half the mutants, despite being largely valid and accompanying implementations that pass most reference tests.Source: §6 · p. 9
  • Providing the reference executable leaves mutation scores unchanged, pointing to test design rather than specification quality as the bottleneck.Generated suites are often too small and miss input combinations needed to expose violated requirements.Source: §6 · p. 9
  • The authors frame closing the verification gap as a prerequisite for fully autonomous development of new software.TestSuiteBench makes progress toward that gap measurable.Source: §6 · p. 9

REPRODUCIBILITY STATEMENT

The paper describes where benchmark construction and evaluation methods, as well as their prompts, are documented. It states that the benchmark data and reproduction code will be released upon publication.

REPRODUCIBILITY STATEMENT

AI USAGE STATEMENT

The appendix discloses the paper’s use of generative AI and reports correlations between generated-suite size, specification length, and mutation score. It also describes the measures shown in a score-and-cost distribution figure.

AI USAGE STATEMENT

  • Generative AI tools helped construct and refine benchmark artifacts and create or edit figures and text, while authors reviewed the resulting work.The disclosure says AI was not used to develop conceptual frameworks, hypotheses, research methods, experiments, or interpret results.Source: AI USAGE STATEMENT · p. 10

A.1 CORRELATION BETWEEN REFERENCE AND GENERATED TEST SUITE

  • Generated suites are generally smaller than reference suites, with Opus generating a similar number; mutation scores are generally higher when reference suites are smaller and generated suites are closer in size.Source: §A.1 · p. 12

A.2 CORRELATION BETWEEN SPEC LENGTH AND MUTATION SCORE

  • Across agents, specification length has a weak negative correlation with mutation score.The passage interprets this as longer specifications with more specified requirements making strong test-suite generation more challenging.Source: §A.2 · p. 12

A.3 SCORE AND COST DISTRIBUTION

  • Figure 10 presents distributions of code-generation reference pass rate and test-generation validity and mutation scores, alongside cost, time, and step count by setting.Source: §A.3 · p. 12

A.4 TOOL USE OVER TRAJECTORIES

The appendix describes tool-use patterns across agent trajectories. Early steps are dominated by reading and searching, while later steps are mostly writing and executing; writing accounts for 60% of Opus’s actions.

A.4 TOOL USE OVER TRAJECTORIES

  • Figure 11 classifies trajectory actions as read, search, write, execute, or other.Source: §A.4 · p. 12, Figure 11 · p. 13
  • Early trajectory steps are dominated by reading and searching, while most later steps are dominated by writing and executing.The authors say models quickly digest the specification, then write code while executing it to check correctness.Source: §A.4 · p. 12
  • 60% of Opus’s trajectory actions are writing actions.Writing dominates the overall trajectory.Source: §A.4 · p. 13

A.5 MORE ABLATION

The model suites overlap incompletely, so combining them adds only modest mutation coverage; the logic-level instruction makes mutants less tied to literal values while preserving the single-test-kill criterion.

A.5 MORE ABLATION

  • Combining three models raises killed mutants from 6,584 (54%) for Opus alone to 7,134 (59%), while 4,996 (41%) remain unkilled.Only 3,080 of 12,130 mutants (25%) are killed by all three models, and each weaker model kills over 200 mutants missed by the others.Source: §A.5 · p. 14
  • All four mutants satisfy the single-test-kill criterion, but without the logic-level instruction they hard-code test literals and behave like the reference when those literals change.The instructed mutants instead break behavior across renamed domains or changed port values.Source: §A.5 · p. 14

A.6 MORE ERROR ANALYSIS

The error analysis shows that tests can miss requirement-specific behavior even when they exercise related features: Opus catches two mutants, while both suites miss two others.

A.6 MORE ERROR ANALYSIS

  • For Figure 14b, Opus alone kills a mutant that aborts after parsing in all-bold mode: its -B test detects the failure, while Sol’s -a -b -r test selects occasional, not all-bold, characters.Figure 14a is the full version of Figure 7.Source: §A.6 · p. 14
  • Both suites miss the Figure 14c and 14d mutants: one ignores JavaScript `await` with no path, and the other emits Zsh completion when Elvish is requested.Opus tests await and nested JavaScript in different contexts, while its Elvish check does not validate script identity; Sol does not test await and does not validate generated Elvish output.Source: §A.6 · p. 14

B BENCHMARK CONSTRUCTION DETAIL

The benchmark construction details describe filtering reference tests, optimally assigning mutation targets, and correcting specifications after manual quality checks.

B.1 DETAILS ON FILTERING

  • Tests are filtered if they exceed resource limits, require internet access, or are flaky during the pipeline.The resource cap is 2 CPU cores, 60 GB RAM, and 1 hour per test-environment run.Source: §B.1 · p. 15

B.2 OPTIMAL SELECTION OF MUTANT TARGETS

  • The target-selection graph links specification sections to tests that check their behavior, then uses min-cost max-flow to favor section-specific tests.Each section receives at most six target tests, while each test is assigned to at most one mutant; test identifiers break remaining ties.Source: §B.2 · pp. 15, 17
  • For `run` and `scan`, omitted paths mean the current directory and multiple paths are accepted; the example mutant omits `b.js` from the reference output.Source: §B.2 · p. 16

B.3 FINDINGS IN MANUAL QUALITY CHECK

  • Manual checks found 20 specifications with identifying error locations and 14 describing behaviors tied to skipped tests, prompting specification edits and test removal.The locations were obfuscated, and behaviors related to skipped tests were dropped from the specifications.Source: §B.3 · p. 17

C EXPERIMENTAL SETUP

This section describes the experimental setup for reimplementing programs and generating test suites, including task prompts, test-failure and mutant classification procedures, and benchmark instance inventory. It also defines how specifications and focused mutants are prepared and evaluated using behavioral metrics.

C EXPERIMENTAL SETUP

  • Agents use default reasoning effort and vendor-recommended harnesses, with 2 CPUs, 256 GB RAM, and a six-hour limit per generation instance.The evaluated models are GPT-5.6 Sol, GPT-5.6 Luna, GPT-6 Astra, Claude Opus 5, Gemini-3.8 Flash, and DeepSeek V4 Flash.Source: C EXPERIMENTAL SETUP · p. 17
  • Code-generation agents build an executable from build.sh, while test-generation agents produce JUnit XML; a pytest hint led all observed submissions to use pytest.Generation uses a shared, preinstalled-library container with internet access disabled except for the API provider endpoint.Source: C EXPERIMENTAL SETUP · p. 17
  • Evaluation measures test suites by Mutation Score and Validity Rate, and implementations by Reference Pass Rate.Only tests that pass on the reference implementation count toward Mutation Score; the test suite is run on the reference three times to avoid trivial test generation.Source: C EXPERIMENTAL SETUP · p. 17

D.1 CONSTRUCTION STEPS

  • Update specification/ so an executable can pass all tests in test-branches/ without outside information; abstractly describe tested behaviors in 10–50 sections, rather than enumerating test functions.Treat functions listed in skipped_tests.json as absent: they are excluded from evaluation and must not influence the specification.Source: §D.1 · p. 18
  • Mutants are assigned to specific specification rules and tests, and should violate the behavior narrowly so only the assigned test fails.The prompt requires plausible behavior-level mutations rather than test-environment detection or changes keyed to copied test inputs.Source: §D.1 · pp. 18–19
  • Failure attribution distinguishes executable errors from specification underspecification or contradiction, while unclear evidence is classified as unknown.Contradiction requires a quoted specification statement that conflicts with the test, confirmed by running the reconstructed invocation on the reference executable.Source: §D.1 · p. 20
  • Generated-test failures were classified as test, spec_underspecification, spec_contradiction, error, or unknown by examining the specification and test and replaying the invocation on the ground-truth executable.Failures attributable to the test, missing or misleading specification guidance, or the execution environment receive distinct labels.Source: §D.1 · p. 21
  • Specification faults were assigned only when clearly supported; unsupported expectations, incidental-value assertions, timing-sensitive tests, and behavior contradicted by ground truth favored a test-fault classification.For contradiction and test classifications, the procedure required a verbatim specification sentence; asynchronous failures with flaky siblings were treated as races unless replay failed deterministically.Source: §D.1 · pp. 21–22
  • Surviving mutants were classified as specification underspecification, contradiction, test, or unknown, distinguishing unclear guidance from behavior already stated clearly but missed by the generated suite.The procedure prohibited changing, rebuilding, removing, strengthening, or narrowing mutants.Source: §D.1 · p. 23

D.2 EVALUATION

  • Reimplementation agents had to build an original codebase from the specification using available dependencies, with an executable compile script and a self-contained executable for a fresh checkout.They were instructed not to retrieve the original project or fetch dependencies from the network.Source: §D.2 · p. 23
  • Agents were asked to create comprehensive suites that kill mutants violating the specification while passing any implementation that follows it, using focused tests for documented requirements.The prompt emphasizes numbered or sectioned requirements and at least one meaningful test per documented behavior.Source: §D.2 · p. 24
  • Evaluation instructions require descriptive parameterized-test IDs, prohibit large strings or object representations as IDs, and place helper files and generated assets under /workspace/eval.Source: §D.2 · p. 24

E LIST OF INSTANCES

Loading 2610.13065v1…