PaperlayerView exact revision on arXiv

Source-linked AI summary

Benchmarking LLM Agents for Behaviorally Adequate Test Generation from Software Requirements

Binhang Qi, Yiming Liu, Yun Lin, Yangjie Zhou, Hailong Sun, Jin Song Dong

arXiv:2610.36584v1cs.SEcs.AI

TL;DR

It is unclear whether agents can generate tests that validate required behavior when its implementation is withheld, despite the need for requirement-derived validation of generated code. Req2Test-Bench evaluates this task using execution-validated BDD artifacts and finds that passing tests often fall short of behavioral adequacy.

  • Problem

    Existing test-generation benchmarks mostly focus on white-box testing, while requirement–test studies often evaluate semantic alignment rather than whether tests validate behavior with its implementation withheld.

    Source: §1 · pp. 1–2

  • Method

    Req2Test-Bench recovers and execution-validates 5,086 developer-authored BDD requirement–test instances and evaluates eight agent–model–reasoning configurations against the developers’ tests.

    Source: §1 · pp. 2–4

  • Results

    82.8% of instances pass with Codex CLI and DeepSeek-V4.1-Flash at high reasoning effort, but only 33.8% are kill-complete.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Ambiguous setup and tacit expected outcomes account for many adequacy gaps, while selecting among generations offers harnesses potential headroom if they can recognize adequate tests.

    Source: §1 · p. 3, §5.5 · p. 18

    Kill-completeness is measured against one developer-written test, so a test may still satisfy a narrower defensible reading of the requirement.

    Source: §6 · p. 18

Abstract

from arXiv · show

As large language model agents increasingly compile natural-language requirements into repository-level code, requirement-to-test generation grows in importance to provide validation constraints (or harness) to guide code generation. Yet it remains unclear whether current agents can generate tests that validate the required behavior when the implementation of that behavior is withheld. Existing test-generation benchmarks mostly focus on white-box testing, whereas the requirement--test-related study usually requires evaluation on semantic alignment between the given documents and the generated tests. We introduce Req2Test-Bench, a benchmark containing 5,086 execution-validated instances recovered from developer-authored behavior-driven development (BDD) artifacts in 8 open-source projects, spanning 5 programming languages and 4 BDD frameworks. Each instance pairs a natural-language requirement in Gherkin with its developer-written Test Implementation. Given the requirement, target interfaces of the required behavior, an agent must generate a complete, potentially multi-file Test Implementation. Evaluation of 8 agent--model--reasoning configurations reveals a substantial gap between execution success and behavioral adequacy, which is measured strictly as kill-completeness: a kill-complete test kills every mutant that the ground-truth test kills. The best-performing configuration, Codex with DeepSeek-V4.1-Flash, produces passing tests for 82.8% of instances, yet only 33.8% are behaviorally adequate. We find that (1) scaling up agents yields sharply diminishing returns in behavioral adequacy relative to generation cost; (2) under-specified requirements cause the information gap, including ambiguous setup and implicit expected outcomes; and (3) requirement-grounded harnesses that steer agent non-determinism offer a promising direction for improvement.

1 Introduction

The introduction asks whether agents can generate complete tests that check required behavior when its implementation is withheld. Req2Test-Bench evaluates this task using developer-authored BDD artifacts and finds a substantial gap between passing and behavioral adequacy.

1 Introduction

  • The study addresses limited evidence on requirement-driven testing: existing benchmarks mostly emphasize white-box testing, while related studies often assess document–test semantic alignment.The target behavior is withheld from the agent, making it unclear whether generated tests validate the required behavior.Source: §1 · pp. 1–2
  • The task asks agents to create complete, potentially multi-file Test Implementations from requirements while withholding the implementation of the target behavior.The benchmark pairs an enclosing Feature with a target Scenario and exposes designated target interfaces and project context through target-hidden or all-hidden views.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Req2Test-Bench contains 5,086 execution-validated requirement–Test Implementation instances from 8 projects, 5 languages, and 4 BDD frameworks.The benchmark recovers developer-written implementations for individual Scenarios and validates each mapping by execution.Source: §1 · pp. 3–4
  • Eight agent–model–reasoning configurations are evaluated with KillComplete@1, which requires a passing test to kill every mutant killed by the ground-truth test.The benchmark also records execution outcomes and compares generated tests with the ground truth on production-API calls, coverage, and mutant kills.Source: Figure 1 · p. 2, §1 · p. 3
  • 82.8% of instances pass with the best-passing configuration, Codex CLI with DeepSeek-V4.1-Flash, but only 33.8% are kill-complete.The introduction also reports sharply diminishing adequacy returns as generation cost increases and identifies ambiguous setup and tacit outcomes as common gaps.Source: §1 · pp. 3–4
  • The authors identify harnesses that exploit LLM-agent generation non-determinism and discriminate non-trivial from trivial differences between tests as a promising direction for future research.Source: §1 · p. 4

2 Related Work

Related work spans requirement-based testing, BDD resources, and test generation from code, defects, or observed applications. Req2Test-Bench’s task unit is a complete Test Implementation for one concrete Scenario.

2 Related Work

  • Existing test-generation benchmarks commonly define objectives through focal code, code changes, defects, or observed applications, unlike Req2Test-Bench’s requirement-defined task.Methods2Test, TestBench, TestGenEval, SWT-Bench, and AutoE2E are examples of those differing task boundaries.Source: §2 · p. 4
  • Requirement–test resources differ in provenance and mapping granularity; Req2Test-Bench recovers developer-authored Scenario artifacts and complete ground-truth tests through native BDD bindings.The related-work discussion contrasts these with resources using generated or adapted artifacts, file-level records, or no individual criterion-to-test mapping.Source: §2 · p. 4, Figure 2 · p. 5
  • AutoGlue is the closest task-level neighbor, generating one glue method for a Gherkin Step, whereas Req2Test-Bench evaluates a complete, potentially multi-file implementation for one Scenario.Other cited systems generate specialized UI artifacts or guide testing around prescribed architectures or focal code.Source: §2 · p. 4, Figure 2 · p. 5

3 Req2Test-Bench

Req2Test-Bench formalizes requirement-to-test generation around BDD requirements, complete project-local test implementations, and execution-validated benchmark instances. Its evaluation separates successful execution from behavioral comparison against developer-written tests and reports generation efficiency.

3.1 Requirement-to-Test Artifact Meta-Model

  • A Test Implementation is the project-local test-side closure of matched Step Definitions and required helpers needed to bind, execute, and check a target Scenario.The BDD meta-model links Features to Scenarios, Scenarios to ordered Steps, and Steps to selected Step Definitions; test implementations may overlap.Source: §3.1 · pp. 5–6

3.2 Dataset Construction

  • Dataset construction recovers each Scenario’s test-side closure from runtime Step bindings and validates it in paired workspaces across three runs.The positive workspace must pass the target Scenario with the closure retained, while the negative workspace must initialize but not pass it when the closure is withheld.Source: §3.2 · p. 7
  • Target interfaces are public production methods directly called by action and check Steps, supplied as declarations without arguments, invocation order, assertions, or test code.This observed testing boundary is not an exhaustive semantic localization; 234 instances with no direct production-method calls were retained.Source: §3.2 · p. 7
  • The benchmark contains execution-validated requirement–test pairs recovered from developer-authored BDD artifacts across eight projects.Table 3 gives a project-level breakdown of instances, per-instance requirement and test measurements, and mutation data.Source: §3.2 · p. 9, Table 3 · p. 9

3.3.1 Stage-based execution outcomes.

  • Stage-based outcomes distinguish generation, build, binding, execution, and assertion failures from a passing Scenario, with infrastructure failures excluded from those categories.Execution success and failure rates use the attempted instances with recorded outcomes after documented infrastructure exclusions.Source: §3.3.1 · pp. 9–10

3.3.2 Ground-truth-based behavioral comparison.

  • Behavioral evaluation compares generated tests with developer-written tests across API-call F1, coverage Jaccard, and mutant-kill recall, while KillComplete@1 requires both passing and killing every ground-truth-killed mutant.A passing test that misses any evaluated ground-truth kill is kill-incomplete, though it may satisfy the requirement under a defensible interpretation.Source: §3.3.2 · p. 10

3.3.3 Efficiency.

  • Efficiency reporting includes generation time, LLM calls, token counts, and recorded monetary cost, including cost per execution pass and per kill-complete pass.Post-generation test execution and offline mutation-analysis costs are reported separately.Source: §3.3.3 · p. 11

4 Experimental Design

The experimental design evaluates requirement-driven test generation through research questions, multiple agent–model configurations, and controlled repository-level generation and evaluation procedures.

4.1 Research Questions

  • The study asks about effectiveness, efficiency trade-offs, associated requirement/test/repository factors, and failure modes across Req2Test tasks.Source: §4.1 · p. 11

4.2 Evaluated Models and Agents

  • Eight configurations combine four LLM backbones—DeepSeek-V4.1-Flash [8], GPT-6 Luna and GPT-6 Sol, and Claude Sonnet 5.5 —with four coding agents and disabled or high reasoning effort.Source: §4.2 · p. 11
  • The overall-effectiveness table reports means across eight projects and counts non-passing tests as zero in similarity columns.Source: Table 4 · p. 12

4.3 Implementation Details

  • Each generation receives the requirement, target interfaces, and a repository view in a sandbox where hidden production bodies prevent exercising target behavior or receiving execution feedback.Source: §4.3 · p. 12
  • Analysis covers 2,051 instances from 780 Features and uses project-clustered feature-file bootstrap confidence intervals plus paired exact McNemar or sign-flip tests.Source: §4.3 · p. 12

5 Results

Across five research questions, agents often pass tests that are not behaviorally adequate, and higher generation cost does not reliably buy better kill-completeness. The study traces shortfalls to missing or tacit requirement information and explores requirement-grounded harnesses as a way to improve generation.

5.1 RQ1: Effectiveness

  • 82.8% of target-hidden instances pass with Codex/DS, but only 33.8% are kill-complete.Across projects and views, only 6.7–66.2% of passing tests are kill-complete.Source: §5.1 · pp. 12–13, Figure 3 · p. 13
  • 77.9% of Codex/DS kill-incomplete tests execute a mutant the developers’ test kills and still pass.Figure 4 illustrates a test that constructs the object itself and kills only 10 of the 110 mutants killed by the ground-truth test.Source: §5.1 · p. 13, Figure 4 · p. 14

5.2 RQ2: Efficiency

  • Along the target-hidden cost frontier, 4.0 times the cost buys 6.1 pp of KillComplete@1; further spending buys no measurable gain.The frontier runs from Codex/Luna to Claude Code/DS; Codex/DS costs 1.5 times as much as Claude Code/DS without a measurable KillComplete@1 difference.Source: §5.2 · pp. 14–15, Figure 5 · p. 15

5.3 RQ3: Associated Factors

  • All-hidden lowers Pass@1 by 2.5–9.1 pp but KillComplete@1 by only 0.1–3.7 pp; longer Scenarios and larger tests are harder on both.From the shortest to longest Scenarios, Pass@1 falls from 82.8% to 70.8%, and KillComplete@1 from 45.8% to 22.7%.Source: §5.3 · pp. 15–16, Figure 6 · p. 16

5.4 RQ4: Failure Modes

  • 90–99% of judged kill-incomplete tests reflect information gaps, including ambiguous setup and tacit expected outcomes.Failing generations mostly deliver no test or misuse project interfaces; in the sampled tests, ambiguity accounts for over half and tacit outcomes for up to 28%.Source: §5.4 · pp. 16–17

5.5 Toward Requirement-Grounded Harnesses

  • Requirement-grounded harness interventions target setup, tacit outcomes, interface misuse, and generation non-determinism.Grounding prompts raised KillComplete@1 by 6.1, 15.0, and 8.0 pp for three Codex configurations; selecting the best of three generations was 8.6–10.2 pp above a single generation.Source: §5.5 · pp. 17–18

6 Threats to Validity

The benchmark’s validity is bounded by choices in its adequacy measure, implementation visibility, project selection, and test-suite context. The authors report that the passing-versus-kill-complete gap persists across several checks and project settings.

Internal validity

  • Up to 24% of pass verdicts and 17% of kill-complete verdicts flipped across two regeneration runs, while Pass@1 and KillComplete@1 changed by at most 4.5 pp.The full benchmark was not regenerated repeatedly for every agent because of time budgets.Source: §6 · p. 18
  • The target-hidden view relies on an LLM-drafted, author-accepted map that may hide slightly more or less than the code realizing the target behavior.The all-hidden view does not depend on that map and shows the same gap between passing and kill-complete tests.Source: §6 · p. 18

External validity

  • Eight purposively selected projects span five languages, four BDD frameworks, six domains, and three test levels, with the reported gap appearing in every project.Because Gherkin is more structured than most informal requirements, the authors expect the information gap to be larger for informal requirements.Source: §6 · p. 18

Construct validity

  • KillComplete@1 measures adequacy against one developer-written test, so a kill-incomplete test may still fit a narrower reading of the requirement.The gap also persists when only check-dependent kills count; most kill-incomplete tests run a faulty line and still pass.Source: §6 · p. 18
  • Mutants are artificial faults, but their detection correlates with real-fault detection, and equivalent mutants do not lower scores because only ground-truth-killed mutants count.Both benchmark views remove tests of all other Scenarios, so results concern writing tests without those tests; extending an existing suite may expose them.Source: §6 · p. 18

7 Conclusion

Req2Test-Bench evaluates requirement-to-test generation using execution-validated instances and developer-written tests. The results show that passing tests often fall short of reproducing developers’ fault detection, especially when requirements leave setup or expected outcomes underspecified.

7 Conclusion

  • 82.8% of instances pass for the best-passing configuration, but only 33.8% are kill-complete.The benchmark contains 5,086 execution-validated requirement-to-test instances from eight real projects and evaluates eight agent–model–reasoning configurations.Source: §7 · p. 18
  • Kill-incomplete tests most often fall short when requirements leave setup ambiguous or expected outcomes tacit.Agents may bypass project fixtures and entry points or check less than the developers’ tests.Source: §7 · p. 18
  • The gap persists with costlier agent configurations, motivating requirement-grounded harnesses for guiding requirement understanding, test validation, and project-specific knowledge acquisition.Source: §7 · p. 18

Related papers

Loading 2610.36584v1…