PaperlayerView exact revision on arXiv

Source-linked AI summary

How effective are traditional test criteria at detecting bugs in large language models generated code?

Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis

arXiv:2609.09315v1cs.SE

TL;DR

The paper asks whether traditional test adequacy criteria remain effective for detecting faults in LLM-generated code, where both code and tests are automatically produced. It evaluates statement coverage, branch coverage, and mutation testing across LLM-generated programs and tests, finding low triggering and very low detection, with limited improvement from prompt-aware oracles. The results point to a continuing need for human validation of test assertions and new adequacy criteria.

  • Problem

    Whether traditional adequacy criteria remain effective for detecting faults in LLM-generated code is unresolved, despite their extensive evaluation on human-written programs, faults, and tests.

  • Method

    The study evaluates statement coverage, branch coverage, and mutation testing in end-to-end workflows using LLM-generated code and tests, including 6,066 challenging faults and sampled test suites.

  • Results

    Fault triggering averaged 32.9% for statement coverage, 38.9% for branch coverage, and 32.2% for mutation testing, while fault detection was approximately 1–2%; prompt-aware oracles improved detection by around 6.6% on average.

  • Takeaways & Limitations

    Traditional coverage- and mutation-based criteria provide limited guidance for fault detection in LLM-generated code, and human oversight remains important for validating test assertions.

Abstract

from arXiv · show

Test adequacy criteria are widely used to evaluate and guide software testing. Although prior research has extensively examined these criteria using human-written programs, faults, and tests, the increasing adoption of Large Language Models (LLMs) for code generation raises important questions about their effectiveness in detecting LLM-induced faults. To investigate this, we conduct an empirical study involving 5 LLMs and 4 benchmarks, simulating end-to-end workflows in which both code and tests are automatically generated. We collect 6,000+ faulty program instances and evaluate the effectiveness and efficiency of 3 widely used adequacy criteria: statement coverage, branch coverage, and mutation testing. Our findings reveal several key insights. First, most faults introduced by LLMs are relatively trivial to catch. Second, the challenging faults are difficult to trigger using either traditional coverage-based or mutation-based criteria. Third, actual fault detection rates remain extremely low, often near zero, because test oracles fail to capture faulty behavior triggered by the generated test prefixes, exposing a critical limitation of automated test generation. Fourth, prompt-aware oracles can improve fault detection, but their overall effectiveness remains limited, highlighting the need for users to manually reason about test assertions. We further observe that mutation testing only marginally outperforms traditional coverage criteria in both triggering and detecting faults, raising questions about whether its significantly higher application cost is justified in this context.

I. INTRODUCTION

Traditional adequacy criteria may not transfer well from human-written to LLM-generated code because LLM faults arise from different sources and are difficult to trigger or detect. The study evaluates these criteria through a controlled, end-to-end experiment and finds weak guidance for strong fault detection.

  • Motivation: Traditional criteria assume coherent code, manageable complexity, and faults near conditional boundaries, assumptions that may not hold for LLM-generated programs.LLM faults can instead stem from underspecified prompts, ambiguous descriptions, or misalignment with intended semantics.
  • Findings: Approximately 99% of examined faults were missed by traditional adequacy criteria, indicating limited power for LLM-generated code.The study reports that traditional criteria fail to trigger and detect almost all faults examined.
  • Study design: The study augments four coding benchmarks with LLM-generated tests and collects 73,785 valid faulty implementations before retaining 6,066 challenging faults.The retained faults are selected to reduce bias from trivial-to-detect faults and task imbalance.
  • Study design: For statement, branch, and mutation criteria, the experiment samples 100 minimal test suites per criterion and evaluates fault triggering separately from fault detection.Fault triggering concerns exercising a faulty path, whereas fault detection concerns whether the oracle identifies the fault.
  • Findings: Average fault-triggering rates were 32.9% for statement coverage, 38.9% for branch coverage, and 32.2% for mutation testing, while detection remained approximately 1–2%.Prompt-aware oracle generation improved detection by around 6.6% on average, but effectiveness remained limited and human validation of assertions was still needed.
  • Implications: LLM-generated faults do not appear to couple with faults produced by traditional fault-injection techniques, suggesting that LLM mistakes differ from human mistakes.The authors argue that testing methodologies may need adaptation for LLM-based and agent-based code generators.

A. Test Adequacy Criteria

Test adequacy criteria quantify whether test suites exercise programs sufficiently, with statement coverage, branch coverage, and mutation testing representing common structural and fault-based approaches. Prior evidence largely comes from developer-written programs and artificial faults, motivating reassessment for LLM-generated code.

  • Foundations: Test adequacy criteria measure whether a test suite is sufficiently thorough, with reliability and validity as foundational requirements.The adequacy concept originated in work asking what constitutes an adequate test.
  • Criteria: Common criteria include statement, branch, and path coverage, specification-based coverage, and mutation testing.This study focuses on statement coverage, branch coverage, and mutation testing as widely adopted criteria.
  • Criteria: Statement coverage requires executing every executable statement but does not guarantee that relevant control-flow paths and input conditions are explored.It is therefore generally considered a relatively weak adequacy criterion.
  • Criteria: Branch coverage requires exercising every control-flow branch, including both outcomes of each decision.It is presented as a stronger structural alternative to statement coverage.
  • Criteria: Mutation testing injects syntactic changes to create mutants and measures the proportion killed by tests through mutation score.Mutation coverage seeks a minimal suite achieving a 100% mutation score and has been reported to subsume structural criteria.
  • Prior evidence: Earlier adequacy studies primarily used developer-written programs and either real faults or artificially injected faults, often under the Clean Program Assumption.Later work challenged that assumption and reported stronger fault detection with strong mutation, but it predates LLM code generation.

C. LLM-generated Code and Test

LLM-generated code and tests create an open question about whether traditional adequacy criteria still predict fault detection. This study addresses that gap by evaluating these criteria in workflows where both code and tests are generated by LLMs.

  • Context: LLMs increasingly generate code, tests, repairs, bug findings, and other software artifacts, while their outputs can be affected by nondeterminism and prompt quality.These uses motivate testing methods aligned with LLM-driven development.
  • Context: Automated test-generation systems use statement, branch, and mutation coverage either to evaluate test suites or to guide generation objectives.Examples include statement-focused CoverUp and branch-focused TestSpark and YATE.
  • Research gap: Prior work had not systematically studied the relationship between traditional adequacy criteria and fault detection when both code and tests are LLM-generated.This study addresses that gap through a large-scale empirical analysis.
  • Contribution: The study is described as the first large-scale evaluation of adequacy criteria and fault detection on LLM-generated code from correct and under-specified prompts.This extends earlier work that examined developer-written code or only parts of the code-and-test generation workflow.
  • Research questions: The study asks how statement, branch, and mutation coverage relate to fault triggering and detection in LLM-generated code.The distinction helps assess whether tests exercise faults but fail to capture them with correct assertions.
  • Research questions: LLM-generated tests can trigger faulty behavior yet show very low fault detection because their oracles are often incorrect, making manual inspection an important testing cost.The study therefore compares criteria by the testing effort required for fault triggering or detection.
  • Research questions: The study also evaluates whether specification-guided oracle generation can improve fault detection when the implementation is hidden.This targets the tendency of LLMs to focus on implementation rather than specification when producing oracles.

IV. EXPERIMENTAL SETUP

The study distinguishes fault triggering from fault detection to evaluate whether tests expose behavioral deviations and whether their oracles recognize them. Detection can lag triggering when oracle quality is insufficient.

  • Fault Triggering and Fault Detection: Fault triggering occurs when a faulty program produces an output different from the correct program.
  • Fault Triggering and Fault Detection: A test suite triggers a fault when at least one test in the suite triggers it.
  • Fault Triggering and Fault Detection: Fault detection requires both a triggered behavioral discrepancy and an oracle that flags the output as incorrect.
  • Fault Triggering and Fault Detection: A test suite detects a fault when at least one test in the suite detects it.
  • Fault Triggering and Fault Detection: Because weak or missing oracles can miss triggered faults, fault detection rates are typically no greater than fault triggering rates.

2) Fault Difficulty:

Fault difficulty measures how rarely a test suite triggers a fault. The scale distinguishes trivial faults from hard faults based on the proportion of triggering tests.

  • Fault Difficulty: Fault difficulty is the complement of the proportion of tests in suite T that trigger fault f.
  • Fault Difficulty: Difficulty ranges from 0 for trivial faults triggered by every test to 1 for hard faults triggered by no tests.

3) Mutation Score:

Mutation score quantifies how many non-equivalent mutants a test suite kills. The study’s mutation-based evaluation is situated within a benchmark and model setup spanning diverse code-generation tasks.

  • Mutation Score: A mutant is killed when at least one test produces an output differing from the original program’s output.
  • Mutation Score: Mutation score is the ratio of killed mutants to all non-equivalent mutants generated.
  • Study Context: The study uses four Python coding benchmarks containing tasks, reference implementations, and tests for evaluating generated solutions.
  • Study Context: The benchmarks include HumanEval+, MBPP, BigCodeBench, and NaturalCodeBench, covering natural-language coding, entry-level problems, complex instructions, and varied application domains.
  • Study Context: Five models provide diversity across commercial and open-source families: GPT-5-mini, GPT-4.1-mini, DeepSeek-V4-Flash, Claude Haiku 4.5, and Llama 3.3 Instruct.

C. Non-Trivial Faults Collection

The fault-collection process varies prompts and generations, augments benchmark tests, and compares generated implementations with references to identify faulty behavior. Faults are then assessed for difficulty and non-trivial faults are retained for evaluation.

  • Code Generation: Prompt variations inject ambiguous, incomplete, and under-specified defects to increase the diversity of faulty implementations.
  • Code Generation: 233,300 implementations were generated from benchmark size, 10 generations, two prompt variants, and five models.
  • Test Augmentation: The generated implementation and reference solution are run on the same tests, retaining cases whose execution behavior differs from the reference.
  • Test Augmentation: LLM-generated differential tests compare outputs from generated and reference implementations to expose subtle incorrect behavior.
  • Fault Selection: The fault-difficulty histogram ranges from easy faults at difficulty = 0 to difficult faults at difficulty ∼1.
  • Fault Selection: The collected non-trivial faults are reported per benchmark and model in Table II.

3) Fault Selection:

The study filters generated implementations to retain challenging LLM-induced faults and constructs test pools for subsequent adequacy analysis, while recognizing oracle-generation bias as a potential threat.

  • Fault filtering: The study focuses on non-trivial faults because faults exposed by many tests provide little information about adequacy criteria.The authors prioritize subtle faults requiring stronger test suites.
  • Fault filtering: Fault selection discards implementations with difficulty scores below 0.75, retaining faults detected by fewer than 25% of available tests.For tasks with multiple faulty implementations, only the implementation with the highest difficulty score is retained.
  • Test-pool construction: LLM-Plain generates the test pool without coverage feedback, providing tests from which criterion-specific suites can later be sampled.The approach generated 4,872 tests for HumanEval before the passage ends.
  • Oracle preparation: The evaluation uses specification-guided oracle generation to reduce bias toward the faulty implementation’s observed behavior.The procedure starts from tests that distinguish faulty and ground-truth implementations and uses benchmark natural-language specifications.

V. EXPERIMENTAL PROTOCOL

The experiment simulates selecting tests to maximize statement, branch, or mutation coverage, then measures fault triggering, detection, and effort-dependent effectiveness across repeated samples.

  • Suite sampling: For each criterion, the study repeatedly samples minimal suites by retaining tests that increase the target coverage until the test-pool coverage is matched.The randomized selection procedure is repeated 100 times per fault and criterion.
  • Outcome measures: The evaluation reports fault triggering rate and fault detection rate for the sampled suites across benchmarks and criteria.The rates are averaged over the repeated sampled suites.
  • Cost analysis: Cost effectiveness is measured by the number of tests executed while tracking how fault triggering and detection change as effort increases from 0 to 100%.The same simulation is reused for this comparison.
  • Oracle analysis: The oracle experiment compares base suites with corrected suites whose assertions are regenerated from distinguishing inputs and natural-language specifications.The comparison records changes in both triggered and detected faults.

A. RQ1 : To what extent do statement, branch, and mutation

Across benchmarks and models, the three adequacy criteria have similar and limited effects: they moderately trigger faults but rarely detect them, while coverage offers weak prioritization over random sampling.

  • Criterion comparison: No adequacy criterion consistently outperforms the others across benchmark-model combinations.Mutation performs best on BigCodeBench, branch coverage on MBPP, and statement coverage on NaturalCodeBench in the reported comparisons.
  • Adequacy limits: High statement, branch, or mutation coverage does not guarantee triggering or detecting LLM-generated faults.This remains true even for faults that fail up to 25% of augmented test suites.
  • Triggering versus detection: LLM-generated tests trigger 25%–78% of faults but detect only 0%–10%, indicating that generated inputs distinguish behaviors without reliably asserting intended behavior.The gap reflects a failure of test oracles rather than merely a lack of execution of faulty paths.
  • Cost-efficiency: Under increasing analysis effort, mutation leads FTR on BigCodeBench and NaturalCodeBench, while branch and statement coverage overtake it on MBPP after about 70%.Branch and statement coverage often behave similarly, and in NaturalCodeBench they can match the random baseline.
  • Cost-efficiency: Traditional adequacy criteria provide weak prioritization signals, with little improvement over random sampling when human analysis effort is considered.The result concerns fault triggering under increasing test-analysis effort.

C. RQ3 : To what extent can specification-guided oracle

Specification-guided oracle generation can improve fault detection, especially for mutation-selected suites, but corrected suites still detect too few faults to eliminate manual oracle validation.

  • Oracle improvement: Replacing generated assertions with specification-guided oracles leaves branch- and statement-coverage detection unchanged but improves mutation-based detection across almost all settings.The largest reported improvement rises from 1% to 28.6%, a gain of up to 27%.
  • Oracle mechanism: The oracle-generation experiment changes assertions while preserving the generated test prefixes, explaining why branch and statement detection rates do not change.The improvement appears when mutation-selected suites receive replacement oracles.
  • Remaining limitations: Corrected suites detect at most approximately 30% of faults even when given fault-triggering inputs and explicit behavioral specifications.Thus, improved oracles do not produce reliably fault-revealing tests.
  • Validity considerations: The study mitigates LLM nondeterminism through repeated generation with five models and repeated simulations.Code generation is repeated 10 times and the simulation 100 times.
  • Validity considerations: The generated faults may not fully represent real-world defects because the study filters trivial faults and retains only challenging instances.This limits how directly the findings transfer to naturally occurring defects.
  • Validity considerations: The results are limited to Python functions, five LLMs, and four benchmarks, so they may not generalize to other languages, systems, models, or datasets.This is the study’s stated external-validity boundary.

VIII. CONCLUSION

Across five LLMs and four Python benchmarks, traditional adequacy criteria were weak predictors of whether LLM-generated tests detected faults in LLM-generated code. The main obstacle was often not exercising faulty behavior, but specifying and verifying expected outcomes through reliable test oracles.

  • 6,066 faults across five mainstream LLMs and four Python benchmarks were used to evaluate traditional adequacy criteria in fully LLM-driven development.
  • Statement coverage, branch coverage, and mutation testing consistently provided limited guidance about the fault-detection capability of LLM-generated test suites.
  • LLM-generated tests often exercised faulty behavior and exposed behavioral differences, yet weak, incomplete, or incorrect oracles still failed to detect the faults.
  • Human involvement remains essential for validating test-oracle correctness despite progress in LLM-based test generation.
  • The study does not definitively establish how test effectiveness should be measured, but it provides evidence that new adequacy criteria may be needed for LLM-driven development.
  • Future work should prioritize criteria that predict fault detection, improve oracle quality, address prompt underspecification, and align testing with LLM-driven workflows.
Loading 2609.09315v1…