PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Coverage and Mutation: A Blind-Spot Analysis of LLM-Generated Unit Tests

Xiaoyu Shi, Wenqiang Luo, Boyang Yang, Haoye Tian

arXiv:2610.32548v1cs.SE

TL;DR

Aggregate coverage and mutation scores do not identify which behavioral obligations tests leave unchecked. BlindSpotEval evaluates generated tests across input boundaries, control flow, and assertions, while ObligBench supports a complexity-stratified study that finds recurring blind spots across models and prompting strategies.

  • Problem

    Aggregate coverage and mutation scores summarize test effectiveness but do not identify whether input boundaries, exceptional paths, or behavioral properties remain unchecked.

    Source: §1 · pp. 1–2

  • Method

    BlindSpotEval assesses obligations across input boundaries, control-flow alternatives, and behavioral assertions, alongside execution reliability.

    Source: §1 · p. 2, §8 · p. 19

  • Results

    Null, negative, and empty inputs are covered in only 34.8%, 23.8%, and 38.8% of applicable cases, while catch blocks and default groups score 0.100 and 0.312.

    Source: §8 · p. 19

  • Takeaways & Limitations

    Construct-level adequacy profiles reveal where generated tests leave behavioral obligations unchecked and guide targeted improvements beyond aggregate scores.

    Source: §8 · p. 19

    Results may differ for very small models, domain-specialized code models, unstudied prompt designs, or generation at non-zero temperature.

    Source: §7 · p. 19

Abstract

from arXiv · show

Existing evaluation protocols for unit tests rely on aggregate metrics such as branch coverage and mutation score. These metrics quantify the extent to which a test suite exercises program behavior, but do not identify the behavioral obligations it systematically leaves unchecked. We propose BlindSpotEval, an obligation-based evaluation framework aligned with the Arrange-Act-Assert pattern. The framework comprises three metrics, Boundary-Value Adequacy, Control-Flow Adequacy, and Oracle Strength, complemented by an assessment of execution reliability. We also introduce ObligBench, a complexity-stratified Java benchmark comprising 895 complex and 700 low-complexity functions with source-side obligation annotations. Evaluating 14,320 tests from four LLMs under four prompting strategies reveals systematic blind spots. Incorrect assertions account for 15.6% of test failures, illustrating that coverage alone does not establish oracle correctness. Among executable tests, coverage of null, negative, and empty arguments reaches only 34.8%, 23.8%, and 38.8%, compared with average coverage above 93% for booleans. Catch blocks and default groups have Control-Flow Adequacy scores of only 0.100 and 0.312. Across both dimensions, LLM-generated tests exhibit the happy-path bias observed in human-written tests, concentrating on normal inputs and expected flows while leaving unusual values and exceptional paths unchecked. Oracle strength is high on average but uneven, with weaker assertions for object return property values and exceptions. These blind spots also occur in low-complexity functions, suggesting that code complexity alone does not explain them. BlindSpotEval provides construct-level adequacy profiles that identify under-tested obligation types and can guide targeted improvements beyond those supported by aggregate scores alone.

1 Introduction

Aggregate coverage and mutation scores do not identify which behavioral obligations remain unchecked. BlindSpotEval evaluates generated tests across input boundaries, control flow, and assertions, alongside execution reliability, and the study finds recurring blind spots across LLMs and prompting strategies.

1 Introduction

  • Aggregate branch coverage and mutation scores summarize test behavior but do not reveal whether input boundaries, exceptional paths, or behavioral properties remain unchecked.Tests using ordinary inputs can match the line coverage of tests that also check boundary values when both traverse the same statements.Source: §1 · pp. 1–2
  • BlindSpotEval assesses input boundaries, control-flow alternatives, and behavioral properties as three obligation-level dimensions aligned with Arrange-Act-Assert.The study also examines compilability, runnable tests, and common execution failure modes before adequacy analysis.Source: §1 · p. 2
  • The study also tests whether observed blind spots persist in low-complexity functions, comparing them with complex functions.RQ5 asks whether the gaps are concentrated in complex code or also occur in simpler functions.Source: §1 · pp. 2–3
  • LLM-generated tests show happy-path adequacy blind spots: null, negative, and empty inputs are covered in 34.8%, 23.8%, and 38.8% of applicable cases, while catch blocks and default groups score 0.100 and 0.312 in control-flow adequacy.Source: §1 · p. 3
  • ObligBench supplies 895 complex and 700 low-complexity Java functions with source-side obligation annotations for a complexity-controlled comparison.The benchmark supports evaluation of four LLMs under four prompting strategies, producing 14,320 tests on complex functions.Source: §1 · p. 3

2 Background and Related Work

Prior test-generation work spans search-based, random, symbolic-execution, and LLM-based methods, commonly evaluated with execution, coverage, or aggregate mutation metrics. BlindSpotEval is positioned as a complement that profiles adequacy by input category, control-flow construct, and assertion target.

LLM-based test generation

  • LLM-based test generation includes direct generation, execution-error refinement, coverage-guided generation, and mutation-feedback approaches such as TestPilot [48], CoverAgent, and MuTAP [13].Panta [19] combines static control-flow analysis with dynamic coverage feedback to guide test generation.Source: §2 · pp. 3–4

Test adequacy criteria

  • Structural coverage and mutation testing provide detailed signals, but their aggregate scores do not distinguish adequacy across input categories, construct types, and assertion targets.A replicability study reports that coverage and mutation scores' usefulness depends on the evaluation setting [59].Source: §2 · p. 4
  • Boundary-Value Adequacy operationalizes boundary-value analysis with statically extracted per-parameter obligations, crediting a category only when its value appears at the relevant parameter position.Control-Flow Adequacy targets individual exception handlers, default branches, and throw paths, unlike extreme mutation's focus on pseudo-tested methods [39] [56].Source: §2 · p. 4
  • Oracle Strength mutates return values and exception behavior, excludes incidental kills from unexpected exceptions or crashes, and reports scores by assertion subcategory.Prior oracle work includes checked coverage and evidence for mutation analysis as a real-fault proxy under studied conditions.Source: §2 · p. 4

Benchmarks for automated test generation

  • Existing benchmarks do not jointly provide source-side obligation annotations and a complexity-stratified comparison set; ObligBench supplies both for real-world Java functions.Its protocol keeps function bodies unchanged so adequacy is assessed against a fixed implementation.Source: §2 · p. 5

3 ObligBench Construction

ObligBench was constructed as a Java benchmark with high- and low-complexity function sets, source-side obligations, and validation for self-containment and leakage. Its treatment functions have more extensive complexity and branch-obligation profiles than the control set.

Complexity-guided non-trivial FUT selection

  • Cyclomatic complexity counts independent execution paths, while cognitive complexity additionally penalizes nesting and flow interruptions.The two measures capture different aspects of control-flow complexity and guide selection of functions with varied test-generation demands.Source: §3 · p. 5
  • The deduplicated treatment set contains 895 functions selected from the top 5% under cyclomatic and cognitive complexity; the control set contains 700 functions below both thresholds.The two high-complexity views overlap on 505 functions, and the control set is used for the RQ5 complexity comparison.Source: §3 · p. 5

Self-containment

  • Tasks were packaged as isolated executable Java functions, with each function body kept unchanged through AST-token signature comparison against the original source.A three-stage filter checks parsing and types, compilation and execution of three invocations, and an LLM wrapper fallback when fixtures fail.Source: §3 · p. 5

Test leakage check

  • The deterministic corpus check found no likely corresponding pre-existing tests for any of the 895 unique tasks.Candidates were flagged using repository and revision, path and content patterns, and heuristic references to the focal method, class, or package.Source: §3 · p. 5

ObligBench profiles

  • Treatment functions have median cyclomatic and cognitive complexity of 10 and 18, compared with 1 and 1 in the control set; median branch items are 9 versus 0.Treatment functions span a median of 35 lines versus 3 in control, and the treatment profile has more extensive source-level branch obligations.Source: §3 · pp. 5–6, Table 1 · p. 6

4 Evaluation Metrics

The evaluation metrics follow the Arrange-Act-Assert pattern, profiling input boundaries, control-flow alternatives, and assertion strength rather than relying only on aggregate scores. Execution failures are classified before adequacy scoring, and obligation-level scores distinguish untested behavior from tooling limitations.

4 Evaluation Metrics

  • Three AAA-aligned metrics assess arranged inputs, reached control-flow alternatives, and assertions about return values and exceptions.Scores are computed per generated-test/FUT instance and aggregated by model and prompting strategy.Source: §4 · p. 6

4.1 Mutation-Based Obligation Scoring Pipeline

  • Mutants are attributed and scored per obligation; unrepresentable obligations are excluded, and incidental kills earn no credit.Table 3 traces the Oracle Strength eligibility funnel through test execution, obligation reachability, mutant viability, and oracle detection.Source: §4.1 · pp. 6–7, Table 3 · p. 7

4.2 Execution Reliability: Pre-Scoring Failure Classification

  • Tests are classified for compilation, execution, wrong-oracle, and placeholder failures before adequacy scoring, with exclusions retained for reporting.Wrong-oracle tests may still record line or branch coverage from code executed before the failing assertion.Source: §4.2 · p. 7

4.3 Boundary-Value Adequacy: the Arrange Phase

  • Boundary-Value Adequacy credits a category only when its value is supplied at the corresponding parameter position.Applicable categories come from parameter types and source constraints; argument resolution is limited to literals, simple local constants, and directly passing single-dispatch helpers.Source: §4.3 · pp. 7–8

4.4 Control-Flow Adequacy: the Act Phase

  • Control-Flow Adequacy measures whether tests detect changes to applicable conditions, switch groups, catch blocks, and explicit throws.Unreached obligations score zero, while obligations without viable mutations are excluded; Table 4 presents obligation-level mutants and scoring.Source: §4.4 · pp. 8–9, Table 4 · p. 9

4.5 Oracle Strength: the Assert Phase

  • Oracle Strength tests assertion obligations by mutating return values and exception behavior; surviving mutants and incidental failures receive no credit.Return mutations use prioritized rules, while broad original exception types may be excluded when replacements cannot be distinguished; selected-mutation detection is not exhaustive verification.Source: §4.5 · pp. 9–10, Table 5 · p. 10

4.6 Metric Development and Validation

  • Metric validation used independently labeled BVA instances and audited assertion-mutation cases, with high reported agreement across checks.BVA sampling covered 780 instances across eleven subtypes; OS audits covered 100 cases.Source: §4.6 · p. 10

5 Experimental Setup

The experiments evaluate four prompting strategies across four LLMs on complex functions, using a low-complexity control set for comparison. Tests are run in isolated Java projects, with adequacy analyses limited to executable tests.

5.1 Benchmark and Execution Protocol

  • 14,320 generated test rows cover 895 treatment functions across four models and strategies; adequacy scores exclude non-executable tests.The 700-function control set is used only for the complexity comparison.Source: §5.1 · p. 11, §5.4 · p. 11

5.2 Prompting Strategies

  • Zero-shot generates tests from source and metadata, intention-planning adds a structured plan, mutation-feedback adds one feedback round, and Panta uses one tool-level repair iteration.All strategies share the same function source, metadata, and JUnit 5 execution contract.Source: §5.2 · p. 11

5.3 Model Suite

  • The suite includes GPT-5.4, Gemma-4-31B, Gemini-3.6-flash, and Qwen3-Coder-30B; RQ5 compares GPT-5.4 and Gemma-4-31B as the suite’s highest and lowest performers.Qwen is excluded from RQ5 because of its elevated failure rate under guided strategies.Source: §5.3 · p. 11

5.4 Decoding and Reproducibility

  • Temperature is fixed at zero, and recorded artifacts support deterministic analysis without rerunning mutations.MuTAP and ChatTester could not be integrated because their API-specific harnesses were incompatible; the study instead uses a shared-contract mutation-feedback strategy.Source: §5.4 · pp. 11–12

6 Experimental Results

Across the evaluated tests, adequacy varies by input boundary, control-flow construct, and assertion target, while execution failures also constrain which tests can be assessed. Comparing complex and low-complexity functions shows that several boundary and control-flow gaps persist across complexity levels, though some comparison categories have limited evidence.

6.1 RQ1: To What Extent Do LLMs Generate Compilable and Runnable Tests?

  • 72.1% of 14,320 generated-test rows were eligible for adequacy analysis; wrong-oracle failures made up 15.6% of excluded rows, while Panta placeholders and Qwen3-Coder-30B had notable failure rates.Intention-planning had the lowest strategy-level failure rate at 21.5%; Panta placeholders accounted for 26.9% of excluded rows, and Qwen3-Coder-30B had the highest model failure rate at 65.3%.Source: §6.1 · pp. 12–13

6.2 RQ2: Which Input Boundary Categories Are Systematically Missed?

  • Negative, null, and empty boundary coverage is only 23.8%, 34.8%, and 38.8%, respectively, while boolean coverage averages above 93%.These low-coverage categories remain weakest across models and strategies; they require selecting values not explicit in the implementation, unlike readily cued boolean alternatives.Source: §6.2 · pp. 13–14
  • Boundary-level analysis distinguishes obligations that branch coverage can group together, such as a missing null input despite tests exercising both outcomes of a null-or-empty guard.The example separates the null obligation from the covered empty-string obligation.Source: §6.2 · p. 14

6.3 RQ3: Which Control-Flow Constructs Are Systematically Blind Spots?

  • Across constructs, obligation-level operators achieve 78.1–99.5% representativeness, while configured PIT operators represent 44.7–45.3% of catch and default obligations and 1.4% of explicit-throw obligations.For other constructs, PIT’s smaller represented obligation populations may contribute to differences in adequacy scores.Source: §6.3 · pp. 14–15
  • Catch blocks score 0.100 and default groups 0.312 on CFA, with weaknesses recurring across models and strategies.Testing these paths may require inducing exceptional states or selector values outside explicit switch cases.Source: §6.3 · p. 15, Figure 2 · p. 15

6.4 RQ4: How Strongly Does the Test Oracle Check Return and Exception Behaviour?

  • Oracle Strength ranges from 0.710 for collections to 0.899 for booleans; concrete-constructor-defaults object mutations score 0.325, and reached, eligible exception obligations score 0.625.The object-category score depends on the mutation rule; broad exception assertions may accept both original and substituted exception types.Source: §6.4 · pp. 16–17, Table 8 · p. 17

6.5 RQ5: Does Complexity Impact the Systematic Weaknesses?

  • Negative and empty inputs remain weak boundary categories across complexity levels, and some low-complexity control-flow configurations also have low CFA scores.The control set has no applicable catch obligations; null coverage is higher there but is based on only 74 applicable nullable parameters.Source: §6.5 · pp. 17–18
  • The complexity comparison has limited evidence for some categories: several control-flow groups contain fewer than ten scorable items, and weak-category Oracle Strength comparisons are inconclusive.Control-set object OS covers 87 obligations from 11 methods under one null-replacement rule; exception scores cover 13 distinct methods.Source: §6.5 · p. 18

7 Threats to Validity

The reported validity threats concern input-resolution scope, CFA mutation operators, and the study’s generalizability beyond its tested models, prompts, and generation setting.

7 Threats to Validity

  • 29.3% of target-call arguments cannot be resolved by BVA’s supported static-analysis rules, and deeper interprocedural reasoning falls outside the stated resolution scope.Manual validation assesses the reliability of the resulting classifications.Source: §7 · pp. 18–19
  • CFA mutation-operator choice may influence measured adequacy across control-flow constructs, although the study applies deterministic construct-specific transformations uniformly.Representativeness rates distinguish mutant availability limits from failures to detect represented behavioral changes.Source: §7 · p. 19
  • Results may differ for very small models, domain-specialized code models, or unstudied prompts; all generation used temperature 0, so non-zero-temperature results may vary more.Blind-spot patterns were consistent across four models from different families and four strategies.Source: §7 · p. 19

8 Conclusion

BlindSpotEval and ObligBench reveal obligation-level blind spots in generated unit tests that aggregate scores do not distinguish. The results include weak coverage of unusual inputs and selected control-flow paths, alongside uneven oracle checking and blind spots in simpler functions.

8 Conclusion

  • BlindSpotEval evaluates Boundary-Value Adequacy, Control-Flow Adequacy, and Oracle Strength, with execution reliability and failure-mode analysis, using ObligBench’s source-side obligation annotations.Its construct-level profiles identify input categories, control-flow constructs, and observable behaviors requiring stronger testing.Source: §8 · p. 19
  • Null, negative, and empty obligations reach only 34.8%, 23.8%, and 38.8% coverage, while catch-block and default-group CFA scores are 0.100 and 0.312.The study reports these patterns as consistent with a happy-path preference; wrong-oracle failures account for 15.6% of excluded rows.Source: §8 · p. 19
  • Oracle Strength distinguishes reaching behavior from checking it: concrete-constructor-defaults obligations score 0.325, and exception obligations score 0.625.Return obligations score 0.842 overall.Source: §8 · p. 19
  • Explicit-throw obligations achieve 98.2% representativeness under obligation-based control-flow mutations, compared with 1.4% under configured PIT analysis.Separate forced-outcome mutations distinguish partial checking of conditional alternatives from an any-kill indicator.Source: §8 · p. 19
  • Selected boundary-value and control-flow blind spots also occur in low-complexity functions, while well-supported Oracle Strength categories remain strongly checked across both complexity sets.The benchmark contains 895 complex and 700 low-complexity Java functions; the evaluation covers 14,320 tests from four LLMs and four prompting strategies.Source: §8 · p. 19

Related papers

Loading 2610.32548v1…