PaperlayerView exact revision on arXiv

Source-linked AI summary

On the Risks of using LLM-Generated Tests for Regression Testing

Mohammadali Charoosaei, Cedric Richter, Mike Papadakis

arXiv:2610.11835v1cs.SE

TL;DR

Regression tests generated from faulty code may encode defects as expected behavior, despite the importance and effort of maintaining reliable tests. The authors generate tests for injected faults in historical pull requests and track their behavior across project histories. Generated tests often enforced faults rather than revealed them, and those faulty expectations frequently persisted or accumulated.

  • Problem

    Regression tests are costly to write, and it is unclear whether tests generated from faulty changes capture intended behavior or encode defects as expectations.

    Source: §1 · pp. 1–2

  • Method

    The authors inject faults into historical pull requests, generate regression tests, classify their outcomes on faulty and reference code, and track fault-enforcing tests through later project changes.

    Source: §1 · p. 3, §8 · p. 18

  • Results

    Across 145 pull requests, 8%–17% of generated tests were fault-enforcing versus 2.4%–4.8% fault-revealing; 83%–91% of enforcing tests persisted through later commits.

    Source: §1 · p. 3, §8 · p. 18

  • Takeaways & Limitations

    When faults accumulated, 83%–92% remained enforced at the history’s end, while the project test suite detected only 14%–30%.

    Source: §1 · p. 3

    Classification uses execution outcomes rather than assertion intent, and the one-mutant-per-file rule makes the RQ3 kill rate a lower bound for denser accumulation.

    Source: §7.3 · p. 17

Abstract

from arXiv · show

Software is under constant evolution: developers continuously add features, fix bugs, and refactor code, and any of these changes may break existing functionality. Regression testing guards against such effects by capturing expected behavior in test cases. LLM-based test generation aims to automate this process by generating regression tests directly from the code under test. This is beneficial when the implementation is correct, but problematic when the code contains faults: the generated tests may then encode and preserve incorrect behavior. To investigate this risk, we apply LLM-based regression test generation to pull requests merged into the main branch of software projects and study the impact of the generated tests on subsequent project evolution. We distinguish between fault-revealing tests, which assert correctly implemented behavior, and fault-enforcing tests, which assert faulty behavior. Across 145 pull requests from SciPy, Qiskit, and pandas, 8%-17% of the generated tests are fault-enforcing, while only 2.4%-4.8% reveal faults. Fault-enforcing tests persist over time: after several subsequent commits, 83%-91% of them are still relevant and pass. They also accumulate: when the faults of all pull requests are combined in one codebase, 83%-92% remain enforced at the end of the commit history, and the developer-written test suite detects only 14%-30% of them. Our results reveal a fundamental risk of LLM-generated regression tests: without manual validation, they may encode faulty behavior as expected behavior, allowing bugs to persist across software revisions and largely evade developer-maintained test suites. LLM-based regression testing can thus give rise to a new form of technical debt.

1 Introduction

Regression tests can reduce the effort of maintaining evolving software, but tests generated from faulty changes may encode defects as expected behavior. This study finds that such tests are common and can persist or accumulate, creating risks for later maintenance.

1 Introduction

  • Fault-enforcing tests may pass unnoticed when added, then fail after a fault is fixed and make the suite less trustworthy as they accumulate.Fixing a fault can require developers to determine whether a failure reflects a regression or an incorrect test.Source: §1 · p. 2
  • The study injects faults into historical pull requests, generates tests without manual validation, and tracks fault enforcement through later project changes.It examines persistence, accumulation across pull requests, and detection by the project test suite.Source: §1 · pp. 2–3
  • Across 145 pull requests, 8%–17% of generated tests enforced faults, while only 2.4%–4.8% revealed them.Fault-enforcing tests passed on faulty code and failed on the intended behavior.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • After later commits, 83%–91% of fault-enforcing tests still passed only on faulty code; when faults accumulated, 83%–92% remained enforced and the project suite detected only 14%–30%.The findings suggest that unvalidated generated tests can lock faults into the suite and contribute to technical debt.Source: §1 · p. 3

2 Background and Related Work

Prior work generates regression tests at change level and improves coverage, while oracle studies examine whether tests reflect intended behavior. This paper addresses the lack of longitudinal evidence on whether faulty expectations persist through project histories.

2.1 LLM-Based Regression Test Generation

  • Change-level systems generate tests for modified code, while other approaches use code-aware prompts, coverage feedback, and iterative refinement to improve tests or coverage.Prior evaluations use coverage, mutant killing, change capture, or engineer acceptance.Source: §2.1 · pp. 3–4

2.2 Change-Aware Mutation Testing

  • Prior change-aware mutation studies use commit-relevant faults, but this study tracks generated tests through real pull-request histories.The authors describe injected commit-aware mutants as proxies for real bugs.Source: §2.2 · p. 4

2.3 The Oracle Problem in LLM-Based Testing

  • Earlier oracle studies mainly assess tests at one time point; this study follows them through project history, where incorrect expectations can pass and cover faulty code.The authors state that their study enables systematic measurement and longitudinal evaluation of this failure mode.Source: §2.3 · p. 4

3 Study Design

The study injects faults into historical pull requests, generates regression tests without access to the fixed reference, and classifies test behavior. It then follows fault-enforcing tests across later changes and simulated histories.

3 Study Design

  • The study asks how often generated tests enforce faults, whether those tests persist across later pull requests, and whether accumulated faults remain enforced or are detected by the project suite.Each pull request is modeled with a fixed reference version and faulty mutants that pass the existing suite.Source: §3 · p. 5

3.1 Behavioral Taxonomy

  • Tests are classified by their outcomes on faulty and reference code as fault-revealing, fault-enforcing, weak, or invalid.Fault-revealing tests fail on faulty code and pass on reference code; fault-enforcing tests do the reverse.Source: §3.1 · pp. 5–6

3.2 Fault Injection

  • Faults are injected into changed code and retained only if they survive the project test suite and are empirically killable by a generated detection test.Mutation is scoped to functions changed by the pull request and other methods in a changed class.Source: §3.2 · p. 6

3.3 Test generation

  • The LLM generates coverage tests from pre-change code, mutated post-change code, and their diff, without seeing the fixed reference.It refines tests using execution feedback for up to six turns; existing acceptance requires a test to pass on the code the generator sees.Source: §3.3 · pp. 6–7

3.4.1 Persistence.

  • For persistence, each fault and its test are propagated through later pull requests; only tests that remain fault-enforcing are counted as survived.Tests are first gated on a clean baseline, and other outcomes—including a removed mutation—are terminal for the analysis.Source: §3.4.1 · p. 7

3.4 Longitudinal Analysis

  • For accumulation, simulations add one randomly chosen fault-enforcing pair at each later pull request and test each fault amid the others.At the history’s end, the authors measure remaining enforcement and whether the project test suite detects the accumulated faults; results average repeated simulations.Source: §3.4.2 · p. 7, §3.4 · p. 7

4 Experimental Setup

The study evaluates generated regression tests across SciPy, Qiskit, and pandas by injecting faults into historical pull requests and tracking generated tests through project evolution.

4.1 Subject Systems

  • The benchmark uses SciPy, Qiskit, and pandas, selecting 145 merged pull requests that change incompletely covered code and meet the study’s scope criteria.All 45 eligible pandas PRs and random samples of 50 each from SciPy and Qiskit were used.Source: §4.1 · pp. 7–8

4.2 Mutation Analysis

  • Mutation analysis targets changed functions and sibling methods, applies standard mutation operators, and retains surviving mutants for which an LLM can generate a killing test.Mutants already killed by the project suite or lacking an informative test run are excluded.Source: §4.2 · p. 8

4.3 Coverage-Guided Test Generation

  • Coverage tests are generated with pull-request-aware prompts using modified code, surrounding context, the prior revision’s diff, and project execution information.Claude Haiku 4.5 can revise a coverage-test attempt for up to six turns.Source: §4.3 · p. 8

4.4 Forward Propagation

  • Mutants and fault-enforcing tests are propagated through later pull requests to assess whether enforced faults persist across project changes.The propagation analysis counts pairs that can be followed to the last selected pull request.Source: §4.4 · p. 9

5.1 RQ1: Behavioral Quality of Generated Tests

Across the three projects, generated tests enforce injected faults more often than they reveal them. The imbalance holds across pull requests, although rates vary widely, and exception assertions do not explain it.

5.1.1 Behavioral Classification.

  • 16.9%, 8.4%, and 16.9% of classified pairs enforce faults, versus 2.6%, 2.4%, and 4.8% that reveal them, respectively across SciPy, Qiskit, and pandas.Fault-enforcing pairs outnumber fault-revealing pairs by 3.5–6.5× in every project.Source: §5.1.1 · pp. 9–10, §5.1.3 · p. 12
  • Weak tests dominate in SciPy (64.6%) and Qiskit (69.5%), while pandas has 41.4% weak and 37.0% invalid tests.Only 5.2% of invalid tests fail to import or collect; the rest run and fail on both original and faulty code.Source: §5.1.1 · p. 10

5.1.2 Per-Pull-Request Variance.

  • Fault-enforcing rates vary widely by pull request, with the highest rates in smaller PRs, yet fault-enforcing shares exceed revealing shares across every size third.The reported highest rates are 77.8% for SciPy #21505, 75.0% for Qiskit #12979, and 84.6% for pandas #59114; many PRs have no classified pairs of either kind.Source: §5.1.2 · p. 11, Figure 6 · p. 11

5.1.3 What Fault-Enforcing Tests Assert.

  • Exception expectations appear in 37–57% of enforcing tests versus 3–9% of revealing tests, but excluding exception oracles still leaves 4.0×, 4.3×, and 2.6× as many enforcing tests.One SciPy example asserts that a default call raises an error on faulty code and fails once the fault is fixed.Source: §5.1.3 · pp. 11–12, Figure 7 · p. 11

5.2 RQ2: Do Fault-Enforcing Tests Persist Through The Project’s Evolution?

The study tracks fault-enforcing tests through later pull requests and finds that most continue accepting the injected faults. Developer-written tests catch only a minority of those faults.

5.2 RQ2: Do Fault-Enforcing Tests Persist Through The Project’s Evolution?

  • The propagation analysis assumes an undetected faulty PR is merged with passing enforcing tests, and that rewritten mutated code leaves the analysis without a verdict.A developer test that fails on the mutant but passes on original code is treated as exposing the fault; such cases are reported separately.Source: §5.2 · p. 12

5.2.2 How Long Do Fault-Enforcing Tests Persist?

  • Survival estimates remain near-complete over time: no SciPy or Qiskit test begins detecting its fault over up to 49 later PRs, while pandas falls once to 99.5%.The pandas drop reflects seven pairs killed at the second or third later PR; none were killed afterward.Source: §5.2.2 · p. 13

5.2.3 Developer Tests Catch a Minority.

  • Developer tests catch faults in 36.3% of SciPy pairs, 6.1% of Qiskit pairs, and 7.6% of pandas pairs.In Qiskit and pandas, over 90% of faults accepted by generated tests are never caught by a developer test within the studied history.Source: §5.2.3 · p. 13

5.2.1 Most Fault-Enforcing Tests Persist.

  • Across projects, 83–91% of 2,480 propagated pairs survive to the end of the studied history with generated tests still accepting the fault.A developer test catches the fault at a later PR for only 6–36% of pairs.Source: §5.2.3 · p. 13, Table 5 · p. 13

5.3 RQ3: Do Faults Accumulate in a Single Codebase?

The study simulates fault accumulation across later pull requests and examines whether other faults disrupt generated tests or project tests detect accumulated faults. Fault-enforcing tests largely persist in these histories, while project tests detect only a minority of the faults.

5.3 RQ3: Do Faults Accumulate in a Single Codebase?

  • The RQ3 histories are a deliberately adverse stress test, not a model of how faults accumulate in real projects.The study completed 246 trials from 91 source pull requests, adding one randomly chosen fault and its test at each later pull request.Source: §5.3 · p. 14

5.3.1 Faults accumulate.

  • About one fault was added per two pull requests, producing chains of 21–24 simultaneous bugs.The eligible-fault line trails the injected-fault line by 8%–17% at chain end; Figure 8 distinguishes injected faults, eligible faults, and the shaded gap.Source: §5.3.1 · p. 14, Figure 8 · p. 15

5.3.2 Other Faults Rarely Change a Test’s Behavior.

  • Only 55 of 44,974 compound evaluations caused a test that accepted its fault alone to fail with the rest of the pool present.The rate was 0 in SciPy, 0.37% in Qiskit, and 0.03% in pandas; tests continued accepting faulty behavior even amid many other faults.Source: §5.3.2 · pp. 14–15

5.3.3 The Project’s Test Suite Detects a Minority.

  • The project test suites confirmed catches for 30.3% of SciPy, 13.8% of Qiskit, and 21.0% of pandas pool members.These confirmed rates are lower bounds; additional failures occurred only with the full pool and could not be attributed to an individual fault.Source: §5.3.3 · p. 15
  • Across pools of up to 24 simultaneous faults, other faults disrupted acceptance in 0.00%–0.37% of cases; 83%–92% remained enforced, while project tests caught 14%–30%.Project tests could have caught at most 34%–57% of pool members.Source: §5.3.3 · p. 15

6 Discussion

The discussion attributes fault-enforcing tests to deriving expectations from faulty code and accepting only tests that pass on that code. It reports that many persist and discusses implications for developers, tool builders, and evaluation.

6.1 Why Fault-Enforcing Tests Arise

  • The oracle problem and an acceptance rule requiring tests to pass on the observed faulty code contribute to fault-enforcing tests.Tests that would reveal a fault may be discarded or revised, partly contributing to the gap between 8%–17% fault-enforcing and 2.4%–4.8% fault-revealing tests.Source: §6.1 · pp. 15–16

6.2 Fault Enforcing Tests Do Not Vanish Over Time

  • 83%–91% of faults remained applicable and accepted by their generated tests, while for 64%–94% of faults no project test had failed by the end of the studied history.The discussion notes that developers might otherwise be expected to uncover or overwrite incorrect assumptions over time.Source: §6.2 · p. 16

6.3 Implications

  • Developers should check generated expected values against specifications or PR descriptions, rather than treating the current code’s output as evidence of correctness.The discussion says a passing generated test is not evidence of correctness.Source: §6.3 · p. 16
  • Tool builders could report failing tests as possible code faults instead of repairing or discarding them, and derive expectations from specifications or documentation when available.The discussion also notes that 2.4%–4.8% of tests are fault-revealing, even after several refinement rounds.Source: §6.3 · p. 16
  • Coverage guidance is weak because 41%–70% of generated tests cover changed code but cannot distinguish faulty from reference behavior.The discussion recommends longitudinal benchmarks that track generated tests across pull-request histories.Source: §6.3 · p. 16

7 Threats to Validity

The study’s conclusions are bounded by its use of artificial mutants, three mature Python projects, and Claude Haiku 4.5. The authors also discuss pipeline, construct, and statistical threats that qualify interpretation of their findings.

7.1 External Validity

  • Artificial mutants may not represent real faults, and the fault-revealing rate may be optimistic; results may also differ for other projects, testing cultures, or stronger models.The three projects span numerical computing, quantum computing, and data manipulation, but all are mature libraries with established regression suites. The study uses Claude Haiku 4.5; stronger models may not yield the same results.Source: §7.1 · pp. 16–17

7.2 Internal Validity

  • The taxonomy depends on correct mutation generation, execution, propagation, and container management; excluding ambiguous propagations leaves survival shares and interaction rates nearly unchanged.Skipped evaluations at SciPy’s baseline gate lower its effective sample, while mutant misplacement is more likely to lead to skipping or killing than erroneous survival.Source: §7.2 · p. 17

7.3 Construct Validity

  • Mutation testing is an imperfect proxy for real defects; pre-commit behavior explains only 0.2% of comparable fault-enforcing pairs, and classification ignores assertion intent.The one-mutant-per-file rule makes RQ3’s kill rate a lower bound on denser accumulation; that analysis measures stability of an incorrect expectation, not classical masking.Source: §7.3 · p. 17

7.4 Conclusion Validity

  • PR-level resampling and paired tests address wide PR variation, but project point estimates are imprecise and PRs lacking propagated pairs were excluded from longitudinal analyses.The reported SciPy fault-enforcing share spans 8.4–26.2%.Source: §7.4 · pp. 17–18

8 Conclusion

The study finds that generated regression tests can enforce faults and that many remain enforced as projects evolve. Such tests can fail after a fault is fixed, so developers should validate expectations against intended behavior.

8 Conclusion

  • Across 145 pull requests, 8%–17% of generated tests were fault-enforcing versus 2.4%–4.8% fault-revealing, and 83%–91% of tracked fault-enforcing pairs still accepted faults later.In the accumulation stress test, a test stopped accepting its fault in at most 0.4% of evaluations; project suites caught faults for 6%–8% of pairs in Qiskit and pandas and 36% in SciPy.Source: §8 · p. 18
  • A fault-enforcing test can pass silently on faulty code, then fail after a fix and make the fix harder.The failure leaves developers to decide whether the code or the test is wrong.Source: §8 · p. 18
  • Developers should check generated tests’ expected values against intended behavior before merging and consider failing tests as possible signals of code faults.The authors also propose deriving expectations from specifications or PR descriptions and evaluating generation approaches that do not require tests to pass.Source: §8 · p. 18

9 Data Availability

The replication package provides the evaluation pipeline, benchmark configuration, generated artifacts, and derived data and scripts.

9 Data Availability

  • The replication package includes the pipeline and its tests, configuration for 145 pull requests, generated mutants and tests with execution records, and scripts and derived data for regenerating tables and figures.The package is available at https://figshare.com/s/bcdd07ff46760fd254ec.Source: §9 · p. 18
Loading 2610.11835v1…