PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Rule-Based Mutation Testing: Test-Aware Mutant Generation Using Large Language Models

Nils Kiele, Zainab Saad, Zirui Wang, Steve Drew, Samira Ebrahimi Kahou

arXiv:2609.35841v1cs.SEcs.AI

TL;DR

Mutation testing can produce many equivalent or trivial mutants, while test-blind LLM approaches lack information about existing test coverage. This paper prompts LLMs with the problem, solution, and tests to generate mutants that pass base tests. Test-aware prompting yields substantially more verified faults than matched test-blind prompts and mutmut, with comparable subtlety and lower cost per verified fault than test-blind prompting.

  • Problem

    Traditional mutation tools and test-blind LLM approaches can produce low-value mutants, while test-blind prompting does not condition generation on existing test coverage.

    Source: §1 · p. 2

  • Method

    The approach gives an LLM the problem, canonical solution, and base tests in one prompt, then uses extended EvalPlus tests to verify faults that survive the base suite.

    Source: §1 · p. 2

  • Results

    Test-aware prompting substantially outperforms matched test-blind prompting and mutmut in verified fault generation across HumanEval and MBPP, with comparable subtlety and lower cost per verified fault than test-blind prompting.

    Source: §1 · p. 3, §7 · pp. 17–18

  • Takeaways & Limitations

    When tests are available, LLMs can use them as a signal of test-suite weaknesses, making test-aware prompting a practical lightweight alternative to exhaustive rule-based mutation and heavier iterative systems.

    Source: §7 · p. 18

    The extended oracle is imperfect, so undetected mutants may be valid code or genuine faults missed by its tests.

    Source: §6 · p. 17

Abstract

from arXiv · show

Mutation testing evaluates test-suite adequacy by injecting synthetic faults into program code. However, traditional rule-based tools often generate large numbers of trivial, redundant, or equivalent mutants that limit their practical use for identifying gaps in a test suite. While recent large language model (LLM)-based approaches generate more realistic faults, most remain test-blind: The model sees only the source code and cannot reason about what existing tests already cover. We propose test-aware mutant generation, in which an LLM receives the problem statement, canonical solution and base tests in a single prompt, and must generate a nontrivial mutant that passes the base unit tests. We evaluate this approach across a set of five LLMs -- Gemini 3.1 Pro, Gemini 3 Flash, GPT 5.1 Codex Mini, GPT 4.1 Mini, Qwen3-32B -- on the HumanEval and MBPP benchmarks. The extended EvalPlus test suites serve as an automated oracle to verify whether surviving mutants represent genuine bugs. Test-aware prompting yields verified fault rates of 87.7% (HumanEval) and 79.1% (MBPP), meaning these mutants pass all base tests but are caught by the oracle. This vastly outperforms the matched test-blind prompting (which yields only 12.2% and 23.0%, respectively) and the traditional rule-based tool mutmut (4.4% and 5.7%). While fault subtlety (the fraction of extended tests a mutant fails) remains comparable across all three methods, test-awareness minimizes the computational cost per verified fault, compared to test-blind prompting. Exposing an LLM to existing unit tests shifts mutant generation from untargeted bug injection toward effective discovery of weaknesses in an existing test suite. Our work establishes a concrete foundation for future research to scale test-aware mutant generation to production-level environments.

1 Introduction

Mutation testing uses injected faults to assess whether existing tests detect defects, but traditional tools can produce equivalent or trivial mutants that limit practical value. Because test-blind LLMs lack information about covered behavior, this work tests whether a single test-aware prompt can generate mutants that expose test-suite gaps.

1 Introduction

  • Traditional mutation tools can generate equivalent or trivially killed mutants, while large mutant pools impose computational and manual-triage costs.Distinguishing equivalent mutants from genuine faults is generally undecidable, and empirical estimates put their share at 5–20%.Source: §1 · p. 2
  • Test-blind LLM prompts omit existing tests, so models cannot directly target coverage gaps and may generate redundant, trivially caught, or equivalent mutants.Such outputs add review or execution costs; iterative test-aware approaches can reduce this gap but are computationally expensive.Source: §1 · p. 2
  • The proposed single-prompt approach gives the LLM the source code and existing tests and asks it to generate a subtle logical bug that survives the base tests.EvalPlus extended suites serve as an automated oracle: a mutant passing base tests but failing an extended test is classified as a verified, non-equivalent fault.Source: §1 · pp. 2–3
  • Across five LLMs, test context increases verified fault generations by up to 7×, while reducing amortized token cost per verified fault relative to test-blind prompting.The evaluation also measures mutant subtlety through the number of oracle tests triggered.Source: §1 · p. 3

2 Related Work

Prior mutation-generation work includes syntactic rule-based tools, test-blind LLM approaches, and iterative adversarial workflows. This paper differs by directly conditioning mutant generation on existing tests and framing the task as evading them.

2 Related Work

  • Rule-based tools such as mutmut use simple syntactic changes and can produce noncompiling, trivial, or semantically equivalent mutants.The paper instead prioritizes fewer, more informative mutants aimed at semantic blind spots and important edge cases.Source: §2 · p. 3
  • Test-blind LLM methods generate mutants without current tests, while LLMut varies context but reportedly performs slightly worse when tests are appended passively.The authors hypothesize that their adversarial prompt succeeds by explicitly instructing the model to bypass tests and target gaps.Source: §2 · p. 3
  • Adversarial and iterative approaches use multiple agents or repeated feedback, whereas this work reports strong results from a single LLM interaction.Those iterative workflows require repeated compilation, execution, and reprompting.Source: §2 · p. 4
  • Mutation- and coverage-guided test-generation systems improve tests rather than generate mutants, but share the premise that test-related context can improve LLM generation.Examples include MuTAP [6], TestGen-LLM [1], CoverUp [2], and SymPrompt [21].Source: §2 · p. 4
  • LLMs have also been studied for detecting equivalent mutants after generation; this paper instead seeks to reduce low-value generation by exposing tests to the model.The cited detection study reports an average F1 score of 35.69% improvement over prior methods.Source: §2 · p. 4
  • The authors identify a gap in empirical study of test-awareness for mutant generation, distinct from test-related context used on the test-generation side.They note that test-blind approaches need expensive iterative feedback to directly find test-suite gaps.Source: §2 · p. 4

3 Method and Experimental Setup

The study evaluates single-shot LLM-generated mutants that must pass existing tests, then uses extended tests to identify verified faults. It compares test-aware and test-blind prompt variants across models and benchmarks, measuring effectiveness, subtlety, and token cost.

3 Method and Experimental Setup

  • Each problem-model-prompt configuration generates one mutant without compilation or test-execution feedback, except in the iterative ablation.The mutant must preserve the function signature, pass base tests, and fail at least one hidden-oracle test.Source: §3 · p. 4
  • The pipeline prompts the model, extracts mutant code, filters mutants that fail or error on base tests, then evaluates survivors with EvalPlus; any oracle failure marks a verified fault.Base-test filtering reduces oracle runs and excludes trivial faults that do not survive developer tests.Source: §3 · p. 5
  • The evaluation uses EvalPlus on 164 HumanEval and 378 MBPP problems, treating the extended suites as an automated oracle for semantic faults overlooked by base tests.The authors select EvalPlus because the extended suites are stronger than the base suites, while acknowledging they are not perfect.Source: §3 · pp. 5–6
  • Five LLMs are compared with matched test-blind prompts and mutmut; experiments use temperature 0, which does not guarantee exact reproducibility.The models span open-source, frontier, and more computationally efficient options.Source: §3 · p. 6
  • The two prompt families each have test-aware and test-blind variants, isolating test context while also comparing free-form reasoning with a structured template.AGENT1 is the primary prompt; its test-aware version asks for blind-spot analysis, a minimal mutation plan, and a mental dry-run against each base test.Source: §3 · p. 6
  • The structured AGENT2 test-aware prompt elicits a function summary, base-test coverage, two to three blind spots, a mutation plan, and a per-test trace.Its test-blind counterpart omits coverage analysis because it has no tests.Source: §3 · p. 7
  • Effectiveness is the success rate of verified faults; quality is fault subtlety measured by oracle kill rate, where lower values indicate narrower faults.The study also reports raw output tokens and amortized input, thinking, and output tokens per successful mutant; mutmut has no token-cost comparison.Source: §3 · pp. 8–9

4 Results

Test-aware prompting consistently raises the share of LLM-generated mutants that evade base tests, while mutant subtlety remains comparable to test-blind generation. Iterative feedback further narrows the test-blind gap, and reviewed undetected mutants often expose edge cases missed by the oracle.

4 Results

  • Mutmut generates about 18 mutants per HumanEval task and 9 per MBPP task on average, unlike the single mutant generated per LLM model-prompt-task combination.Mutmut’s exhaustive generation yields many Invalid or Rejected mutants.Source: §4 · p. 9, Figure 2 · p. 9, Table 3 · p. 10
  • The move from test-blind to test-aware prompting shifts outcomes from Rejected toward Success, with Gemini 3.1 Pro producing 144 Success mutants (87.8%) on HumanEval.Smaller models also produce more Invalid mutants; the prompt has less effect on status distribution than model capability.Source: §4 · pp. 9–10, Figure 3 · p. 11

4.1 Effectiveness of Test-Aware Prompting

  • 87.8% vs. 12.2% and 81.1% vs. 11.6%: agent1 test-aware Success rates exceed test-blind rates for Gemini 3.1 Pro and GPT 5.1 Codex Mini on HumanEval.Across models, test-aware prompting outperforms test-blind prompting; mutmut reaches around 4.4% on HumanEval and 5.7% on MBPP.Source: §4.1 · p. 10, Figure 4 · p. 11

4.2 Subtlety and Fault Quality

  • Test-aware mutants had median oracle kill rates of roughly 10–30%, comparable to test-blind mutants, and the study could not establish a subtlety difference from mutmut.Lower oracle kill rates indicate subtler edge-case bugs; the figure reports medians and 1.5× IQR whiskers.Source: §4.2 · pp. 11–12, Figure 5 · p. 12
  • 49 of 65 undetected mutants (75.4%) were serious edge-case bugs, while 9 (13.8%) were equivalent mutants.The remaining 7 (10.8%) were ambiguous faults; the review was conducted by one author without measuring inter-rater agreement.Source: §4.2 · pp. 12–13, Table 5 · p. 13

4.3 Cost Efficiency

  • Test-aware prompting uses fewer tokens per verified mutant than test-blind prompting for every evaluated model except GPT 4.1 Mini.GPT 5.1 Codex Mini required 36.6 seconds per verified HumanEval mutant and 22.5 seconds per verified MBPP mutant, versus mutmut’s 54.0 and 41.9 seconds.Source: §4.3 · pp. 13–14, Figure 6 · p. 14

4.4 Iterative Prompting

  • With iterative feedback, Gemini 3.1 Pro agent1-tb rises from 12.2% to 83.5% on HumanEval, approaching the iterative test-aware rate of 98.2%.The test-blind prompts use around twice the generation cost in the iterative setting, while single-shot test-aware prompting captures most achievable gain.Source: §4.4 · pp. 14–15, Figure 7 · p. 15, Table 6 · p. 15

5 Discussion

The discussion argues that test context makes LLM-generated mutants target existing semantic blind spots, while practical value depends on review cost, model choice, base-test quality, and the availability of a real-world oracle.

Human Time vs. Compute Time Trade-off

  • Test-aware mutation wins on cost per useful, verified fault rather than cost per generated candidate, reducing the human effort needed to review low-value mutants.The authors connect this trade-off to the goal of producing productive mutants rather than strengthening tests against every unproductive one.Source: §5 · pp. 15–16

Model Selection

  • Model choice affects success rate, fault subtlety, and token cost, with GPT 4.1 Mini agent2-ta underperforming the cited test-blind alternative.GPT 5.1 Codex Mini is suggested for strict compute budgets, while Gemini 3.1 Pro is recommended for subtle edge cases despite higher API costs.Source: §5 · p. 16

Why Single-Prompt Test-Awareness Helps

  • Test-aware prompting shifts outputs from Rejected to Success by helping models avoid trivial mutations and target gaps in existing tests.The authors describe this as the main value of test-awareness, beyond simply generating bugs.Source: §5 · pp. 15–16

The Dependency on Base Test Quality

  • Even sparse base suites—7.7 tests per HumanEval problem and 3 per MBPP problem—improve effectiveness over test-blind prompting, but the study did not vary suite quality.The approach can exploit only shortcomings expressed in the supplied tests.Source: §5 · p. 16

No Oracle in the Real World

  • Because real-world development usually lacks an oracle, surviving mutants should be treated as suggestions for stronger tests and developer inspection, not ground-truth defects.The evaluation uses extended EvalPlus tests to verify non-equivalence in a controlled setting.Source: §5 · p. 16

Practical Implications

  • Test-aware LLMs could assist CI pull-request review by generating targeted mutants and actionable suggestions for stronger unit tests.Source: §5 · p. 17

6 Threats to Validity

The authors qualify validity across sparse base tests, automated mutant classification, benchmark scope, and model and provider variability.

Internal Validity: Sparse Base Tests

  • Sparse base tests may seem like a low bar, but the authors view them as plausible for codebases with only basic assertions.The study aims to show that LLMs can uncover test-suite gaps, not that they outperform humans at mutant generation.Source: §6 · p. 17

Construct Validity: Lack of Human Review

  • Automated classification leaves uncertainty for Undetected mutants, which may be valid code or bugs missed by the extended EvalPlus oracle.Qualitative analysis found most such cases in the best model-prompt combination to be serious faults; EvalPlus remains a proxy for industrial ground truth.Source: §6 · p. 17

External Validity: Benchmark Limitations

  • The study uses self-contained Python benchmarks, while production systems add larger suites, stateful functions, and multi-file dependencies that may strain context and focus.The authors suggest iterative prompting and retrieval-augmented generation as possible needs for production-level datasets.Source: §6 · p. 17

External Validity: Model Constraints

  • LLM performance varies across models and provider-side policies, although benefits of test-aware prompting generally held across the five evaluated models.Source: §6 · p. 17

7 Conclusion & Future Work

Test-aware prompting substantially improves verified fault generation over test-blind prompting and mutmut, while producing mutants of comparable subtlety. The paper identifies greater efficiency per verified mutant and outlines future evaluation in industrial repositories and of computational and environmental trade-offs.

7 Conclusion & Future Work

  • Test-aware prompting substantially improves verified fault generation over matched test-blind prompting and mutmut across HumanEval and MBPP.The authors describe this as an increase in the number of useful mutants produced at comparable subtlety.Source: §7 · pp. 17–18
  • The largest gain is more verified, useful mutants at comparable subtlety, not a universal reduction in oracle kill rate.Test-aware prompting is also more cost-effective per verified mutant because less computation is spent on outputs that do not become useful faults.Source: §7 · p. 18
  • Future work could evaluate the pipeline on industrial repositories by splitting tests into base and oracle sets.The authors hypothesize this could replicate the observed increase in success rate.Source: §7 · p. 18
  • Further research is needed to quantify computational and environmental trade-offs, including local versus cloud inference energy use and emissions.The authors also propose studying whether next-generation models with extended reasoning can further optimize amortized cost per fault.Source: §7 · p. 18

Data Availability Statement

The paper makes its code and data publicly available and provides instructions for reproducing its plots.

Data Availability Statement

Related papers

Loading 2609.35841v1…