Source-linked AI summary
Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing
Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, Michel C. Desmarais
TL;DR
MuTAP addresses the weak relationship between test coverage and bug detection by augmenting LLM prompts with initial tests and surviving mutants. It improves bug-detection effectiveness across human-written and synthetic buggy programs, while using mutation killing and bug detection as evaluation measures.
Problem
LLM-generated tests may contain ineffective assertions, while coverage is weakly correlated with their ability to detect bugs.
Method
MuTAP augments LLM prompts with initial test cases and surviving mutants, after repairing syntax, return-value, and assertion errors in generated tests.
Results
MuTAP detects up to 28% more human-written buggy snippets and achieves a 93.57% Mutation Score on synthetic buggy code, outperforming evaluated alternatives.
Takeaways & Limitations
LLMs can generate useful tests, but MuTAP's mutation-testing and post-processing steps improve their effectiveness in revealing bugs.
Takeaways & Limitations
Mutation killing and bug detection may not agree with coverage when ranking different testing strategies.
Abstract
from arXiv · showhide
One of the critical phases in software development is software testing. Testing helps with identifying potential bugs and reducing maintenance costs. The goal of automated test generation tools is to ease the development of tests by suggesting efficient bug-revealing tests. Recently, researchers have leveraged Large Language Models (LLMs) of code to generate unit tests. While the code coverage of generated tests was usually assessed, the literature has acknowledged that the coverage is weakly correlated with the efficiency of tests in bug detection. To improve over this limitation, in this paper, we introduce MuTAP for improving the effectiveness of test cases generated by LLMs in terms of revealing bugs by leveraging mutation testing. Our goal is achieved by augmenting prompts with surviving mutants, as those mutants highlight the limitations of test cases in detecting bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs). We employ different LLMs within MuTAP and evaluate their performance on different benchmarks. Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets. Among these, 17% remained undetected by both the current state-of-the-art fully automated test generation tool (i.e., Pynguin) and zero-shot/few-shot learning approaches on LLMs. Furthermore, MuTAP achieves a Mutation Score (MS) of 93.57% on synthetic buggy code, outperforming all other approaches in our evaluation. Our findings suggest that although LLMs can serve as a useful tool to generate test cases, they require specific post-processing steps to enhance the effectiveness of the generated test cases which may suffer from syntactic or functional errors and may be ineffective in detecting certain types of bugs and testing corner cases PUTs.
1. Introduction
MuTAP addresses the weak relationship between test coverage and bug detection by using mutation testing to improve LLM-generated unit tests. It augments prompts with surviving mutants and reports stronger fault-detection results than the evaluated alternatives.
- Coverage is weakly correlated with the effectiveness of tests in detecting bugs.
- Mutation testing evaluates tests by injecting artificial faults and identifying surviving mutants that expose test weaknesses.
- 93.57% average Mutation Score was achieved, outperforming Pynguin and zero-shot/few-shot LLM techniques.
- 28% more human-written buggy snippets were detected, including 17% missed by all compared methods.
- MuTAP augments prompts with initial tests and surviving mutants to generate more effective tests without requiring natural-language PUT descriptions.
2. Motivating Example
The motivating example shows MuTAP iteratively using surviving mutants to augment prompts and generate additional assertions. In the example, this raises mutation score from 60% to 90%, exceeding Pynguin's 40%.
- MuTAP seeks to maximize the number of mutants killed in tests generated for a PUT.
- Initial tests leave four mutants surviving, revealing missing assertions for injected bugs.
- MuTAP augments the prompt with refined initial tests and a surviving mutant to target the uncovered behavior.
- The augmentation process iterates with additional surviving mutants until all are killed or no unused survivors remain.
- 90% mutation score was reached, compared with 60% initially and 40% for Pynguin.
3. Approach
MuTAP prompts an LLM to generate tests, repairs syntax and intended-behavior errors, evaluates tests through mutation testing, and re-prompts using surviving mutants. The process supports zero-shot or few-shot initialization and optional oracle minimization.
- 3.1. Initial Prompt: MuTAP begins with zero-shot or few-shot prompts containing the PUT, then sends them to an LLM for test generation.
- 3.2. Test Refinement: Generated tests are refined by repairing syntax errors and correcting assertion outputs that do not match the PUT's returned behavior.
- 3.2.2. Intended Behavior Repair: MuTAP verifies assertion oracles by executing the PUT and replaces incorrect expected outputs, omitting tests with invalid input types.
- 3.3. Mutation Testing: Mutation testing returns mutation score and surviving mutants, which determine whether MuTAP augments the prompt or proceeds to oracle minimization.
- 3.4. Prompt Augmentation: The pipeline stops when mutation score reaches 100% or when no unused surviving mutants remain for prompt augmentation.
4. Evaluation
The evaluation investigates MuTAP's effectiveness against automated test-generation tools, the performance of its components, and performance across mutation types.
- RQ1: RQ1 compares MuTAP-generated tests with tests produced by automatic test-generation tools.
- RQ2: RQ2 examines how the different parts of MuTAP perform.
- RQ3: RQ3 evaluates MuTAP's performance for each mutation type.
4.1. Experimental Setup
The experiments compare MuTAP with Pynguin and LLM baselines across synthetic and human-written Python benchmarks. They use two LLMs, repeated prompting, syntax handling, mutation testing, and defined benchmark configurations.
- Prompting and preprocessing: Initial prompts and syntax-fixing prompts are each run for up to 10 iterations to obtain valid candidate tests and repair syntax errors.Tasks remain problematic when valid test cases cannot be generated or syntax errors remain unresolved after the allowed iterations.
- Automated baseline: Pynguin 0.17.0 with DynaMOSA is used as the fully automated comparison tool with a 600-second generation timeout.Pynguin generates tests through search-based algorithms targeting branch coverage.
- Language models: The study employs Codex and llama-2-chat as MuTAP’s language-model components.Codex is configured for code generation and llama-2-chat for dialog-oriented use, with both used as test-generation models.
- Baselines: MuTAP is compared with Pynguin and two LLM baselines: Before-refining and After-refining.Before-refining uses initial outputs without syntax fixing or intended-behavior repair, whereas After-refining applies refining and oracle minimization.
- Mutation testing: MutPy 2.0 generates Python mutants by applying mutation operators one at a time when applicable to the program under test.Mutation testing is used to assess whether generated tests detect injected faults.
- Benchmark datasets: The evaluation uses HumanEval’s 164 programming problems and Refactory’s 1710 human-written buggy submissions as benchmark datasets.HumanEval uses reference solutions as programs under test, while Refactory provides real buggy code for comparison with Pynguin and the baselines.
4.2. Experimental Results
MuTAP improves LLM-generated test effectiveness through syntax and behavior repair plus surviving-mutant prompt augmentation. Across synthetic and real buggy programs, it generally surpasses Pynguin and conventional LLM approaches in mutation killing and bug detection.
- Synthetic buggy programs: 89.13% and 91.98% MS are obtained with zero-shot MuTAP using Codex and llama-2-chat, rising to 92.02% and 93.57% with few-shot prompts.The few-shot results correspond to Codex and llama-2-chat, respectively.
- Synthetic buggy programs: 93.57% MS is achieved by few-shot MuTAP with llama-2-chat, killing 1179 of 1260 mutants and reaching MS=100% for up to 70% of PUTs.Pynguin killed 649 mutants and reached MS=100% for 28.22% of PUTs.
- Post-processing: Prompt augmentation reduces unintended-behavior assertions from 63.63% to 19.38% for Codex and from 63.25% to 10.75% for llama-2-chat in the reported settings.Intended Behavior Repair further improves assertion alignment with the PUT’s intended behavior.
- Post-processing: Post-processing is necessary: syntax fixing and intended behavior repair correct up to 95.94% and 89.86% of syntax and functional errors, respectively.MuTAP also reduces unintended behavior in outputs by 44.36% with Codex and 52.5% with llama-2-chat.
5. Discussion
MuTAP combines LLM-based test generation with post-processing and mutation-guided prompt augmentation to improve fault detection. The discussion highlights stronger bug-detection performance than Pynguin, while noting operational constraints and the importance of mutation score as the evaluation focus.
- Approach: MuTAP uses prompt-based LLM test generation without computationally expensive fine-tuning.The approach leverages code synthesis and prompt-based learning to assist developers.
- Effectiveness: MuTAP generates more effective bug-revealing tests than Pynguin, whose tests use random integers and mutated inputs.The paper characterizes LLM-generated tests as more natural-looking and more correlated with input/output types and PUT functionality.
- Implications: MuTAP can support developer-facing test generation and may integrate into GitHub Copilot lab through automatically generated mutants and prompt augmentation.The proposed use is generating effective tests in terms of fault detection.
- Execution time: 39.75 seconds and 42.11 seconds are the average MuTAP processing times per HumanEval task with llama-2-chat using zero-shot and few-shot prompts, respectively.These totals include prompt construction, syntax fixing, behavior repair, mutation-score calculation, augmentation, and greedy optimization.
- Evaluation: Mutation Score evaluates test-oracle quality, whereas test coverage assesses another quality aspect and is weakly correlated with bug-detection effectiveness.MuTAP executes tests on both the PUT and its mutants and reports mutation score.
- Approach: MuTAP improves generated tests through syntax correction, intended-behavior repair, prompt augmentation, and greedy optimization.Surviving mutants guide repeated re-prompting for new tests, while post-processing addresses syntactic and functional issues.
6. Threats to Validity
The study identifies limitations involving prompt choices, syntax-error repair, oracle minimization, assertion types, validity metrics, datasets, and reproducibility. MuTAP is currently implemented for Python, and its results may not extend directly to other languages.
- Internal validity: Changing prompt instructions or few-shot examples could potentially improve MuTAP’s results.The study did not investigate these prompt variations.
- Internal validity: The syntax-repair process omits error messages and error lines, which could potentially enhance performance if included.
- Internal validity: The greedy oracle-minimization algorithm may not optimally minimize test oracles while maximizing Mutation Score.
- Internal validity: The study generates only primitive assertions, leaving other assertion types for future investigation.
- Construct validity: Coverage and bug-finding can correlate without agreeing on the ranking of different testing strategies.The authors therefore evaluate effectiveness using mutant killability and bug detection.
- External and reliability validity: The current implementation and evaluation use executable Python programs, so the findings cannot be extended directly to other programming languages.The design is conceptually adaptable beyond Python, but existing results are Python-specific.
7. Related work
Prior LLM-based test-generation studies explored few-shot prompting, retrieved examples, fine-tuning, and hybrid tools, but generally emphasized coverage, lexical match, or test validity. This work distinguishes itself by using mutation testing to enhance generated tests’ fault-revealing effectiveness.
- LLM-based test generation: Prior studies used few-shot learning, retrieved examples, or fine-tuned models to generate tests and assertions.Examples include Codex, CEDAR, ATHENATEST, and TOGA across Java, JavaScript, and other benchmarks.
- Bug-reproducing tests: LIBRO used issue reports as few-shot prompts and improved correct bug-reproducing test generation over state-of-the-art tools.
- Coverage-oriented approaches: CODAMOSA combined Codex- and Pynguin-generated tests to improve coverage, but its tests lacked assertion oracles for bug detection.
- Tests for code filtering: Other studies generated tests from problem descriptions to filter buggy code suggestions produced by Codex.
- Research gap: Prior work assessed LLM-generated tests using coverage or lexical matching, without specifically using mutation testing to improve their effectiveness.
8. Conclusion
MuTAP generates tests, repairs errors, evaluates them with mutation testing, and re-prompts with surviving mutants to target undetected faults. Across real and synthetic Python programs, it reports strong bug-detection performance, while its current implementation remains Python-specific.
- Conclusion: MuTAP generates tests, repairs syntax and return-value errors, evaluates them through mutation testing, and re-prompts using surviving mutants.
- Conclusion: 86.72% of buggy code was identified on average by MuTAP with Codex on a bug-repairing benchmark.
- Conclusion: 94.06% of buggy code was detected by MuTAP with llama-2-chat, outperforming Pynguin and zero-shot and few-shot LLM techniques.
- Future scope: MuTAP’s design and evaluation methodology are adaptable to other programming languages and models, although the current implementation uses Python programs.