Source-linked AI summary
No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation
Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng
TL;DR
Manually written unit tests are costly, and traditional generators often produce tests that are difficult for developers to read and adopt. This paper evaluates ChatGPT-generated tests across multiple quality dimensions and proposes ChatTester, whose two-stage refinement improves correctness over default ChatGPT.
Problem
Manual unit-test writing is time-consuming, while traditional automated tests often have poor readability and meaningfulness despite reasonable coverage.
Method
The paper conducts quantitative and user-study evaluations of ChatGPT-generated tests and introduces ChatTester with an initial generator and iterative refiner.
Results
34.3% more tests are compilable and 18.7% more tests pass execution with ChatTester than with default ChatGPT.
Takeaways & Limitations
Passing ChatGPT-generated tests achieve comparable coverage and readability to manually written tests and are sometimes preferred by developers.
Takeaways & Limitations
The findings may not generalize to other datasets.
Abstract
from arXiv · showhide
Unit testing is essential in detecting bugs in functionally-discrete program units. Manually writing high-quality unit tests is time-consuming and laborious. Although traditional techniques can generate tests with reasonable coverage, they exhibit low readability and cannot be directly adopted by developers. Recent work has shown the large potential of large language models (LLMs) in unit test generation, which can generate more human-like and meaningful test code. ChatGPT, the latest LLM incorporating instruction tuning and reinforcement learning, has performed well in various domains. However, It remains unclear how effective ChatGPT is in unit test generation. In this work, we perform the first empirical study to evaluate ChatGPT's capability of unit test generation. Specifically, we conduct a quantitative analysis and a user study to systematically investigate the quality of its generated tests regarding the correctness, sufficiency, readability, and usability. The tests generated by ChatGPT still suffer from correctness issues, including diverse compilation errors and execution failures. Still, the passing tests generated by ChatGPT resemble manually-written tests by achieving comparable coverage, readability, and even sometimes developers' preference. Our findings indicate that generating unit tests with ChatGPT could be very promising if the correctness of its generated tests could be further improved. Inspired by our findings above, we propose ChatTESTER, a novel ChatGPT-based unit test generation approach, which leverages ChatGPT itself to improve the quality of its generated tests. ChatTESTER incorporates an initial test generator and an iterative test refiner. Our evaluation demonstrates the effectiveness of ChatTESTER by generating 34.3% more compilable tests and 18.7% more tests with correct assertions than the default ChatGPT.
1 INTRODUCTION
The paper examines ChatGPT-generated unit tests across correctness, sufficiency, readability, and usability, then introduces ChatTester to improve their correctness. ChatTester combines initial test generation with iterative refinement and outperforms default ChatGPT on compilability and execution passing.
- Manual unit-test writing is laborious, while traditional automated tests often sacrifice readability and meaningfulness despite reasonable coverage.
- LLM-based techniques generate more human-like and meaningful tests by translating focal methods into test prefixes and assertions.
- The study evaluates ChatGPT-generated tests for syntactic, compilation, and execution correctness, coverage and assertions, readability, and developer adoption.
- Only 24.8% of ChatGPT-generated tests pass execution; 57.9% encounter compilation errors and 17.3% compile but fail during execution.
- ChatTester uses an initial test generator and an iterative test refiner to improve ChatGPT-generated tests, while evaluating component contributions and generalization to other LLMs.
- ChatTester improves ChatGPT-generated tests by 34.3% in compilable rate and 18.7% in execution passing rate.
2 BACKGROUND
The background contrasts conventional coverage-oriented unit-test generation with LLM-based approaches designed to produce more readable and meaningful tests. It also defines the structure of unit tests and motivates evaluating their practical adoption.
- Unit tests validate whether functionally discrete program units behave correctly and typically contain a test prefix plus a test oracle.
- Traditional generators use search-based, random-based, or constraint-based strategies, commonly optimizing test-suite coverage.
- Traditional generated tests can achieve reasonable coverage but generally have lower readability and meaningfulness than manually written tests, limiting direct developer adoption.
- LLM-based methods formulate test generation as translating focal methods into test prefixes or assertions, often through fine-tuning.
- More recent approaches use prompt strategies with instructed LLMs to generate test inputs and assertions.
3 STUDY SETUP
The study builds an executable Java benchmark, designs a practical ChatGPT prompt, compares traditional and learning-based baselines, and evaluates generated tests through compilation, execution, and human assessment.
- The benchmark addresses prior datasets’ limited code context by including complete, executable Java projects alongside focal methods.
- The researchers filter 4,685 CodeSearchNet Java projects for maintenance, popularity, Maven buildability, and local compilation, obtaining 185 projects.
- The final empirical-study benchmark samples 1,000 data pairs from 1,748 extracted pairs, covering focal and test methods with diverse scales and structures.
- The basic prompt combines a natural-language task description with code context containing the focal method, focal class, fields, and method signatures.
- EvoSuite and AthenaTest serve as traditional and learning-based baselines for JUnit test generation.
- The workflow queries gpt-3.5-turbo for each benchmark pair, places each generated test with its focal class, and attempts compilation and execution.
4 STUDY RESULTS
ChatGPT-generated tests show strong sufficiency, readability, and practical usability among passing cases, but many tests still fail compilation or execution. The study identifies diverse compilation errors and assertion-related execution failures as key correctness problems.
- Correctness: 42.1% of ChatGPT-generated tests compiled successfully, while 24.8% executed successfully without execution errors.The evaluation analyzed 1,000 generated tests.
- Correctness: ChatGPT-generated tests encounter diverse compilation errors, and most failed executions result from incorrectly generated assertions.Compilation failures include symbol-resolution, type, and access errors; assertion errors account for 85.5% of failed executions.
- Readability and Usability: ChatGPT-generated tests resemble manually written tests in assertion-count distributions and have reasonable, comparable readability.Most generated tests receive decent readability assessments, sometimes exceeding manually written tests.
- Sufficiency: ChatGPT-generated tests achieve comparable coverage to manually written tests and the highest coverage among the compared existing techniques.Coverage is calculated using executable tests to avoid bias from unexecutable tests.
- Readability and Usability: Participants assess ChatGPT-generated tests as comparably usable to manually written tests, with a considerable portion willing to adopt them directly.Passing tests could alleviate manual test-writing effort, provided correctness issues are addressed.
- Limitations: ChatGPT-based unit test generation remains limited by compilation and execution failures linked to incomplete code knowledge and weak understanding of focal-method intent.The paper suggests supplying deeper code knowledge and improving intent understanding to reduce these errors.
5 APPROACH OF CHATTESTER
ChatTester improves ChatGPT-based unit test generation through an intention-guided initial generator and an iterative compiler-feedback refiner.
- Overview: ChatTester contains an initial test generator and an iterative test refiner that improve the correctness of ChatGPT-generated tests.The workflow is designed to address incorrect assertions and compilation errors.
- Initial Test Generator: The initial test generator first infers the focal method’s intended functionality, then generates a unit test using that intention.This decomposes test generation into intention understanding and test construction.
- Iterative Test Refiner: The iterative test refiner validates generated tests, constructs prompts from compilation errors and additional code context, and repeatedly queries ChatGPT for refined tests.Refinement continues through controller decisions about successful, valid, or invalid refinements.
- Initial Test Generator: Intention-guided generation can replace an incorrect assertion produced by the basic prompt with a correct assertion for the same focal method.The paper illustrates this contrast using setCharAt().
- Iterative Test Refiner: The prompt constructor combines parsed error information with code analysis to add relevant context from project files.It extracts error type, buggy location, and buggy elements, then uses those elements to locate related classes and public method signatures.
6 EVALUATION OF CHATTESTER
The evaluation studies ChatTester’s effectiveness, component contributions, generalization, and project-level performance. ChatTester improves correctness over default ChatGPT and remains effective across other LLMs and project-level measures.
- Evaluation Setup: The evaluation examines ChatTester’s overall effectiveness, component contributions, generalization to other LLMs, and project-level performance.The study uses separate data for RQ5 and evaluates alternative LLMs and projects.
- RQ5: Effectiveness: 34.3% more tests compile and 18.7% more tests pass execution with ChatTester than with default ChatGPT.The comparison is based on compilation rates of 73.3% versus 39.0% and passing rates of 41.0% versus 22.3%.
- RQ5: Effectiveness: Both ChatTester components contribute: the initial generator improves assertions, while the iterative refiner fixes compilation errors.Removing components reduces performance, and the complete approach adds further fixes beyond the initial generator.
- RQ5: Effectiveness: ChatTester’s iterative refinement requires repeated compilation and execution checks across refinement iterations.Table 8 reports tests remaining under refinement and tests successfully compiled or executed at each iteration.
- RQ6: Generalization: ChatTester improves compilation and passing-execution rates for CodeLlama-Instruct-34B and CodeFuse-34B.The reported improvements are 21.0%/23.0% in compilation rate and 11.0%/18.0% in passing execution rate, respectively.
- RQ7: Project-level Evaluation: In project-level evaluation, ChatTester achieves comparable or better coverage and consistently outperforms baselines in coverage, readability, and usability.Coverage includes statement and branch coverage.
7 THREATS TO VALIDITY
The study identifies randomness in ChatGPT and benchmark selection as threats to validity, while using repeated experiments, diverse data, separate evaluation data, and an additional Python evaluation.
- Threats to Validity: ChatGPT randomness is a threat because generated-test results may vary across runs.Automatic evaluations repeat experiments three times and report averages, whereas the user-study experiments are not repeated.
- Threats to Validity: Findings may not generalize to other datasets despite using diverse projects and focal methods.A separate evaluation dataset is used to reduce potential overfitting, and HumanEval provides an additional Python-language evaluation.
8 RELATED WORK
Related work includes fine-tuned and prompt-based LLM techniques for unit test generation. ChatTester differs in scope and can complement search-based test generation.
- LLMs for Test Generation: Fine-tuned LLM approaches formulate test generation as translating focal methods into test prefixes, assertions, or complete test cases.The paper reports that ChatGPT outperforms AthenaTest in correctness and coverage.
- LLMs for Test Generation: Prompt-based techniques use instructed LLMs without fine-tuning, including strategies that decompose complex tasks into smaller subtasks.ChatTester is presented as complementary to CODAMOSA.
- Comparison with Related Techniques: ChatTester generates one test for a focal method, whereas CODAMOSA generates a test set for a module using LLMs and search-based algorithms.ChatTester could provide higher-quality seed tests for CODAMOSA’s search procedure.
- Comparison with Related Techniques: Unlike LIBRO, which generates tests for bug reports, this work targets unit test generation without bug reports.The distinction concerns the test-generation scenario addressed by each technique.
9 CONCLUSION
The study finds that ChatGPT-generated tests have substantial correctness problems, while passing tests can match manually written tests in several developer-relevant qualities. ChatTESTER improves generated-test quality over default ChatGPT.
- ChatGPT-generated tests suffer from diverse compilation errors and execution failures, often caused by incorrect assertions.
- Passing ChatGPT-generated tests achieve comparable coverage and readability to manually written tests and are sometimes preferred by developers.
- ChatTESTER generates 34.3% more compilable tests than default ChatGPT.
- ChatTESTER generates 18.7% more tests with correct assertions than default ChatGPT.