PaperlayerView exact revision on arXiv

Source-linked AI summary

No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation

Zhiqiang Yuan, Yiling Lou, Mingwei Liu, Shiji Ding, Kaixin Wang, Yixuan Chen, Xin Peng

arXiv:2305.04207v3cs.SE

TL;DR

Automated unit-test techniques can achieve reasonable coverage but often produce tests developers find insufficiently readable and meaningful for direct adoption. This paper evaluates ChatGPT-generated tests and introduces ChatTester, which improves correctness over default ChatGPT by 34.3% in compilable rate and 18.7% in execution passing rate.

  • Problem

    Traditional automated unit-test techniques can achieve reasonable coverage but often produce tests lacking the readability and meaningfulness needed for direct developer adoption.

    Source: §1 · p. 2

  • Method

    The paper evaluates ChatGPT-generated tests quantitatively and through a user study, then proposes ChatTester, combining an initial test generator with an iterative test refiner.

    Source: §1 · p. 4, §5 · p. 12

  • Results

    34.3% improvement in compilable rate and 18.7% improvement in execution passing rate over default ChatGPT.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Passing ChatGPT-generated tests are often high quality and could be put into direct use to alleviate manual test-writing effort.

    Source: §4.5 · p. 12

    The findings might not generalize to datasets beyond the benchmarks used in this work.

    Source: §7 · p. 19

Abstract

from arXiv · show

Unit testing is essential in detecting bugs in functionally-discrete program units. Manually writing high-quality unit tests is time-consuming and laborious. Although traditional techniques can generate tests with reasonable coverage, they exhibit low readability and cannot be directly adopted by developers. Recent work has shown the large potential of large language models (LLMs) in unit test generation, which can generate more human-like and meaningful test code. ChatGPT, the latest LLM incorporating instruction tuning and reinforcement learning, has performed well in various domains. However, It remains unclear how effective ChatGPT is in unit test generation. In this work, we perform the first empirical study to evaluate ChatGPT's capability of unit test generation. Specifically, we conduct a quantitative analysis and a user study to systematically investigate the quality of its generated tests regarding the correctness, sufficiency, readability, and usability. The tests generated by ChatGPT still suffer from correctness issues, including diverse compilation errors and execution failures. Still, the passing tests generated by ChatGPT resemble manually-written tests by achieving comparable coverage, readability, and even sometimes developers' preference. Our findings indicate that generating unit tests with ChatGPT could be very promising if the correctness of its generated tests could be further improved. Inspired by our findings above, we propose ChatTESTER, a novel ChatGPT-based unit test generation approach, which leverages ChatGPT itself to improve the quality of its generated tests. ChatTESTER incorporates an initial test generator and an iterative test refiner. Our evaluation demonstrates the effectiveness of ChatTESTER by generating 34.3% more compilable tests and 18.7% more tests with correct assertions than the default ChatGPT.

1 INTRODUCTION

The paper evaluates ChatGPT-generated unit tests and finds substantial correctness problems despite promising quality among tests that pass. It introduces ChatTester, which uses ChatGPT to generate and iteratively refine tests, and reports improved correctness across ChatGPT and two open-source LLMs.

Introduction

  • Traditional automated test-generation methods can achieve reasonable coverage but often produce tests less readable and meaningful than manually written tests, limiting developers’ willingness to adopt them.Prior LLM-based techniques fine-tune models on test generation to produce more human-like and meaningful tests.Source: §1 · p. 2
  • The first empirical study evaluates ChatGPT-generated tests for 1,000 Java focal methods across correctness, sufficiency, readability, and usability using quantitative analysis and a user study.The study examines syntactic, compilation, and execution correctness; coverage and assertions; and, for correct tests, readability and developers’ willingness to adopt them.Source: §1 · pp. 2–4
  • Only 24.8% of ChatGPT-generated tests pass execution; 57.9% have compilation errors and 17.3% compile but fail, mostly because of incorrect assertions.Passing tests achieve comparable coverage and readability to manually written tests and sometimes receive developers’ preference; the paper identifies code knowledge and focal-method intent as ways to reduce errors.Source: §1 · p. 3
  • ChatTester first prompts ChatGPT to infer a focal method’s intention and generate a test, then iteratively fixes compilation errors using error messages and additional code context.Its validate-and-fix refiner works on tests from the initial generator.Source: §1 · p. 3
  • ChatTester improves ChatGPT-generated tests’ compilable rate by 34.3% and execution passing rate by 18.7%, and generalizes to CodeLlama-Instruct and CodeFuse.The initial test generator produces more tests with correct assertions, while the iterative refiner fixes compilation errors.Source: §1 · p. 4

2 BACKGROUND

The background introduces LLMs and ChatGPT’s training, then situates unit test generation through its test components, traditional limitations, and newer learning-based approaches.

2.1 Large Language Models

  • LLMs are pretrained on large text corpora with self-supervised objectives and commonly use Transformer architectures categorized as encoder-only, decoder-only, or encoder-decoder.Examples include CodeBERT, CodeGen, and CodeT5, respectively.Source: §2.1 · p. 4
  • ChatGPT builds on GPT through instruction tuning followed by reinforcement learning from human feedback to improve task performance and alignment with human intent.Open-source instructed models such as CodeLlama-Instruct and CodeFuse are also emerging.Source: §2.1 · p. 4

2.2 Unit Test Generation

  • A unit test for a focal method combines a test prefix that drives it to a testable state with a test oracle, such as assertions, that checks behavior against the specification.Source: §2.2 · p. 4
  • Traditional search-, random-, and constraint-based techniques generate unit tests automatically, but their tests may be difficult for developers to adopt because of low readability and meaningfulness despite reasonable coverage.EvoSuite is a search-based example that uses evolutionary algorithms to maximize coverage for Java classes.Source: §2.2 · pp. 4–5
  • Learning-based approaches use deep learning, including LLMs, to generate tests; methods range from translating focal methods into prefixes or assertions to prompting instructed LLMs for inputs and assertions [24] [51] [54] [73] [43] [48].Source: §2.2 · p. 5

3 STUDY SETUP

The study constructs a benchmark of complete, executable Java projects and evaluates ChatGPT-generated tests using a defined prompt, baselines, and correctness, sufficiency, readability, and usability measures.

3.1 Benchmark

  • The benchmark contains 1,000 focal-method/test-method pairs sampled from 1,748 pairs extracted from 185 maintained, compilable Java projects.Projects were selected from CodeSearchNet using maintenance, star-count, and Maven-build criteria; the final sample includes tests and focal methods of diverse scales and structures.Source: §3.1 · p. 5

3.2 Basic Prompt Design

  • The basic prompt combines a task description and role-playing instruction with the focal method, class name, class fields, and signatures of all class methods.The natural-language task requests a JUnit test for the named focal method using the specified JUnit version.Source: §3.2 · p. 6

3.3 Baselines

  • The study compared ChatGPT with EvoSuite and AthenaTest, reproducing AthenaTest with fine-tuned CodeT5 and removing benchmark overlaps from its training data.EvoSuite used its default settings; the study excluded Teco because it completes statements rather than generating complete methods, and excluded Codex-based approaches because they target different scenarios.Source: §3.3 · pp. 6–7

3.4 Experimental Procedure

  • The study queried gpt-3.5-turbo via the official API, then compiled and executed each generated test and assessed correctness, sufficiency, readability, and usability.Correctness covered syntax, compilation, and execution; sufficiency covered focal-method statement and branch coverage and assertion count. The user study compared 248 passing ChatGPT tests with corresponding manually written tests, rated by five Java developers with 4–5 years of experience.Source: §3.4 · pp. 7–8

4 STUDY RESULTS

ChatGPT-generated tests often fail compilation or execution, but passing tests show coverage, readability, and usability comparable to manual tests. The authors identify limited code knowledge and understanding of method intent as correctness gaps to address.

4.1 RQ1: Correctness

  • 42.1% of ChatGPT-generated tests compiled successfully, while 24.8% executed successfully without errors.ChatGPT outperformed AthenaTest in syntactic, compilation, and execution correctness, while Evosuite had higher compilable and passing rates partly because it prunes invalid code and derives assertions from execution; the authors therefore do not conclude it is inherently better at generating correct tests.Source: §4.1 · pp. 8–9
  • ChatGPT’s failed tests included diverse compilation errors, and 85.5% of failed executions were caused by incorrect assertions.Compilation errors included unresolved symbols, type and access errors, invalid abstract-class instantiation, and unsupported operators; manual inspection found the assertion errors came from the generated assertions rather than bugs in the focal methods.Source: §4.1 · pp. 9–10

4.2 RQ2: Sufficiency

  • Among executable tests on focal methods where both ChatGPT and Evosuite generated executable tests, ChatGPT achieved the highest coverage among techniques and coverage comparable to manual tests.ChatGPT’s assertion-count distribution most resembled manual tests, whereas Evosuite tended to generate fewer assertions and AthenaTest sometimes generated more than 15 per test.Source: Table 5 · p. 10, §4.2 · p. 10, Figure 4 · p. 11

4.3 RQ3: Readability

  • Participants rated most ChatGPT-generated tests as reasonably readable, with readability comparable to and sometimes better than manual tests.The assessment was based on participants’ score distributions.Source: §4.3 · p. 11, Figure 5 · p. 11

4.4 RQ4: Usability

  • ChatGPT-generated tests showed usability comparable to manual tests, and participants were willing to directly adopt them in a considerable portion of cases.Most were assessed as having high-quality assertions suitable for immediate use.Source: §4.4 · p. 11, Figure 6 · p. 12

4.5 Enlightenment

  • The authors characterize ChatGPT-based test generation as promising but say correctness issues require further work on code knowledge and method-intent understanding.They connect deeper code knowledge to reducing compilation errors and better understanding the focal method’s intent to reducing assertion errors; passing tests often have strong sufficiency, readability, and practical usability.Source: §4.5 · pp. 11–12

5 APPROACH OF CHATTESTER

ChatTester uses an initial test generator and an iterative test refiner to improve the correctness of ChatGPT-generated unit tests. The generator infers a focal method’s intention before test generation, while the refiner uses compiler feedback and relevant code context to fix compilation errors.

5 APPROACH OF CHATTESTER

  • ChatTester combines an initial test generator with an iterative refiner to improve the correctness of ChatGPT-generated unit tests.The workflow uses ChatGPT for both components.Source: §5 · p. 12

5.1 Initial Test Generator

  • The initial test generator asks ChatGPT to infer the focal method’s intention, then generates a test using that intention.For setCharAt(), the paper contrasts an incorrect assertion from the basic prompt with a correct assertion after intention inference and generation.Source: §5 · p. 12, §5.1 · p. 14

5.2 Iterative Test Refiner

  • The iterative refiner compiles each generated test and prompts ChatGPT with compilation errors and relevant code context until compilation succeeds or the iteration limit is reached.An EM parser extracts error type, buggy location, and buggy element; a code analyzer retrieves relevant class declarations and public method signatures, including from other project files. Refinement targets compilation errors, not execution failures; the controller accepts a refinement only when the compilation-error count decreases and can stop after repeated invalid refinements.Source: §5.2 · pp. 15–16

6 EVALUATION OF CHATTESTER

The evaluation examines ChatTester’s effectiveness, generalization to other LLMs, and project-level performance. It reports improved test correctness, coverage, readability, and usability across these settings.

6.1 Evaluation Setup.

  • The evaluation used a separate 100-pair dataset, repeated RQ5 experiments three times, tested two open-source LLMs, and assessed three Java projects.The RQ5 dataset was sampled from 748 remaining pairs to avoid reusing the previously analyzed benchmark. The project-level user study compared 219 tests per technique, scored by five participants.Source: §6.1 · p. 16

6.2 Evaluation Results

  • ChatTester produced 34.3% more compilable tests and 18.7% more passing tests than default ChatGPT.The reported rates were 73.3% versus 39.0% for compilation and 41.0% versus 22.3% for passing execution.Source: Table 7 · p. 17, §6.2.1 · p. 17

6.2.1 RQ5: Effectiveness Evaluation.

  • ChatTester-Ite improved compilation and passing rates by 11.7% and 7.4% over ChatGPT, while the refiner fixed a further 22.6% and 11.3% of tests, respectively.ChatTester-Ite corrected 12.5% of ChatGPT tests with incorrect assertions; removing the initial generator decreased ChatTester’s compilation and passing rates by 12.7% and 7.0%. Iterative refinement averaged 99.0 seconds overall, including 15.0 seconds for initial generation and 30.0 seconds per refinement iteration.Source: §6.2.1 · p. 17

6.2.2 RQ6: Generalization Evaluation.

  • ChatTester improved compilation rates by 21.0% for CodeLlama-Instruct-34B and 23.0% for CodeFuse-34B, and passing execution rates by 11.0% and 18.0%, respectively.The results indicate that ChatTester generalizes beyond ChatGPT to the two studied open-source LLMs.Source: Table 9 · p. 18, §6.2.2 · p. 18

6.2.3 RQ7: Project-level Evaluation.

  • At project level, ChatTester achieved comparable or better coverage than baselines, and participants judged its tests better in readability and usability.Participants cited more intuitive naming, more structured code, better assertions, and less adoption effort; the comparison covered statement and branch coverage across three projects.Source: §6.2.3 · p. 18, Table 10 · p. 18, Figure 10 · p. 19

7 THREATS TO VALIDITY

The authors identify randomness, benchmark generalizability, and possible training-data leakage as threats, and describe replication, diverse and separate evaluations, and edit-distance analysis as mitigations.

7 THREATS TO VALIDITY

  • On 164 HumanEval Python methods, ChatTester raised ChatGPT-generated tests’ successful execution rate from 31.7% to 43.2%, indicating cross-language generalization.The authors report this as an additional evaluation of ChatTester.Source: §7 · p. 19
  • To address ChatGPT’s randomness, the authors repeated automated ChatTester evaluations three times and report averages, while the unrepeated empirical study showed similar correctness results across two datasets.The empirical study was not repeated because of the large manual effort required for the user study.Source: §7 · p. 19
  • To limit benchmark concerns, the authors included high-quality projects and diverse focal and test methods, evaluated ChatTester on a separate dataset, and added a Python dataset evaluation.They note that findings might not generalize to other datasets.Source: §7 · p. 19
  • Against potential training-data leakage, ChatTester-generated tests had an average 1,027-character and median 609-character Levenshtein distance from manual tests; only 5% were under 200 characters.The authors interpret this as suggesting direct copying from training data may be less prevalent in their benchmark.Source: §7 · p. 19

8 RELATED WORK

The section situates ChatGPT’s unit-test generation among prior software-engineering and LLM-based test-generation work, highlighting ChatTester’s focus on complete tests and iterative refinement.

8.1 ChatGPT for SE tasks

  • This work presents the first study of ChatGPT for unit-test generation and proposes ChatTester to improve the quality of its generated tests.ChatGPT had also been applied to software-engineering tasks including program repair [71] [82] and code generation [25] [32] [66].Source: §8.1 · p. 19

8.2 LLMs for Test Generation

  • ChatGPT outperforms AthenaTest [73] in test correctness and coverage; AthenaTest fine-tunes BART to generate complete tests from focal methods and code context.Source: §8.2 · p. 20
  • Unlike fine-tuned models, newer approaches prompt instruction-tuned LLMs; ChatTester draws on task decomposition, while Nashid et al. use few-shot Codex prompts for assertions.ChatTester instead uses zero-shot prompting to generate complete tests, including prefixes and assertions.Source: §8.2 · p. 20
  • ChatTester complements CODAMOSA [46] by generating one test per focal method that can seed its search, unlike CODAMOSA’s LLM-and-search generation of test sets per module.Unlike LIBRO [43], ChatTester does not require bug reports; unlike Li et al. [48], it generates complete tests rather than only failure-inducing inputs.Source: §8.2 · p. 20
  • Unlike Ring System [42]’s one-turn generation for a code snippet, ChatTester iteratively refines tests using added code context and error-message information, unlike TestPilot [69]’s code-snippet search feedback.Source: §8.2 · p. 20

9 CONCLUSION

The study finds that ChatGPT-generated unit tests have correctness problems, although passing tests resemble manually written tests; ChatTester improves test quality over default ChatGPT.

9 CONCLUSION

  • ChatTester generated 34.3% more compilable tests and 18.7% more tests with correct assertions than default ChatGPT.The approach leverages ChatGPT itself to improve the quality of its generated tests.Source: §9 · p. 20
  • ChatGPT-generated tests have diverse compilation errors and execution failures, mostly caused by incorrect assertions.Source: §9 · p. 20
  • Passing ChatGPT-generated tests achieve coverage and readability comparable to manually written tests and sometimes receive developers’ preference.Source: §9 · p. 20
  • The study systematically evaluates ChatGPT-generated tests for correctness, sufficiency, readability, and usability.The paper describes this as the first empirical study of ChatGPT’s capability in unit test generation.Source: §9 · p. 20

Related papers

Loading 2305.04207v3…