PaperlayerView exact revision on arXiv

Source-linked AI summary

An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation

Max Schäfer, Sarah Nadi, Aryaz Eghbali, Frank Tip

arXiv:2302.06527v4cs.SEcs.AI

TL;DR

Manual unit-test creation is laborious, and the paper evaluates whether off-the-shelf LLMs can automate it without additional training. TestPilot combines function context, documentation examples, and adaptive failure-driven re-prompting, achieving high coverage across 25 npm packages while producing mostly non-duplicative tests. The approach remains bounded by the evaluation’s assertion metric and by cases where generated tests fail because of domain-specific issues.

  • Problem

    Manual unit-test creation is laborious, motivating evaluation of automated generation using LLMs without additional training or manual effort.

  • Method

    TestPilot prompts off-the-shelf LLMs with function signatures, implementations, documentation, and usage examples, then re-prompts with failing tests and error messages.

  • Results

    TestPilot achieved 70.2% median statement coverage and 52.8% branch coverage with gpt3.5-turbo, while 92.8% of tests had no more than 50% similarity to existing tests.

  • Takeaways & Limitations

    The approach achieves state-of-the-art statement coverage on 25 npm packages, and most generated tests contain non-trivial assertions.

  • Takeaways & Limitations

    The non-trivial-coverage measure is a lower bound because the assertion criterion is simple and some apparently non-trivial assertions may not be meaningful.

Abstract

from arXiv · show

Unit tests play a key role in ensuring the correctness of software. However, manually creating unit tests is a laborious task, motivating the need for automation. Large Language Models (LLMs) have recently been applied to this problem, utilizing additional training or few-shot learning on examples of existing tests. This paper presents a large-scale empirical evaluation on the effectiveness of LLMs for automated unit test generation without additional training or manual effort, providing the LLM with the signature and implementation of the function under test, along with usage examples extracted from documentation. We also attempt to repair failed generated tests by re-prompting the model with the failing test and error message. We implement our approach in TestPilot, a test generation tool for JavaScript that automatically generates unit tests for all API functions in an npm package. We evaluate TestPilot using OpenAI's gpt3.5-turbo LLM on 25 npm packages with a total of 1,684 API functions. The generated tests achieve a median statement coverage of 70.2% and branch coverage of 52.8%, significantly improving on Nessie, a recent feedback-directed JavaScript test generation technique, which achieves only 51.3% statement coverage and 25.6% branch coverage. We also find that 92.8% of TestPilot's generated tests have no more than 50% similarity with existing tests (as measured by normalized edit distance), with none of them being exact copies. Finally, we run TestPilot with two additional LLMs, OpenAI's older code-cushman-002 LLM and the open LLM StarCoder. Overall, we observed similar results with the former (68.2% median statement coverage), and somewhat worse results with the latter (54.0% median statement coverage), suggesting that the effectiveness of the approach is influenced by the size and training set of the LLM, but does not fundamentally depend on the specific model.

1 INTRODUCTION

The paper evaluates whether off-the-shelf LLMs can generate useful unit tests through careful prompting, without additional training or manual effort. TestPilot combines contextual prompts with adaptive repair and achieves high coverage across a large JavaScript evaluation.

  • Manual unit-test creation is labor-intensive, motivating automated techniques for checking individual functions.
  • Traditional generators often produce tests that lack meaningful assertions or contain spurious assertions, limiting their direct usability.
  • Off-the-shelf LLMs may generate more natural-looking tests with assertions, but their ability to cover complex edge cases remains uncertain.
  • TestPilot generates tests by prompting an LLM with function signatures, implementations, documentation, and usage examples, then re-prompts with failures to repair tests.
  • 70.2% median statement coverage and 52.8% branch coverage were achieved by TestPilot using gpt3.5-turbo.
  • 92.8% of generated tests had no more than 50% similarity to existing tests, and none were exact copies.

2 APPROACH

TestPilot explores package APIs, mines documentation, constructs prompts, validates generated tests, and adaptively refines prompts after failures. Its prompts provide progressively richer function context, while tests are generated and validated individually.

  • TestPilot uses Mocha test scaffolding, Node.js assertions, and a completion-based LLM workflow to construct executable tests.
  • API Exploration: The API Explorer dynamically traverses a package’s object graph to identify functions and records each function’s access path, signature, and source definition.
  • Documentation Mining: The Documentation Miner extracts comments and code snippets from package documentation and associates them with relevant API functions.
  • Test Generation: The system processes functions one at a time and generates one test at a time, enabling individual validation without interference from other tests.
  • Validation: The test validator repairs simple syntax errors, rejects invalid code, and runs syntactically valid tests to determine whether they pass.
  • Prompt Refinement: Prompt refiners independently add function bodies, documentation comments, usage snippets, or failure information to create additional prompts.
  • Examples: Generated examples use ordinary usage scenarios and assertions, producing tests that look similar to tests a human developer might write.

3 RESEARCH QUESTIONS & EVALUATION SETUP

The evaluation asks how well TestPilot generates JavaScript unit tests, how its coverage compares with Nessie, and whether results depend on prompt information or the underlying LLM. It uses 25 npm packages, multiple LLM configurations, and coverage measurements based on passing tests.

  • Research Questions: TestPilot’s evaluation covers statement and branch coverage, comparison with Nessie, assertion quality, failing tests, prompt refiners, test similarity, and dependence on the underlying LLM.The research questions span generated-test effectiveness, comparisons, test characteristics, prompt information, memorization, and model choice.
  • Metrics: Coverage is measured with Istanbul/nyc for passing tests only, reporting both package-level and function-level statement and branch coverage.The setup uses Mocha’s default two-second time limit per test and reports median values across repeated runs.
  • Benchmark: The benchmark contains 25 npm packages spanning GitHub and GitLab, with varied popularity, size, domains, programming styles, documentation, and API-function counts.The benchmark combines the 10-package Nessie set, 10 additional GitHub packages, and 5 GitLab packages.
  • Experimental Procedure: TestPilot generates tests for API functions using gpt3.5-turbo, with five completions of up to 100 tokens per prompt at temperature zero.Returned tests are deduplicated, and experiments are repeated 10 times to account for response nondeterminism.

4 EVALUATION RESULTS

TESTPILOT generated passing tests with substantial statement and branch coverage across 25 npm packages, generally outperforming Nessie while producing meaningful and diverse tests. Coverage varied by package and function, with failures often arising from timeouts, assertion errors, or domain-specific filesystem issues.

  • Coverage: 70.2% median statement coverage and 52.8% median branch coverage were achieved by TESTPILOT’s passing tests across packages.Statement coverage ranged from 33.9% to 93.1%, while branch coverage ranged from 16.5% to 71.3%.
  • Coverage per function: 77.1% median statement coverage per function was achieved, with no statistically significant correlation between coverage and function size.Coverage per function ranged from 0.0% to 100.0%, indicating performance was not limited to smaller functions.
  • Comparison with Nessie: TESTPILOT outperformed Nessie on 17 of 24 packages, with a median coverage increase of 30.0% among those packages.It achieved higher median statement coverage than Nessie, 70.2% versus 51.3%, and higher branch coverage, 52.8% versus 25.6%; both differences were statistically significant.
  • Test quality: 61.4% median of generated tests contained non-trivial assertions, and these tests achieved 61.6% median coverage.The small median coverage difference from all generated tests indicates that much coverage came from assertions exercising package functionality.
  • Failing tests: Timeouts were the most common failure reason, while assertion errors reflected difficulty determining correct expected oracle values.Timeouts accounted for a median 22.7% of failing tests, assertion errors for 19.2%, and the refiner fixed 15.4% of timeout errors and 11.1% of assertion errors on average.

5 THREATS TO VALIDITY

The paper identifies threats involving documentation-example matching, proxy measures for oracle quality, readability assessment, evaluation scope, and language coverage.

  • Internal Validity: Documentation snippets are matched to functions by name, so same-named functions with different access paths may receive inaccurate examples.The authors report no precise alternative and state that high coverage and passing-test rates suggest this was not limiting in practice.
  • Construct Validity: Non-trivial assertions are used as a proxy for oracle quality, but dynamic JavaScript makes precise function-use attribution difficult.This limits precise non-trivial coverage measurement for individual functions, while package-level reported coverage is unaffected.
  • Construct Validity: Non-trivial coverage measures full coverage of tests containing at least one non-trivial assertion rather than only assertion-checked coverage.Other calls in those tests may contribute to the reported coverage.
  • Construct Validity: The non-triviality criterion sets a low bar, so reported non-trivial coverage is a lower bound on true non-trivial coverage.Assertions classified as trivial are not meaningful, but some assertions classified as non-trivial may still be weak.
  • Construct Validity: The study does not formally assess readability, leaving comparisons of generated-test readability for future user studies.Examples suggest programmer-like variable names, but this evidence is informal.
  • External Validity: The evaluation covers 25 npm packages and may not generalize to other JavaScript code bases or proprietary code absent from the LLM’s training data.The authors also state that the current implementation targets JavaScript, preventing generalization of results to other languages.

6 RELATED WORK

Related work spans neural and traditional test-generation techniques, differing in goals, training requirements, inputs, and JavaScript support. TESTPILOT is positioned as an off-the-shelf LLM approach that also outperforms Nessie on coverage.

  • Neural Techniques: Prior neural test-generation efforts differ from TESTPILOT in their goals and in requiring fine-tuning or additional training data.These differences can make direct experimental comparison meaningless or impossible.
  • Neural Techniques: Other LLM techniques use generated tests to support search, generate assertions for assertion-less tests, or fine-tune models for test production.These approaches position LLMs as complements to, rather than replacements for, traditional generation techniques.
  • Neural Techniques: TESTPILOT uses prompt context without few-shot example tests, whereas Bareiß et al. include a related function, its test, and helper signatures.Their limited evaluation found slightly better coverage than Randoop, with coverage depending strongly on closely related examples.
  • Test Generation Techniques for JavaScript: TESTPILOT’s execution-feedback prompt refinement is inspired by feedback-directed random generation and achieves higher statement and branch coverage than Nessie.Nessie is described as state of the art for feedback-directed random test generation in JavaScript.
  • Test Generation Techniques for JavaScript: Traditional JavaScript test-generation research includes dynamic symbolic execution, event-focused exploration, type inference, search-based methods, and invariant-guided inputs.The cited systems target varied JavaScript testing goals, including client-side event sequences and input-space exploration.
  • Test Generation Techniques for JavaScript: JavaScript tools such as JSART generate assertions by inferring likely invariants and removing invalid assertions.This represents a traditional approach capable of producing assertion-containing tests.

7 CONCLUSIONS AND FUTURE WORK

The paper presents TESTPILOT, an adaptive LLM-based unit-test generator that uses contextual prompts and execution feedback without fine-tuning or parallel function-test corpora. It reports strong JavaScript results while identifying scope and quality limitations and several future directions.

  • Conclusions: TESTPILOT generates adaptive unit tests from function context and failure feedback without fine-tuning or a parallel corpus of functions and tests.Prompts include signatures, documentation, usage examples, and source code; failed tests are re-prompted with their failure messages.
  • Conclusions: TESTPILOT achieves state-of-the-art statement coverage on 25 npm packages, with most generated tests containing non-trivial assertions.Experiments across gpt3.5-turbo, code-cushman-002, and StarCoder outperform Nessie on key metrics.
  • Future Work: Future work includes evaluating useful failing tests, improving non-trivial assertion generation, and varying sampling temperature beyond zero.Usage examples may help assertions, while temperature zero provides stable but less exploratory completions.
  • Future Work: Hybrid systems could combine LLM-generated starting tests with feedback-directed techniques such as Nessie to uncover difficult edge cases.The proposed combination uses LLMs for initial tests and traditional feedback-directed generation for extension.
  • Future Work: Although conceptually language-agnostic, adapting TESTPILOT to other languages requires language-specific prompts, testing frameworks, and documentation-example mining.Effectiveness would also depend on how much code in the target language appears in the LLM’s training set.
Loading 2302.06527v4…