PaperlayerView exact revision on arXiv

Source-linked AI summary

TESTEVAL: Benchmarking Large Language Models for Test Case Generation

Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, Lei Ma

arXiv:2406.04531v2cs.SE

TL;DR

LLM test-case generation lacks public benchmarks for fair comparison across capabilities. TESTEVAL evaluates models on 210 Python programs across three coverage tasks; results show that targeting specific program elements remains challenging, highlighting limits in comprehending program logic and execution paths.

  • Problem

    Public benchmarks are lacking for comparing LLM test-generation capabilities, including generating high-coverage tests and targeting specified lines, branches, or execution paths.

    Source: §1 · p. 1

  • Method

    TESTEVAL evaluates LLM test generation on 210 LeetCode Python programs using overall coverage, targeted line/branch coverage, and targeted path coverage tasks.

    Source: §1 · p. 1

  • Results

    Current LLMs can achieve high overall coverage with diverse tests, but generating tests for a specific element remains challenging and reflects difficulty comprehending program logic.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The findings point to developing advanced LLM reasoning frameworks to improve understanding of program behavior during testing.

    Source: §1 · p. 2

    TESTEVAL uses online programming problems, which may differ from real-world scenarios.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Testing plays a crucial role in the software development cycle, enabling the detection of bugs, vulnerabilities, and other undesirable behaviors. To perform software testing, testers need to write code snippets that execute the program under test. Recently, researchers have recognized the potential of large language models (LLMs) in software testing. However, there remains a lack of fair comparisons between different LLMs in terms of test case generation capabilities. In this paper, we propose TESTEVAL, a novel benchmark for test case generation with LLMs. We collect 210 Python programs from an online programming platform, LeetCode, and design three different tasks: overall coverage, targeted line/branch coverage, and targeted path coverage. We further evaluate sixteen popular LLMs, including both commercial and open-source ones, on TESTEVAL. We find that generating test cases to cover specific program lines/branches/paths is still challenging for current LLMs, indicating a lack of ability to comprehend program logic and execution paths. We have open-sourced our dataset and benchmark pipelines at https://github.com/LLM4SoftwareTesting/TestEval.

1 Introduction

TESTEVAL addresses the lack of public benchmarks for comparing LLM test-generation capabilities across overall, targeted line/branch, and targeted path coverage. Experiments show that specific coverage remains challenging even as models generate executable and diverse tests.

1 Introduction

  • The benchmark addresses a lack of public comparisons and asks whether LLMs can generate diverse high-coverage tests or target specified lines, branches, and execution paths.The targeted tasks require understanding branch conditions and execution paths, rather than merely simulating numerical operations.Source: §1 · pp. 1–2
  • TESTEVAL benchmarks LLM test generation on 210 LeetCode Python programs through overall coverage, targeted line/branch coverage, and targeted path coverage.The targeted tasks assess whether models can reason about program execution behavior and follow tester intent.Source: §1 · p. 1
  • 12 of 16 LLMs improve by no more than 5% on targeted line coverage when given the target line, versus results without that information.Commercial LLMs such as GPT-4 generally outperform open-source LLMs across overall and targeted line/branch/path coverage.Source: §1 · p. 2
  • The paper contributes a benchmark, new performance metrics and experiments, and a systematic analysis of LLM test-generation capabilities.The analysis discusses challenges and opportunities in using LLMs for test case generation.Source: §1 · p. 2

2 Approach

TESTEVAL evaluates LLM test generation through overall coverage and targeted line, branch, and path tasks, using 210 LeetCode Python programs. Its benchmark design specifies task prompts, coverage measures, and procedures for selecting and characterizing targets.

2.1 Task Description

  • Programs are filtered for syntactic, execution, and assertion correctness before coverage is evaluated.Assertion errors do not make a test fail execution correctness because such tests can still cover the program.Source: §2.1 · p. 2
  • cov@k measures line or branch coverage across subsets of k executable test cases, with k set to 1, 2, and 5.The increase in cov@k as k grows measures test-case diversity.Source: §2.1 · p. 3
  • Targeted tasks ask for a test covering a specified line or branch, or a specified execution path spanning multiple branches.For each target line, branch, or path, the LLM is prompted to generate one test case; path evaluation checks complete coverage and path similarity.Source: §2.1 · p. 3

2.2 Benchmark Dataset

  • The dataset starts from 3,123 LeetCode solutions and retains 210 Python programs after filtering for cyclomatic complexity of at least 10 and removing similar problems.The final set contains 9 easy, 100 medium, and 101 hard problems; accompanying example tests are removed to prevent direct copying.Source: §2.2 · pp. 3–4
  • Programs are reformatted so each statement maps to one physical line and each inline conditional becomes a multiline block for coverage measurement.The dataset preprocessing also adds required imports and removes natural-language comments.Source: §2.2 · p. 4
  • Target branches are if, elif, and else branches; statements inside them are target lines, yielding 983 branches and 1,312 lines across 210 programs.The averages are 4.7 target branches and 6.2 target lines per program.Source: §2.2 · p. 4
  • Branches are labeled easy, medium, or hard by coverage frequency across 100 uniformly sampled valid inputs, using thresholds of 99–100%, 40–<99%, and 0–<40%.The respective branch counts are 498, 225, and 260; GPT-4-generated input generators are manually inspected and corrected.Source: §2.2 · p. 4
  • Target paths come from LeetCode example tests: two random subpaths of five consecutive branches are sampled per execution path, averaging 4.1 targets per problem.The benchmark samples paths rather than collecting all program paths, which may be enormous or undecidable.Source: §2.2 · pp. 4–5

3 Evaluation

TESTEVAL evaluates 17 instruction-following LLMs on overall and targeted coverage tasks; broad coverage is stronger than reliably reaching specified lines, branches, or paths. Two-step chain-of-thought prompting often improves targeted line coverage, but generated tests can still fail to satisfy correctly identified conditions.

3.1 Experiment Setup

  • The evaluation covers 17 commercial and open-source instruction-following LLMs, using 20 test-generation rounds per program for overall coverage.Open-source models range from 1.3B to 34B parameters; experiments use pytest-cov for coverage measurement.Source: §3.1 · p. 5, §3.2 · p. 5

3.2 Overall Coverage

  • Most models cover over 80% of program lines and branches, and GPT-4o achieves the best overall line coverage (98.65%) and branch coverage (97.16%).DeepSeek-coder-33b exceeds Gemini-1.0-pro on both overall measures.Source: §3.2 · p. 5, Table 1 · p. 6
  • GPT-4o has the best cov@1, and every model’s cov@2 and cov@5 exceed its cov@1, indicating that additional generated tests add coverage.Gemma-7b has the largest cov@5 gains over cov@1: +9.67% for line and +13.14% for branch coverage.Source: §3.2 · p. 5, Table 1 · p. 6

3.3 Targeted Line and Branch Coverage

  • GPT-4o reaches 80.97% average targeted line coverage, while six of seventeen models do not beat baseline and seven improve by less than 5%.Reaching a target line inside a branch requires identifying that branch and generating an input that invokes it.Source: §3.3 · p. 6
  • GPT-4o covers 80.87% of targeted branches; eight models show only marginal improvement and four do not improve over baseline.Twelve of sixteen models show their largest baseline improvement on hard branches, suggesting target information can help reach branches that random testing seldom covers.Source: §3.3 · p. 6

3.4 Targeted Path Coverage

  • GPT-4o and Gemini-1.0-pro reach 56.67% and 56.09% average targeted path coverage, respectively, but neither outperforms its baseline; nine of sixteen models fail to do so.No model shows an obvious improvement greater than 5% over baseline, and path-similarity gains are also not large.Source: §3.4 · p. 7

3.5 Advanced Prompting

  • For most tested models, two-step chain-of-thought prompting improves targeted line coverage, but it costs significantly more than basic prompting.The study omits expensive proprietary and large open-source models; in one GPT-4o example, the model identified correct target-line conditions but generated a test that did not satisfy them.Source: §3.5 · pp. 7–8

4 Related Work

Prior LLM research covers code generation, other software-development tasks, and several approaches to unit-test generation, but test case generation has received less benchmark-specific evaluation. TESTEVAL distinguishes itself by focusing on coverage-guided testing and a broader range of studied models.

Code-related Benchmarks for LLMs

  • Earlier coding benchmarks evaluate capabilities such as generating code from natural-language descriptions, rather than specifically targeting test case generation.The related-work discussion identifies this as a distinct software-engineering lifecycle phase.Source: §4 · p. 8
  • Existing code-related benchmarks address code generation and other development tasks, while DevBench includes testing among broader software-development stages.TESTEVAL instead provides coverage-guided tasks tailored specifically to test case generation and includes a broader range of studied models than DevBench.Source: §4 · p. 8

LLMs for Software Testing

  • LLM-based software-testing work includes fine-tuning, prompt design, and integration with traditional testing tools, with unit-test generation as a primary focus.Fine-tuning approaches are described as effective but potentially costly and difficult to scale.Source: §4 · pp. 8–9

5 Conclusion

TESTEVAL provides a benchmark and standardized pipelines for evaluating LLM-generated Python test cases across overall and targeted coverage tasks. Experiments find that high overall coverage can coexist with difficulty generating tests for specific elements, pointing to challenges in program-logic comprehension.

5 Conclusion

  • TESTEVAL provides a Python test-generation benchmark with three tasks and standardized evaluation pipelines for overall, targeted line/branch, and targeted path coverage.The targeted tasks assess understanding of program logic and execution paths and generation of tests that follow tester intent.Source: §5 · p. 9
  • LLMs can achieve high overall coverage with diverse tests, yet generating tests for specific program elements remains challenging.The paper reports a common shortfall in comprehending program logic despite promising performance on other code-related tasks.Source: §5 · p. 9

Limitations

TESTEVAL is limited to Python and to programs drawn from online programming problems, which may differ from real-world scenarios. The authors aim to verify whether results transfer to other LeetCode solution languages.

Limitations

  • The benchmark covers Python only; whether LLM behavior transfers to other LeetCode solution languages remains for future verification.The authors believe behavior may be similar across languages.Source: Limitations · p. 9
  • TESTEVAL uses online programming problems, which may differ from real-world testing scenarios.The authors note that these programs have more complex control flow than real-world test-generation datasets.Source: Limitations · p. 9

Ethical Discussion

The dataset is derived from MIT-licensed user-written LeetCode solutions, and the authors report using rule-based automated tools. They describe limits on industry impact and note that the dataset's single-file scope excludes exploitable security vulnerabilities.

Ethical Discussion

  • The source LeetCode solutions are in a GitHub repository licensed under MIT, permitting the authors to build their dataset from it.The dataset-creation tools are rule-based, and the authors report no bias introduced by them.Source: Ethical Discussion · p. 9
  • The authors judge the paper's industry impact limited because current LLMs still struggle with correct test cases and accurate target coverage.They also note that real-world software testing may raise questions not discussed in the paper.Source: Ethical Discussion · p. 9
  • The single-file, self-contained programs in the dataset contain no security vulnerabilities that LLM-generated tests could exploit.The authors say security should be considered if the dataset is extended to real-world software.Source: Ethical Discussion · p. 10

A Prompt Templates

The appendix provides primary prompts for overall and targeted coverage tasks, then describes a two-step chain-of-thought prompt that first reasons about conditions and then generates a test case.

A Prompt Templates

  • The primary prompts use no few-shot examples or chain-of-thought, and the authors encourage future prompt designs for TESTEVAL.For targeted tasks, line numbers are added to the program and target information to locate the requested line, branch, or path.Source: A Prompt Templates · p. 13

A.1 Prompt Template for Overall Coverage

  • The overall-coverage prompt requests one test input at a time and asks subsequent tests to cover different statements and branches.It supplies the program and function description and specifies the test-method format.Source: §A.1 · p. 13

A.2 Prompt Template for Targeted Line Coverage

  • The targeted line prompt asks for a test case that covers a specified target line.It otherwise provides the program and function description and specifies the test-method format.Source: §A.2 · p. 13

A.3 Prompt Template for Targeted Branch Coverage

  • The targeted branch prompt asks for a test case that covers a specified branch.It provides the program and function description and specifies the test-method format.Source: §A.3 · p. 13

A.4 Prompt Template for Targeted Path Coverage

  • The targeted path prompt defines the target as a sequence of branch conditions that must be satisfied sequentially during execution.It supplies the program, function description, target path, and required test-method format.Source: §A.4 · p. 13

A.5 Prompt Template for Two-step COT

  • The two-step COT prompt first derives conditions needed to execute a target line, then generates a test case satisfying those conditions.Its example explains that executing the twoSum return requires two different elements whose sum equals the target.Source: §A.5 · pp. 14–15

B Targeted Line/Branch Identification

The algorithm identifies targeted branches and lines by parsing a program's abstract syntax tree and recording branch spans and their constituent lines.

B Targeted Line/Branch Identification

  • The algorithm parses the program and treats conditional branches beginning with if, elif, or else as targeted branches.It records each branch's first and last line numbers as the branch span.Source: B Targeted Line/Branch Identification · p. 16
  • For each identified branch, it records the line numbers of its lines as targeted lines, excluding a line containing only else.Parsing continues until all branches in the program have been processed.Source: B Targeted Line/Branch Identification · p. 16

C Error Analysis

The error analysis examines failed test cases generated by Llama 3. It identifies two recurring forms of malformed output, both represented among Llama 3-8b’s execution errors for targeted line coverage.

C Error Analysis

  • Llama 3-8b’s targeted-line execution errors all came from a trailing redundant comma or an endlessly repeated statement pattern.The post-processing step removes an uncompilable final statement, leaving an empty test case counted as an execution error.Source: C Error Analysis · p. 16
  • A redundant trailing comma can make a generated test case syntactically invalid.Figure 6(a) illustrates this error type.Source: C Error Analysis · p. 16, Figure 6 · p. 16
  • Repeated simple patterns can produce an endless statement in a generated test case.Figure 6(b) illustrates this error type.Source: C Error Analysis · p. 16, Figure 6 · p. 16

D Data Leakage Analysis

The authors compare GPT-4o’s results on problems released before and after its training-data cutoff to assess potential data leakage. Coverage metrics were similar across the two subsets, which they interpret as evidence that leakage is not a major concern for TESTEVAL.

D Data Leakage Analysis

  • The leakage analysis uses 21 problems released after October 2023 and a comparison subset of the 21 oldest problems released before that cutoff.The cutoff reflects GPT-4o’s training data, which covers up to October 2023.Source: D Data Leakage Analysis · p. 16
  • None of the 49 official test cases for post-October 2023 problems appeared in GPT-4o’s generated tests, while 35 of 52 appeared for older problems.GPT-4o generated 20 test cases per problem, or 420 across the 21 older problems; the authors characterize copying as minor.Source: D Data Leakage Analysis · p. 16
  • Coverage metrics were similar for problems released before and after October 2023, indicating potential data leakage is not a major concern for TESTEVAL.The comparison uses 21 problems from each release period.Source: D Data Leakage Analysis · p. 16
Loading 2406.04531v2…