PaperlayerView exact revision on arXiv

Source-linked AI summary

Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM

Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, Baishakhi Ray

arXiv:2402.00097v2cs.SEcs.LG

TL;DR

Generating high-coverage tests for complex focal methods challenges both SBST and LLM-based approaches. SymPrompt uses a structured, code-aware sequence of prompts aligned with execution paths. It improves CodeGen2’s correct-test-generation ratio by a factor of 5 and relative coverage by 26%, and improves GPT-4 relative coverage by 105% over baseline prompting.

  • Problem

    Generating high-coverage tests for complex focal methods challenges both SBST and LLM-based approaches.

    Source: §1 · p. 2

  • Method

    SymPrompt generates tests through a structured sequence of code-aware prompts that use focal-method information to target individual execution paths.

    Source: §1 · p. 3

  • Results

    SymPrompt improves CodeGen2's correct-test-generation ratio by a factor of 5 and relative coverage by 26%, and GPT-4 relative coverage by 105% over baseline prompting.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The study concludes that structured, code-aware prompting can improve comprehensive LLM-generated test suites in the evaluated setting.

    Source: §7 · p. 19

    The evaluation assumes focal-method implementations are correct, so generated tests may not uncover implementation bugs and may be unsuitable when implementations are unfinished.

    Source: §5 · p. 17

Abstract

from arXiv · show

Testing plays a pivotal role in ensuring software quality, yet conventional Search Based Software Testing (SBST) methods often struggle with complex software units, achieving suboptimal test coverage. Recent works using large language models (LLMs) for test generation have focused on improving generation quality through optimizing the test generation context and correcting errors in model outputs, but use fixed prompting strategies that prompt the model to generate tests without additional guidance. As a result LLM-generated testsuites still suffer from low coverage. In this paper, we present SymPrompt, a code-aware prompting strategy for LLMs in test generation. SymPrompt's approach is based on recent work that demonstrates LLMs can solve more complex logical problems when prompted to reason about the problem in a multi-step fashion. We apply this methodology to test generation by deconstructing the testsuite generation process into a multi-stage sequence, each of which is driven by a specific prompt aligned with the execution paths of the method under test, and exposing relevant type and dependency focal context to the model. Our approach enables pretrained LLMs to generate more complete test cases without any additional training. We implement SymPrompt using the TreeSitter parsing framework and evaluate on a benchmark challenging methods from open source Python projects. SymPrompt enhances correct test generations by a factor of 5 and bolsters relative coverage by 26% for CodeGen2. Notably, when applied to GPT-4, SymPrompt improves coverage by over 2x compared to baseline prompting strategies.

1 INTRODUCTION

SymPrompt decomposes test-suite generation into path-specific prompts informed by static code analysis and focal-method context. On 897 challenging Python methods, it improved test-generation correctness and relative coverage with CodeGen2, and improved relative coverage with GPT-4 over baseline prompting.

Motivation

  • In regression testing, the current code is assumed correct; generated tests are intended to detect bugs introduced by later development.Manual test writing can take more than 15% of development time on average.Source: §1 · pp. 1–2
  • SBST can miss branches requiring specific inputs, while LLMs often generate tests for easy branches and common scenarios, leaving complex conditions uncovered.The paper illustrates the contrast with filesystem-object paths: random generation misses specialized device strings, while zero-shot LLM tests favor files and directories.Source: §1 · p. 2, Figure 1 · p. 4

Approach

  • SymPrompt gathers approximate path constraints and return values, focal-method types and dependencies, then iteratively prompts for a test input per execution path, appending earlier tests to later prompts.The approach is implemented for Python using TreeSitter and integrates with open-source transformers and GPT models.Source: §1 · p. 3
  • Rather than resolve complex dependencies or use a solver, SymPrompt retains approximate constraints, targets paths with unique branching conditions, and relies on the LLM to produce inputs.Static analysis identifies covered and uncovered branches so the LLM can focus on generating the next test case.Source: §1 · p. 3

Results

  • On 897 challenging Python focal methods, SymPrompt improved correct test generations by a factor of 5 and relative coverage by 26% with CodeGen2, and relative coverage by 105% with GPT-4 over baseline prompting.The GPT-4 relative-coverage improvement is also described as more than 2×.Source: §1 · p. 3

2 WORKING EXAMPLE

The working example shows how SymPrompt combines path constraints with type- and dependency-aware context to target branches missed by Pynguin and standard LLM prompting. For `exists_as`, path-specific prompts helped CodeGen2 generate correct tests for uncommon filesystem-object cases.

SBST-based Test Generation

  • Pynguin generated directory and nonexistent-device inputs but missed other use cases without special configuration, and its tests were less maintainable because they diverged from developer patterns [36].The example attributes this limitation to Pynguin’s heuristic-based random input generation.Source: §2 · pp. 4–5

LLM-based Test Generation

  • Standard prompting used by Lemieux et al. let CodeGen2 test common directory and file cases, but not less common paths such as block devices, even across sampled generations.The example notes that LLMs can infer plausible inputs for some branches, such as `./tmp` for `path.is_file()`, without seeing dependency definitions.Source: §2 · p. 5

Collecting Approximate Path Constraints

  • SymPrompt traverses the method AST in preorder, records branch conditions and their inversions along each path, and prompts for the corresponding return value.Each recorded path and return value becomes a prompt targeting a test case for that execution path.Source: Figure 2 · p. 5, §2 · p. 6

Context Construction

  • Each prompt includes the focal method signature and relevant types and dependencies, such as `_PATH` and `normalize_path` for `exists_as`.This context helps the model make correct method calls, initialize input types, and use relevant external APIs.Source: §2 · p. 6, Figure 3 · p. 6

Test Generation

  • CodeGen2 generated correct inputs for three of four prompted device and special-file cases, more than doubling the tested branches covered in its testsuite.The targeted cases were `is_block_device()`, `is_char_device()`, `is_fifo()`, and `is_socket()`.Source: §2 · p. 6

SymPrompt

  • Path constraints and type-aware focal context combine meaningful input derivation with coverage-aware prompting for a set of high-coverage tests.The approach uses LLMs to initialize complex input types, draw on relevant external APIs, and address difficult branch constraints.Source: §2 · p. 6

3 METHODOLOGY

SymPrompt turns regression test generation into a static, three-stage process: collecting path constraints, capturing relevant code context, and prompting an LLM to generate tests for specific execution paths.

3.1 Overview

  • SymPrompt approximates symbolic path constraints, focuses on paths with unique branching conditions, and uses an LLM rather than a solver to generate test cases.It leaves unresolved conditions approximate and performs the process statically, without executing the focal method.Source: §3.1 · p. 7

3.2 Step-I: Collecting Approximate Path Constraints

  • Step I traverses the focal method’s AST preorder, collecting branch and loop constraints per path and recording constraints and return expressions for paths ending in a return.The path set is minimized during traversal to retain paths that increase overall branch coverage.Source: §3.2.1 · p. 7

3.2.2 Path Minimization.

  • For three sequential if-else branches, path minimization reduces 8 possible paths to at most 4, each covering a unique branch condition.The algorithm is applied after collecting constraints from each if or while statement during AST traversal.Source: Figure 5 · p. 8, §3.2.2 · p. 9

3.2.1 Path Constraint Collection.

  • To limit exponential path growth and redundant tests, SymPrompt retains a linearly independent subset of basis path constraints.Path minimization keeps paths that add branch constraints not already represented in the linearized constraint set.Source: §3.2.2 · pp. 8–9

3.3 Step-II. Context Construction

  • Step II selectively exposes type definitions and dependencies, including imports, globals, used types, class initialization, called class methods, and the focal method.SymPrompt constructs test execution context from the focal context to override model-generated imports and type definitions.Source: §3.3 · pp. 9–10, Figure 6 · p. 9

3.4 Step-III. Test Generation

  • Step III prompts for each path specify the method signature, path constraints, and expected return behavior, then iteratively generate tests while including earlier tests in subsequent prompts.Generated code is trimmed from the end until it parses, and duplicate model-generated imports are removed from the execution context.Source: §3.4 · pp. 10–11, Figure 7 · p. 10

4 EVALUATION

The evaluation tests SymPrompt on hard-to-test focal methods from open-source projects, measuring generated-test correctness and coverage with CodeGen2 and GPT-4. Across four research questions, it examines overall performance, unseen training data, design choices, and large-model performance.

4 EVALUATION

  • Evaluation covered 897 focal methods from 26 projects selected for Pynguin’s failure to reach full coverage, using CodeGen2 and GPT-4.Metrics included Pass@1, FM Call@1, Correct@1, and line and branch coverage.Source: §4 · p. 11

4.1 RQ1: Performance Improvement

  • SymPrompt produced 5× more correct generations and 10% higher line coverage and 4% higher branch coverage than baseline prompts.Its pass rate improved by nearly 4×; baseline generations called the focal method in 34% of cases but were correct in only 3%.Source: §4.1 · p. 12, Table 1 · p. 13
  • Filtered SymPrompt achieved 77% line coverage and 66% branch coverage, compared with Pynguin’s 72% and 64%; case studies show improved path coverage and correctly typed inputs.In one case, SymPrompt called the focal method correctly and tested both filename paths, while baseline prompting used an invalid parameter and missed one path; type hints in comments also helped the model avoid errors that stopped SBST tests.Source: §4.1 · pp. 12–13

4.2 RQ2: Training Data Memorization

  • On three projects excluded from CodeGen2’s training data, SymPrompt achieved 4× higher correct-generation rates and 12.5% higher line coverage than baseline prompts.Both prompting approaches had lower correct-generation rates and coverage on this subset than on the full benchmark.Source: Table 1 · p. 13, §4.2 · p. 14

4.4 RQ4: Large Model Performance Impact

  • With GPT-4, SymPrompt improved Correct@1 by 178% and raised average coverage from 36% to 74% over the two-stage baseline.Calling context helped prevent import and undefined-object errors; 6 of 10 manually examined tests followed their specified paths, while deeply nested branches and exception paths caused misses.Source: Table 2 · p. 15, §4.4 · p. 16

4.3 RQ3: Design Choices

  • On 100 sampled methods, unablated SymPrompt achieved 26% correct generations and 53% coverage; removing path constraints reduced pass rate, focal-method calls, and coverage more than removing calling context.The findings indicate that calling context contributes, but path-constraint guidance is the stronger component of the gains.Source: §4.3 · p. 15, Table 2 · p. 15

5 THREATS TO VALIDITY & DISCUSSION

The study’s findings may not generalize beyond the evaluated Python projects and CodeGen2/GPT-4, and its metrics and regression-testing assumption limit what the evaluation can establish. The authors consider memorization risk minimal but acknowledge it remains possible.

Model and Benchmark Validity; Memorization Validity

  • The evaluation covers open-source Python projects and CodeGen2/GPT-4, limiting generalizability, while possible CodeGen2 exposure to focal or similar methods remains a memorization threat.The authors expect findings to hold elsewhere because the models are state of the art and the methods are not Python-specific; they consider memorization risk minimal because the models were not explicitly trained for test generation.Source: §5 · p. 16

Metric Validity

  • Pass@1, FM Call@1, Correct@1, and Line Coverage may not capture the full complexity or usefulness of generated tests.Source: §5 · p. 17

Test Generation in a Regression Setting

  • Assuming the focal method implementation is correct means generated tests may not uncover implementation bugs, and the approach is limited before implementations are finalized.The authors note that unfinished implementations are common in continuous development environments.Source: §5 · p. 17

6 RELATED WORK

Related work spans coverage-guided and symbolic testing, LLM test generation, ChatGPT-based context and repair methods, and hybrid SBST–LLM systems. SymPrompt is positioned as code-aware prompting that guides test generation along execution paths.

Search Based Software Testing (SBST) and Symbolic Approaches

  • SymPrompt relates to coverage-guided SBST and concolic testing, but guides the LLM along execution paths using focal-method source rather than symbolic constraint solving.Source: §6 · p. 17

LLM Regression Test generation

  • Prior LLM test-generation work includes fine-tuning on method-test pairs [36], one-shot Codex tests [7], postprocessing [3], project-suite fine-tuning [19], and mutation-guided generation [11].Source: §6 · pp. 17–19

ChatGPT test generation

  • Testpilot [32], ChatUnitTest [40], and ChatTester [43] construct focal context and/or repair generations using error feedback, whereas SymPrompt guides tests across execution paths.Source: §6 · p. 19

Hybrid SBST-LLM regression test generation

  • Codamosa combines Pynguin with iterative Codex generation for low-coverage methods, while SymPrompt focuses on improving LLM-generated test suites.Source: §6 · p. 19

7 CONCLUSION

SymPrompt structures LLM test-suite generation as a code-aware sequence of prompts, improving test-generation correctness and coverage. With a path-constraint instruction, GPT-4 generates its own path-constraint prompts, raising test-suite coverage 2× over recent prior-work prompting strategies.

7 CONCLUSION

  • With a path-constraint analysis instruction, GPT-4 generates its own path-constraint prompts, improving test-suite coverage by 2× over recent prior-work prompting strategies.Source: §7 · p. 19
  • SymPrompt significantly improves comprehensive test-suite generation with CodeGen2, including the ratio of correct test generations and coverage.Source: §7 · p. 19
  • SymPrompt decomposes test-suite generation into a structured sequence of code-aware prompts.Source: §7 · p. 19

Related papers

Loading 2402.00097v2…