Source-linked AI summary
Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM
Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, Baishakhi Ray
TL;DR
Existing SBST and fixed-prompt LLM methods can produce low-coverage tests for complex software units. SymPrompt decomposes testsuite generation into path-specific, code-aware prompts using static path information and type/dependency context. It improves correct CodeGen2 generations fivefold and relative coverage by 26%, while GPT-4 achieves a 105% relative coverage improvement over baseline prompting.
Problem
SBST and fixed-prompt LLM test generation struggle to achieve high coverage for complex methods and uncommon execution paths.
Method
SymPrompt uses a multi-stage, code-aware prompting strategy that targets execution paths with approximate constraints, return values, and relevant type and dependency context.
Results
5× more correct test generations and 26% higher relative coverage are reported for CodeGen2, while GPT-4 achieves 105% higher relative coverage over baseline prompting.
Takeaways & Limitations
Structured path-focused prompts enable pretrained LLMs to generate more comprehensive testsuites without additional training.
Takeaways & Limitations
The evaluation covers open-source Python projects and CodeGen2 and GPT-4, limiting generalizability to other languages or models.
Abstract
from arXiv · showhide
Testing plays a pivotal role in ensuring software quality, yet conventional Search Based Software Testing (SBST) methods often struggle with complex software units, achieving suboptimal test coverage. Recent works using large language models (LLMs) for test generation have focused on improving generation quality through optimizing the test generation context and correcting errors in model outputs, but use fixed prompting strategies that prompt the model to generate tests without additional guidance. As a result LLM-generated testsuites still suffer from low coverage. In this paper, we present SymPrompt, a code-aware prompting strategy for LLMs in test generation. SymPrompt's approach is based on recent work that demonstrates LLMs can solve more complex logical problems when prompted to reason about the problem in a multi-step fashion. We apply this methodology to test generation by deconstructing the testsuite generation process into a multi-stage sequence, each of which is driven by a specific prompt aligned with the execution paths of the method under test, and exposing relevant type and dependency focal context to the model. Our approach enables pretrained LLMs to generate more complete test cases without any additional training. We implement SymPrompt using the TreeSitter parsing framework and evaluate on a benchmark challenging methods from open source Python projects. SymPrompt enhances correct test generations by a factor of 5 and bolsters relative coverage by 26% for CodeGen2. Notably, when applied to GPT-4, SymPrompt improves coverage by over 2x compared to baseline prompting strategies.
1 INTRODUCTION
SymPrompt addresses low coverage in automated test generation by decomposing testsuite generation into path-focused prompts enriched with code context. Across CodeGen2 and GPT-4 evaluations, it improves correct generations and relative coverage.
- Motivation: Automated testing matters for software quality, but SBST and fixed-prompt LLM approaches struggle to achieve high coverage on complex methods.SBST can miss branch conditions requiring specific values or states, while LLMs tend to generate tests for common use cases.
- Approach: SymPrompt collects approximate path constraints, return values, argument types, dependencies, and code context before constructing prompts for each path.Its static analysis approximates unresolved behavior instead of fully resolving complex types and dependencies.
- Results: 5× more correct test generations and 26% higher relative coverage are reported for CodeGen2.These results are reported on the benchmark of methods challenging for existing SBST approaches.
- Results: 105% relative coverage improvement is reported for GPT-4 over baseline prompting.This corresponds to more than a twofold relative improvement in the cited comparison.
- Approach: SymPrompt introduces Path Constraint Prompting, decomposing testsuite generation into separate test inputs targeting each execution path.The strategy uses a multi-stage procedure rather than directly asking an LLM to generate an entire testsuite.
- Evaluation: The implementation evaluates 897 challenging focal methods from widely used Python projects and integrates with open-source transformers and GPT models.The prototype uses TreeSitter and includes integrations for CodeGen2 and GPT models.
2 WORKING EXAMPLE
The working example shows how SymPrompt converts a branching filesystem utility method into path-specific prompts. It combines AST-derived constraints with type and dependency context so the LLM can generate tests for uncommon branches.
- Challenge: The exists_as method is difficult for both SBST and standard LLM prompting because its branches require specific filesystem-object inputs.Common prompting tends to cover files and directories while missing less common cases such as block devices and sockets.
- Path Constraint Collection: SymPrompt traverses the focal method’s AST in preorder and records branch constraints and return values for each execution path.When a return statement is reached, the collected conditions needed to reach it are paired with the returned value.
- Path Constraint Collection: Each path prompt specifies the constraints and return behavior required for the target test case.This directs generation toward paths that ordinary prompting may overlook.
- Context Construction: SymPrompt adds the focal method signature, input-type definitions, and relevant external method dependencies to each prompt.For exists_as, the context includes _PATH and normalize_path.
- Test Generation: CodeGen2 generated correct inputs for three of four uncommon filesystem cases when specifically prompted.The resulting tests more than doubled the number of tested branches in the example testsuite.
- Test Generation: Path constraint prompting with type-aware context combines coverage-aware path targeting with the LLM’s ability to derive meaningful inputs from code context.The approach targets complex input initialization, external APIs, and difficult branch constraints.
3 METHODOLOGY
SymPrompt decomposes test generation into static path analysis, relevant context construction, and iterative path-specific prompting. It minimizes redundant paths while exposing constraints, return behavior, types, and dependencies to guide the LLM toward broader coverage.
- Motivation: Traditional symbolic analysis struggles with complex data types, external dependencies, and computational overhead from numerous nested branches.These limitations motivate approximate constraints and path minimization.
- Path Constraint Collection: Path minimization retains paths with unique branch conditions instead of enumerating every possible execution path.The method collects constraints through AST traversal and reduces redundant paths that add no line or branch coverage.
- Framework: SymPrompt statically collects approximate path constraints and return expressions, captures relevant code context, then generates path-specific prompts for the LLM.The three-step process is performed without executing the focal method.
- Context Construction: Generation context includes imports and globals, type definitions, focal-class information, called methods, and the focal method itself.These definitions help the model attend to relevant dependencies and provide the execution context for generated tests.
- Path Minimization: For three sequential if-else branches, path minimization reduces 8 possible paths to at most 4 paths.The retained paths comprise one initial path plus one additional path for each branch.
- Prompt Construction: Each path prompt combines the focal method signature, required path constraints, and expected return behavior to guide inputs and assertions.Prompts are iteratively passed to the model, with generated tests included in subsequent generations.
4 EVALUATION
The evaluation tests SymPrompt on challenging Python methods using multiple baselines, metrics, and ablations. Across CodeGen2 and GPT-4 settings, it improves correctness and coverage, while results also identify the roles of path constraints and calling context.
- Experiment Setting: The evaluation covers 897 focal methods from 26 open-source projects and measures pass rate, focal-method calls, correct tests, and line and branch coverage.CodeGen2 experiments use 10 generations per focal method and compare SymPrompt with Pynguin, baseline prompts, and No-Op tests.
- RQ1: Performance Improvement: SymPrompt improves line coverage by 10% and branch coverage by 4% over baseline prompts.Relative to No-Op tests, baseline prompts improve absolute coverage by 5%, whereas SymPrompt improves it by 15%.
- RQ1: Performance Improvement: 5× more correct test generations and 10% higher coverage are reported for SymPrompt than the baseline test-completion prompt on CodeGen2.The result concerns tests that both call the focal method and pass.
- RQ1: Performance Improvement: CodeGen2 SymPrompt reaches 77% line coverage and 66% branch coverage when test suites without working tests are filtered.Unfiltered SymPrompt remains below Pynguin’s 72% line and 64% branch coverage averages in the benchmark.
- RQ2: Training Data Memorization: On projects excluded from CodeGen2’s training data, SymPrompt produces more correct tests and 12.5% higher line coverage than the LLM baseline.The reported correct-generation improvement is approximately 4×.
- RQ3: Design Choices: Ablating calling context reduces pass and focal-method call rates, but ablating path-constraint prompting causes larger performance losses.The ablation results identify symbolic path guidance as crucial to SymPrompt’s improvements in correct generations and coverage.
- RQ4: Large Model Performance Impact: With GPT-4, calling context reduces hallucinated imports and undefined objects, while path constraints improve coverage when combined with that context.Path-constraint prompts alone do not significantly outperform the baseline describe-and-generate prompts.
5 THREATS TO VALIDITY & DISCUSSION
The evaluation is bounded by its Python-project and model choices, by metric coverage, and by its regression-setting assumption. The paper also illustrates how GPT-4-generated path descriptions are transformed into test-generation prompts.
- Model and Benchmark Validity: Evaluations use open-source Python projects and CodeGen2 and GPT-4, restricting generalizability to other languages and models.The authors expect findings to transfer because the methods are not Python-specific, but this remains a validity boundary.
- GPT-4 can generate precise execution-path descriptions, which SymPrompt extracts with a markdown parser and embeds as test docstrings.
- Metric Validity: Pass@1, FM Call@1, Correct@1, and Line Coverage may not capture the full complexity or usefulness of generated tests.
- Test Generation in a Regression Setting: Regression-setting evaluation assumes the focal method is correct, so generated tests may not uncover implementation bugs.This assumption is limiting when the focal method is not yet finalized, as in continuous development environments.
6 RELATED WORK
SymPrompt relates to coverage-guided, symbolic, LLM-based, and hybrid test-generation approaches. Unlike prior fixed-prompt methods, it guides LLMs toward tests following specified execution paths.
- Search Based Software Testing and Symbolic Approaches: SymPrompt formulates test generation as an LLM constraint-solving problem guided toward a specific execution path.It is conceptually related to coverage-driven SBST and concolic execution, but uses an LLM instead of symbolic constraint solving.
- Search Based Software Testing and Symbolic Approaches: Prior approaches include SBST, randomized generation, concolic execution, and formal-specification-based testing across Java, Python, and C.
- Hybrid LLM-SBST Test Generation: GPT generations with path constraint prompts are presented as a case study related to SymPrompt's execution-path guidance.
- LLM Test Generation: Mutation-guided and fine-tuned LLM approaches improve test quality or bug detection, while SymPrompt focuses on code-aware prompts for high coverage.
- ChatGPT Test Generation: Testpilot and ChatUnitTest use iterative or adaptive context strategies for ChatGPT-based test generation.
7 CONCLUSION
The paper introduces SymPrompt as a structured, code-aware prompting strategy for LLM test generation. It reports improved correctness and coverage with CodeGen2, and more than doubled GPT-4 coverage over prior prompting strategies.
- SymPrompt decomposes test-suite generation into a structured, code-aware sequence of prompts.The approach targets comprehensive test suites through staged prompting.
- GPT-4-generated path constraint prompts improve test-suite coverage by 2× over prompting strategies from recent prior work.