Source-linked AI summary
HITS: High-coverage LLM-based Unit Test Generation via Method Slicing
Zejun Wang, Kaibo Liu, Ge Li, Zhi Jin
TL;DR
Complex focal methods remain difficult for automatic test generation, and ChatUniTest coverage drops on them. HITS decomposes methods into slices and generates tests slice by slice; the authors report higher line and branch coverage than the compared methods, while noting limitations in dataset size and tested LLMs.
Problem
Automatic test-generation tools perform poorly on complex methods, and ChatUniTest’s line and branch coverage significantly drops on them.
Source: §1 · p. 1Method
HITS decomposes focal methods into slices and uses an LLM to generate tests separately for each slice.
Source: §1 · pp. 1–2Results
HITS overperforms all baseline methods in line and branch coverage by 10 to 20 percent.
Source: §1 · p. 2Takeaways & Limitations
The ablation studies confirm the contribution of generating tests for a slice to HITS’s effectiveness.
Source: §6 · p. 10The evaluation uses a limited dataset and gpt-3.5-turbo, so generalization to other datasets and LLMs remains future work.
Source: §5.4 · p. 10
Abstract
from arXiv · showhide
Large language models (LLMs) have behaved well in generating unit tests for Java projects. However, the performance for covering the complex focal methods within the projects is poor. Complex methods comprise many conditions and loops, requiring the test cases to be various enough to cover all lines and branches. However, existing test generation methods with LLMs provide the whole method-to-test to the LLM without assistance on input analysis. The LLM has difficulty inferring the test inputs to cover all conditions, resulting in missing lines and branches. To tackle the problem, we propose decomposing the focal methods into slices and asking the LLM to generate test cases slice by slice. Our method simplifies the analysis scope, making it easier for the LLM to cover more lines and branches in each slice. We build a dataset comprising complex focal methods collected from the projects used by existing state-of-the-art approaches. Our experiment results show that our method significantly outperforms current test case generation methods with LLMs and the typical SBST method Evosuite regarding both line and branch coverage scores.
1 INTRODUCTION
HITS addresses poor coverage on complex methods by decomposing them into code slices and generating tests for each slice. Its reported evaluation finds higher line and branch coverage than the compared baselines.
1 INTRODUCTION
- Complex methods, defined as having cyclomatic complexity greater than 10, remain challenging for automatic test generation.The paper notes that existing tools perform poorly on these methods.Source: §1 · p. 1
- ChatUniTest’s line and branch coverage scores significantly drop when it is applied to complex methods.The evaluation distinguishes complex methods from all methods in the tested projects.Source: §1 · p. 1
- HITS decomposes a focal method into slices and generates tests separately for them, combining those tests into a complete suite.The paper argues that per-slice analysis reduces the branches and execution paths the LLM must consider at a time.Source: §1 · pp. 1–2
- HITS retrieves method dependencies, uses chain-of-thought prompting for sequential decomposition and test generation, and applies Self-Debug to failed tests.The approach is designed specifically for complex methods.Source: §1 · p. 2
- The reported results show that HITS overperforms all baseline methods in both line and branch coverage scores, with gains ranging from 10 to 20 percent.Source: §1 · p. 2
- The paper contributes a slice-by-slice test-generation approach, the HITS tool for complex Java methods, and an evaluation of its effectiveness.The evaluation compares HITS with existing methods on complex focal methods.Source: §1 · p. 2
2 RELATED WORK
Prior unit-test-generation research includes search-based, symbolic-execution, deep-learning, and LLM-based approaches, each with stated trade-offs. Program decomposition has also been used to narrow analysis, and this paper applies it to test generation for complex methods.
2.1 Unit Test Generation
- Search-based testing faces a broad search space on complex methods, symbolic execution faces path explosion, and deep-learning-generated code may not compile or execute.Deep learning can generate more varied inputs than SBST by learning from human-written tests.Source: §2.1 · p. 3
- LLM-based test-generation work includes ChatTester, ChatUniTest, Codamosa, and SymPrompt; this paper targets complex Java methods where prior methods have low coverage.The cited approaches include Java test-generation tools and an approach that invokes an LLM when SBST fails.Source: §2.1 · p. 3
2.2 Program Decomposition
- Program decomposition narrows analysis by dividing programs into slices, and prior work has used it for program analysis and bug-report matching.This paper uses decomposition to simplify generating tests for a complex method as several simpler code slices.Source: §2.2 · p. 3
3 METHODOLOGY
HITS decomposes focal methods into slices, generates slice-specific tests, and then executes and repairs the resulting suite. Its prompts supply retrieved context and staged instructions for decomposition and test generation, followed by formatting, isolation, and rule-based fixes.
3.1 Overview
- HITS first generates tests for slices to form an initial suite, then executes the tests and repairs those that are not executable.The intended slice tests cover all lines and branches.Source: §3.1 · p. 3
3.2 Decompose Method into Slices
- HITS retrieves dependent classes’ fields and method declarations, invoked-method bodies, and available Javadocs as context for the prompts.The focal method’s source code and this context information form the prompt prefix.Source: §3.2 · p. 3
- Chain-of-thought prompting decomposes a focal method by problem-solving steps, with each slice corresponding to one step.The LLM summarizes the method, recites invoked non-local variables and methods, then produces JSON descriptions and corresponding code segments for the slices.Source: §3.2 · p. 3
3.3 Generate Tests for a Slice
- For each slice, the LLM recites inputs and invoked methods, enumerates feasible input scenarios, infers an execution environment, and generates a test class.Manually crafted examples guide scenario analysis and environment inference; prompts also include Java and JUnit 5 guidance.Source: §3.3 · p. 4
- HITS applies Self-Debug to failed tests using the test and JVM error report, with a staged plan to diagnose, describe, and fix the error.Few-shot examples cover common compilation and execution failures, but not all causes.Source: §3.3 · p. 4
3.4 Post-Process of LLM’s Generation
- Post-processing formats LLM output for extraction, isolates independent test cases, and applies rule-based fixes such as balancing brackets and importing common packages.The paper presents these post-processing techniques as effective and potentially generalizable to other LLM-based work.Source: §3.4 · p. 5
4 EXPERIMENTS
Across ten projects, the experiments compare HITS with LLM-based methods and EvoSuite on complex Java methods, assessing coverage, test execution, and component contributions. HITS leads the average coverage comparison, while the pass-rate and ablation analyses qualify how that performance is achieved.
4.1 Experiment Setup
- The ten-project dataset contains complex methods from projects selected for prior-work relevance, method counts, and project variety; six projects predate the GPT training-data cutoff and four postdate it.Large projects were excluded because of the token budget, and the experiments use gpt-turbo-3.5-0125 with comparable test-suite generation settings for the baselines.Source: §4.1 · p. 5
4.2 RQ1: Coverage Scores Comparison
- HITS outperforms all LLM-based baselines and EvoSuite on average coverage, ranking first on seven projects and second on the remaining three.It performs below LLM baselines on Commons-collections, while EvoSuite leads some individual projects, including Datafaker.Source: §4.2 · p. 6, Table 4 · p. 7, Table 5 · p. 7
4.3 RQ2: Execution Correctness Check
- HITS has higher average pass rates than the LLM-based baselines, but a high pass rate does not guarantee high coverage.Low pass rates are associated with low coverage; executable tests can still repeat coverage of the same branches and lines.Source: §4.3 · pp. 6–7, Table 6 · p. 8
4.4 RQ3: Ablation Study
- Slicing, prompt instructions and examples, and post-processing each contribute to average coverage improvements over ChatUniTest.The slicing workflow contributes more than instructions and examples in the reported comparison. The analysis also notes that results vary across projects and treats branches equally, leaving prioritization of branches with more lines for future work.Source: §4.4 · pp. 7–8
5 DISCUSSION
The discussion illustrates how slice-focused generation addresses a missed loop block, examines prompt refinement and non-executable tests, and identifies dataset and model generalizability as threats. The case study’s improvement is specific to the shown Parser.parse() example.
5.1 Case Study
- In the Parser.parse() case, HITS partially covers the block missed by ChatUniTest by generating tests focused on Slice #2.ChatUniTest’s suite did not cover the while-loop block; HITS inferred input conditions, including a Java-recognized short option, that reached it.Source: §5.1 · p. 8
5.2 Methodology for Optimizing Prompts
- Prompt Self-refine helped the LLM follow the required output format and perform the instructed analysis.The authors report that early prompts sometimes produced invalid JSON, zero slices, empty code, or irrelevant analysis.Source: §5.2 · pp. 8, 10
5.3 Non-executable Test Distribution
- HITS and SymPrompt have a higher proportion of runtime errors among non-executable tests than ChatUniTest and ChatTester.The authors attribute the pattern to more demanding prompts and complicated focal-class constructions, which can produce improper test structures.Source: §5.3 · p. 10
- The authors identify more effective LLM instruction-following as future work.They also identify a limited dataset and generalization to other LLMs as threats; the evaluation excludes projects with more than 30 complex methods and uses gpt-3.5-turbo.Source: §5.3 · p. 10, §5.4 · p. 10
5.4 Threats to Validity
- The evaluation’s validity is constrained by a limited dataset and limited generalization to other LLMs.Projects with more than 30 complex methods were excluded, and gpt-3.5-turbo was selected partly to balance performance and API cost.Source: §5.4 · p. 10
6 CONCLUSION
The conclusion presents slice-by-slice test generation as a response to low coverage on complex methods. It reports that HITS outperformed the compared LLM-based methods and Evosuite, and that ablations support the contribution of generating tests for slices.
6 CONCLUSION
- HITS outperformed state-of-the-art LLM-based test-generation tools and Evosuite in the reported evaluation.The authors state that ablation studies also confirm a contribution from generating tests for a slice.Source: §6 · p. 10
- The paper proposes decomposing a focal method into slices and generating tests for each slice using a divide-and-conquer approach.HITS implements this approach for complex Java methods using an LLM.Source: §6 · p. 10
- The conclusion identifies low coverage on complex methods as the problem addressed by HITS.Source: §6 · p. 10