CoverUp: Effective High Coverage Test Generation for Python

1 Introduction

• Automated test generation addresses labor-intensive manual test creation and can help uncover bugs missed by manually written tests.Generated tests support regression testing by seeking to prevent future bugs as software changes.Source: §1 · p. 1
• 80% per-module median line+branch coverage versus CodaMosa’s 47%, and 89% overall line+branch coverage versus MuTAP’s 77%.The evaluation used challenging code derived from open-source Python projects.Source: §1 · p. 2
• CoverUp combines coverage analysis and LLMs to generate high-coverage Python regression tests, with an ablation study finding contributions beyond the LLM alone.Its prompts target uncovered code and use additional source context and feedback when tests fail or do not improve coverage.Source: §1 · pp. 1–2

2 Related Work

• Prior automated test-generation research includes specification-based, random, feedback-directed random, concolic, search-based, and transformer-based approaches.The related-work discussion focuses on approaches most closely related to CoverUp.Source: §2 · p. 2

SBST approaches.

• CodaMosa tracks Pynguin’s search progress and prompts an LLM to reseed the search when it stalls.Pynguin’s genetic search can get stuck when mutations repeatedly follow the same execution paths.Source: SBST approaches · p. 2

LLM-based Approaches.

• Nearly 40% of CoverUp’s successful test generation is attributed to iterative dialogue after build failures, failing tests, or lack of coverage.CoverUp also focuses prompts on uncovered code and checks whether each new test increases coverage.Source: LLM-based Approaches · pp. 2–3
• Unlike Vikram et al. [Vikram et al. 2023] and TiCoder [Lahiri et al. 2022], CoverUp prompts from source code and coverage rather than API documentation or intended-functionality descriptions.CoverUp accepts property-based tests if generated, but does not explicitly request them; TiCoder generates tests to clarify and formalize user intent.Source: LLM-based Approaches · p. 3
• Unlike TestPilot [Schäfer et al. 2024] and ChatUniTest [Chen et al. 2024], CoverUp requests coverage improvements and uses a tool function for additional context.Both compared systems check or repair generated tests, but neither uses coverage to identify uncovered lines or branches and seek improved coverage.Source: LLM-based Approaches · p. 3

Concurrent Work Using LLM-based Approaches.

• Fuzz4All [Xia et al. 2024] generates test inputs from documentation or examples and relies on a user-provided oracle, while CoverUp targets coverage using source code and measurements.Fuzz4All repeatedly mutates prompts after initial input distillation and prompt selection.Source: Concurrent Work Using LLM-based… · p. 3
• 74% line coverage is SymPrompt’s reported result; CoverUp obtains 82% median per-module line coverage and 64% overall line coverage on a different, 5x larger suite.The paper calls this a coarse comparison because SymPrompt’s exact evaluated functions were undisclosed.Source: Concurrent Work Using LLM-based… · p. 3
• Like TestGen-LLM [Alshahwan et al. 2024], CoverUp rejects failing or non-improving tests, but continues dialogue to request repairs and uses coverage prompts and tool-based context discovery.TestGen-LLM presents remaining tests to a developer for approval during code review.Source: Concurrent Work Using LLM-based… · p. 4
• Unlike MuTAP [Dakhel et al. 2024], CoverUp uses coverage information in prompts and supports context discovery; its implementation can be applied to any Python package.MuTAP uses mutation testing to prompt for assertions for surviving mutants, and its hardcoded evaluation datasets hamper evaluation on other code.Source: Concurrent Work Using LLM-based… · p. 4

3 Approach

• CoverUp measures existing coverage, targets uncovered code segments, prompts an LLM, verifies generated tests, and continues the dialogue when tests fail or do not increase coverage.The process can be repeated so later iterations refocus on segments still lacking coverage.Source: §3 · pp. 4–5

3.1 Code Segmentation

• Code segments center on uncovered lines or branch endpoints; oversized classes are narrowed to smaller AST objects while retaining class context.CoverUp keeps comments to aid code comprehension, trading shorter excerpts for generation quality.Source: §3.1 · p. 5

3.2 Initial Prompting

• Initial prompts identify missing coverage, request pytest tests, add imports, and tag uncovered lines and branches; prompts that only give a starting line do not improve coverage.The prompt also discourages disruptive top-level pytest calls and asks tests to avoid state pollution.Source: §3.2 · pp. 6–8

3.3 Verification and Continued Chat

• CoverUp saves tests only when they pass and increase coverage; otherwise, it continues the chat with the problem and a request for improvements.Tests are executed and coverage measured again; SlipCover's near-zero overhead avoids the up-to-260% overhead reported for coverage.py.Source: §3.3 · p. 8

3.4 Tool Functions

• The get_info tool lets the LLM request definitions for names in the excerpt, providing context such as constructor usage that can prevent incorrect test calls.CoverUp searches source definitions and abbreviates classes or modules to avoid overly long responses.Source: §3.4 · pp. 8–10

3.5 Integration Check

• To address state pollution, CoverUp can isolate tests, disable polluting tests, or disable failing tests.Isolation is the default used in the empirical evaluation; disabling failing tests can lose coverage, disable tests unnecessarily, and still leave a polluting test enabled.Source: §3.5 · p. 11

3.6 Handling Flaky Tests

• CoverUp reruns each new test several times to make flaky failures more likely to surface, then asks the LLM to correct a failing test.One example fails when random blood types are O+ or O-, which the LLM mistakenly assumes use zero rather than the letter O.Source: §3.6 · p. 12

3.7 Other Technical Challenges

• Asynchronous prompting and verification yielded a 500x speedup over serial execution during evaluation; the implementation also handles API limits and checkpoints progress.Requests are rate-limited with a leaky-bucket scheme, and checkpoint files allow interrupted runs to resume.Source: §3.7 · pp. 12–13

4 Evaluation

• The evaluation asks five questions covering state-of-the-art comparisons, direct LLM prompting, continued dialogues, cost, and component importance.The questions are addressed in Sections 4.2–4.6.Source: §4 · p. 13

4.1 Experimental Setup

• The benchmarks comprise CM (425 modules from 35 projects), PY (84 modules), and MT (163 HumanEval-derived functions).PY includes modules on which Pynguin already performs well; MT enables comparison with MuTAP, whose implementation does not support arbitrary Python packages.Source: §4.1 · p. 13
• CM excludes mimesis because random data complicates non-flaky tests, and excludes sanic and thef*ck because required modules are unavailable or conflict with CoverUp.The original CodaMosa evaluation included mimesis, but its generated tests lacked assertions against randomly generated values.Source: §4.1 · p. 13

Baselines.

• CodaMosa is compared in its original Codex configuration and an adapted GPT-4o version intended to control for differences in the LLM.Because Codex is unavailable, the original CodaMosa tests come from its prior evaluation.Source: Baselines · p. 13
• MuTAP comparisons include Codex few-shot and zero-shot versions plus newly generated GPT-4o few-shot and zero-shot versions.The adaptations use OpenAI's chat API and prompts requesting output in the expected format.Source: Baselines · pp. 13–14
• The authors could not evaluate Ryan et al. because the work was unavailable and its authors said it was not publicly available.Source: Baselines · p. 14
• The CoverUp ablation removes coverage cues, computed imports, the extra-context tool, and continued chats for errors or missing coverage.The ablated version otherwise uses a nearly identical prompt.Source: Baselines · p. 14

Metric.

• Line, branch, and combined line-and-branch coverage are computed across benchmark modules, with combined coverage also computed per module.Source: Metric · p. 14
• Both line and branch coverage are necessary because Python exceptions can be thrown without being recorded as branches.Source: Metric · p. 14
• Generated tests run in isolation, and coverage is measured using SlipCover.Source: Metric · p. 14

Execution Environment.

• CodaMosa and CoverUp ran in a Debian 11-based Docker image with Python 3.10.2 on a Linux 5.6 host with 10 Intel i9 cores and 64GB RAM.The image was based on codamosa-docker and modified to install SlipCover and disable its default entrypoint.Source: Execution Environment · p. 14
• Before measurements, benchmark-specific requirements were installed with pip, but installation failures were ignored to replicate CodaMosa’s original conditions.Source: Execution Environment · p. 14
• MuTAP and its generated tests used Python 3.9.12 because its MutPy mutation-testing dependency requires older Python versions.Source: Execution Environment · p. 14

CoverUp options.

• CoverUp used GPT-4o at temperature zero, a default 50-line target segment, and repeated executions to check for flaky tests.It ran first without a pre-existing test suite, then twice more building on earlier runs, and automatically installed missing modules assumed by generated tests.Source: CoverUp options · p. 14

4.2 [RQ1] Comparison to the previous state of the art

• 60% overall line+branch coverage (vs. 49% and 45%) and 80% per-module median line+branch coverage (vs. 55% and 47%) beat both CodaMosa baselines.Per-module improvements over CodaMosa GPT-4o and Codex were statistically significant (p-value 2.0 × 10^-5 for both).Source: Figure 12 · p. 15, §4.2 · p. 15
• 89% overall line+branch coverage (vs. 78% to 10%) exceeded MuTAP baselines, while CoverUp and most baselines reached 100% median per-module coverage.MuTAP (gpt4o few-shot) was an exception, with per-module coverage of 17%, 0%, and 11% across the three metrics.Source: Figure 15 · p. 16, §4.2 · p. 16, Table 2 · p. 18

4.3 [RQ2] How effective is CoverUp compared to simply prompting an LLM for tests?

• 64% line, 49% branch, and 60% line+branch coverage exceeded the LLM-only ablation’s 39%, 19%, and 34%, respectively.CoverUp also exceeded the ablation on per-module medians: 82% vs. 44% line, 67% vs. 7% branch, and 80% vs. 39% line+branch coverage.Source: Figure 16 · p. 17, §4.3 · p. 17

4.5 [RQ4] How does the cost of running CoverUp compare to CodaMosa?

• CoverUp ran about 18 times faster than CodaMosa (4 vs. 71 hours), with higher coverage but 48% more token use.The authors note that the runtime comparison is limited by different execution setups and by CoverUp’s parallelized, asynchronous implementation.Source: Table 3 · p. 18, §4.5 · p. 18

4.4 [RQ3] How effective are CoverUp’s continued dialogues?

• About 40% of successful tests came from continued dialogue: 27.2% from the second prompt and 12.5% from the third.The first prompt generated 60.3% of successes; evaluation allowed up to two additional prompts.Source: §4.4 · p. 18

4.6 [RQ5] How important are CoverUp’s components to its performance?

• Ablating error fixing reduced coverage by 14%–37%, code context by 7%–16%, and coverage information by up to 2%.As coverage grew, the coverage-ablated prompt produced fewer functions that increased coverage; it generated over 50% more functions overall (11,222 vs. 7,366). The components are not entirely independent.Source: Figure 17 · p. 19, §4.6 · pp. 19–20, Figure 18 · p. 19

Benchmark selection.

• CM and PY were selected to evaluate CoverUp on challenging and less challenging code, while MT was included because MuTAP supports it.Source: Benchmark selection · p. 20
• Selecting a different benchmark set could produce different results, and some benchmark prerequisite modules were missing in the replicated execution environment.The authors suggest both CodaMosa and CoverUp might generate better tests if those modules were installed.Source: Benchmark selection · p. 20

LLM model dependency.

• CoverUp was developed and evaluated using OpenAI’s GPT-4, GPT-4 Turbo, and GPT-4o models.Source: LLM model dependency · p. 20
• The CoverUp approach is independent of the LLM, and the paper reports that it significantly outperforms an ablated version relying only on LLM capabilities.Source: LLM model dependency · p. 20
• CoverUp’s ultimate performance depends on the model’s ability to interpret its prompts and generate tests as requested.Source: LLM model dependency · p. 20

• Combining coverage analysis, code context, and feedback, CoverUp produces test suites with higher coverage than previous approaches.Source: §6 · p. 20
• Integrating coverage information into prompts directs the LLM’s attention to uncovered code.Source: §6 · p. 20
• Iterative refinement through error and coverage feedback benefits high-coverage test generation.Source: §6 · p. 20