PaperlayerView exact revision on arXiv

Source-linked AI summary

VibeCheck: Assessing the Quality of LLM-Generated Unit Tests: A Multi-agent Empirical Study across Heterogeneous Repositories

Anika Tabassum, Mushahid Intesum, Md. Fahim Arefin, Tarannum Shaila Zaman

arXiv:2609.05978v2cs.SE

TL;DR

Common execution and coverage metrics can miss whether generated tests meaningfully validate behavior. VibeCheck evaluates IDE-generated tests across repositories with a five-dimensional rubric and peer review, finding that runnable suites often remain behaviorally weak.

  • Problem

    Execution success or coverage alone can miss weak assertions, missing edge cases, poor isolation, and maintainability problems in generated tests.

    Source: §1 · p. 1

  • Method

    VibeCheck evaluates three AI-assisted IDEs and standalone Claude Sonnet 4.5 across 15 repositories using a five-dimensional rubric and leave-one-out cross-agent peer evaluation.

    Source: §1 · pp. 1–2

  • Results

    Across 15 heterogeneous repositories, tests from both the IDEs and the CS-4.5 baseline often run successfully but retain weak assertions, missing edge cases, isolation issues, and maintainability problems.

    Source: §1 · p. 1, §7 · p. 8

  • Takeaways & Limitations

    Execution alone is an incomplete quality signal, so generated tests should also be assessed for behavioral adequacy and testing practices.

    Source: §1 · p. 1

    Peer assessment relied entirely on cross-agent LLM reviews, with no human-annotated validation of rubric scores.

    Source: §6 · p. 7

Abstract

from arXiv · show

LLM-based IDE agents are increasingly used to generate repository-grounded unit tests, yet common evaluations often rely on execution success or coverage. These metrics can miss deeper quality issues such as weak assertions, missing edge cases, poor isolation, and limited maintainability. This paper presents VibeCheck, an empirical study of unit test generation across 15 student-developed Python and JavaScript/TypeScript repositories. We evaluate Kiro, Antigravity, and Cursor with Claude Sonnet 4.5 as the underlying agent, under repository-only, zero-shot conditions using a five-dimensional rubric covering runnability, assertion strength, logic and edge-case coverage, isolation/determinism, and maintainability. We also apply leave-one-out cross-agent peer evaluation to compare tools and identify failure patterns. Results show a clear execution-adequacy gap: generated tests are often runnable but frequently lack strong assertions and meaningful behavioral coverage. Weak assertions and missing edge cases occur more often than blocking failures, showing that runnable tests can still be shallow. VibeCheck provides a reliability-oriented framework for evaluating LLM-generated tests beyond pass/fail outcomes.

1 Introduction

VibeCheck addresses the gap between tests that execute and tests that meaningfully validate behavior, using a reusable framework to assess generated suites. The study finds this gap across IDEs and the standalone model.

1 Introduction

  • Execution success alone is an incomplete quality signal because runnable suites can still have weak assertions, miss edge cases, behave flakily, or be difficult to maintain [27].Source: §1 · p. 1
  • Executable tests are common, but both IDEs and the CS-4.5 baseline show recurring weaknesses in assertion quality, edge-case coverage, isolation, and maintainability.VibeCheck also compares evaluator consistency and confidence-quality alignment relative to the baseline.Source: §1 · pp. 1–2
  • VibeCheck is a repository-grounded framework that combines a five-dimensional adequacy rubric, developer-defined behavioral obligations, and leave-one-out cross-agent peer evaluation.The study evaluates Kiro, Antigravity, and Cursor against standalone CS-4.5 across 15 heterogeneous Python and JavaScript/TypeScript repositories.Source: §1 · pp. 1–2
  • The study releases prompts, generated test suites, rubric scores, peer-evaluation reports, and analysis scripts in a public replication package.The prompts and scoring rubric are fixed and versioned to support reproducible comparison and potential DevOps integration.Source: §1 · p. 2

2 Motivation and Background

The paper motivates VibeCheck with a practical mismatch: passing execution and high coverage can coexist with tests that fail to check important behavior. IDEs also differ in how they work with repository context.

2 Motivation and Background

  • The paper distinguishes IDE approaches: Antigravity emphasizes multi-agent orchestration, Kiro spec-driven governance, and Cursor semantic codebase context with graduated autonomy.Source: §2 · p. 2
  • A suite may compile, pass, and achieve high coverage yet rely on non-null checks, omit invalid-input scenarios, reuse mutable fixtures, and duplicate setup code.Such weaknesses can leave regressions undetected despite satisfying execution-based metrics.Source: §2 · p. 2
  • The mismatch between execution success and behavioral adequacy, along with differing results from IDEs built on the same model, motivates repository-grounded evaluation.Source: §2 · p. 2

3 Related Work

Prior test-generation work has moved from coverage-oriented automated techniques toward LLM-based generation and richer context or feedback. The paper identifies a gap in evaluating IDE agents across broader dimensions of test quality.

3 Related Work

  • Earlier automated approaches emphasized statement and branch coverage, while LLM-based generation can produce natural tests but still yield superficial or incorrect assertions and tests needing repair [26].Repository context and validation-and-repair loops have also been used to improve coverage and assertion quality.Source: §3 · p. 2
  • Later approaches use repository context, prompt engineering, retrieval, execution feedback, iterative refinement, and multi-agent collaboration to improve executability, coverage, and test quality [2] [10].Source: §3 · p. 2
  • Existing evaluations focus mainly on execution, coverage, or mutation scores, with less attention to IDE agents under repository-only constraints, joint test-quality dimensions, and cross-agent evaluator confidence.VibeCheck addresses this gap with a five-dimensional rubric, developer-defined obligations, and leave-one-out peer evaluation.Source: §3 · p. 2

4 Methodology

VibeCheck evaluates repository-grounded unit-test generation through a staged framework, combining peer review with developer-defined behavioral obligations and aggregation across diverse projects. Its methodology is designed to assess more than whether tests execute.

4 Methodology

  • VibeCheck assesses whether existing agents produce executable, behaviorally meaningful tests under repository-only constraints, through a six-stage pipeline from benchmark construction to result aggregation.The stages include generation, ground-truth coverage checking, metrics, and cross-agent peer evaluation.Source: §4 · pp. 2–3, Figure 1 · p. 3

4.1 Benchmark

  • The benchmark contains 15 student-developed Python and JavaScript/TypeScript repositories spanning diverse domains and nontrivial multi-file projects.The original repository source code is not included in the replication package because of privacy and academic-integrity/licensing constraints.Source: §4.1 · p. 3

4.2 Unit-Test Generation

  • Kiro, Antigravity, Cursor, and standalone CS-4.5 generate tests with the same zero-shot repository-only instruction, without external documentation or production-code changes.Generation proceeds file-by-file in a generate-run-check cycle; outputs are preserved without manual editing, repair, or filtering.Source: §4.2 · p. 3

4.3 Ground-Truth Coverage

  • Authors independently derive repository-specific behavioral obligations from source, documentation, and implementation logic before examining generated suites.An obligation counts only when a test exercises the behavior with a meaningful assertion, so superficial checks do not qualify.Source: §4.3 · pp. 3–4

4.4 Evaluation Metrics

4.5 Cross-Agent Peer Evaluation

  • Each generated suite receives three independent leave-one-out reviews, with evaluators scoring quality and reporting diagnostic findings and confidence in structured reports.Source: §4.5 · p. 4

4.6 Aggregation and Robustness Analysis

  • Peer evaluations are aggregated across reviewers, then summarized with repository-level rankings and macro-level averages across all 15 repositories.The aggregation is intended to assess consistency and robustness across repositories that vary in size, domain, and implementation characteristics.Source: §4.6 · p. 4

5 Experimental Evaluation

Across 15 repositories, the evaluation compares test-suite breadth, peer-evaluator agreement, test adequacy, and confidence-quality alignment. Results show that agent rankings vary by measure, and runnable tests do not necessarily provide strong behavioral validation.

5.1 RQ1: Test-Suite Breadth and Behavioral Coverage

  • 182 files from Kiro exceeded CS-4.5’s 83, but none of the IDEs matched the baseline’s 2,848 test cases.Kiro alone surpassed the baseline in file count; CS-4.5 produced denser suites in case count.Source: Table 1 · p. 5, §5.1 · p. 5
  • Kiro showed the broadest sector-level reach, while specialized behaviors were generally less covered than common backend- and validation-oriented sectors.Cursor and Antigravity were below the baseline in most sectors; Kiro exceeded it in several specialized categories.Source: Figure 2 · p. 5, §5.1 · p. 5

5.2 RQ2: Cross-Agent Evaluator Reliability

  • Cursor had the strongest agreement with CS-4.5 and Antigravity the weakest, but shared-model agreement indicates harness consistency rather than evaluator correctness.CS-4.5 agreed more consistently with each IDE than the IDEs agreed among themselves.Source: §5.2 · pp. 5–6, Figure 3 · p. 6

5.3 RQ3: Runnability, Adequacy, and Failure Modes

  • Kiro ranked first on 10 of 15 repositories, versus three first-place finishes for CS-4.5; Cursor never ranked first and Antigravity ranked first on two.The comparison uses overall scores out of 100 across repository–IDE pairs.Source: Table 2 · p. 6, §5.3 · p. 6
  • Missing behavioral cases were the most frequent failure signal for every model; Kiro had the most favorable rate relative to CS-4.5.Kiro also had lower blocking-error and flakiness rates than the baseline, while Antigravity and Cursor had higher blocking-error rates.Source: Table 3 · p. 6, §5.3 · pp. 6–7
  • CS-4.5 led all models on maintainability, while Kiro exceeded the baseline on runnability and the first three rubric dimensions.Both Kiro and Antigravity exceeded the baseline on isolation and determinism; Cursor trailed it on that dimension and maintainability.Source: Table 4 · p. 7, §5.3 · p. 7
  • A runnable test with no assertion could pass when approveItem did not throw, illustrating how execution success can coexist with weak behavioral validation.Evaluators marked the example as having both a weak assertion and a missing behavioral obligation.Source: §5.3 · p. 7

5.4 RQ4: Confidence-Quality Alignment

  • Pearson r=0.21 showed a weak positive confidence–quality association, and substantial score dispersion meant confidence was only a limited proxy for adequacy.The pooled estimate was 95% CI [−0.05, 0.44], p=0.11, n=60.Source: Figure 4 · p. 7, §5.4 · p. 7

6 Limitations

The study’s findings are bounded by its specific IDE–model pairings, peer-only evaluation, student-project benchmark, and zero-shot repository-only prompting setup.

6 Limitations

  • Testing each IDE with one underlying LLM means results reflect specific IDE–model pairings, not the full range of configurations.Source: §6 · p. 7
  • Peer assessment used no human feedback or human-annotated validation of rubric scores, despite safeguards intended to reduce calibration and self-bias effects.Source: §6 · p. 7
  • The 15 student-developed repositories are smaller and less complex than industrial projects, leaving real-world-scale generalizability unclear.Source: §6 · p. 8
  • Only zero-shot, repository-only prompting was tested; other prompting strategies may change IDE performance, leaving intrinsic weaknesses versus setup effects unresolved.Source: §6 · p. 8
  • The observed inadequacy of current LLM-based IDE tests is limited to repository-only, zero-shot conditions.Source: §6 · p. 8

7 Conclusion

VibeCheck evaluates repository-grounded IDE test generation against standalone Claude across 15 repositories. Tests often run but remain behaviorally weak, and Kiro leads overall project-level performance.

7 Conclusion

  • Across 15 heterogeneous repositories, tests often ran successfully but still showed weak assertions, missing edge cases, isolation issues, and maintainability problems.Source: §7 · p. 8
  • Kiro had the strongest overall project-level performance, exceeding the Claude baseline on most repositories, while Claude generated denser suites than any IDE.Source: §7 · p. 8
  • The findings support evaluating AI-generated tests beyond pass rates and coverage, using structured adequacy criteria and diagnostic failure analysis.Source: §7 · p. 8

Related papers

Loading 2609.05978v2…