PaperlayerView exact revision on arXiv

Source-linked AI summary

VIBEPASS: Can Vibe Coders Really Pass the Vibe Check?

Srijan Bansal, Jiao Fangkai, Yilun Zhou, Austin Xu, Shafiq Joty, Semih Yavuz

arXiv:2603.15921v1cs.SEcs.AI

TL;DR

Autonomous coding requires models to diagnose latent faults and repair them, but these connected capabilities have not been systematically evaluated. VIBEPASS jointly measures discriminative test generation and repair across 12 frontier models using model-generated buggy solutions. Fault-hypothesis generation is the dominant bottleneck, while self-generated tests can outperform external tests when both expose valid corner cases.

  • Problem

    Prior test-generation and repair work evaluated components separately, leaving the connective capability to expose latent faults and use diagnosis for repair unmeasured.

    Source: §1 · p. 1

  • Method

    VIBEPASS evaluates discriminative FT-Test generation and repair under no-test, external-test, and self-generated-test conditions on model-generated buggy solutions.

    Source: §1 · p. 2, §2 · p. 2, §2.2 · p. 3

  • Results

    Across 12 frontier models, fault-hypothesis generation is the dominant bottleneck, and self-generated tests outperform external tests by 6.4 points when both yield valid corner cases.

    Source: §1 · p. 2, §3.1 · p. 5

  • Takeaways & Limitations

    Fault-targeted reasoning remains deficient across frontier models, making fault localization—not code or test synthesis—the binding bottleneck for autonomous debugging.

    Source: §1 · p. 2, §5 · p. 9

    The benchmark treats solutions passing all provided tests as silver, although those test suites may be incomplete and do not guarantee absolute correctness.

    Source: §2.2 · p. 3

Abstract

from arXiv · show

As Large Language Models shift the programming toward human-guided ''vibe coding'', agentic coding tools increasingly rely on models to self-diagnose and repair their own subtle faults -- a capability central to autonomous software engineering yet never systematically evaluated. We present \name{}, the first empirical decomposition that jointly evaluates two coupled tasks: \emph{Fault-Triggering Test Generation (FT-Test)} constructing a discriminative witness that exposes a latent bug, and \emph{Fault-targeted Program Repair (FPR)}, repairing it under varying diagnostic conditions. \name{} pairs competitive programming problems with LLM-generated solutions that pass partial test suites but fail on semantic edge cases, enabling controlled identification of where the diagnostic chain breaks down. Evaluating 12 frontier LLMs, we find that fault-targeted reasoning does not scale with general coding ability. Models produce syntactically valid test inputs at near-ceiling rates yet collapse on discriminative generation, with fault hypothesis generation -- not output validation -- as the dominant bottleneck. Test-guided repair reveals a complementary insight: when self-generated tests successfully witness a fault, the resulting repair matches or outperforms repair guided by externally provided tests, but tests that fail to witness the fault actively degrade repair below unguided baselines. Together, these results reframe the challenge of autonomous debugging: the binding bottleneck is not code synthesis or test validity but fault-target reasoning, a capability that remains deficient across all frontier models. As Large Language Models shift the programming toward human-guided ''vibe coding'', agentic coding tools increasingly rely on models to self-diagnose and repair their own subtle faults -- a capability central to autonomous software engineering yet never systematically evaluated.

1 Introduction

Established code-generation benchmarks measure success from clear specifications, while autonomous debugging also requires models to expose latent faults and repair them. VIBEPASS addresses the unmeasured connection between test generation and repair, finding fault-hypothesis generation to be the binding bottleneck.

1 Introduction

  • VIBEPASS jointly evaluates fault-triggering test generation and targeted repair, addressing the connective diagnostic capability that prior test-generation and repair work assessed separately.Prior test-generation work used validity and coverage as proxies for fault exposure, while repair work assumed the fault location was known.Source: §1 · pp. 1–2
  • The benchmark pairs competitive-programming problems with model-generated buggy solutions that pass 10–90% of official tests, alongside accepted human-authored solutions.Its 173 instances span 76 problems and support repair under no-test, externally tested, and self-generated-test conditions.Source: §1 · p. 2
  • Across 12 frontier models, fault-targeted reasoning does not scale with general coding ability; fault-hypothesis generation is the dominant bottleneck, not output validation.Inputs are syntactically valid 86% of the time but fault-triggering only 61%; the hypothesis-generation gap is 2.7 times larger than the output-validation gap.Source: §1 · p. 2

2 VIBEPASS

VIBEPASS decomposes fault-targeted reasoning into discriminative test generation and repair under different diagnostic conditions. It builds a verified benchmark from semantic bugs in model-generated solutions and evaluates 12 frontier models.

2 VIBEPASS

  • VIBEPASS evaluates FT-Test, which exposes semantic bugs with discriminative tests, and FPR, which measures repair under varying guidance.The benchmark emphasizes diverse semantic bugs, execution-based verification, and multiple evaluation settings.Source: §2 · p. 2, §2.2 · p. 3

2.1 Benchmark Construction

  • The benchmark contains 173 instances across 76 problems, pairing each problem with official tests, a validity checker, a verified silver solution, and a model-generated buggy solution.Buggy solutions pass 10%–90% of official tests; candidate solutions with runtime errors are discarded as trivially detectable.Source: §2.1 · p. 3

2.2 Task 1: Fault-Triggering Test Generation (FT-Test)

  • FT-Test requires a valid input and expected output that the buggy solution fails while the silver solution passes.Bug-aware evaluation tests generation for known faults; bug-discovery evaluation first asks the model to judge whether the solution is buggy.Source: §2.2 · pp. 3–4

2.3 Task 2: Fault-targeted Program Repair (FPR)

  • FPR measures whether a corrected solution passes all official tests, comparing unguided repair with externally supplied or self-generated fault-triggering tests.Pass@1 and success rate are primary metrics; self-generated test quality is also assessed using Task 1 metrics.Source: §2.3 · p. 4

2.4 Evaluated Models

  • The evaluation covers 12 frontier commercial and open-source models, including OpenAI, Gemini, Claude, GPT-OSS, and Nemotron systems.The lineup includes GPT-5 variants, Gemini-3 models, Claude Sonnet-4.6 and Opus-4.6, and two open-source systems.Source: §2.4 · p. 4

3 Results and Analysis

Across test generation and repair, results expose fault localization and test-to-repair as key bottlenecks, despite high input validity. Self-generated tests can outperform external tests when both are valid corner cases, but average test guidance does not consistently improve repair.

3.1 RQ1: How effectively do frontier LLMs generate fault-triggering test cases?

  • 86.4 average VI contrasts with 61.3 DIO, and the fault-hypothesis gap averages about 23 percentage points versus about 8 for output validation.Table 1 reports a 25.5% gap between validity and discriminative test generation; the hypothesis gap is 2.7 times larger than the validation gap, with DIO/DI ≈ 0.96.Source: Table 1 · p. 5, §3.1 · p. 5
  • Bug-discovery end-to-end J + DIO is 49.9%, versus 61.3% Bug-Aware DIO; 60% of bug-discovery failures trace to misjudgment.Under bug-aware evaluation, average DIO is 71.2% when judgment is correct versus 33.6% when incorrect, a 37.6-point gap.Source: §3.1 · pp. 5–6, Table 2 · p. 6

3.2 RQ2: How effectively do frontier LLMs debug subtle semantic bugs?

  • On average, test-guided repair does not beat unguided repair; among valid corner-case tests, INTTEST reaches 64.2% success versus 57.8% for EXTTEST, a 6.4-point advantage.The controlled comparison covers valid samples; INTTEST beats EXTTEST with p < 0.05, while the broader Table 3 results show average test augmentation underperforming NoTest.Source: §3.2 · pp. 6–8, Figure 2 · p. 7, Table 3 · p. 7

3.3 RQ3: How correlated are code-reasoning skills, and what limits bug repair?

  • FT-Input and FT-IO correlate at r = 0.988 and predict Judge performance at r ≥ 0.86, while Valid Input alone has a weak correlation of r = 0.311.Figure 3 identifies the largest drops at Valid IO → FT-Input (14.7 percentage points) and FT-IO → Repair (21.2 percentage points); OpenAI models reach 54.3% final success and open-source models 12.1% in repair.Source: Figure 3 · p. 8, §3.3 · pp. 8–9

4 Related Works

Related work spans competition-level code generation, software-engineering workflows, and debugging, testing, and repair. The paper identifies fault-targeted reasoning as underexplored relative to generation from scratch.

Competition-Level Code Generation

  • Competition-level benchmarks assess algorithmic code generation, with LiveCodeBench and LiveCodeBench Pro periodically updating problems to address pretraining contamination.EvoCodeBench instead evolves datasets aligned with real repositories.Source: §4 · p. 9

Software Engineering Code Generation

  • Software-engineering benchmarks target long-context understanding and maintainable implementations, including repository completion, bug fixing, and repository synthesis.CodeChain explores modular self-revision for correctness on complex generation tasks.Source: §4 · p. 9

Automatic Debugging, Testing, and Repair

  • Debugging research includes execution-feedback self-correction, repair benchmarks such as SWE-bench, agent training over software histories, patch ranking, and test synthesis.Related efforts also extend repair evaluation to vulnerabilities through CVE-Bench.Source: §4 · p. 9

4 Related Works

  • Most benchmarks emphasize generation from scratch, leaving the connected work of fault identification, test generation, and repair underexplored.The paper terms these capabilities fault-targeted reasoning and motivates dedicated benchmarks for realistic coding workflows.Source: §4 · p. 9

5 Conclusion

VIBEPASS evaluates fault-targeted reasoning across 173 instances and 12 frontier models. Its findings identify fault hypothesis generation as the dominant bottleneck and show that contextual alignment matters for test-guided repair.

5 Conclusion

  • Across 173 instances and 12 frontier models, fault-targeted reasoning does not scale with general coding ability, with fault hypothesis generation the dominant bottleneck.The bottleneck is not code or test synthesis.Source: §5 · p. 9
  • Self-generated tests match or exceed external tests when they successfully witness a fault, highlighting contextual alignment over raw test quality.Source: §5 · p. 9
  • Performance cliffs at fault localization and the test-to-repair transition indicate that causal program reasoning remains unsolved even for state-of-the-art systems.Source: §5 · p. 9

A.1.1 Key Design Principles

VIBEPASS uses three design principles to evaluate fault-targeted reasoning: semantic bugs, execution-based verification, and varied information conditions.

A.1.1 Key Design Principles

  • The benchmark prioritizes non-trivial semantic bugs that pass some official tests, favoring higher pass rates and diverse failure patterns over syntax errors.Diversity is measured with Hamming distance.Source: §A.1.1 · p. 14
  • Execution-based metrics verify solutions against official tests, while a fault-triggering test requires the buggy solution to fail and the accepted solution to succeed.Source: §A.1.1 · p. 14
  • Models are evaluated across bug-aware and bug-discovery test generation, plus unguided and test-guided repair, to isolate reasoning capabilities and guidance effects.Source: §A.1.1 · p. 14

A.2 Benchmark Construction Details

The benchmark is constructed from filtered algorithmic problems, validated input checkers, and verified silver and buggy solutions.

A.2 Benchmark Construction Details

  • Filtering 1,055 LiveCodeBench problems yields 170 retained problems, with 89% rated medium or hard and coverage across several algorithmic domains.Problems solved within three attempts by cost-effective frontier models are excluded; the retained set represents 16% of the source problems.Source: §A.2 · p. 14
  • Input-validity checkers are regenerated if they reject any official input; 98.8% of problems yield valid checkers with 100% acceptance on official tests.Validation prevents false negatives on official inputs but does not guarantee coverage of all implicit constraints.Source: §A.2 · p. 14
  • Candidates passing all official tests become silver solutions, while only wrong-answer failures are retained as bugs, prioritized for subtlety and distinct failure patterns.Subtlety is favored through higher official pass rates, and diversity through Hamming distance over official-test pass/fail patterns.Source: §A.2 · p. 14
  • The collected set contains 2,184 candidate solutions across 168 problems, drawn from human submissions and model-generated candidates spanning multiple model families.Source: §A.2 · p. 15

A.3 Experimental Setup– LLM Generators for Bug and Solution Generations

The experimental setup combines broad model coverage, structured task-specific prompts, standardized inference settings, and sandboxed evaluation. It reports test-generation accuracy and debugging Pass@1 over benchmark instances.

  • The evaluation spans OpenAI, Google, Anthropic, and open-source models, enabling comparisons across model families, sizes, and architectures.GPT-5-Mini builds input-validity checkers, while GPT-4o, Gemini-2.5-Pro, and Claude Sonnet-4 support capability-based filtering.Source: §A.3 · p. 15
  • API models use temperature T=1 when available, and all models use high reasoning effort; open-source models receive 128K-token contexts and API models nearly 1M+ tokens.Source: §A.3 · p. 16
  • Structured prompts provide problem descriptions and input-output specifications, with task-specific instructions for bug discovery and test-guided or self-guided debugging.Bug-discovery prompts request a correctness judgment and, if the solution is buggy, a test case; debugging prompts either provide failing tests or ask models to generate tests internally.Source: §A.3 · p. 16
  • Generated solutions and tests run in isolated sandboxes with time limits relaxed to 5× the original limits to emphasize semantic correctness over efficiency.Test validity is checked with generated input-validity checkers; debugging uses Pass@1 and test generation uses FT-Test accuracy, with each experiment run once.Source: §A.3 · p. 16
  • Figure 4 provides an example prompt and corresponding model output for input-validity generation.Source: Figure 4 · p. 17

Related papers

Loading 2603.15921v1…