PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Generate Unit Tests for Automated Debugging

Archiki Prasad, Elias Stengel-Eskin, Justin Chih-Yao Chen, Zaid Khan, Mohit Bansal

arXiv:2502.01619v3cs.SEcs.AIcs.CLcs.LG

TL;DR

Automated debugging needs failing unit tests with correct expected outputs, but generating revealing inputs and predicting their outputs present a trade-off. UTGEN trains models to generate such tests, and UTDEBUG scales output prediction and validates edits against multiple tests. The paper reports improved test quality and debugging outcomes, while noting that repeated test generation makes the process costly on frontier models.

  • Problem

    Automated debugging feedback is often bottlenecked by scarce failing unit tests, which require both inputs that expose errors and correct expected outputs.

    Source: §1 · p. 2

  • Method

    UTGEN trains models to generate error-revealing input-output tests from task descriptions and buggy code, while UTDEBUG uses test-time scaling and multiple-test validation with backtracking.

    Source: §1 · pp. 2–3

  • Results

    UTGEN improves the joint attack-and-output metric by 7.59% absolute over prompted failing-test generation, and UTGEN feedback improves pass@1 by 15.07% absolute over no UT feedback on MBPP+Fix (Hard).

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Generated UTs can provide useful debugging feedback, including for frontier-model debugging, and can serve as a stronger code-correctness judge than a trained reward model in the reported evaluation.

    Source: §1 · p. 3

    UTDEBUG can be costly on frontier models because it generates multiple tests with self-consistency across several debugging rounds.

    Source: §5.4 · p. 10

Abstract

from arXiv · show

Unit tests (UTs) play an instrumental role in assessing code correctness as well as providing feedback to large language models (LLMs), motivating automated test generation. However, we uncover a trade-off between generating unit test inputs that reveal errors when given a faulty code and correctly predicting the unit test output without access to the gold solution. To address this trade-off, we propose UTGen, which teaches LLMs to generate unit test inputs that reveal errors along with their correct expected outputs based on task descriptions. Since model-generated tests can provide noisy signals (e.g., from incorrectly predicted outputs), we propose UTDebug that (i) scales UTGen via test-time compute to improve UT output prediction, and (ii) validates and backtracks edits based on multiple generated UTs to avoid overfitting, and helps LLMs debug effectively. We show that UTGen outperforms other LLM-based baselines by 7.59% based on a metric measuring the presence of both error-revealing UT inputs and correct UT outputs. When used with UTDebug, we find that feedback from UTGen's unit tests improves pass@1 accuracy of Qwen2.5 32B on HumanEvalFix and our own harder debugging split of MBPP+ by over 3.17% and 12.35% (respectively) over other LLM-based UT generation baselines. Moreover, we observe that feedback from Qwen2.5 32B-based UTGen model can enhance debugging with frontier LLMs like GPT-4o by 13.8%. Lastly, we demonstrate that UTGen is a better judge for code correctness, outperforming a state-of-the-art trained 8B reward model by 4.43% on HumanEval+ with best-of-10 sampling using Qwen2.5 7B.

1 Introduction

The paper addresses the scarcity of failing unit tests with correct expected outputs, which limits feedback for debugging, and identifies a trade-off between revealing errors and predicting outputs. It introduces UTGEN and UTDEBUG to generate such tests and use their feedback in debugging.

1 Introduction

  • Automated debugging is bottlenecked by the availability of failing tests, while test collection requires both error-triggering inputs and known expected outputs.Existing benchmark suites are typically small because human annotation is laborious; prior approaches often sample inputs randomly or use the gold solution for outputs.Source: §1 · p. 2
  • 7.59% absolute improvement over prompted failing-test generation is reported for UTGEN on the metric requiring both error-revealing inputs and correct outputs.Source: §1 · p. 2
  • 15.07% absolute improvement over debugging without UT feedback on MBPP+Fix (Hard) accompanies a 4.61% improvement over zero-shot UT generation on MBPP+Fix.UTGEN feedback also improves pass@1 by 4.43% over a reward model on HumanEval+ with best-of-10 Qwen2.5 Coder 7B sampling.Source: §1 · p. 3
  • UTDEBUG scales output prediction with self-consistency and validates edits against multiple generated tests, backtracking when the suite’s pass rate does not improve.Source: §1 · p. 3
  • Nearly 25% absolute improvement over GPT-4o self-generated feedback is reported for UTGEN feedback from Qwen2.5 32B on the hardest MBPP+ subset.Source: §1 · p. 3

2 Related Work

Prior LLM test-generation approaches often rely on gold solutions or treat tests as a byproduct of code generation. This paper instead targets test quality directly and positions generated tests as feedback where human-written tests are scarce.

2 Related Work

  • Unlike prompted approaches that assume access to the gold solution, UTGEN generates input-output test pairs from task descriptions without relying on the gold implementation.Sch¨afer et al. (2023) and Liu et al. (2024b) focus on generating inputs, while Chen et al. (2023a) primarily focuses on code generation.Source: §2 · p. 3
  • Unlike work that treats tests as part of code generation, this paper directly models unit-test quality and evaluates its utility for code generation and debugging.Source: §2 · p. 3
  • Prior LLM debugging methods rely on gold unit tests, so UTGEN complements them when human-written tests are scarce or unavailable.The paper presents UTDEBUG as a way to handle noisy feedback from generated tests through test-time scaling and backtracking.Source: §2 · p. 3

3 Unit Test Generation and Automated Debugging

The method defines useful tests as valid input-output pairs that can expose errors in buggy code, then trains generators to produce them from task descriptions and buggy candidates. UTDEBUG addresses noisy generated-test feedback through output scaling and edit validation.

3.1 Background and Task Setup

  • A valid unit test pairs an input allowed by the task description with the output produced by any correct implementation; useful generators should also expose errors in buggy code.The setup focuses on generators conditioned on both the task description and buggy code because description-only generators lack sufficient context for effective error-revealing tests.Source: §3.1 · p. 4

3.2 UTGEN: Training LLMs for Unit Test Generation

  • UTGEN bootstraps training data from code-generation datasets by perturbing gold code, sampling inputs that fail against the buggy candidate, and using reference outputs with rationales as supervision.A sampled test fails when the candidate’s output differs from the reference output; reference-code outputs are used to ensure output accuracy.Source: §3.2 · pp. 4–5
  • UTGEN trains models to improve attack rate and output accuracy jointly, using nearly 30K instances for 8B-scale models and roughly 70K for 32B models.Source: §3.2 · p. 5

3.3 UTDEBUG: Generated Unit Tests for Debugging

  • Generated tests can mislead debugging when inputs do not fail on buggy code or predicted outputs are wrong, potentially misclassifying faulty code or introducing errors into correct code.Source: §3.3 · p. 5
  • UTDEBUG uses self-consistency to select outputs and validates edits on multiple tests, accepting changes only when the suite’s overall pass rate improves and otherwise backtracking.For each input, it samples eight output completions and retains tests whose majority answer receives at least four votes.Source: §3.3 · pp. 5–6

4 Experimental Setup

The experiments evaluate UT generation and downstream debugging with several coding models, benchmark-derived faulty-code datasets, intrinsic test metrics, and code pass@1. Baselines vary in whether they provide no test feedback, random tests, or prompted failing tests.

4 Experimental Setup

  • Experiments include Llama3 8B, Llama3.1 8B, Qwen2.5 Coder 7B, and Qwen2.5 Coder 32B models across HumanEval- and MBPP-based debugging benchmarks.Source: §4 · p. 6
  • MBPP+Fix contains 325 faulty solutions, while MBPP+Fix (Hard) retains 170 problems whose faulty code passes 50%–95% of tests.HE+Fix contains 158 problems with human-introduced errors and private evaluation tests.Source: §4 · p. 6
  • Intrinsic metrics measure attack rate, output accuracy, and their intersection, while downstream utility is measured by pass@1 code accuracy.Source: §4 · pp. 6–7
  • Baselines include self-generated feedback without unit tests, randomly sampled valid tests, and prompted failing tests; all UT-generation baselines use UTDEBUG’s scaling and backtracking.Source: §4 · p. 7

5 Results and Analysis

UTGEN-generated tests provide stronger feedback for debugging and code selection than the reported baselines, including with a frontier debugger. Results also show that debugging difficulty varies across datasets and that repeated test generation has a cost.

5.1 Intrinsic Evaluation of UT Generation

  • 5.3%: On Qwen2.5 32B, UTGEN improves Acc. ∩Attack over prompted failing UTs, and improves it by 23.34% over randomly generated UTs.Across the evaluated models, UTGEN-trained models rank highest on the joint metric; Llama3.1 improves the joint score by up to 7.65%.Source: Table 1 · p. 7, §5.1 · p. 8

5.2 Generated UTs for Code Debugging

  • 12.35%: On MBPP+Fix (Hard), UTGEN tests improve Qwen2.5 32B pass@1 over debugging without UT feedback.On MBPP+Fix, UTGEN improves over randomly sampled UTs by 4.61%, failing UTs by 3.38%, and no UT feedback by 15.07%.Source: Table 2 · p. 8, §5.2 · p. 8
  • MBPP+Fix (Hard) is harder to debug than HumanEvalFix, with up to a 53.5% final-accuracy difference for UTGEN with Qwen2.5 7B.The hard split starts with an approximately 75% pass rate, while HE+Fix starts at 0%; the paper links the contrast to subtle flaws and corner cases.Source: §5.2 · pp. 8–9

5.3 Generated UTs for Code Generation

  • UTGEN-based best-of-10 selection outperforms an external reward model by 4.36% on HE+ with Llama3 8B, with similar trends on MBPP+ and Qwen2.5 7B.Generated UTs outperform the trained 8B reward model in all four model-and-dataset settings.Source: Table 3 · p. 9, §5.3 · p. 9

5.4 Comparison to Unit Test Generation by Frontier LLMs

  • UTGEN beats GPT-4o and DeepSeek-V3 by 1.38% on Acc. ∩Attack and also has a higher attack rate on MBPP+Fix (Hard).Its output accuracy is better than DeepSeek-V3's and slightly worse than GPT-4o's.Source: Table 4 · p. 9, §5.4 · p. 9
  • UTDEBUG's multiple UT generations, self-consistency, and debugging rounds make its use costly, especially with frontier models.The authors therefore describe relatively smaller models trained with UTGEN as more attractive options.Source: §5.4 · p. 10

5.5 Importance of Unit Tests with a Frontier Debugger

  • UTGEN improves GPT-4o debugging over the prompted UT baseline by 13.8% on MBPP+Fix (Hard).Both prompted UT feedback and UTGEN feedback outperform the no-UT baselines; UTGEN feedback yields an approximately 25% absolute improvement over them.Source: Table 5 · p. 10, §5.5 · p. 10

6 Conclusion

The conclusion presents UTGen as a response to the trade-off between finding error-revealing inputs and predicting their outputs correctly, and UTDebug as a way to use generated tests for debugging. It reports consistent debugging gains across models and notes their potential use in code generation.

6 Conclusion

  • UTDebug combines test-time scaling with cross-validation and backtracking to improve output prediction and avoid overfitting to narrow or incorrect tests.The conclusion also reports that gains persist as the number of generated unit tests scales across datasets.Source: §6 · p. 10
  • The paper identifies a trade-off between generating error-revealing unit-test inputs and correctly predicting their outputs.Source: §6 · p. 10
  • UTGen is a data-creation and training method for teaching models to generate unit tests, assessed using attack rate and output accuracy.The authors also introduce a subset with challenging, hard-to-diagnose errors because existing data contains many easy errors.Source: §6 · p. 10
  • UTGen and UTDebug yield consistent increases in debugging performance across models, and the approach can also improve code generation through best-of-N sampling.Source: §6 · p. 10
  • The authors describe UTGen as complementary to work on debugging real-world GitHub issues and suggest that unit-test generation could identify such issues in the future.Source: §6 · p. 10

A.1 Debugging Datasets

The paper evaluates debugging on HumanEvalFix and two MBPP+Fix splits, using expanded unit-test suites and faulty solutions with different levels of difficulty. It also evaluates generated-test utility for code generation on HumanEval+ and MBPP+.

A.1 Debugging Datasets

  • The debugging datasets are based on HumanEval and MBPP, together with their extended evaluation suites containing more unit tests.Source: §A.1 · p. 14
  • 158 HumanEvalFix problems use faulty human-written code and expanded private tests, averaging 775.18 gold tests per problem rather than 8.17 in the original suite.The split filters for overlap with EvalPlus and has an initial pass rate of 0%.Source: §A.1 · pp. 14–15
  • MBPP+Fix contains 325 problems with faulty model-generated code, an initial pass rate of 24.21%, and an average of 107.45 gold tests per problem.One incorrect solution is sampled per problem from solutions generated by multiple 7–8B models.Source: §A.1 · pp. 14–15
  • MBPP+Fix (Hard) contains 170 partially correct faulty solutions that pass 50–95% of tests, with an initial pass rate of 74.83%.The split averages 107.49 gold tests per problem and targets subtler errors.Source: §A.1 · pp. 14–15
  • For code-generation evaluation, the authors use HumanEval+ and MBPP+ problems with extractable private tests, and compare against an 8B Llama3.1-based reward model.Source: §A.1 · p. 15

A.2 Evaluation Metrics

The evaluation separates unit-test generation quality into attack rate, output accuracy, and their conjunction, then measures debugging utility using pass@1 code accuracy. A test-generation procedure uses repeated output sampling and voting before retaining generated tests.

A.2 Evaluation Metrics

  • The evaluation reports attack rate, output accuracy, and Accuracy ∩Attack as intrinsic test-generation metrics, alongside pass@1 for debugging.Source: §A.2 · p. 15
  • Output accuracy measures whether a generated test's predicted output matches the reference solution, whether or not the test fails.Source: §A.2 · p. 15
  • BuildUT samples up to three times the requested number of tests, takes a majority vote over output samples, and retains an output when it receives at least half the votes.It returns once it has collected the requested number of tests or finishes sampling.Source: §A.2 · p. 16
  • Accuracy ∩Attack measures how often a generated test both fails the target buggy code and predicts the correct output.Source: §A.2 · p. 16
  • Pass@1 code accuracy is the percentage of debugged code passing all private human-annotated tests after three debugging rounds.Source: §A.2 · p. 16

B UTGEN Training

UTGen training data is built from filtered coding examples, corrupted code, and generated failing tests, then rationalized for training. UTDebug uses generated tests to guide iterative edits and accepts edits only when pass rate improves.

B UTGEN Training

  • Filtering the Tulu-3 SFT mixture for valid Python coding prompts and executable code yields 48.3K instances for UTGen data construction.The process creates two corrupted versions per code example, attempts five failing-test generations per target, and keeps examples with such tests.Source: B UTGEN Training · p. 16
  • The resulting training data contains roughly 30K instances per 7–8B model and nearly 70K for Qwen2.5 32B.Source: B UTGEN Training · p. 17
  • Models train for 10 epochs with batch size 16 and learning rate 5e-6; the 7–8B models use LoRA, while Qwen2.5 32B uses 4-bit QLoRA.Training takes about 20 GPU hours for the smaller models and about 40 GPU hours for Qwen2.5 32B.Source: B UTGEN Training · pp. 17–18

C Overall Pipeline for UTDEBUG

D Additional Intrinsic Evaluation

The additional intrinsic evaluation examines how unit-test properties relate to debugging utility and compares their balance across benchmarks. Both failing inputs and correct outputs together align most strongly with successful debugging.

D Additional Intrinsic Evaluation

  • On MBPP+Fix, UTGEN yields the highest Acc. ∩Attack score, while on HE+Fix the random baseline scores highest because almost any test fails the faulty code.Across both datasets, randomly sampled tests show higher output accuracy and prompted tests higher attack rates; UTGEN best balances the trade-off.Source: D Additional Intrinsic Evaluation · p. 18
  • The presence of both a failing test and a correct output has the strongest correlation with successful debugging, exceeding either property individually.The correlation analysis uses Qwen2.5 7B-generated tests for MBPP+Fix, with n = 1 test and one debugging round; all reported correlations are statistically significant (p < 0.05).Source: D Additional Intrinsic Evaluation · p. 18, Table 7 · p. 18

E Additional Analysis and Results with UTDEBUG

The section evaluates UTDEBUG’s safeguards against noisy generated tests and compares UT-generation methods for downstream debugging. UTGEN tests outperform CodeT tests, while removing test-time scaling or backtracking reduces performance.

E Additional Analysis and Results with UTDEBUG

  • Removing test-time scaling decreases performance by 4% for randomly sampled tests and 11.4% for UTGEN; removing backtracking drops UTGEN performance by up to 3.2%.The ablation uses Qwen2.5 7B on MBPP+Fix with n = 3 generated tests over 3 debugging rounds.Source: E Additional Analysis and Results with… · p. 19, Table 8 · p. 19
  • Without test-time scaling, UTGEN’s larger performance drop is consistent with models struggling to predict outputs for its more frequent failing, challenging inputs.The study attributes overfitting to feedback from a single test to the absence of validation against other generated tests.Source: E Additional Analysis and Results with… · p. 19
  • UTGEN improves over CodeT by up to 7.54% (absolute) on MBPP+Fix and 15.88% (absolute) on MBPP+Fix (Hard) with Qwen2.5 32B.The comparison uses n = 3 tests; CodeT generates tests independently of the code being debugged, while UTGEN conditions generation on buggy code.Source: Table 9 · p. 19, E Additional Analysis and Results with… · p. 19

F Scaling with Number of Generated UTs

The section reports that UTGEN retains an advantage as the number of generated tests increases, discusses the challenge of predicting correct outputs from task descriptions, and gives examples of the prompts used.

F Scaling with Number of Generated UTs

G Discussion

  • The discussion presents UTGEN as a source of unit-test feedback that complements work on verifiers and methods for incorporating debugging feedback.It states that debugging with LLM-generated feedback remains challenging without unit tests that indicate correctness or how faulty code fails.Source: G Discussion · p. 20
  • UTGEN targets the upstream discovery of potential issues, whereas SWEBench approaches fix known GitHub issues; both require predicting expected behavior without a correct implementation.The discussion links this challenge to LLMs’ difficulty predicting correct outputs from specifications alone, while noting that UTDEBUG helps account for noise and boost UTGEN accuracy.Source: G Discussion · pp. 20–21

H Prompts

  • The prompts ask for valid inputs that expose errors, then require reasoning about the error pattern and a test input and output consistent with it.They also assess whether inputs are valid, outputs faithful to the task description, and tests break the supplied code.Source: H Prompts · pp. 21–22

Related papers

Loading 2502.01619v3…