Source-linked AI summary
Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction
Sungmin Kang, Juyeon Yoon, Shin Yoo
TL;DR
General bug-report-to-test generation remains difficult because existing approaches largely focus on crashes, despite the importance of semantic bug reproduction. LIBRO uses an LLM to generate, process, and rank tests, reproducing bugs in Defects4J and a later dataset while identifying likely successful tests. The authors report that LIBRO can reproduce a substantial share of bugs, though its performance does not generalize well to tests relying on external files or atypical testing infrastructure.
Problem
Existing test-generation techniques largely miss semantic bug reproduction, while crash-focused reproduction covers only a subset of bug reports despite the recurring report-to-test task.
Method
LIBRO prompts an LLM to generate prospective tests, processes them for executability and likely reproduction, and ranks candidates using behavior matching, output consensus, and test length.
Results
LIBRO reproduced 33.5% of studied Defects4J bugs and 32.2% of bugs in a distinct later dataset, with a reproducing test ranked first for 149 Defects4J bugs.
Takeaways & Limitations
The results indicate that LLM-based generation can automatically produce and prioritize bug-reproducing tests for a substantial portion of general bug reports.
Takeaways & Limitations
LIBRO does not generalize well to tests relying on external files or substantially different testing infrastructure, such as Closure’s atypical Defects4J tests.
Abstract
from arXiv · showhide
Many automated test generation techniques have been developed to aid developers with writing tests. To facilitate full automation, most existing techniques aim to either increase coverage, or generate exploratory inputs. However, existing test generation techniques largely fall short of achieving more semantic objectives, such as generating tests to reproduce a given bug report. Reproducing bugs is nonetheless important, as our empirical study shows that the number of tests added in open source repositories due to issues was about 28% of the corresponding project test suite size. Meanwhile, due to the difficulties of transforming the expected program semantics in bug reports into test oracles, existing failure reproduction techniques tend to deal exclusively with program crashes, a small subset of all bug reports. To automate test generation from general bug reports, we propose LIBRO, a framework that uses Large Language Models (LLMs), which have been shown to be capable of performing code-related tasks. Since LLMs themselves cannot execute the target buggy code, we focus on post-processing steps that help us discern when LLMs are effective, and rank the produced tests according to their validity. Our evaluation of LIBRO shows that, on the widely studied Defects4J benchmark, LIBRO can generate failure reproducing test cases for 33% of all studied cases (251 out of 750), while suggesting a bug reproducing test in first place for 149 bugs. To mitigate data contamination, we also evaluate LIBRO against 31 bug reports submitted after the collection of the LLM training data terminated: LIBRO produces bug reproducing tests for 32% of the studied bug reports. Overall, our results show LIBRO has the potential to significantly enhance developer efficiency by automatically generating tests from bug reports.
I. INTRODUCTION
Bug reports are an important but difficult source for automated test generation because many involve semantic behavior rather than crashes. LIBRO addresses this gap with LLM-generated tests, post-processing, and ranking, reproducing bugs across two datasets.
- Problem: Existing bug-reproduction techniques mostly target crashes, while general bug reports often describe semantic issues.General report-to-test generation requires understanding natural and programming language plus deduction.
- Motivation: 96% of Defects4J cases lacked bug-revealing tests before their bug reports were filed.This absence can limit the use of automated debugging techniques that rely on tests or dynamic information.
- Contribution: LIBRO prompts an LLM to generate tests and suggests solutions only after processing results to assess reproduction reliability.The framework is designed to address developers’ difficulty determining when LLM-generated code fulfills their intent.
- Results: 251 bugs, or 33.5% of studied Defects4J bugs, had at least one LIBRO-generated reproducing test.LIBRO also achieved 71.4% accuracy in identifying successful reproductions and ranked a reproducing test first for 149 bugs.
- Results: 32.2% of bugs were reproduced in a distinct report-test dataset, while analyses across two datasets supported the robustness of observed patterns.The additional dataset was constructed to provide further validation beyond Defects4J.
- Motivation: 28.4% was the median ratio of tests added by issue-referencing commits to test-suite size across 300 repositories.The analysis indicates that report-to-test activity significantly contributes to test-suite evolution, without showing that 28.4% of existing tests originated from reports.
III. APPROACH
LIBRO turns a bug report into executable test candidates by prompting an LLM, processing generated code, and selecting and ranking likely reproducing tests. Its prompt variations include examples and contextual information such as stack traces or class constructors.
- Pipeline: The framework generates an initial candidate set by querying the LLM multiple times, then makes the tests executable in the target program.It subsequently curates likely bug-reproducing tests and ranks them to reduce developer inspection effort.
- A. Prompt Engineering: LIBRO constructs a Markdown prompt from a bug report, asks for a self-contained reproducing example, and uses a partial test declaration to induce test generation.The prompt includes a reproduction command, a Markdown code block, and the partial snippet public void test.
- A. Prompt Engineering: LIBRO varies the number and origin of report-test examples in prompts to test whether in-project or cross-project examples influence performance.The prompt experiments are motivated by prior findings that LLMs benefit from question-answer examples.
- A. Prompt Engineering: Stack traces for crash bugs and constructors of faulty classes are additional prompt inputs used to simulate richer bug-report contexts.The prompt format has no real restriction, allowing these contextual variations.
B. Querying an LLM
LIBRO generates multiple test candidates from an LLM prompt, injects them into existing test classes, resolves dependencies, and executes them for evaluation.
- LIBRO queries the LLM with a prompt designed to generate a test method, stopping at the end of the Markdown code block.
- Generating multiple candidates with temperature 0.7 enables distinct tests from the same prompt and supports later assessment of reproduction likelihood.
- LIBRO injects generated test methods into existing test classes, matching developer workflow while providing dependencies and context.
- The framework selects the lexically most similar test class using shared tokens, methods, and classes as indicators of suitable context.
- A sanity check found that injected developer-added tests executed normally 89% of the time in the Math and Lang Defects4J projects.
2) Resolving remaining dependencies:
LIBRO postprocesses generated tests by inferring imports and resolving dependencies before execution, although compilation is not guaranteed in every case.
- LIBRO heuristically infers packages to import when generated tests introduce constructs absent from the selected test class.
- The dependency resolver parses generated methods, identifies referenced types and classes, and filters already imported names through lexical matching.
- When a type has exactly one matching public class, LIBRO derives its classpath and adds an import; otherwise, it searches project imports by class name.
- The postprocessing pipeline does not guarantee compilation, but its heuristics resolve most unhandled dependencies before tests are executed.
D. Selection and Ranking
LIBRO filters and ranks executable failing tests using failure agreement, report-matched behavior, and test length, while diversifying results across failure clusters.
- A Bug Reproducing Test fails specifically because of the reported bug, whereas an FIB test merely compiles and fails in the buggy program.
- LIBRO groups FIB tests with identical error types and messages, showing results when cluster agreement exceeds a configurable threshold.
- The ranking heuristics prioritize report-matched behavior, agreement among generations, and shorter tests in increasing discriminative strength.
- LIBRO selects tests from diverse clusters because tests sharing failure output are expected to have similar reproduction status.
IV. EVALUATION
This section provides evaluation details for the experiments.
- The section provides evaluation details for the experiments.
A. Dataset
The study evaluates LIBRO on curated Defects4J and a post-training-cutoff GHRB dataset, using bug/fixed-version execution criteria and comparisons with crash reproduction. It also defines research questions around reproduction performance, efficiency, and generalization.
- Dataset: Defects4J 2.0 provides 814 bugs with paired reports, reduced to 750 evaluable bugs after excluding incorrect pairings and directory mismatches.The benchmark contains real-world bugs from 17 Java projects.
- Dataset: GHRB contains 31 reproducible bugs gathered from post-July-2021 GitHub changes, mitigating concerns that Codex encountered the cases during training.The dataset was filtered from 581 pull requests through test-introduction, merge, issue-association, and reproduction checks.
- Evaluation criteria: A Bug Reproducing Test fails on the buggy version and passes on the fixed version; a bug is reproduced when LIBRO generates at least one such test.Defects4J uses pre-fix and post-fix revisions, while GHRB uses pre-merge and post-merge versions.
- Research design: The evaluation compares LIBRO with EvoCrash under the same execution-based criterion and with a Copy&Paste baseline extracted from report code snippets.The study also measures ranking using acc@n and precision, alongside developer inspection effort.
- Research questions: The research questions cover Defects4J performance, resource efficiency, developer inspection effort, and generalization to bugs observed in the wild.The questions include query counts, runtime, ranking reliability, and qualitative reproduction success or failure.
B. RQ2: Efficiency
LIBRO’s efficiency study examines query volume, runtime, and inspection effort while also evaluating prompt settings and reproduction across projects. Results indicate scalable gains from more attempts and a pipeline runtime within a commonly used search budget.
- Evaluation dimensions: The efficiency evaluation measures Codex queries, time for querying, execution, and ranking, and the number of suggestions developers must inspect.It also evaluates wasted effort and reproduction within 1, 3, and 5 suggestions.
- Prompt settings: Adding stack traces helps reproduce crash bugs, whereas providing constructors does not help significantly across prompt settings.The study varies the number and source of report-test examples, including project-specific examples.
- Reproduction performance: LIBRO reproduces 251 of 750 Defects4J bugs, or 33.5%, under the two-example n=50 setting.Performance varies by project, with stronger results for Lang and Jsoup and weaker results for Closure.
2) RQ1-2:
LIBRO reproduces bugs beyond crash-only baselines and ranks candidate tests using agreement-based selection and test prioritization. More generation attempts improve coverage logarithmically, while thresholding and ranking reduce inspection effort.
- Baseline comparison: LIBRO reproduces 91 more unique bugs than EvoCrash, including 19 crash bugs, showing coverage of non-crash bugs.Copy&Paste reproduces some additional cases, but long tests and complex helper dependencies can exceed LIBRO’s generation capacity.
- RQ2-1: More generation attempts yield logarithmic increases in median bug reproduction, with no observed plateau.The analysis samples generated tests repeatedly to estimate reproduction as attempts increase.
- Selection: At agreement threshold 1, LIBRO selects 350 of 570 bugs with a FIB and reproduces 219, achieving precision 0.63 and recall 0.87.Threshold 10 raises precision to 0.84 while reducing recall to 0.42.
- Ranking: LIBRO’s ranking finds 30, 14, and 7 more BRTs than random at n=1, 3, and 5, respectively.BRTs appear in 43% of top-ranked first suggestions and 57% of selected bugs within five suggestions.
- Developer effort: LIBRO reduces inspection effort by discarding 33% of bugs while preserving 87% of successful reproductions, with 80% of all reproductions found within five inspections.Its ranking also saves up to 14.5% of wasted effort versus random ordering.
C. RQ3. How well would LIBRO work in practice?
On recent bug reports, LIBRO reproduced bugs at a rate similar to Defects4J, and its selection and ranking indicators remained useful across datasets.
- RQ3-1: 32.2% of 31 GHRB bug reports were reproduced by LIBRO using 50 trials.LIBRO reproduced 10 bugs, with success observed in AssertJ, Jsoup, Gson, and sslcontext.
- RQ3-1: LIBRO can reproduce bugs in recent data, suggesting its performance is not simply due to training-data memorization.
- RQ3-2: The maximum output cluster size remains a useful reproduction indicator because bugs without BRTs typically have values below ten in both datasets.The measure captures agreement among FIB tests.
- RQ3-2: Ranking results on GHRB are consistent with Defects4J, indicating that LIBRO’s ranking features continue to predict successful reproduction.
3) RQ3-3:
Case studies show that LIBRO can synthesize a functional reproducing test from a sparse report, but generated tests may fail when required external files or environments are unavailable.
- RQ3-3: LIBRO reproduced an AssertJ locale bug by generating a test that compares I and i with containsIgnoringCase.The report contained no executable code, yet the generated test used the method named in the report.
- RQ3-3: A Checkstyle test initially failed because it referenced a nonexistent Java file, exposing LIBRO’s inability to create external working environments.After the reported file content was supplied, the test successfully reproduced the bug.
VII. DISCUSSION
LIBRO demonstrates report-to-test feasibility across benchmark and recent real-world data, but its success is constrained by project-specific helpers, external resources, report quality, and generalization limits.
- Manual Analysis of LIBRO Failures: 13 of 40 analyzed failures required helper definitions that LIBRO-generated tests could not adequately use.Other failure causes included low report quality, incorrect expected behavior, external resources, and insufficient synthesis length.
- Discussion: LIBRO successfully reproduced bugs from reports with full, partial, or no code snippets, indicating both extraction and test synthesis capabilities.Among successful cases, 25.1% had full snippets, 37.9% partial snippets, and 37.1% none.
- Threats to Validity: Generalization beyond the studied Java projects and languages remains difficult to assess, and Codex may have seen Defects4J during training.
- Threats to Validity: LIBRO does not generalize well to tests requiring external files or infrastructure unlike typical JUnit tests.The Closure project is given as an example of substantially different test structure.
- Discussion: The paper concludes that LIBRO can reproduce many benchmark and real-world bugs while identifying successful tests to reduce developer inspection effort.