PaperlayerView exact revision on arXiv

Source-linked AI summary

SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents

Niels Mündler, Mark Niklas Müller, Jingxuan He, Martin Vechev

arXiv:2406.12952v3cs.SEcs.AIcs.LG

TL;DR

SWT-BENCH addresses the limited study of LLM-based Code Agents for generating tests from real-world issue reports. It benchmarks issue reproduction and changed-code coverage, finding that code-repair agents outperform specialized test-generation approaches and that generated tests can improve code-fix selection.

  • Problem

    Automatic test generation, particularly by Code Agents for reproducing real-world issues, has received comparatively little study despite testing’s importance for software reliability.

  • Method

    SWT-BENCH transforms over 1,900 Python GitHub issues with golden patches and tests into a benchmark using fail-to-pass reproduction and golden-patch change coverage.

  • Results

    Code Agents designed for code repair outperform non-agent test-generation methods, and an ideal ensemble of four methods solves 71% more samples than the best single method.

  • Takeaways & Limitations

    Generated tests provide a useful signal for evaluating proposed code fixes, while issue reproduction and code repair remain distinct tasks.

  • Takeaways & Limitations

    SWT-BENCH is limited to Python and popular GitHub repositories, may not represent common development practices, and focuses on bug reproduction rather than edge-case detection or global coverage.

Abstract

from arXiv · show

Rigorous software testing is crucial for developing and maintaining high-quality code, making automated test generation a promising avenue for both improving software quality and boosting the effectiveness of code generation methods. However, while code generation with Large Language Models (LLMs) is an extraordinarily active research area, test generation remains relatively unexplored. We address this gap and investigate the capability of LLM-based Code Agents to formalize user issues into test cases. To this end, we propose a novel benchmark based on popular GitHub repositories, containing real-world issues, ground-truth bug-fixes, and golden tests. We find that LLMs generally perform surprisingly well at generating relevant test cases, with Code Agents designed for code repair exceeding the performance of systems designed specifically for test generation. Further, as test generation is a similar but more structured task than code generation, it allows for a more fine-grained analysis using issue reproduction rate and coverage changes, providing a dual metric for analyzing systems designed for code repair. Finally, we find that generated tests are an effective filter for proposed code fixes, doubling the precision of SWE-Agent. We release all data and code at https://github.com/logic-star-ai/SWT-Bench

1 Introduction

SWT-BENCH addresses limited research on Code Agents for issue-reproducing test generation by transforming real-world GitHub issues, codebases, patches, and tests into an evaluation benchmark. Its evaluation finds that adapted code-repair agents perform strongly and can outperform methods designed specifically for test generation.

  • Research gap: Code Agents were largely unexplored for automatic test generation, especially for reproducing issues from detailed natural-language reports.Prior work included symbolic execution, specialized transformers, and general-purpose LLMs, but lacked broad Python test-generation datasets.
  • Benchmark motivation and design: SWT-BENCH provides real-world issues, codebases, golden patches, and tests for evaluating issue-reproducing test generation.A test reproduces an issue when it fails before the golden patch and passes afterward.
  • Evaluation design: The benchmark evaluates issue reproduction through fail-to-pass tests and coverage of code changed by the golden patch.These metrics connect generated tests to both reproduction success and the tested scope of the fix.
  • Main findings: Code Agents adapted from code repair outperform non-agent test-generation methods in issue reproduction and coverage, while different approaches remain complementary.An ideal ensemble of the four best methods solves 71% more samples than the best single method.
  • Contributions: The study proposes adapting Code Agents to test generation and reports that they perform well despite issue reproduction remaining difficult.The benchmark is presented as an extensive evaluation of real-world repositories, issues, patches, and test cases.

2 Related Work

Existing code benchmarks and test-generation datasets provide limited coverage of real-world repository-level issue reproduction, particularly for Python. SWT-BENCH extends this setting with larger, more recent Python issues and evaluates both traditional test-generation methods and Code Agents.

  • Code datasets: Function-level benchmarks often use interview-style tasks and insufficient tests, failing to capture real-world repository complexity.This limits their ability to assess correctness of generated code in realistic codebases.
  • Repository-level benchmarks: Repository-level benchmarks such as SWE-BENCH assess code generation and repair, but were not designed to evaluate test generation.This leaves issue-reproducing test generation without a corresponding benchmark in that setting.
  • Bug-reproduction datasets: Defects4J is the main real-world issue-reproduction dataset, but it focuses on outdated, limited Java data with short bug descriptions.SWT-BENCH instead targets Python, uses more recent issue reports, and is substantially larger.
  • Automated unit-test generation: Automated unit-test generation methods use symbolic execution, specialized transformers, and general-purpose LLMs for coverage, edge-case discovery, or issue reproduction.These approaches differ in both technique and intended testing objective.
  • Code Agents: Code Agents use tools to observe, search, read, and edit software across multiple turns, supporting complex code repair and synthesis tasks.Their interactive capabilities motivate adapting them to test generation.

3 Benchmarking Test Generation

SWT-BENCH transforms real-world Python code-repair tasks into issue-reproducing test-generation instances and evaluates generated tests through reproduction success, change coverage, and patch well-formedness.

  • Notation and Definitions: A test reproduces an issue when it fails on the original codebase and passes after the golden patch, while the benchmark also tracks other pass/fail transitions.The execution function classifies each test as passing or failing, enabling F →P, F →F, P →P, and P →F analyses.
  • Benchmark Construction: SWT-BENCH is constructed from SWE-BENCH by scraping, filtering, and validating Python pull requests with issue-resolving test changes.The resulting instances include a GitHub issue, golden patch, and golden reference tests.
  • Benchmark Construction: 1,983 instances remain in SWT-BENCH after excluding 311 whose golden patches were unevaluable or unreliable.The excluded cases involved flaky test suites, erroneous setup, or timeouts; SWT-BENCH-LITE contains 276 issues for cheaper evaluation.
  • Benchmark Overview: SWT-BENCH instances contain complex repositories and issue-specific test changes, with golden tests adding 2.9 new cases on average and roughly two-thirds being F →P.Repositories average over 1,500 files and 300,000 lines of code, while test patches edit 31.8 lines in one or two files on average.
  • Metrics: Success rate measures issue reproduction, change coverage measures executable golden-patch lines exercised by generated tests, and well-formedness measures whether generated patches apply.Change coverage includes removed or modified lines in the original codebase and added or modified lines in the patched codebase.

4 Automatic Test Generation

The paper reframes issue reproduction as test generation and introduces a fault-tolerant patch format alongside direct LLM and Code Agent methods. Code Agents originally designed for repair are adapted to generate tests, with explicit execution checks improving SWE-AGENT.

  • 4.1 Test Generation vs Code Repair: Test generation predicts tests that fail on the original repository and pass after the golden fix, rather than predicting a repair patch.The task also differs because generated tests may be correct but irrelevant, incompletely expose the bug, or fail in other ways.
  • 4.1 Test Generation vs Code Repair: The custom diff format lets models insert, replace, or delete complete functions or classes at fault-tolerant code locations.It is designed to reduce sensitivity to exact line numbers and verbatim context required by unified diffs.
  • 4.3 Direct LLM Generation of Tests: The evaluation compares unified-diff prompting, the custom-format baseline, oracle-selected PASS@5, and heuristic-selection LIBRO.All methods are instructed to add tests reproducing and covering the described issue.
  • 4.4 Code Agents for Test Generation: SWE-AGENT, AIDER, and AUTOCODEROVER are adapted from repair to test generation by changing their instructions to create issue-covering unit tests.The underlying agents retain their distinct search, editing, indexing, validation, or staged-context workflows.
  • 4.4 Code Agents for Test Generation: 18.5% success rate is achieved by SWE-AGENT+ versus 15.9% for SWE-AGENT when agents are instructed to execute generated tests before submission.The instruction provides no test-running command, unlike the LIBRO setting.

5 Experimental Evaluation

The evaluation finds that Code Agents, especially SWE-AGENT+, outperform direct and specialized test-generation methods, while performance depends on model capability and retrieved context. Generated tests also improve code-fix filtering precision, although test generation and repair solve largely distinct instances.

  • 5.2 Automatic Test Generation: 18.5% success rate for SWE-AGENT+ exceeds SWE-AGENT at 15.9% and LIBRO at 14.1%, while PASS@5 reaches 20.3%.ZEROSHOTPLUS raises well-formed patches from 48.6% to 89.5% and reaches 9.4% success; AUTOCODEROVER and AIDER reach 9.1% and 12.7%.
  • 5.2 Automatic Test Generation: SWE-AGENT+ achieves higher coverage on successful instances than SWE-AGENT, while LIBRO has lower coverage than most Code Agents.Coverage is higher on successful instances across methods, indicating that change coverage is correlated with test quality but more granular than success rate.
  • 5.2 Automatic Test Generation: SWE-AGENT performance is highly sensitive to the underlying LLM, with Claude 3.5 Sonnet and Mistral Large 2 performing on par with GPT-4 and Mistral Large 2 outperforming it.Haiku, GPT-4o mini, and Mixtral perform significantly worse on both success rate and applicability.
  • 5.3 Code Repair and Test Generation: 15.1% success rate results when ZEROSHOTPLUS receives the test file to modify, versus 8.1% without it; providing patch context raises success only to 10.5%.GPT-4 also increases relevant-line coverage from 12.5% to 18.4% when given the correct patch.
  • 5.3 Code Repair and Test Generation: 47.8% precision is achieved by filtering SWE-AGENT fixes with generated tests, more than doubling precision at 20% recall.The filter retains fixes for which all generated tests are F →P or P →P.
  • 5.4 Analysis: The overlap between solved SWE-BENCH and SWT-BENCH instances is small, with no statistical evidence of correlation for ZEROSHOTPLUS or SWE-AGENT.The reported p-values are 80.4% and 72.8%, respectively, under the stated independence and uniform-hardness null hypothesis.
  • 5.4 Analysis: Combining four test-generation approaches solves 87 instances, compared with 51 solved by SWE-AGENT+ alone.The result indicates complementarity among methods.

6 Limitations and Future Work

SWT-BENCH is constrained by language, dataset representativeness, issue scope, and possible knowledge-cutoff contamination. Future work targets broader evaluation and monitoring recurring failures in the strongest agent.

  • Python-only evaluation may limit how well SWT-BENCH findings generalize to other programming languages.
  • Popular GitHub repositories may not represent common software development practices and prevent a private holdout test set.
  • The benchmark covers bug reproduction issues amenable to added tests, not edge-case detection or global coverage increase.
  • Most issues predate model knowledge cutoffs, creating a risk of data contamination.
  • Future monitoring could address passing non-reproducing tests, edit-command loops, test-environment failures, syntax errors, and invalid variables.

7 Conclusion

The paper introduces SWT-BENCH and evaluates LLM-based test-generation approaches for reproducing GitHub issues. It concludes that Code Agents perform strongly and that generated tests can help assess proposed fixes.

  • SWT-BENCH evaluates generated tests by checking whether they fail before a golden patch and pass afterward, while measuring reproduction and patch coverage.
  • Code Agents outperform other evaluated approaches at generating reproducing tests with only minor task-specific adaptations.
  • The benchmark repurposes SWE-BENCH data with issue descriptions, codebases, golden patches, and reference tests for test generation.
  • Generated tests can filter proposed code fixes, doubling Code Agents’ precision when patches pass previously failing self-generated tests.

C.1 Ablation on number of LIBRO samples

LIBRO improves as its sample count increases, but gains become marginal after approximately five samples, which is the selected default for cost comparability.

  • LIBRO’s performance improves with more samples, but additional gains are marginal from around five samples.
  • The study uses five LIBRO samples to enable comparison at a cost similar to Code Agents.

C.2 Ablation on Interaction Rounds for Code Agents

Increasing Code Agent interaction rounds generally improves performance until saturation at five to ten iterations. AutoCodeRover is the exception, continuing to gain through the tested maximum of twenty iterations.

  • Performance generally improves with more interaction rounds until saturation at 5–10 iterations, while AutoCodeRover continues gaining through 20.
  • The ablation varies the number of agent interactions used before submitting a solution.

C.3 Ablation on Temperature

At increasing temperature, ZEROSHOTPLUS performance tends to decline while metric variance rises. Greedy decoding at T = 0 shows minimal test-environment variability across runs.

  • C.3 Ablation on Temperature: Performance decreases and variance increases across all metrics as temperature rises from T = 0 to T = 0.7.ZEROSHOTPLUS uses GPT-4 with 25 samples; T = 0 is used for ZEROSHOT, ZEROSHOTPLUS, and agent settings, while LIBRO uses T = 0.7.
  • C.3 Ablation on Temperature: At T = 0, repeated test-environment runs show minimal variability.The observed variability is smaller than the variation at higher temperatures discussed in the passage.
  • Prompt configurations: The appendix displays prompts for ZEROSHOT, ZEROSHOTPLUS, SWE-AGENT, AUTOCODEROVER, and AIDER.These figures document the prompting setups used in the evaluated methods.

H Execution times

Execution time is measured across five instances for the evaluated methods. Token processing contributes to runtime, while environment setup usually dominates interactive methods.

  • H Execution times: Interactive methods are usually dominated by clean, reproducible environment setup rather than model token processing.The environments are dockerized for clean and reproducible evaluation.
  • H Execution times: Execution time is partly related to the number of tokens processed and generated by the language model.The token-related contribution is reported alongside execution-time results.
  • H Execution times: The methods are evaluated on five instances, with average execution times reported in Table 11.The passage states that execution-time results are listed in Table 11.
Loading 2406.12952v3…