PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark

Boxi Yu, Yang Cao, Yuzhong Zhang, Liting Lin, Junjielong Xu, Zhiqing Zhong, Qinghua Xu, Guancheng Wang, Jialun Cao, Shing-Chi Cheung, Pinjia He, Lionel Briand

arXiv:2603.00520v1cs.SE

TL;DR

SWE-Bench Verified success rates are inflated because weak tests allow semantically incorrect patches to pass. SWE-ABS strengthens evaluation through coverage-driven augmentation and mutation-driven adversarial testing, rejecting accepted patches and lowering the top agent's score from 78.80% to 62.20%.

  • Problem

    Weak benchmark test suites leave coverage gaps and semantic blind spots, allowing semantically incorrect solutions to pass and inflating apparent code-agent performance.

  • Method

    SWE-ABS uses program slicing to generate tests for patch-affected regions and synthesizes plausible incorrect mutants to generate adversarial tests for semantic blind spots.

  • Results

    19.78% of accepted SWE-Bench Verified patches are rejected, while 50.2% of instances are strengthened and the top agent's score falls from 78.80% to 62.20%.

  • Takeaways & Limitations

    SWE-ABS shows that leaderboard performance can substantially change when benchmark tests are strengthened against coverage and semantic failures.

  • Takeaways & Limitations

    Augmented tests can overfit to gold-patch-specific behavior, producing a 10.6% false-negative rate; intraprocedural slicing and gold-patch dependence also constrain applicability.

Abstract

from arXiv · show

The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we show that this performance is inflated. Our re-evaluation reveals that one in five "solved" patches from the top-30 agents are semantically incorrect, passing only because weak test suites fail to expose their errors. We present SWE-ABS, an adversarial framework that strengthens test suites through a two-stage pipeline: (1) coverage-driven augmentation using program slicing to target untested code regions, and (2) mutation-driven adversarial testing that synthesizes plausible but incorrect patches to expose semantic blind spots. On SWE-Bench Verified (500 instances), SWE-ABS strengthens 50.2% of instances, a 25.1x improvement over prior work, and rejects 19.71% of previously passing patches. As a result, the top agent's score decreases from 78.80% to 62.20%, leading to significant leaderboard reshuffling, with the previous top-ranked agent dropping to fifth place.

1. Introduction

SWE-Bench test suites can accept semantically incorrect patches because they inherit coverage gaps and semantic blind spots from development-oriented tests. SWE-ABS addresses these weaknesses with coverage-driven and mutation-driven strengthening, substantially changing evaluation outcomes and leaderboard rankings.

  • Motivation: 78.80% success by the top SWE-Bench agent suggests saturation, but 19.78% of accepted patches from top-30 agents are semantically incorrect.Strengthened tests reject 2,184 of 11,041 patches that passed the original SWE-Bench Verified tests.
  • Motivation: Coverage gaps and semantic blind spots arise because pull-request tests verify specific patches rather than discriminate among correct and incorrect solutions.Coverage gaps miss patch-affected code, while semantic blind spots accept superficially correct behavior without checking deeper requirements.
  • Motivation: 25 patches omitting str() conversion pass the original Django tests but fail augmented tests using non-string passwords.The example shows how testing only string passwords leaves the subprocess.run string-type requirement untested.
  • SWE-ABS: SWE-ABS combines coverage-driven augmentation using program slicing with mutation-driven adversarial testing of plausible but incorrect patches.The two stages target untested patch-affected regions and semantic blind spots, respectively.
  • Results: 50.2% of SWE-Bench Verified instances are strengthened, a 25.1× improvement over UTBoost.The evaluation covers 500 SWE-Bench Verified instances and includes strengthened suites for future evaluation.
  • Results: 19.78% of accepted patches are rejected, reducing the top agent from 78.80% to 62.20% and moving it from first to fifth place.The re-evaluation also induces 30 rank changes among the top-30 leaderboard agents.
  • Results: Comparable strengthening rates on SWE-Bench Verified and the more challenging SWE-Bench Pro indicate that task difficulty and test discriminativeness are orthogonal.The top system's resolve rates are 78.80% on Verified and 45.89% on Pro.

2. Related Work

Code-generation evaluation has progressed toward repository-level benchmarks, but plausible patches can still pass insufficient tests. Prior augmentation and mutation-testing approaches motivate SWE-ABS's explicit focus on patch-relevant regions and semantic correctness.

  • Benchmarks for Code Generation: Repository-level benchmarks such as SWE-Bench extend code-generation evaluation beyond function-level datasets like HumanEval and MBPP.Variants improve test validity, language coverage, contamination resistance, and task difficulty.
  • Test Augmentation: Plausible program-repair patches may pass insufficient tests without genuinely fixing the underlying bug.EvalPlus addressed test insufficiency in HumanEval and MBPP, while UTBoost exposed 345 previously undetected incorrect SWE-Bench patches.
  • Test Augmentation: Prior SWE-Bench augmentation generates tests without explicitly analyzing patch-relevant regions or targeting semantic correctness.This limits systematic detection of both coverage gaps and semantic blind spots.
  • Mutation Testing: SWE-ABS extends mutation testing by synthesizing LLM-based semantic mutations representing plausible but incorrect patches.Unlike traditional syntactic operators, these mutations target realistic semantic failures in repository-level repair.

3. Method

SWE-ABS strengthens SWE-Bench test suites against coverage gaps and semantic blind spots through coverage-guided augmentation and mutation-driven adversarial testing. It decouples generated tests from gold-patch implementation details, targets uncovered patch-relevant code, and adds tests that distinguish semantically incorrect mutants from correct alternatives.

  • Framework overview: SWE-Bench tests can miss patch-affected code and accept superficially correct behavior, so SWE-ABS targets both coverage gaps and semantic blind spots.The framework seeks an augmented suite that rejects incorrect patches while accepting correct alternative implementations.
  • Coverage-Driven Augmentation: Stage I uses program slicing to identify patch-relevant regions and generates tests for uncovered lines after decoupling tests from gold-patch-specific behavior.The slice captures upstream dependencies and downstream dependents within the same function or class, while coverage-guided generation addresses uncovered branches.
  • Coverage-Driven Augmentation: Test decoupling generalizes over-specialized generated tests so they verify issue resolution rather than enforcing implementation-specific details.The refined tests are checked to ensure the gold patch still passes them before entering the coverage-oriented suite.
  • Mutation-Driven Adversarial Strengthening: Stage II synthesizes plausible but incorrect mutants that pass existing tests, then filters them for issue relevance and semantic non-equivalence before generating targeted tests.Three independent LLM judgments per dimension are combined by majority voting in the reported experiments.
  • Mutation-Driven Adversarial Strengthening: The final suite fixes false negatives by generalizing overly strict tests and fixes false positives by adding tests that reject non-equivalent mutants.This combines generalized tests with new discriminative tests, complementing coverage-driven tests to improve coverage and semantic robustness.
  • Final Test Suite: The combined adversarial augmentation yields a suite that is comprehensive in coverage and robust against plausible but incorrect fixes.

4. Experiments

Experiments show that SWE-ABS strengthens benchmark tests, rejects many previously passing patches, reshapes agent rankings, generalizes across benchmarks and models, and benefits from combining coverage- and mutation-driven augmentation. The evaluation also identifies semantic-error patterns and an overfitting trade-off.

  • RQ1: Effectiveness: 50.2% of SWE-Bench Verified instances were strengthened, compared with 2% under UTBoost, while 19.78% of previously passing patches were rejected.SWE-ABS strengthened 251/500 instances and rejected 2,184 of 11,041 previously passing patches.
  • RQ1: Effectiveness: 14.56 percentage points was the average resolve-rate drop under SWE-ABS, compared with 0.70 points under UTBoost, and rank correlation fell from 0.98 to 0.82.TRAE dropped from first to fifth, while live-SWE-Agent rose from second to first.
  • RQ2: Generalization: SWE-ABS produced comparable strengthening across SWE-Bench Verified and the harder SWE-Bench Pro despite top-system resolve rates of 78.80% and 45.89%, respectively.Average resolve-rate drops were 14.56 points on Verified and 16.46 points on the SWE-Bench Pro subset.
  • RQ2: Generalization: SWE-ABS showed comparable behavior across GPT-5 and GLM-4.7, strengthening 31 of 50 instances with each model.The average resolve-rate drop was 19.40 points for GPT-5 and 15.93 points for GLM-4.7.
  • RQ3: Test Quality: 53 of 500 instances contained overfitting tests, yielding a conservative 10.6% false-negative rate before manual correction.The authors revised or removed gold-patch-specific assertions in these cases; 447 instances reflected legitimate strengthening.
  • Error Analysis: Logic errors accounted for 47% and incomplete fixes for 35% of identified error types, while syntactic errors accounted for 14%.The results characterize many failures as shallow solutions that satisfy explicit tests without semantic completeness.
  • RQ4: Ablation Study: Coverage augmentation improves structural adequacy, while mutation-driven augmentation exposes executable behaviors that violate intended semantics.Together, the stages operate along complementary dimensions of test adequacy.

5. Conclusions, Limitations, and Future Work

The paper concludes that adversarial strengthening exposes inflated code-agent performance and that benchmark difficulty does not guarantee test strength. It also identifies overfitting, intraprocedural analysis scope, and gold-patch dependence as limitations while proposing broader language and continuous-evaluation extensions.

  • Conclusions: SWE-ABS strengthens tests through coverage-driven generation and mutation-driven strengthening, revealing 19.78% semantically incorrect accepted patches.On SWE-Bench Verified, it strengthened 50.2% of instances and reduced the top agent’s score from 78.80% to 62.20%.
  • Conclusions: Task difficulty and test strength are orthogonal, so current leaderboard rankings may not reflect true code-agent capabilities.The conclusion connects cross-benchmark findings to the reliability of leaderboard comparisons.
  • Limitations: The approach can overfit to gold-patch behavior, miss cross-module dependencies through intraprocedural slicing, and require reference implementations.These limitations constrain robustness, analysis scope, and applicability without gold patches.
  • Future Work: Future work includes extending language coverage, integrating continuous benchmark co-evolution, and using strengthened tests as training signals.The proposed extensions target Java, C++, Rust, periodic test re-strengthening, and environments such as SWE-Gym.
  • Broader Applicability: The methodology may apply beyond program repair to domains with automated correctness checks, including code translation, text-to-SQL, and robotic control.The paper frames this as potential applicability rather than a demonstrated result in those domains.

Impact Statement

The paper argues that reliable test-based evaluation is necessary for meaningful decisions about AI coding systems. Its findings show that weak tests can distort research and deployment judgments, while adversarial strengthening can reveal failures before deployment and may extend to other automatically testable domains.

  • Reliable Evaluation: 19.78% of patches previously labeled “solved” were semantically incorrect, showing that weak tests can mislead evaluation decisions.The paper links this issue to researchers prioritizing future directions and practitioners making deployment decisions.
  • Reliable Evaluation: Harder benchmarks do not automatically provide more discriminative evaluations, because task difficulty and test strength are orthogonal.The paper therefore treats principled test strengthening as a distinct concern.
  • AI Safety: Strengthened benchmarks can surface omitted input validation or error handling before AI-generated patches reach production environments.The paper frames rigorous evaluation as a safety concern for deployed coding assistants.
  • Broader Applicability: Adversarial strengthening may apply to code translation, text-to-SQL, robotic control, and other domains with automated testing.The broader implication is maintaining sufficient discriminative power when test-based benchmarks measure AI systems.

A. Appendix

The appendix provides implementation details, supplementary evaluations, leaderboard results, test-quality analyses, case studies, computational-cost analysis, and a discussion of threats to validity.

  • Organization: Appendix A.1 lists the prompt templates used in the pipeline.
  • Organization: Appendix A.2 provides test-decoupling case studies, while A.3 details implementation, hyperparameters, and program slicing.
  • Organization: Appendix A.4 covers mutation cases and evaluation, and A.9 presents generalization analysis.
  • Organization: Appendices A.5 through A.7 report full leaderboard results, Bash-only results, and repository- and language-level experiments.
  • Organization: Appendix A.8 analyzes computational cost, while A.12 discusses limitations and threats to validity.
  • Organization: Appendices A.10 and A.11 cover test quality, failure case studies, and agent-generated patch failures.

A.1. Prompt Templates

The prompt templates guide staged test generation, test decoupling, coverage-guided augmentation, and mutation generation through constrained repository interactions. They emphasize behavior-focused, robust tests and surviving mutants that expose weaknesses in existing test suites.

  • A.1.1. STAGE I: INITIAL TEST GENERATION PROMPT: Generated tests must verify behavioral correctness, pass with the gold patch, and be consolidated into a new test file rather than modifying the original test file.The prompt explicitly prioritizes intended behavior over matching the gold patch’s exact output form.
  • A.1.1. STAGE I: INITIAL TEST GENERATION PROMPT: Initial test generation asks agents to understand the issue, inspect relevant source and tests, and expand the original test suite with general edge and stress cases.The workflow uses the gold patch and original test patch as references while requiring new tests to avoid overfitting to the implementation.
  • A.1.2. STAGE I: TEST DECOUPLING PROMPT: Test decoupling removes hard-coded error strings, unjustified ordering assumptions, and other unnecessary dependencies on the gold patch implementation.The revised assertions should preserve essential error semantics rather than exact formatting.
  • A.1.3. STAGE I: COVERAGE-GUIDED AUGMENTATION PROMPT: Coverage-guided augmentation identifies missed lines, updates existing tests to exercise uncovered paths, and iterates until coverage improves while all tests remain stable and passing.The workflow forbids creating new test files during augmentation and uses coverage feedback to direct revisions.
  • A.1.4. STAGE II: MUTANT GENERATION PROMPT: The mutation stage uses the supplied test patch and test execution command as the validation context for generated mutants.The prompt frames mutations as wrong but test-passing alternatives to the correct fix.
  • A.1.4. STAGE II: MUTANT GENERATION PROMPT: Mutation generation produces subtle semantic behavior changes that pass the supplied tests but do not implement the correct gold-patch fix.A surviving mutant is intended to expose weaknesses in the existing tests.

A.2. Test Decoupling Case Studies

The case study examines a TimeSeries error-message fix where exact-string tests were replaced with assertions over semantic message content. The decoupled test preserves the intended behavior while reducing sensitivity to legitimate formatting changes.

  • Problem: The issue concerns misleading errors when a required non-time column is removed from a TimeSeries object.The reported message incorrectly claims that the first column is invalid and can compare identical values.
  • Gold Fix: The gold fix reports the full required-column list and the actual columns found instead of hard-coding only the first required column.This makes the exception semantically accurate for multiple required columns.
  • Original Test: The original test asserted the exact exception string, tightly coupling validation to a particular formatting.Such coupling makes the test fragile under legitimate error-message refactoring.
  • Initial Generated Test (Overfitted): The generated overfitted test similarly assumed a specific phrasing and column representation despite being functionally correct.Its assertions therefore depended on details beyond the intended error semantics.
  • Decoupled Test: The decoupled test checks for semantic elements including expected and found markers, first-column wording, and the relevant column names.This preserves behavior validation while tolerating legitimate changes in message formatting.

A.3.1. HYPERPARAMETERS

The configuration sets bounded generation, mutation, augmentation, execution, and dependency-analysis budgets for the framework.

  • Generation Settings: The configured LLM temperature is 1.0 by default.This is the stated generation setting for the framework.
  • Generation and Mutation Budgets: The framework allows 3 test-generation attempts and generates at most 2 mutants per instance.These limits bound both initial test synthesis and adversarial mutation generation.
  • Generation and Mutation Budgets: Each adversarial mutant receives at most 3 test-augmentation attempts.The augmentation budget limits iterative strengthening after a mutant is identified.
  • Execution and Analysis Limits: Test execution is limited to 120 seconds, while dependency analysis is limited to 5 graph hops.These settings constrain runtime and program-dependency expansion.

A.3.2. PROGRAM SLICING IMPLEMENTATION

The implementation uses AST-based, scope-limited, bidirectional program slicing to identify executable code regions relevant to a patch. The resulting slice combines upstream and downstream dependencies within bounded propagation limits.

  • Parsing and Scope: The implementation performs intraprocedural slicing across Python, JavaScript, Go, and TypeScript using Tree-sitter parsing.The design targets data-flow and control-flow dependencies while supporting multiple source languages.
  • Parsing and Scope: AST parsing builds a line-to-scope mapping so slicing can remain limited to relevant functions, classes, or global scopes.Scope limitation prevents excessive propagation across unrelated code regions.
  • Executable Line Extraction: Executable-line extraction retains statements such as assignments, returns, and definitions while filtering comments, docstrings, and blank lines.Modified lines inside multi-line statements are mapped to the statement’s starting line.
  • Modification Filtering: Global modification filtering removes semantically minor imports and simple assignments while retaining changes such as definitions and control-flow statements.The filter focuses slicing on modifications with substantial semantic impact.
  • Dependency Analysis: Def-use analysis records variable definitions and uses at each line to construct data-dependency information.The analysis is implemented through an AST visitor pattern.
  • Bidirectional Slicing: Forward slicing follows def→use chains, whereas backward slicing follows use→def chains from modified lines.Together they capture both downstream uses and upstream definitions relevant to the patch.
  • Bidirectional Slicing: Propagation is bounded by k hops, typically k = 1 or k = 5, and may be restricted to the modified lines’ scope.These constraints balance dependency coverage against uncontrolled slice growth.
  • Slice Union: The final relevant-line set is the executable-line intersection of the union of forward and backward slices: Lrel = (Lfwd ∪ Lbwd) ∩ Lexecutable.This combines both dependency directions while excluding non-executable lines.

A.3.3. COVERAGE CALCULATION

Coverage-driven augmentation measures the percentage of patch-relevant lines executed by a test suite, using program slicing to identify those lines. Language-specific coverage tools are parsed into a standardized representation so the same formula applies across instances.

  • Coverage is computed as the percentage of patch-relevant lines executed by a test suite.Patch-relevant lines are identified through program slicing.
  • SWE-Bench instances use trace.py from the SWT-Bench repository to collect line-level execution traces.
  • SWE-Bench Pro standardizes coverage outputs across Python, JavaScript, Go, and TypeScript using language-specific tools.
  • All parsers produce CoverageResult records containing executed and missing lines per file before the common coverage formula is applied.

A.3.4. SPEARMAN CALCULATION

SWE-ABS reveals substantial instability in agent rankings and identifies limitations that constrain the reliability and generalizability of its evaluation.

  • Spearman ρ falls from 0.98 under UTBoost to 0.79 under SWE-ABS, indicating substantial ranking instability across 30 agents.The comparison suggests that augmented tests can materially change leaderboard ordering.
  • SWE-ABS can expose semantic failures that ordinary tests miss, including incomplete handling of mixed-case inputs and non-string passwords.These examples show how patches can pass existing tests while violating broader behavioral requirements.
  • 10.6% of instances contain overfitting tests that incorrectly reject valid alternative patches by encoding gold-patch-specific behavior.The paper reports 53 such instances out of 500 despite using test decoupling.
  • Intraprocedural slicing may miss cross-module dependencies, limiting coverage identification when patches interact with utilities used under different preconditions.Interprocedural analysis could improve coverage but may be computationally prohibitive at benchmark scale.
  • SWE-ABS requires gold patches for test generation, decoupling, and mutation synthesis, limiting use where only issue descriptions are available.The stated boundary includes real-time bug triage and fully automated repair pipelines.
  • The evaluation covers four languages and strong-copyleft repositories, so effectiveness may not generalize to other languages, proprietary codebases, or different programming paradigms.The paper specifically identifies functional and low-level systems languages as outside the evaluated scope.
Loading 2603.00520v1…