PaperlayerView exact revision on arXiv

Source-linked AI summary

TestGRAD: Evolving Test Suites via Failure Pattern Momentum for SWE-Agent Ensemble

Pengfei He, Jiayuan Zhou, Shaowei Wang, Ruiqi Pan

arXiv:2610.10242v1cs.SEcs.AI

TL;DR

SWE-agent ensembles need tests that distinguish candidate patches by behavior, but existing test-generation approaches often lack an explicit selection objective, full repository-test updates, and iterative use of failures. TestGRAD evolves executable test suites using differential loss, CRUD edits, and mined failure momentum. On SWE-bench Verified, it reaches 84.2% Pass@1 with a 4-agent ensemble, 3.6 percentage points above the strongest baseline, while compressing failure-history context by over 100×.

  • Problem

    Test-generation methods often lack an explicit objective for separating competing patches, full test-suite update directions, and feedback from repeated failures.

    Source: Introduction · p. 2

  • Method

    TestGRAD iteratively evolves repository test suites using execution-defined differential loss, full CRUD updates, and Failure Pattern Momentum mined from failed updates.

    Source: Introduction · pp. 1–2

  • Results

    84.2% Pass@1 on SWE-bench Verified with a 4-agent ensemble exceeds TRAE’s 80.6% by 3.6 percentage points.

    Source: Introduction · p. 2, Conclusion · p. 8

  • Takeaways & Limitations

    The reported results support test-suite optimization as an approach to selecting among SWE-agent ensemble patches, with failure history retained in compressed form.

    Source: Conclusion · p. 8

Abstract

from arXiv · show

SWE-agent ensembles improve issue resolution by combining candidate patches from different agents with complementary strengths. The central problem is therefore test-based selection: generate tests, execute candidate patches, and identify the best patch. We formulate this process as test-space optimization: evolving an executable repository test suite until it distinguishes competing patches. Existing test-generation methods are limited optimizers. They usually lack an explicit loss for ensemble selection, optimize through incomplete directions that mostly create new tests or delete old ones, and perform one-off generation without feedback from repeated failures. Inspired by gradient descent with momentum, we introduce TestGRAD, a framework for automatic test optimization. TestGRAD centers on three concepts. Differential loss gives the optimizer an explicit execution-defined target: useful tests should separate candidate patches by behavior. Full CRUD gradients expand the update direction from merely creating or deleting tests to reading existing test infrastructure, creating new tests, updating stale assertions, and deleting only obsolete tests. Failure Pattern Momentum mines frequent failure sequences from memory, allowing the optimizer to avoid repeated non-discriminative directions while compressing the failure-history context. On SWE-bench Verified, TestGRAD achieves 84.2% Pass@1 with a 4-agent ensemble, outperforming the strongest baseline (80.6%) by an absolute improvement of 3.6 percentage points, while compressing failure-history context by over $100\times$.

Introduction

TestGRAD treats SWE-agent ensemble selection as test-space optimization: repository tests are evolved to distinguish candidate patches through execution. It combines differential loss, full CRUD updates, and Failure Pattern Momentum to guide that evolution.

Introduction

  • Test-based selection executes candidate patches against repository tests and uses behavioral differences to evolve the suite, rather than relying on patch plausibility.TestGRAD frames the suite as the optimization state, differential execution as the loss, CRUD edits as update directions, and the LLM agent as optimizer.Source: Introduction · p. 1, Table 1 · p. 2
  • Differential loss reaches zero when at least one test passes on some candidate patches and fails on others, seeking behavioral contrast rather than the highest pass count.The loss remains nonzero otherwise, and the failed update guides the next iteration; this prunes candidates indistinguishable under the current suite.Source: Introduction · pp. 1–2
  • Full CRUD updates read repository testing knowledge, create discriminative tests, revise stale assertions or test bodies, and delete only tests confirmed obsolete.This direction reuses fixtures and conventions, preserves tests that need assertion updates, and avoids rejecting a correct patch because an old assertion remains unchanged.Source: Introduction · p. 2
  • Failure Pattern Momentum mines frequent failed-update sequences from memory to steer later iterations away from recurring non-discriminative directions.The deterministic mechanism compresses failure-history context by over 100×.Source: Introduction · p. 2
  • 84.2% Pass@1 on SWE-bench Verified beats TRAE’s 80.6% by 3.6 percentage points, with gains consistent across all three base LLMs.Ablations report drops of 3.0% without Read, 2.4% without Update, 3.4% when replacing differential loss with pass-count selection, 1.6% without Failure Pattern Momentum, and 9.2% without iterative optimization.Source: Introduction · p. 2

Related Work

Prior test-generation methods often use incomplete Create/Delete updates, lack a differential objective, or do not turn failed attempts into iterative test-suite updates. TestGRAD connects repository-native test editing with execution-defined optimization and mined failure history.

Test Generation for SWE-Agent Ensemble

  • Earlier approaches create issue-based tests or standalone scripts, but often omit repository-native updates, differential optimization, or iteration from non-discriminative failures.LIBRO and Agentless use Create/Delete-style pipelines; AEGIS generates standalone scripts, while TRAE adds browsing but retains Create/Delete updates without differential loss or momentum.Source: Test Generation for SWE-Agent Ensemble · p. 3

Reflection and Text-Space Optimization

  • Reflection methods use LLM-generated signals, while AgentCoder feeds raw execution results back without abstracting recurring failures across attempts.The cited discussion describes LLM evaluation as costly and potentially hallucinatory for fine-grained code correctness.Source: Reflection and Text-Space Optimization · p. 3
  • TextGrad-style frameworks optimize general textual variables, whereas TestGRAD constrains feedback to execution of candidate patches and CRUD edits to repository tests.The supplied discussion distinguishes general text-optimization frameworks from test-time SWE-agent ensemble optimization.Source: Reflection and Text-Space Optimization · p. 3
  • Sequential mining extracts recurring failed strategies across superficially different patches and carries them into the next optimizer step, compressing memory by over 100×.Unlike LLM summaries or raw feedback, Failure Pattern Momentum mines maximal frequent subsequences from failure memory.Source: Reflection and Text-Space Optimization · p. 3

TestGRAD Framework

TestGRAD casts ensemble selection as test-suite optimization: it executes a repository suite across candidate patches, uses differential behavior and recurring failures to guide updates, and iterates until tests distinguish candidates or the budget runs out.

TestGRAD Framework

  • The framework treats the test suite as the optimization state, differential execution as loss, CRUD edits as gradient directions, and recurring failure sequences as momentum.Its loop executes the current suite, updates failure memory and momentum, then sends that signal into the next test update.Source: TestGRAD Framework · p. 3

Problem and Test-Space State

  • The optimization state is the current repository test suite, initialized from the original suite before issue-specific updates.The instance comprises an issue description and initial repository, with candidate patches applied to that repository.Source: Problem and Test-Space State · p. 3

Forward Pass: Execute Candidate Patches

  • A useful suite must contain a differential test that passes on some candidate patches and fails on others; uniform outcomes provide no selection signal.The forward pass executes the current suite against every candidate patch.Source: Forward Pass: Execute Candidate Patches · pp. 3–4

Loss: Differential Execution

  • The execution-defined loss is zero when the suite contains a differential test and one otherwise; zero stops optimization and invokes a separate checker for patch selection.When loss is one, the failed update guides the next iteration.Source: Loss: Differential Execution · p. 4

Memory Update and Momentum

  • Non-differential updates enter failure memory, where maximal frequent failure sequences form compact momentum that suppresses one-off edits and records repeatedly explored directions.The default support threshold is β = 2; lowering it retains more recurring sequences, while raising it keeps more stable ones.Source: Memory Update and Momentum · p. 4

Backward Pass: CRUD Test Update

  • The backward-pass optimizer receives the current suite, execution-defined loss, and mined momentum, then returns a CRUD edit as the next test-space gradient.The new suite enters another forward pass, continuing until a differential test is found or the iteration budget is exhausted.Source: Backward Pass: CRUD Test Update · p. 4

Experimental Setup

The experiments evaluate ensemble selection on SWE-bench Verified using publicly available patches from six leaderboard submissions. TestGRAD is compared with three selectors that differ in how they select among the same candidate patches.

Metrics

  • Pass@1 measures the fraction of instances resolved after the checker selects one patch; Ensemble Gain is the absolute Pass@1 improvement over the best single-agent source, Livesweagent at 79.2%.All methods receive the same candidate patches and differ only in their selection module.Source: Experimental Setup · pp. 4–5

Dataset

  • SWE-bench Verified contains 500 human-validated real-world GitHub issues, with candidate patches drawn incrementally from six leaderboard submissions to form ensembles of one to six agents.The patches are publicly available, supporting reproducibility.Source: Experimental Setup · p. 4

Baselines

  • Augment selects the most plausible patch with an LLM judge, while Agentless generates an issue-based test and uses name-matching deletion without repository inspection or iterative optimization.TRAE adds repository browsing to the Create-Delete pipeline but lacks Update and does not refine failing tests.Source: Experimental Setup · p. 5

Implementation

  • TestGRAD uses Minimax-m2.7, DeepSeek-v3.2, or GLM-5.1 as optimizer models, with at most eight optimization steps and a failure-sequence support threshold β = 2.Runs execute in official SWE-bench Docker images.Source: Experimental Setup · p. 5

Results

TestGRAD improves ensemble patch selection across three base LLMs, with performance peaking at four agents. Ablations and token comparisons report contributions from differential loss, CRUD updates, and Failure Pattern Momentum.

Impact of Ensemble Size

  • At N=4, Pass@1 peaks at 84.2% for Minimax, 84.0% for DeepSeek, and 83.6% for GLM, then declines slightly at N=5 and N=6.For Minimax, the gap over TRAE widens from +1.2% at N=2 to +3.6% at N=4; at N=4, the Oracle is 87.0%, 2.8% above TestGRAD.Source: Impact of Ensemble Size · pp. 5–6, Figure 4 · p. 6

Overall Effectiveness of TestGRAD

  • 84.2% Pass@1 with Minimax-m2.7, 84.0% with DeepSeek-v3.2, and 83.6% with GLM-5.1; TestGRAD surpasses Agentless and TRAE by 3.4–3.6%.Table 2 reports the performance comparison, with gains calculated relative to the 79.2% single-agent baseline.Source: Overall Effectiveness of TestGRAD · p. 5, Table 2 · p. 6

Contribution of Differential Loss and CRUD Gradients

  • Removing differential loss, Read, Update, or both Read and Update reduces Pass@1 by 3.4%, 3.0%, 2.4%, and 4.8%, respectively.The combined Read-and-Update ablation reaches 79.4%; the reported drops are from the 84.2% complete system.Source: Contribution of Differential Loss and… · p. 7

Impact of Failure Pattern Momentum and Token Efficiency

  • Removing Failure Pattern Momentum reduces Pass@1 by 1.6%; removing iterative optimization reduces it by 9.2%, to 75.0%.Momentum mines recurring failure sequences to steer later updates away from dead ends.Source: Impact of Failure Pattern Momentum and… · p. 7
  • Momentum compresses failure-history context by over 100× for DeepSeek at later iterations and by 25×–40× for GLM and Minimax.At iteration 5, raw memory reaches about 1.9K–13.6K tokens, while momentum remains from tens to about a hundred tokens.Source: Impact of Failure Pattern Momentum and… · p. 7

Conclusion

TestGRAD iteratively evolves repository test suites using differential execution, CRUD edits, and Failure Pattern Momentum. On SWE-bench Verified, it reaches 84.2% Pass@1 with a four-agent ensemble while reducing failure-history context by over 100×.

Conclusion

  • TestGRAD treats the test suite as an optimization state, differential execution as loss, CRUD edits as gradients, and recurring failures as retained momentum.The checker is invoked for final patch selection only after zero-loss differential execution.Source: Conclusion · pp. 7–8
  • 84.2% Pass@1 on SWE-bench Verified with a four-agent ensemble, outperforming the strongest baseline by 3.6%.Failure Pattern Momentum reduces raw failure-history context by over 100×.Source: Conclusion · p. 8

Motivation

Test-based selection is needed to evaluate candidate patches through repository behavior. The paper motivates full CRUD test-suite updates and compact memory of recurring failures as alternatives to incomplete test-generation and failure-history strategies.

CRUD Gradient over Test Suites

  • C/D-only generation can produce fragile, non-discriminative tests and delete valid tests by name; CRUD instead reads project conventions, updates stale assertions, and deletes obsolete tests.In the Django example, reading existing infrastructure avoids hand-written setup, while updating test_render preserves a test that can distinguish the correct patch.Source: Figure 1 · p. 10, CRUD Gradient over Test Suites · p. 11

Failure Pattern Momentum

  • LLM summaries can omit or distort failure details, while raw histories grow redundantly; TestGRAD mines maximal frequent failure sequences as compact momentum for later updates.Repeated failures may reflect the same underlying strategy despite superficial differences.Source: Failure Pattern Momentum · p. 11

Human–LLM Collaboration for Aspect Discovery in Test Suite Understanding

The Read step uses a six-category checklist of repository-specific testing knowledge, derived from sampled SWE-bench instances. Human–LLM relabeling and aspect-frequency findings support the taxonomy’s reliability and prevalence.

Human–LLM Collaboration for Aspect Discovery in Test Suite Understanding

  • The Read checklist covers imports and APIs, assertion conventions, fixtures, test base classes, expected values, and helper functions.The six categories were formed by retaining aspects present in at least 50% of sampled instances.Source: Human–LLM Collaboration for Aspect… · p. 11
  • One author independently relabeled 40 instances, with human–LLM agreement reaching Cohen’s κ=0.82.The 200-instance sample excluded SWE-bench Verified to avoid data leakage.Source: Human–LLM Collaboration for Aspect… · p. 11
  • All six retained aspects appear in over 60% of test files; project APIs/imports and assertion conventions each exceed 90%.The taxonomy comes from 200 sampled SWE-bench instances, excluding SWE-bench Verified.Source: Human–LLM Collaboration for Aspect… · p. 12, Table 1 · p. 12

Overhead Analysis

TestGRAD spends substantially more tokens than the baselines, but runs faster than TRAE and yields higher Pass@1 than the single-agent and next-best methods. Table 2 reports average per-instance overhead across methods and base LLMs, with baseline costs estimated from logs and traces.

Overhead Analysis

  • Table 2 reports average per-instance time and token use across four methods and three base LLMs; TestGRAD costs come from run logs, while baseline costs are estimates.Time is measured in minutes and tokens in thousands.Source: Table 2 · p. 12, Overhead Analysis · p. 12
  • TestGRAD uses 670K–1,670K tokens per instance because Read explores the repository each round and optimization can run up to Tmax = 8 steps.The paper attributes most of the accuracy gain to holistic test-suite understanding and iterative momentum optimization.Source: Overhead Analysis · p. 12
  • TestGRAD is faster than TRAE on all three base LLMs, including 12.2 vs. 19.3 minutes on Minimaxm2.7.Optimization runs sequentially only when needed and stops early once it finds a discriminative test.Source: Overhead Analysis · p. 12
  • +5.0% Pass@1 over the single-agent baseline and +3.6% over the next-best method accompany TestGRAD’s higher token cost.The paper frames the additional cost as a tradeoff for applications where selection quality is critical.Source: Overhead Analysis · p. 12
Loading 2610.10242v1…