PaperlayerView exact revision on arXiv

Source-linked AI summary

ExecCritic: Learn to Test, Test to Improve for Coding Agents

Leitian Tao, Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, Sharon Li, Jianfeng Gao

arXiv:2609.09133v1cs.AIcs.CLcs.SE

TL;DR

Coding-agent feedback is useful only when generated tests capture the behavior requested by an issue, since shared patch-and-test errors can create false confidence. ExecCritic separates test construction from source repair with independently generated, qualified, fixed tests and role-specific training; on SWE-bench Verified, the composed agents reach 72.6%, 11.4 points above the original no-test baseline.

  • Problem

    Agent-generated tests can miss required behavior or share a mistaken interpretation with the patch, making successful checks insufficient evidence of task correctness.

  • Method

    ExecCritic uses a test–verify–revise scaffold with a Test agent that generates qualified fixed tests and a Repair agent that revises source code from their feedback, trained with role-specific objectives.

  • Results

    72.6% composed-agent success on SWE-bench Verified is 11.4 points above the original no-test baseline without stronger-model or Oracle feedback at evaluation.

  • Takeaways & Limitations

    Executable feedback helps or harms repair according to test quality, supporting independently generated fixed tests and learned specialization for both roles.

  • Takeaways & Limitations

    The framework trains Test and Repair as separate policies because their action spaces, trajectory structures, and learning signals differ; qualification also does not by itself prove semantic alignment.

Abstract

from arXiv · show

Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence. We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it. The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. In Learn to Test, the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches. In Test to Improve, the Repair agent learns both direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%. Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%; composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline without stronger-model or Oracle feedback at evaluation time. Code is publicly available at https://github.com/MSR-Orchard/execcritic.

1 INTRODUCTION

ExecCritic separates test construction from source-code repair so execution feedback measures the issue’s requested behavior rather than a shared mistaken interpretation. Its role-specific training improves test quality and repair, with composed agents reaching 72.6% on SWE-bench Verified.

  • Motivation: Agent-written tests can omit decisive edge cases or share a mistaken interpretation with the patch, creating false validation.This motivates independently improving test quality and separating test validation from source-code revision.
  • Results: Test quality determines whether executable feedback helps or harms repair.Holding the Qwen Repair agent fixed, Qwen-generated tests reduce resolved rate from 61.2% to 57.3%, while GPT-5.6-generated tests raise it to 65.3%.
  • Approach: The scaffold separates independently generated test construction from source-code revision and keeps qualified tests fixed during repair.A Test agent defines the behavioral target, while the harness validates and freezes it before the Repair agent receives execution feedback.
  • Training: Role-specific reinforcement learning trains the Test agent to generate behaviorally valid, discriminative tests and the Repair agent to revise source code from feedback.The Repair agent also learns direct task resolution, while the Test agent learns repository-native testing behavior.
  • Results: 72.6% composed-agent success on SWE-bench Verified is 11.4 points above the original no-test baseline without stronger-model or Oracle feedback at evaluation.The reported composed gain includes additional test-generation and revision computation.

2 MOTIVATION: REPAIR AND VALIDATION ARE COUPLED

Coding agents may submit patches without task-specific validation, while self-generated tests can share the patch’s incomplete interpretation of the issue. ExecCritic separates test generation from repair and freezes independently generated checks during revision.

  • Without local task-specific validation, a patch can reach the official evaluator without evidence targeting the requested behavior.
  • A test covering only the common case can pass while the original bug remains when an overlooked condition triggers the failure.The example involves a bug that appears only for an empty list, while the generated test covers a nonempty case.
  • Shared repair and validation can let one misinterpretation shape the patch, its apparent supporting evidence, and the decision to stop.A successful check establishes that the patch satisfies its assertions, not necessarily that it resolves the task.
  • Independent, fixed tests reduce trajectory-level coupling by generating checks without access to the candidate repair and holding them fixed during source revision.Independence concerns generation context and write permissions, not statistical independence of Test and Repair errors.
  • ExecCritic implements this separation with a Test agent, a harness that executes and freezes checks, and a Repair agent that changes only source code.

3 METHODOLOGY: LEARN TO TEST, TEST TO IMPROVE

ExecCritic separates test construction from source repair: a harness qualifies and freezes a repository-native test before the Repair agent revises patches using its bounded execution feedback. Learn to Test audits tests against Gold and candidate patches, while Test to Improve preserves the test across revision rounds and limits stopping claims to local verification.

  • Scaffold: ExecCritic assigns test construction and source-code revision to separate agents, reducing coupling between repair and validation.The Test agent constructs the test bundle, while the Repair agent writes source-only patches.
  • Learn to Test: The Test agent submits a test patch, exact command, and behavior contract, and the harness retains the first valid bundle that fails cleanly on Base.Invalid submissions, passes on Base, and failure to meet the generation budget prevent feedback-guided Repair from launching.
  • Learn to Test: Base-to-Gold success requires rejecting Base behavior while accepting the reference-repaired behavior; offline candidate-patch results support training and analysis only.Gold outcomes do not filter Test samples or select inputs for Repair, and deployment admission uses only a clean Base failure.
  • Learn to Test: Qualification does not prove semantic alignment because artifact validity, node binding, and a clean Base failure may still accompany an issue-misaligned test.The harness can verify execution protocol properties without establishing that the behavioral expectation matches the issue.
  • Test to Improve: Test to Improve resets each isolated workspace, applies the current source patch and fixed test bundle, and returns bounded PASS/FAIL feedback for revision.The retained history includes prior interaction, and operational errors are reported separately rather than treated as behavioral verdicts.
  • Test to Improve: A PASS immediately submits the patch, but it is only a local stopping condition; after five revisions without a pass, the latest candidate is force-submitted.The Repair agent may also submit after FAIL when it judges the generated test incomplete, overly specific, or incorrect.

4 LEARNING TO TEST AND REPAIR WITH EXECCRITIC

ExecCritic trains complementary Test and Repair agents around executable feedback: tests are qualified for behavioral validity and discrimination, while repairs learn to use frozen test outcomes for source revision. The framework combines supervised repository-native demonstrations with reinforcement rewards tied to test and repair outcomes.

  • Framework and training: Test training teaches repository-native regression-test construction, while Repair training teaches initial fixes and targeted revision from executable feedback.The two capabilities are optimized separately using rewards tied to observable test and repair outcomes.
  • Learning to test: Reinforcement learning rewards Test bundles that fail the buggy repository, pass the reference behavior, and discriminate correct from incorrect candidate repairs.Candidate-patch evaluation uses balanced accuracy, while operational errors are excluded from policy updates rather than counted as behavioral failures.
  • Framework and training: Supervised fine-tuning demonstrates the complete repository workflow, including exploration, behavioral interpretation, infrastructure discovery, and executable artifact generation.The demonstrations come from fixed stronger-model trajectories; Gold outcomes do not filter the trajectories or enter the student input.
  • Learning to test: A valid Test submission receives zero reward unless it fails cleanly on the Base repository and passes on the Gold repository; higher rewards require stronger candidate discrimination.The reward reaches 1.0 when behavioral validity holds and balanced accuracy equals 1, with intermediate levels for lower discrimination.
  • Learning to revise repairs: Repair training uses execution feedback from a fixed verifier while hiding the test source, and permits at most five feedback-guided revision rounds after the initial patch.Terminal evaluation also uses the full official test suite, distinguishing focused-test success from official repair success.
  • Learning to revise repairs: Passing the focused training test without official success earns 0.2, whereas passing both the selected test and the full official evaluator earns 1.0 after revision or 1.5 at Round 0.This reward gap separates satisfying the focused test from additionally passing the complete official evaluation suites.

5 EXPERIMENTS

Experiments show that executable feedback helps repository repair only when generated tests validly target requested behavior. Role-specific training improves test reliability and repair, with complementary agents reaching 72.6% resolved rate on SWE-bench Verified.

  • Experimental scope: The experiments evaluate feedback quality, Test-agent reliability, Repair-agent training, and the combined system on SWE-bench Verified and related settings.The evaluation addresses whether fixed tests improve repair, whether Test training improves valid behavioral checks, and whether Repair training improves initial and feedback-conditioned resolution.
  • Generated-test feedback: GPT-5.6-generated tests raise Qwen Repair resolution from 61.2% to 65.3%, while Qwen-generated tests reduce it to 57.3%.Oracle F2P feedback reaches 69.4%, showing that feedback benefit depends strongly on the generated test source.
  • Test-agent training: Test-agent Base-to-Gold success rises from 22.2% to 39.6% with SFT and to 62.2% after RL.RL adds 22.6 points after SFT, bringing the trained Test agent near Codex-5.3’s 61.0% while remaining below stronger references.
  • Test-agent training: On-policy RL reaches 62.2% after 5K SFT trajectories, whereas expanding imitation to 10K yields no further Base-to-Gold gain.The results distinguish behavioral success from reward and suggest diminishing returns from additional imitation at this scale.
  • Test-agent training: SFT initialization produces steadily rising Test-agent reward, while Base initialization remains low and oscillatory.SFT teaches repository exploration, test construction, and structured submission behavior, enabling more informative on-policy outcomes.
  • Repair-agent training: Repair training raises no-test Round-0 resolution from 61.2% to 68.3% and increases feedback-guided final resolution to 72.6%.With RL-35B feedback, gains are 2.9 points for the Base Repair agent and 4.3 points for the trained agent.
  • System composition: Combining trained Test and Repair agents reaches 72.6%, 11.4 points above the original no-test baseline without GPT-5.6 or Oracle feedback at evaluation.Each trained component improves the system with either version of the other, while a remaining 5.0-point Oracle gap suggests room for better test quality and coverage.
  • System composition: Feedback-guided revision raises trained-system resolution from 68.3% to 72.6% with an average of 13 additional Repair-agent turns among 120 repair trajectories.Test generation is a separate one-time cost per issue and Test source, with bundles reused across three Repair runs.

6 RELATED WORK

Prior work uses execution feedback for testing, repair, debugging, and candidate assessment. EXECCRITIC instead factors test generation and repair into separately trained policies composed through a harness that keeps repair evidence fixed.

  • Automated testing and repair systems use execution feedback to generate inputs, optimize coverage, reject overfitted patches, compare behaviors, and validate repairs.
  • Coding agents also revise programs from execution or verbal feedback, while other systems use learned preferences, executable environments, and reward signals for post-training.
  • EXECCRITIC trains the Test policy for behavioral change and candidate discrimination, while training the Repair policy separately for direct solving and revision from fixed tests.
  • At evaluation, a harness composes the learned roles while preventing the Repair trajectory from modifying its evidence.

7 LIMITATIONS

The framework currently trains Test and Repair as separate policies and composes them only after training. Shared-weight training remains future work because the roles have different action spaces, trajectory structures, and learning signals.

  • EXECCRITIC trains the Test and Repair agents as separate policies and composes them only after training.
  • A shared-weight design could simplify deployment, transfer knowledge between capabilities, and enable end-to-end optimization of the test–verify–revise loop.
  • Joint training must balance heterogeneous signals, assign credit across trajectories, and prevent degenerate coordination between Test and Repair roles.

8 CONCLUSION

EXECCRITIC separates test generation from source repair by freezing independently generated behavioral checks while the Repair agent revises only source code from bounded feedback. Under the evaluated SWE-bench Verified protocol, strong tests help repair while weak tests can hurt.

  • EXECCRITIC separates test generation from source repair, freezing the Test agent’s behavioral check while the Repair agent revises only the source patch.
  • The official evaluator retains final authority while local generated-test outcomes guide bounded repair revisions.
  • Strong Test patches improve off-the-shelf Repair agents, whereas weak ones can reduce their performance under the shared evaluation protocol.
  • Future work targets multi-check harnesses, stronger Test-agent weights, and lower cost and latency.

A EXPERIMENTS AND ANALYSIS

The appendix documents the separately trained Test and Repair configurations and the evaluation protocol that composes them while distinguishing local generated-test outcomes from official resolution.

  • The appendix reports Test-agent and Repair-agent configurations separately because their trajectory formats, contexts, rollout budgets, verification signals, and reward functions differ.
  • The evaluation protocol composes the independently trained roles while keeping local generated-test outcomes distinct from official resolution.

A.1 EXPERIMENTAL SETUP

The Test policy uses Qwen-3.5-35B-A3B, initialized from teacher trajectories and optimized with execution-based rewards. Its configuration covers data, repository interaction, verification, rewards, and runtime settings.

  • Table 7 organizes the configuration into supervised initialization, actor optimization, repository interaction and verification, executable rewards, and trainer hardware settings.
  • The Qwen Test policy is initialized from teacher trajectories and then optimized with execution-based rewards.
  • Test-agent training uses 5,000 teacher trajectories for supervised initialization, followed by GRPO sampling eight trajectories per issue in independent verifier sandboxes.
  • The fail-closed training setup admits bundles requiring a clean Base failure, while Gold behavior is not used as an SFT admission filter.

A.2 REPAIR-AGENT TRAINING CONFIGURATION

Repair training optimizes a separate feedback-conditioned policy using bounded revision episodes and fixed tests. Its reward and advantage rules distinguish valid behavioral outcomes from operational failures and preserve the source-only repair boundary.

  • The Repair policy starts from the same model family but is trained separately as a feedback-conditioned policy.
  • Each trajectory allows a 200-turn direct-repair phase followed by up to five 40-turn revision rounds with one fixed Oracle fail-to-pass test.
  • Operationally invalid terminal executions receive zero advantage and do not provide a selected-test-failure reward.

A.3 EVALUATION CONFIGURATION

Evaluation composes generated Test and Repair agents under a protocol that keeps Gold hidden from deployment decisions and separates local gate outcomes from official resolution. It also reports revision usage, benchmark-specific metrics, and reproducibility boundaries.

  • Generated-test evaluation composes Test and Repair agents without Oracle feedback, using Gold only for the offline Test audit.
  • SWE-bench Verified reports official resolved rate, while SWE-bench Multilingual reports Test-generation reliability through the Base-to-Gold criterion.
  • Local gate outcomes and full official outcomes are recorded separately, with only the fresh full evaluator determining whether a patch is resolved.
  • Across 120 feedback-guided trajectories, additional Repair-agent turns average 13 and have a median of 11.
  • The release includes launchers, rewards, repair loops, submission protocols, and evaluation drivers, while datasets, checkpoints, installations, and credentials remain external.
  • The Test submission contains a repository-native diff, an exact execution command, and a JSON behavior contract.

A.5 QUALITATIVE ANALYSIS OF FEEDBACK-GUIDED REPAIRS

Qualitative cases show that useful frozen tests expose the failure boundary and redirect source repair, while fail-closed semantics distinguish behavioral failures from operational invalidity. The resulting patches can pass both generated and official verification, though successful repairs need not be minimal.

  • Case 1: Recursive XOR compilation (Django #16901): In Django #16901, a recursion trace exposed reuse of the XOR connector, and replacing it with AND passed the frozen test and fresh official verification.
  • Case 2: Repair targets the wrong parser (Sphinx #9230): In Sphinx #9230, a rendering mismatch redirected repair from the unrelated Napoleon parser to docfields.py, where the revised split preserved the complete type.
  • Case 2: Repair targets the wrong parser (Sphinx #9230): The terminal Sphinx patch passed the frozen parenthesized-type test and fresh official verification, including a bracketed-type case, but retained the unrelated initial edit.
  • Shared pattern: Useful generated tests identify a causal failure boundary rather than merely reporting that a patch is wrong.
  • Fail-closed harness semantics: A valid test run yields PASS or FAIL from behavioral execution, whereas missing dependencies, timeouts, and infrastructure errors are operationally invalid.
  • Repair controller: A valid local pass forwards the patch and ends repair, while budget exhaustion force-submits the latest candidate rather than reverting to an earlier patch.

B.2 HARNESS UPDATES REQUIRE MATCHED EVIDENCE

The paper specifies evidence-gated harness evolution as a controlled, matched comparison rather than an autonomous runtime update. Each challenger is isolated, evaluated on paired instances under predeclared gates, and promoted only when quality, regression, infrastructure, and cost requirements all pass.

  • Evidence-gated maintenance: Evidence-gated harness evolution proposes one declared challenger change, compared with a retained parent on matched instances and aggregated evidence.The maintenance loop is separate from the evaluated runtime and records generation, qualification, audit, and challenger comparison.
  • Harness qualification: The evaluated harness qualifies each generated test on a buggy Base, performs an isolated offline Gold audit, and preserves the test patch, command, and contract unchanged.Gold does not alter the Test-agent episode or online Base-gate feedback.
  • Matched evidence: Paired trials freeze instance IDs, strata, metrics, thresholds, regression sets, execution policies, budgets, and infrastructure-error handling before either version runs.Both versions execute on the same IDs with independently recreated repository state.
  • Promotion gates: Promotion requires every predeclared gate to pass, including the paired quality threshold, regression checks, infrastructure policy, and cost budget.Otherwise the retained parent remains released; revisions become new immutable challengers and reversions preserve the decision record.
  • Test-agent protocol: The Test-agent protocol requires a repository-native regression test that fails on the buggy checkout, asserts corrected public behavior, and uses the established project workflow.The agent must infer the test entrypoint, inspect adjacent tests and boundary cases, consider an alternative interpretation, and submit the test patch through the harness.
Loading 2609.09133v1…