PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-VERIFIER: Selective Verification for Cost-Efficient Coding Agents

Kaijian Zou, Vaibhav Chennareddy, Yunxiang Zhang, Naihao Deng, Longju Bai, Jiaxin Pei, Rada Mihalcea, Lu Wang

arXiv:2610.30284v1cs.LGcs.AI

TL;DR

Poor coding-agent actions can waste computation, while verifying alternatives at every step adds overhead. SWE-VERIFIER selectively triggers candidate sampling and ranks alternatives using a lightweight trained verifier. It improves resolution on both evaluated policy models at modestly higher cost while verifying fewer than 4% of steps on MiniMax-M2.7.

  • Problem

    Poor actions can send coding agents into low-value trajectories that require more exploration and corrective work, while all-step verification incurs candidate-generation and scoring overhead.

    Source: §1 · pp. 1–2

  • Method

    SWE-VERIFIER scores a proposed action to decide whether to sample alternatives, then ranks candidates using a verifier trained on execution-aware preference data and teacher supervision.

    Source: §1 · p. 2

  • Results

    77.20% resolution versus 75.47% for MiniMax-M2.7 at 1.09× cost, and 74.27% versus 72.60% for DeepSeek-V4-Flash at 1.13× cost.

    Source: §1 · p. 2, §8 · p. 8

  • Takeaways & Limitations

    Selective candidate sampling and lightweight action verification improve coding-agent performance while controlling inference overhead in the reported evaluations.

    Source: §8 · p. 8

    The observed accuracy–cost tradeoffs may differ across benchmarks, repository distributions, serving systems, and agent frameworks, and teacher-labeled preferences may be less reliable on underrepresented failure modes or domains.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Coding agents spend substantial computation generating actions and processing long interaction histories. Poor action choices can steer agents into low-value states, wasting computation and increasing API costs. Sampling and verifying alternative actions can improve the agent's decisions, but applying this procedure at every step adds substantial overhead. We introduce SWE-Verifier, a lightweight generative model that selectively triggers candidate sampling and verification. It scores the coding agent's proposed action to determine when additional candidate sampling is needed, then ranks the resulting actions before execution. We train SWE-Verifier from Qwen3.5-4B using preference data that groups contrasting actions with their resulting environment observations and rubric-guided teacher scores. Using the mini-swe-agent harness, SWE-Verifier improves resolution from 75.47% to 77.20% on MiniMax-M2.7 at 1.09 times cost and generalizes to DeepSeek-V4-Flash without training on its actions, improving resolution from 72.60% to 74.27% at 1.13 times cost. Across both models, SWE-Verifier verifies fewer than 4% of steps and achieves comparable or higher resolution than the trajectory and all-step baselines at substantially lower cost. These results demonstrate that selective candidate sampling and lightweight verification provide complementary gains in coding-agent performance and inference efficiency. Our model, code, and data are available at https://github.com/zkjzou/SWE-Verifier.

1 Introduction

Coding-agent actions affect both task success and inference cost, while trajectory-level verification cannot prevent unproductive intermediate choices and all-step verification is costly. SWE-VERIFIER selectively samples and ranks alternative actions, improving the performance–cost tradeoff in the reported experiments.

1 Introduction

  • Poor actions can steer agents into low-value trajectories that require more exploration and corrective work, while expanding histories increase processing costs.The cited motivation identifies improving action quality before execution as an opportunity to improve task success and inference efficiency.Source: §1 · p. 1
  • Trajectory-level selection happens only after rollouts are complete, so it cannot prevent an individual rollout from taking an unproductive branch.This motivates checking actions at intermediate decision points before the agent commits to its next action.Source: §1 · p. 2
  • All-step verification adds candidate-generation and scoring overhead throughout a trajectory, so the central challenge is choosing which actions warrant that cost.The proposed selective procedure first assesses the initial action, then generates and compares alternatives when additional scrutiny is useful.Source: §1 · p. 2
  • SWE-VERIFIER scores the proposed action to decide whether to sample alternatives, then scores candidates to select the action to execute.It is instantiated with Qwen3.5-4B and trained using execution-aware preference data with replayed environment feedback and rubric-guided teacher supervision.Source: §1 · p. 2
  • 77.20% resolution versus 75.47% on MiniMax-M2.7, with 3.43% of steps verified at 1.09× cost; DeepSeek-V4-Flash rises from 72.60% to 74.27% at 1.13× cost.On MiniMax-M2.7, it also outperforms trajectory verification and both evaluated all-step verifiers.Source: §1 · p. 2

2 Preliminary

The section formalizes trajectory-, step-, and selective step-level action selection, then defines cost terms for comparing verification strategies. Selective verification gates candidate comparison to selected states rather than applying it throughout the trajectory.

2.1 Conceptual Formulation

  • Trajectory-level verification samples complete trajectories and selects one with an outcome verifier, but selection after completion cannot guide intermediate actions.The policy samples n trajectories and the verifier selects the one with the highest outcome score.Source: §2.1 · p. 2
  • At each step, a step-level verifier compares n policy-generated candidate actions using an action verifier, potentially improving local decisions but adding overhead throughout the trajectory.The candidates are generated conditional on the current history.Source: §2.1 · pp. 2–3
  • Selective step-level verification uses a gate to choose between comparing candidate actions and executing a policy action directly.The realized verification rate is denoted c.Source: §2.1 · p. 3

2.2 Cost Formulation

  • The cost formulation sets baseline trajectory cost to C0 = x + y, with x and y denoting baseline input and output costs.For selective verification, c is the fraction of verified steps, z the trajectory-length reduction, α the cached-input price multiplier, and β verifier-output cost relative to policy output; the accounting includes a gate cost.Source: §2.2 · p. 3
  • Trajectory-level verification incurs the cost of generating all n complete trajectories before selection.Its cost also includes the outcome-verifier call.Source: §2.2 · p. 3

3 SWE-VERIFIER: Selective Step-Level Verification for Coding Agents

The section motivates selective verification with a preliminary accuracy–cost tradeoff, then presents a gated verifier and an execution-aware preference-data and training recipe. The verifier uses the same model to decide when to sample alternatives and which candidate to execute.

3 SWE-VERIFIER: Selective Step-Level Verification for Coding Agents

  • 76.6% resolution at 1.29× cost with verification on 4% of steps, versus 76.0% at 2.32× cost for full verification; fixed-frequency verification did not match closeout-only verification’s benefit.The preliminary study motivates concentrating verification on useful decisions and using a smaller model for action evaluation.Source: §3 · p. 4

3.1 Gate and Selector Modes

  • SWE-VERIFIER scores the policy’s proposed action to decide whether to trigger verification, then ranks candidates to choose which action to execute.The same verifier score supports gate and selector modes; candidate assessments include likely repository effects and task progress.Source: §3 · p. 4, §3.1 · p. 4

3.2 Preference Data Construction

  • The execution-aware data recipe samples contrasting actions at trajectory steps, replays them for environment feedback, and uses a strong teacher for rubric-guided preference labels.The collected successful trajectories come from Opus 4.7 and GPT-5.5; candidates are filtered by pairwise Jaccard similarity and duplicate non-empty observations are removed.Source: §3 · p. 4, §3.2 · p. 4
  • GPT-5.6 labels candidate steps with expected outcomes, anticipated task progress, a shared evaluation rubric, and final candidate scores.The teacher receives the trajectory prefix, candidate actions, and immediate observations.Source: §3.2 · p. 4

3.3 Training

  • Qwen3.5-4B is fine-tuned on 30,000 action labels from 6,000 agent steps to generate outcome, progress, rubric, and score assessments.The variants are Score Only, Joint Rubric, and Two-Stage Rubric, which separates rubric generation from scoring.Source: §3.3 · pp. 4–5

4 Experimental Setup

The experiments evaluate verification strategies on SWE-Bench Verified with two policy models, reporting resolution, trajectory steps, verification rate, and normalized cost. The setup includes trajectory-level, all-step, and threshold-triggered selective verification.

4.1 Task and Metrics

  • The main strategies are evaluated on the full 500-task SWE-Bench Verified set using mini-swe-agent with MiniMax-M2.7 and DeepSeek-V4-Flash.Gate/selector and threshold analyses use a 202-task subset.Source: §4.1 · p. 5
  • Reported metrics are task-resolution rate, average trajectory steps, verify rate, and cost normalized to each policy’s no-verifier baseline.The local lightweight verifier is assigned zero inference cost in the accounting.Source: §4.1 · p. 5

4.2 Verification Settings

  • Trajectory verification selects among three complete rollouts, while all-step verification generates three candidates at every step and uses either the policy model or Qwen3.5-4B to select.The no-verification policy action defines the 1.00× cost reference; Oracle@3 is an upper bound that selects a successful rollout whenever one exists.Source: §4.2 · p. 5
  • SWE-VERIFIER triggers candidate sampling when the initial-action score falls below a threshold, then selects the highest-scoring valid candidate.MiniMax-M2.7 thresholds are chosen on a 202-task subset; DeepSeek-V4-Flash experiments share a 0.2 threshold without variant-specific tuning.Source: §4.2 · p. 5

5 Main Results

SWE-VERIFIER improves resolution over no verification on MiniMax-M2.7 and DeepSeek-V4-Flash with selective calls and modest added cost. Training and verification frequency affect this performance–cost tradeoff.

5 Main Results

  • 77.20% resolution at 1.09× total cost on MiniMax-M2.7, up from 75.47%, while verifying 3.43% of steps and outperforming the evaluated trajectory and all-step baselines.Whether the gain over no verification is worthwhile depends on the deployment budget.Source: §5 · p. 6
  • Two-Stage Rubric reaches 77.20% resolution at 1.09× cost, while Score Only matches the untrained variant’s 76.93% at 1.05× cost and verifies only 0.45% of steps.The comparison suggests training concentrates verification on consequential decisions; generating a rubric before scoring may further improve action selection.Source: §5 · p. 6
  • 74.27% resolution at 1.13× cost on DeepSeek-V4-Flash, up from 72.60%, nearly matching trajectory-level LLM-as-a-Verifier@3 at 74.30% and approximately 4.19× cost.All-step Qwen3.5-4B costs 2.03× with no improvement over the no-verifier baseline.Source: §5 · p. 6
  • All-step Self-Verification reduces executed steps on both policies but raises total costs to 3.35× on MiniMax-M2.7 and 3.99× on DeepSeek-V4-Flash.Two-Stage Rubric keeps trajectory lengths close to baseline while improving resolution at much lower cost, primarily by avoiding unnecessary candidate generation and verification.Source: §5 · p. 6

6 Additional Analyses

The analyses examine how gate and selector choices, verification frequency, serving costs, and changes to the policy workflow affect selective verification. Across these settings, more verification does not consistently improve resolution, and cost depends on both call frequency and trajectory length.

6.1 Gate/Selector Ablation

  • 73.27% resolution at τ = 0.2 with the trained Score Only gate, versus 71.12% for the Untrained gate, while verifying 5.13% rather than 5.41% of steps.Pairing the trained gate with MiniMax-M2.7 as selector reaches 73.27% versus 71.95% with the verifier selecting at τ = 0.2; their total costs are nearly identical at τ = 0.3.Source: §6.1 · p. 7

6.2 Threshold Sensitivity

  • 72.28% resolution at 1.02× cost is Score Only’s best selective setting, with verification on 0.47% of steps; its resolution falls to 67.16% under all-step verification.The Untrained verifier is less sensitive to increased verification frequency, but its best selective setting still outperforms its all-step result.Source: §6.2 · p. 7, Figure 3 · p. 7

6.3 Cost Sensitivity

  • 3.0–3.9× baseline cost for all-step Self-Verification and 2.1–2.3× for all-step Qwen3.5-4B persist across pricing settings.Matching baseline cost requires trajectory reductions that rise with critical-step rate; at c = 0.15, the requirements are 22% and 10%, versus observed maximum reductions of 17% and 19% for the two policies.Source: §6.3 · pp. 7–8, Figure 4 · p. 7

6.4 Workflow Changes

  • Above 93% phase agreement, ordinary step-level verification largely preserves the baseline workflow, while verifier-as-tool diverges most during the early transition from exploration to fixing.Verifier-as-tool resolution falls below baseline despite shorter trajectories; the authors suggest that prompting for alternatives can interfere with exploration and repair.Source: §6.4 · p. 8

7 Related Work

Related work covers coding agents, test-time scaling and verification, and inference-efficiency methods. SWE-VERIFIER combines selective invocation with a trained generative action verifier.

Coding Agents

  • Coding-agent research includes repository-level issue resolution, interactive tool use, structured repair pipelines, and trajectory search.This paper studies verification of intermediate decisions about what to inspect, execute, edit, and submit.Source: §7 · p. 8

Test-time Scaling

  • Test-time scaling uses verifiers to score sampled solutions or intermediate steps, including process supervision and generative reward modeling.For interactive agents, scores can assess progress toward task completion.Source: §7 · p. 8
  • Prior coding-agent work includes executable-environment training, rubric-conditioned process rewards, learned action values, trajectory reranking, and course correction.SWE-VERIFIER uses action scores both to identify steps warranting additional verification and to rank candidate actions before execution.Source: §7 · p. 8

Inference Efficiency and Selective Verification

  • Selective verification avoids candidate generation and scoring at every step, while distilling teacher supervision into a smaller verifier further lowers evaluation cost.The two approaches target the frequency of candidate comparison and the cost of each evaluation, respectively.Source: §7 · p. 8

8 Conclusion

SWE-VERIFIER selectively samples and ranks alternative coding-agent actions using execution-aware teacher supervision. It improves resolution on MiniMax-M2.7 and generalizes to DeepSeek-V4-Flash, while its analyses favor selective over uniform verification.

8 Conclusion

  • 77.20% resolution on MiniMax-M2.7, up from 75.47%, with verification on 3.43% of steps at 1.09× total cost.On DeepSeek-V4-Flash, it raises resolution from 72.60% to 74.27% at 1.13× cost.Source: §8 · p. 8
  • The analyses find that selective verification works best by targeting decisions that benefit from verification, while uniform verification adds substantial cost without consistent gains.The approach combines selective sampling and ranking with execution-aware teacher supervision.Source: §8 · p. 8

Limitations

Full verification can shorten trajectories, but its added candidate-generation and scoring costs outweigh those savings. The reported scope and cost tradeoffs depend on the evaluation harness, serving assumptions, and teacher-labeled training data.

Limitations

  • Results are limited to the mini-swe-agent harness, MiniMax-M2.7 and DeepSeek-V4-Flash, and SWE-Bench-Verified; tradeoffs may differ across benchmarks, repositories, serving systems, and agent frameworks.The passage particularly notes trajectory length and whether candidate actions share a prompt as factors that may change observed tradeoffs.Source: Limitations · p. 9
  • Cost advantages depend on cached-prompt discounts and inexpensive verifier calls, while teacher-labeled preferences and successful-trajectory states may leave judgments less reliable on underrepresented failure modes or domains.Different provider cache discounts or batching behavior may change the break-even point.Source: Limitations · p. 9

A MiniMax-M2.7 Verification Strategies

  • The MiniMax-M2.7 comparison on SWE-Bench-Verified reports resolution, trajectory length, verification rate, and cost normalized to a three-seed no-verifier baseline.Verification rate is the fraction of steps where the verifier is invoked; actions are verified only when at least two candidates are unique.Source: A MiniMax-M2.7 Verification Strategies · p. 12

A.1 Full Verification Improves Step Efficiency but Remains Costly

  • 76.0% resolution with five-candidate full verification shortens trajectories to 35.9 steps but costs 2.32× baseline, so shorter rollouts do not offset all-step overhead.With three candidates, full verification reaches 76.2% resolution in 38.2 steps at 2.16× total cost.Source: §A.1 · p. 12

A.2 Verifier Awareness Can Disrupt the Workflow

Verifier-as-tool can alter the agent’s workflow and reduce resolution, while selective verification performs best when aimed at particular decision states. The formulation connects action values to task resolution and separates policy from verifier costs.

A.2 Verifier Awareness Can Disrupt the Workflow

  • Verifier-as-tool phase agreement falls to 83–86% at 10–40% of trajectory position, then rises above 95%, while its resolution is 70.2% and 69.2% versus 76.2% and 76.0% for ordinary verification.Figure 5 shows the early and middle phase divergence before later convergence.Source: §A.2 · p. 12, Figure 5 · p. 13

A.3 Verification Value Depends on the Decision State

  • 76.6% resolution at 4% verification and 1.29× baseline cost for closeout checks; periodic checks every five steps use 21% of steps yet reach 75.0%, showing sparsity alone is insufficient.Fix-phase verification reaches 76.2% resolution with a 9% verification rate and 1.35× total cost.Source: §A.3 · p. 13

A.4 A Small Model Verifier Reduces Verification Cost

  • Qwen3.5-4B reaches 76.0% resolution at 1.28× policy and total cost, matching frontier self-verification accuracy under near-zero verifier accounting.The result motivates retaining larger-model judgment quality while keeping serving cost close to that of a small model.Source: §A.4 · p. 13

B Detailed Formulation

  • The formulation decomposes sparse rollout-level success into state-action values: Q estimates whether committing to an action from a history leads to eventual task resolution, and step-level scaling samples alternatives for verifier ranking.The verifier approximates the ordering of actions using task context, prior interactions, and the proposed action.Source: B Detailed Formulation · pp. 13–14

B.1 Cost Formulation

  • Cost accounting separates policy and verifier costs; step-level cost depends on trajectory shortening, critical-step rate, cached-input pricing, and verifier overhead.The formulation defines baseline input and output costs as x and y, and cached-input price as a fraction α of standard input price.Source: §B.1 · p. 14

C Gate and Selector Ablation

This section examines how verifier prompts, candidate scoring, and gating support selective action verification, and how candidate selection affects resolution and cost. It also presents a repository-detection case study and a cookiecutter example contrasting a risky broad overwrite with a safer diagnostic action.

C Gate and Selector Ablation

  • At τ = 0.2, Score Only gating with MiniMax selection reaches 73.27% resolution at 1.29× cost versus 71.95% at 1.24× when Score Only also selects.At τ = 0.3, Untrained selection reaches 71.95% resolution at 1.11× cost, versus 70.13% at 1.45× with MiniMax selection; these are descriptive comparisons across unpaired runs.Source: C Gate and Selector Ablation · p. 14

D Verifier Prompts

  • The appendix documents five verifier configurations, including an action classifier that labels action type, problem-solving phase, uncertainty and risk, and whether verification should run.The prompt set covers candidate selection, critical-step classification, and teacher-label generation.Source: D Verifier Prompts · p. 16

D.2 Single-action verifier prompt

  • The single-action prompt uses the same selection objective but asks for candidate-local reasoning, concrete next-state predictions, and a scalar score for each candidate.It prioritizes safe, minimal actions and favors lower-risk options when candidates are equivalent.Source: D Verifier Prompts · p. 16, §D.2 · pp. 16–17

D.1 Multi-action verifier prompt

  • The multi-action prompt scores each candidate on its own merits, then compares the strongest candidates in a final reasoning block to select one action.Its criteria prioritize correct, safe, minimal, instruction-compliant actions likely to succeed.Source: §D.1 · p. 16

D.3 Teacher-distillation prompt

  • The teacher-distillation prompt uses trajectory summaries and generated checklists alongside immediate observations or future evidence to produce candidate predictions, checklist scores, reasoning, and scalar scores.Candidate-local outputs must stand alone because the distilled student sees one action at a time and not the immediate observation.Source: §D.3 · pp. 17–18

D.4 Two-stage verifier prompts

  • The Two-Stage Rubric variant generates an action-conditioned checklist and then scores the proposed action against it, with neither call observing execution results.Each proposed action receives its own checklist; the scorer expresses predicted outcomes and expected progress in its reasoning.Source: §D.4 · p. 18
  • The action-scoring prompt judges one unexecuted action by its likely outcome, safety, usefulness, and checklist scores, then assigns an overall score.It directs the verifier to assess the action rather than its reasoning and to predict, not assume, its execution result.Source: §D.4 · p. 19

D.5 Action classifier prompt

  • Verification is triggered when candidate uncertainty, step importance, or harm risk makes extra sampling and selection worthwhile.The classifier separates action type, solve phase, uncertainty, importance, and risk from its final decision to run the verifier.Source: §D.5 · p. 20

E.1 Framing and verifier setup

  • The case compares blind, observation-aware, and checklist-based verifiers, which differ in access to tool outputs and explicit task criteria.The decision concerns whether to patch repository-detection logic immediately or first validate preservation constraints.Source: §E.1 · p. 21

E.2 Case overview

  • The reference decision validates the reported Mercurial URL failure before editing, because broad edits could break valid Git URLs or unknown-URL errors.The preservation constraints include GitHub URLs without .git remaining valid and malformed or unknown URLs still raising UnknownRepoType.Source: §E.2 · p. 21

E.3 Evidence shown before candidate selection

  • The candidate slate includes direct edits, diagnostics, and validation actions, with Candidates A and B identified as broad full-file overwrites.The trajectory exhibit shows the test evidence and candidate actions supplied to all teacher verifiers.Source: §E.3 · p. 23
  • The checklist-based teacher selected Candidate D, although the excerpt gives scores only for Candidates A and C.Candidate A scored 0.33 and Candidate C scored 0.58; no score or rationale for D is included in the supplied passages.Source: §E.3 · p. 26
  • Candidate C’s diagnostic scored 0.58 versus 0.33 for Candidate A’s broad overwrite, which was judged risky and liable to regress existing behavior.The diagnostic checked malformed URLs but did not directly test the reported Mercurial cases or apply a fix; the overwrite could appear to fix those cases while silently regressing other behavior.Source: §E.3 · p. 26

E.4 Analysis

In E.4, checklist-conditioned verification favors safe progress that preserves surrounding repository behavior over narrow bug alignment. The astropy-14369 runs show that agents can find and validate a repair yet diverge later when constructing the final patch.

E.4 Analysis

  • Candidate D was the safest immediate action: it reproduced the reported Mercurial cases before edits, avoiding concrete regression risks in the available edit candidates.Candidate A retained fallback-to-Git behavior for malformed URLs, while Candidate B would reject valid GitHub URLs without a .git suffix.Source: §E.4 · pp. 26–27

F MiniMax State-wise Final-Patch Divergence: astropy-14369

  • Across four astropy-14369 runs, unresolved trajectories reached passing validation but submitted a different repository state after restoring or omitting the regenerated parser table.Resolved seeds preserved both modified files; unresolved seeds submitted a cds.py-only patch.Source: F MiniMax State-wise Final-Patch… · p. 27, §F.1 · p. 27

F.1 Case overview

  • Resolved seeds 0 and 1 preserved cds.py and the regenerated tracked cds_parsetab.py, while unresolved seeds 3 and 5 submitted cds.py-only patches.Figure 10 summarizes this divergence.Source: §F.1 · p. 27, Figure 10 · p. 28

F.2 Shared path to passing validation

  • All four runs located and repaired the repeated-division grammar issue, regenerated parser artifacts, and passed local checks before their final-state outcomes diverged.The shared path separates repair discovery from later patch management.Source: §F.1 · p. 27, §F.2 · p. 28

F.3 Interpretation: final-state consistency

A final patch must match the workspace state that passed validation. Removing generated files can be safe only when the resulting file set is revalidated.

F.3 Interpretation: final-state consistency

  • A checkout is harmful here only when it removes cds_parsetab.py from the last validated workspace and the resulting patch is submitted without revalidation.Some earlier cleanup was followed by regeneration and another successful check; unresolved runs occurred when the final patch omitted or restored the parser table after validation.Source: §F.3 · pp. 29–30
Loading 2610.30284v1…