PaperlayerView exact revision on arXiv

Source-linked AI summary

SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review

Ruoyu Wang, Jierun Chen, Shaowei Wang, Chaofan Tao, Sidi Yang, Yuxin Jiang, Kim-Hui Yap, Lifeng Shang, Xiaohui Li, Haoli Bai

arXiv:2607.06065v1cs.SE

TL;DR

One-shot coding-agent PRs lack systematic review, diagnosis, and revision. SWE-Review closes this loop with repository-grounded agentic review, and experiments show improved decisions and post-review revision outcomes over fixed-context review.

  • Problem

    One-shot PR generation lacks reliable mechanisms to determine issue resolution and diagnose revisions, while systematic benchmarks, metrics, training data, and end-to-end evaluations remain limited.

  • Method

    SWE-Review has a reviewer inspect the repository, issue, and candidate PR, make an approve/request-changes decision, and provide structured diagnostic feedback for revision.

  • Results

    Across three PR-generator splits, agentic review outperforms fixed-context review in decision accuracy and post-revision resolve rate, raising Qwen3-30B-A3B from 27.5% to 56.9%.

  • Takeaways & Limitations

    SWE-Review provides a practical route for moving AI coding agents from one-shot PR generation toward closed-loop issue resolution.

  • Takeaways & Limitations

    The study focuses on AI-generated PRs for SWE-style issue resolution and evaluates functional outcomes rather than style, maintainability, security, performance, or other non-functional criteria.

Abstract

from arXiv · show

Coding agents increasingly generate pull requests (PRs) for real-world software issues, yet one-shot PR generation remains open-loop: the PR is proposed without systematic review, diagnosis, or revision. We introduce \textbf{SWE-Review}, a framework for closing this loop with agentic code review. Given an issue and an AI-generated PR, a reviewer agent explores the repository, decides whether the PR should be accepted, and provides structured feedback for revision. We evaluate this setting with our proposed \textbf{SWE-Review-Bench} to measure both review correctness and downstream revision usefulness. We further curate \textbf{SWE-Review-Traj} dataset to study broader applications of agentic review and fill the data-scarcity gap for open reviewer training. Experiments show that agentic review continuously improves PRs through a generate-review-revise loop, outperforms single-turn fixed-context review in both decision accuracy and resolve rate after revision, transfers beyond review to improve issue-resolution models, and enables effective and efficient test-time scaling. These results position agentic code review as a practical mechanism for moving AI coding agents from one-shot PR generation toward closed-loop issue resolution.

1 Introduction

SWE-Review closes the open loop of one-shot AI-generated pull requests by using agentic review to decide acceptance and provide revision guidance. Its benchmark, trajectory dataset, and experiments show that review improves PRs, transfers to issue resolution, and supports effective test-time scaling.

  • Motivation: One-shot PR generation lacks reliable mechanisms to verify issue resolution or diagnose revisions when candidate changes fail.This motivates treating code review as the mechanism for closing the issue-resolution loop.
  • SWE-Review: SWE-Review has a reviewer explore the repository and produce an accept/request-changes decision plus structured revision diagnosis.The framework takes a repository, issue, and AI-generated pull request as input.
  • Evaluation resources: SWE-Review-Bench contains 1,384 candidate PRs from 500 SWE-bench Verified issues and three PR-generator quality distributions, while SWE-Review-Traj contains 8,914 review trajectories.The benchmark and dataset support systematic evaluation, training, and study of open reviewers.
  • Empirical findings: The generate-review-revise loop raises resolve rate from 27.5% to 56.9% for Qwen3-30B-A3B, from 50.9% to 68.8% for Qwen3-Coder-30B-A3B, and from 72.2% to 75.4% for GLM-5.These improvements are reported on SWE-bench Verified.
  • Empirical findings: Agentic review outperforms single-turn fixed-context baselines in DA and RRR across all three PR-generator splits, with largest gains on harder non-local reasoning tasks.The comparison is reported on SWE-Review-Bench.
  • Empirical findings: SFT on SWE-Review-Traj raises an 8B reviewer’s completion rate from approximately 4% to 71–84%, while mixed training improves direct resolve rate by up to 5.6 points and final resolve rate by up to 10.6 points.Review trajectories also raise decision accuracy from near-random to 67–72%. At test time, review-guided revision raises resolve rate from 22.9% to 38.4% over 4 rounds.

2 Related Work

Prior work studies repository-level issue-resolution agents, single-turn automated code review, and test-time selection of candidate patches. SWE-Review differs by making review agentic: the reviewer gathers repository evidence, diagnoses patches, and supports revision rather than reducing each attempt to a fixed-context judgment or score.

  • SWE issue resolution: Repository-level coding agents interact with development environments, invoke tools, and edit repositories to resolve real software issues.Examples include Cursor, Claude Code, Codex, Aider, and Devin.
  • Automated code review and evaluation: Existing automated code-review systems usually inspect a diff, optionally with retrieved context, and produce comments or accept/reject labels in one turn.This formulation assumes the diff contains sufficient evidence for judgment.
  • Automated code review and evaluation: SWE-Review treats review as an agentic capability that gathers repository-level evidence, traces call chains, checks related implementations, and verifies behavior before deciding and diagnosing.The approach addresses evidence outside edited hunks that cannot be pre-specified in a fixed context window.
  • Test-time scaling and trained code scorers: Prior test-time scaling methods select among candidate patches using either explicit verdicts or learned scalar ranking scores.SWE-Review’s reviewer can serve the same candidate-selection role while providing structured diagnosis for revision.

3 Agentic Code Review

Section 3 frames agentic code review as repository-grounded evidence gathering within a generate-review-revise loop, evaluated by whether reviews support issue resolution. Compared with fixed-context single-turn review, agentic review improves decision reliability and revision usefulness, especially as task difficulty increases.

  • Review mechanism: Repository exploration helps reviewers trace non-local bugs to upstream causes, avoiding symptom-only fixes that leave incorrect behavior unresolved.In the sympy-13877 example, the reviewer follows the call chain, identifies a discarded cancel(ret) result, and requests an upstream fix.
  • Task formulation: Each review receives a repository checkout, issue, and candidate PR, while the reviewer may browse files, inspect dependencies, and execute repository commands before reporting.The reviewer does not receive the golden patch or hidden test results.
  • Evaluation: The evaluation measures Completion Rate, Decision Accuracy, and Resolve Rate after Revision, tying review quality to executable patch correctness and feedback-conditioned revision.DA evaluates approve/request-changes decisions against true resolve status, while RRR additionally requires a standardized revision attempt.
  • Comparative results: Agentic review outperforms single-turn diff-only and diff-plus-context review in both Decision Accuracy and Resolve Rate after Revision across PRs from GLM-5, Qwen3-Coder-30B-A3B, and Qwen3-30B-A3B.The reviewer model is held fixed to Claude Opus 4.6, and the gains indicate stronger decisions and more useful diagnoses for downstream revisions.
  • Difficulty analysis: On 1,384 benchmark instances stratified into easy, medium, and hard tertiles, the performance gap between agentic and single-turn review widens monotonically with difficulty.Difficulty uses candidate-patch file-level divergence from the golden patch and the issue’s number of fail-to-pass tests.

4 Learning from Agentic Review Trajectories

SWE-Review-Traj captures agentic reviewers’ repository exploration, evidence, decisions, and diagnoses for training and broader evaluation. Distilling these trajectories improves review quality and issue resolution, while review-guided iterative revision enables efficient test-time scaling.

  • Dataset construction: SWE-Review-Traj records agentic reviewers’ exploration traces, evidence, decisions, and diagnoses to support training and test-time scaling research.The dataset addresses the broader value of agentic review beyond inference-time reviewing.
  • Dataset construction: Approximately 6k verified SWE-rebench issues remain after filtering and repository-level leakage prevention, with candidate PRs generated by GLM-5, Qwen3-Coder-30B-A3B, and Qwen3-30B-A3B.Candidate PRs with empty or oversized patches are also removed.
  • Trajectory validation: 8,914 trajectories remain after filtering for correct decisions on resolving and non-resolving patches, forming the default training set.Correct decisions alone do not guarantee accurate diagnoses, motivating separate semantic and functional validation.
  • Reviewer distillation: SFT on SWE-Review-Traj raises Qwen3-8B completion rates from approximately 4% to 71–84% and decision accuracy by 18–21 points across PR-generator splits.Final resolve rates also increase from 50.9% to 52.8% and from 27.5% to 35.1% on the two Qwen splits.
  • Mixed issue-resolution and review training: Mixed training improves issue-resolution RR from 27.6% to 28.4% at 1k data scale, from 31.2% to 36.8% at 2k, and from 34.0% to 37.8% at 3k.The comparison is against issue-resolution-only SFT, showing that review trajectories provide useful issue-resolution signal.
  • Test-time scaling: 38.4% resolve rate is reached within a maximum budget of five samples, averaging 2.44 samples, versus 32.3% for reviewer-gated resampling with 16 attempts.Review-guided iterative revision uses structured diagnosis and stops when the reviewer approves, outperforming approval-only gating.

5 Conclusion · A Additional Test-Time Scaling Results · A.1 Single-Model Iterative Review-Revision

SWE-Review closes the loop on AI-assisted issue resolution by combining repository-aware agentic review with structured feedback and revision. Additional results show that a single mixed-trained model can iteratively review and revise its own patches, steadily improving resolve rates.

  • 5 Conclusion: SWE-Review uses a reviewer agent to inspect the repository, issue, and candidate PR before deciding approval or requesting changes and producing structured diagnostic feedback.The framework targets closed-loop issue resolution rather than one-shot comment generation.
  • 5 Conclusion: SWE-Review-Bench evaluates agentic review using 1,384 AI-generated PRs from 500 SWE-bench Verified issues.The supplied conclusion passage introduces the benchmark and its evaluation setting.
  • 5 Conclusion: The study focuses on AI-generated PRs for SWE-style issue resolution, excluding broader scenarios such as feature implementation, refactoring, documentation, migration, and architectural changes.Its functional metrics measure issue resolution and revision usefulness, not qualities such as maintainability, security, performance, or project-specific conventions.
  • A Additional Test-Time Scaling Results: Appendix scaling results extend the main analysis with a single-model iterative loop and broader selection-rule comparisons across two coding agents.The main-text setup uses separate issue-resolution and review models, whereas the appendix examines additional test-time scaling configurations.
  • A.1 Single-Model Iterative Review-Revision: Qwen3-8B generates, reviews, and revises its own patch using mixed issue-resolution and review training at the 3k+3k scale, for up to 5 samples or until approval.The loop comprises one initial generation and up to four revision rounds driven by structured self-feedback.
  • A.1 Single-Model Iterative Review-Revision: 44.0% RR at sample 5, up from 34.8% at sample 1, while cumulative oracle coverage reaches 47.2% (236/500) on SWE-bench Verified.RR denotes the resolve rate of the reviewer-approved patch at each sample; cumulative oracle RR credits any instance with at least one resolving patch.

A.2 Comparisons Across More Coding Agents and Selection Methods · B Extended Analysis of Reviewer Behavior

The extended comparison shows reviewer-gated resampling consistently outperforms alternative selectors across two coding agents while using fewer samples. Additional analyses examine reviewer exploration, failure modes, and token consumption.

  • A.2 Comparisons Across More Coding Agents and Selection Methods: The evaluation draws up to K independent issue-resolution trajectories and compares five selection rules under a maximum budget K ∈{1, . . . , 16}.Oracle reports Pass@K, while Reviewer uses sequential reviewer-gated resampling.
  • A.2 Comparisons Across More Coding Agents and Selection Methods: With Qwen3-30B-A3B, Reviewer dominates Verifier and Critic across all test-time budgets, benefiting from early stopping.Reviewer-gated resampling stops when a candidate is approved or when K attempts are exhausted.
  • A.2 Comparisons Across More Coding Agents and Selection Methods: Results report standard deviations over random candidate orderings and compare reviewer-gated sample consumption against the fixed-budget identity line y=K.This analysis highlights efficiency differences between adaptive resampling and fixed best-of-K methods.
  • A.2 Comparisons Across More Coding Agents and Selection Methods: 32.3% resolve rate at K=16 is achieved by Reviewer, versus 25.6% for Verifier and 18.8% for Critic.These results use Qwen3-30B-A3B as the coding agent.
  • A.2 Comparisons Across More Coding Agents and Selection Methods: 44.4% reduction in samples is achieved by Reviewer at K=16, consuming 8.9 samples on average instead of the fixed budget of 16.The comparison reports TTS@K as the SWE-bench Verified resolve rate at budget K.
  • A.2 Comparisons Across More Coding Agents and Selection Methods: With Qwen3-Coder-30B-A3B, Reviewer remains the best practical selector, preserving the relative ordering despite higher absolute resolve rates.The reviewer is the distilled SWE-Review-8B model in both coding-agent comparisons.
  • A.2 Comparisons Across More Coding Agents and Selection Methods: 48.1% resolve rate at K=16 is achieved by Reviewer, versus 44.4% for Verifier, while Reviewer uses 6.0 samples on average.This corresponds to a 62.5% reduction in samples; Critic underperforms Random at larger K.
  • B Extended Analysis of Reviewer Behavior: Extended reviewer-behavior analyses examine exploration behavior, failure modes, and token consumption across the agentic review pipeline.These analyses are organized as Sections B.1, B.2, and B.3.

B.1 Reviewer Exploration Behavior

Reviewer quality depends more on exploration efficiency and reliable verification than on interaction volume. Excessive exploration tracks uncertainty, while difficulty-adaptive exploration and step count reveal diminishing returns and potential confidence signals.

  • Cross-model comparison: 81.8% DA: Opus 4.6 uses 24.1 steps, reads 1.3 files, and consumes 148K tokens, outperforming SWE-Review-8B’s 69.1% DA despite 57.3 steps and 2.36M tokens.The comparison indicates that exploration precision, rather than volume, predicts review quality.
  • Cross-model comparison: 99.9% reproducer execution: Opus 4.6 substantially exceeds SWE-Review-8B’s 76.7%, while distilled models compensate with 3.5–3.8× more files and 2.6–3.3× more grep searches.Despite this additional exploration, distilled models achieve 10–13 percentage points lower DA, indicating a verification-test competency gap.
  • DA-correct versus DA-incorrect exploration: DA-incorrect cases consume significantly more steps, code executions, and tokens across all reviewers, with all differences significant at p < 0.001.Incorrect Opus reviews use 186K versus 140K tokens, while incorrect SWE-Review-8B reviews use 2.91M versus 2.05M.
  • DA-correct versus DA-incorrect exploration: 93.0% reproducer rate: SWE-Review-30B-A3B’s DA-incorrect cases exceed its DA-correct cases at 80.8% (p < 0.05).For this model, reproducer output can mislead reasoning when ambiguous test outcomes are interpreted incorrectly.
  • Difficulty stratification: 86.7% to 76.8% DA: Opus 4.6 increases exploration from 18.3 to 31.8 steps and beyond-diff rate from 18.4% to 55.1% across difficulty tertiles.Exploration rises with difficulty, but accuracy declines despite increased computation, indicating diminishing returns.
  • Difficulty stratification: 90.8% DA: the fastest step quartile (≤17 steps) outperforms the slowest (>28 steps) at 69.4%, with odds ratio 4.32 (p < 0.001).Step count may serve as a lightweight confidence proxy for abstention or escalation strategies.

B.2 Failure Mode Analysis

Claude Opus 4.6 made 272 review errors, comprising 167 false approvals and 105 false rejections, with distinct root causes for each error type. Errors concentrated in difficult tasks, while test execution could create false confidence or misattribute failures to patches.

  • Error Overview: 272 total errors comprised 167 false approvals and 105 false rejections; root-cause annotation successfully categorized 259 errors.The annotation used the issue description, candidate patch, golden patch, and reviewer report.
  • Error Distribution by Split: 83% of errors were false approvals on the 72.2% resolve-rate GLM-5 split, whereas false rejections slightly predominated at 57% on the 27.5% Qwen3-30B-A3B split.The passage attributes this shift to near-correct patches being harder to distinguish from resolving ones and correct rejections being easier on weaker generators.
  • Root Cause Breakdown: False approvals mainly arose from subtle logic bugs (37.7%), insufficient reproducer coverage (32.1%), and cross-file reasoning gaps (29.0%), while false rejections primarily reflected code misreading (44.3%).No-reproducer errors were absent because Opus ran reproducers in 99.9% of trials, indicating that false approvals reflected test quality rather than test absence.
  • Error Distribution by Difficulty: 80 false approvals concentrated in the hard tertile, while false-rejection rates peaked non-monotonically at 9.5% in the medium tertile.False-approval rate more than doubled from easy to hard, rising from 7.6% to 16.7%.

B.3 Token Consumption

Token use in the closed-loop pipeline shifts with generator quality: review remains nearly fixed-cost while generation and revision vary. Review-guided iterative revision also delivers the strongest test-time scaling efficiency.

  • Closed-loop pipeline: As generator quality decreases, review’s budget share rises from 36% to 66%.For GLM-5, generation, review, and revision each consume roughly one-third of the budget.
  • Closed-loop pipeline: Review consumes 2.19–2.46M tokens across generators, while generation and revision costs scale with generator capability.This indicates that review functions as a fixed-cost understanding phase.
  • Review tokens by difficulty: Hard instances require 38% more review tokens than easy instances for GLM-5, rising from 150K to 207K.The difficulty premium is larger for high-quality generators than for weaker ones, which show an 18% increase.
  • Reviewer token efficiency: GLM-5 uses 1,685K tokens per correct decision, versus 3,190K and 3,423K for the SWE-Review models.Its advantage combines 45% fewer tokens per review with higher decision accuracy: 77.3% versus 72.0% and 69.1%.
  • Test-time scaling token efficiency: Review-guided iterative revision reaches a 38.4% resolve rate and 2.28 pp/Mtok efficiency, 6.5× higher than reviewer-gated resampling’s 0.35 pp/Mtok.It uses 2.44 samples on average within a maximum budget of K=5.

C SFT Training Details … D.2 Raw Candidate PR

The appendix specifies the supervised fine-tuning setup and preprocessing pipeline, then examines a SymPy candidate PR whose guarded comparison removes a crash symptom without establishing determinant correctness. The case study contrasts this incomplete patch with the deeper issue behavior that motivates agentic review.

  • C SFT Training Details / C.1 Infrastructure: The appendix provides full reproduction details for the paper’s supervised fine-tuning experiments on distributed multi-node accelerator infrastructure.Training uses high-memory accelerator nodes, with larger-memory devices for larger MoE models.
  • C.2 Hyperparameters: The default SFT configuration uses learning rate 10^-4, effective batch size 64, BF16 precision, and a 131,072-token YaRN-scaled context.It also uses cosine scheduling, gradient clipping at 1.0, weight decay 0.01, warmup ratio 0.1, and qwen3_nothink.
  • C.3 Training Optimizations: Training combines Liger fused kernels, Unsloth gradient checkpointing, Flash Attention 2, YaRN scaling, and loss masking for unsuccessful assistant turns.These optimizations reduce memory or extend usable context while restricting loss to successful assistant responses.
  • C.4 Training Time and Compute: Training takes about 30 hours for Qwen3-8B and about 70 hours for Qwen3-30B-A3B, with variance dominated by samples longer than 64K tokens.Each run uses approximately 9,000 training examples, and MoE models activate only a subset of experts per token.
  • C.5 Data Preprocessing: Raw trajectories are cleaned, rendered with the native chat template, split by turn for masking, and serialized as ShareGPT JSON with per-turn annotations.The tokenizer must exactly match the target base model.
  • D Case Study: sympy-13877: The sympy-13877 case compares single-turn and agentic Opus review of a candidate PR that fails the hidden determinant test because the determinant still returns nan.Single-turn review approves the patch, whereas agentic review requests changes and is scored correct by the evaluation harness.
  • D.1 Raw Issue: The reported issue is an Invalid NaN comparison during symbolic matrix determinants, including f(5) returning nan and f(6) raising a TypeError.The issue questions whether the Bareiss algorithm is valid for matrices whose entries cannot be assumed to be integers.
  • D.2 Raw Candidate PR: The candidate PR edits factor_terms in sympy/core/exprtools.py to guard comparisons involving non-finite coefficients, changing f(6) from a crash to nan.Because it edits a downstream comparison rather than the Bareiss determinant path, the patch removes the crash symptom but does not establish restored determinant correctness.

D.3 Single-Turn Review: Opus, Diff+Context

The single-turn reviewer approves the patch as a safe, targeted fix for the immediate NaN-comparison crash, while explicitly noting that the deeper Bareiss NaN issue remains unresolved.

  • Single-Turn Review: The reviewer recommends approval with 0.88 confidence, judging the crash guard a safe targeted fix despite the unresolved deeper Bareiss NaN issue.The final report records no defects and states that the patch introduces no regressions for normal cases.
  • Single-Turn Review: The patch prevents TypeError from invalid NaN comparisons by validating coefficients before evaluating whether they are negative.The helper returns False for NaN, infinite, or non-real coefficients, skipping common-negative-factoring only when coefficients are invalid.
  • Single-Turn Review: The review identifies the deeper Bareiss algorithm problem—producing NaN coefficients during symbolic matrix determinant computation—as unaddressed.The report gives f(5) already returning nan as an example of the remaining issue.

D.4 Agentic Review: Opus

The agentic reviewer correctly rejects the patch because it is a downstream symptom fix rather than a root-cause correction, leaving determinant computations incorrect. It identifies the discarded cancel() result in the Bareiss algorithm and recommends assigning it back to ret.

  • Review decision: The reviewer requests changes, determining that the patch remains incorrect and does not resolve the underlying bug.The verifier reports verdict=correct, reward=1, gt_resolved=false, and bug_fixed=false.
  • Root-cause diagnosis: The root cause is _eval_det_bareiss discarding cancel(ret), causing unsimplified zero expressions to become nonzero pivots and produce division by zero and NaN results.The correct fix is to change cancel(ret) to ret = cancel(ret) in sympy/matrices/matrices.py.
  • Patch assessment: The patch only guards against NaN comparisons in factor_terms, suppressing a TypeError without correcting the determinant computation.The reviewer characterizes it as a symptom fix at a downstream location.
  • Impact: f(5) and f(6) still return nan instead of the correct value 0, making the issue high-severity and a correctness defect.The incorrect outputs persist after the patch.

D.5 Agentic Trajectory · E System Prompts

The agentic-review trajectory shows how root-cause tracing, patch comparison, and behavioral testing identified a symptom fix and led to a request for changes. The system prompts operationalize this workflow with constrained exploration, verification rules, structured reports, and distinct single-turn and oracle-review settings.

  • D.5 Agentic Trajectory: The 21-step trajectory traced the determinant failure to discarded cancellation in matrices.py, where unsimplified zero expressions could survive as nonzero pivots.The reviewer identified that cancel(ret) should be assigned back to ret.
  • D.5 Agentic Trajectory: The candidate patch added _is_neg_coeff in exprtools.py, but the reviewer classified it as a downstream symptom fix because matrices.py remained unchanged.The patch guarded NaN comparison rather than correcting the upstream determinant computation.
  • D.5 Agentic Trajectory: After applying the patch, the reproducer still returned f(5) = nan and f(6) = nan, so the final report saved OK: request_changes.The final rationale emphasized that incorrect nan outputs remained despite preventing the observed exception.
  • E.1 Patch Generation Instruction: The patch-generation prompt gives the agent full repository access in Docker and requires creating /pr_content.json with a pull-request title and body before completion.The task is incomplete if the file is not created.
  • E.2 Agentic Review Instruction: The review prompt requires independent reconstruction before patch inspection, root-cause commitment, patch comparison, and complete behavioral verification within a 100-iteration budget.The workflow allocates caps for reading and hypothesis, exploration, testing, and report writing.
  • E.2 Agentic Review Instruction: The prompt hard-codes request_changes for symptom fixes, failed tests, incomplete code paths, high-severity defects, or incorrect outputs, while approval requires root-cause coverage and passing verification.Verification must include reproduction, exact output or return-value checks, and regression behavior.
  • E.3 Single-Turn Review Prompts: Single-turn review removes code execution and compares diff-only input with a source-context setting that additionally supplies affected pre-change files, while retaining the same JSON schema.The diff-only setting provides the issue, patch, and PR metadata; the source-context setting adds surrounding code.
  • E.4 Oracle Review Prompt: The oracle prompt internally uses golden patches and tests for calibration but requires a blind, self-contained code review that reports defects without reference leakage.Its three phases are independent analysis, internal validation, and report generation.
Loading 2607.06065v1…