PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarial Review: Structured Disagreement for Grounded Agentic Code Review

Eric S. Qiu, Joyce Gill

arXiv:2608.18167v1cs.AIcs.SE

TL;DR

Coding agents need reliable ways to cooperate without the overhead and diminishing returns of larger teams. Adversarial Review uses a reviewer–critic exchange with structured, evidence-grounded disagreement before the main agent edits, achieving the highest measured score among tested methods across three benchmarks.

  • Problem

    The paper asks how multiple agents should be organized to work together reliably on coding and review tasks despite communication overhead and scaling limits.

  • Method

    Adversarial Review freezes the artifact while a reviewer and critic iteratively examine and challenge the review before the main agent edits.

  • Results

    AR achieves the highest measured score among tested methods on all three benchmarks while remaining a compact interactive structure.

  • Takeaways & Limitations

    Structured, explicit, evidence-grounded disagreement may improve coding and code-review quality more effectively than naively increasing agent count.

  • Takeaways & Limitations

    AR exhibits false consensus on SWE-PRBench, where agents can agree without sufficient evidence and code-generation performance does not transfer to code review.

Abstract

from arXiv · show

Early multi-agent LLM systems often used role-separated teams, yet scaling agent count yields diminishing returns on repository-level coding tasks. Recent alternatives treat agents as passive tools (subagents), yet this removes the benefits of agent interaction entirely. We study whether a subagent paradigm can support a middle ground: minimal agentic cooperation without the overhead of large multi-agent teams. We introduce Adversarial Review (AR), a minimal cooperative code-review protocol in which a main coding agent works with a reviewer and a critic agent. The reviewer evaluates code, while the critic audits the review through structured disagreement before the main agent edits. On LiveCodeBench, AR achieves the highest pass rate among tested methods, outperforming a five-agent baseline while using only three agents. On SWE-PRBench, naive AR exposes a false-consensus failure mode, where agents converge on agreement without sufficient evidence, but a single prompt iteration that adds disagreement explicitly achieves the highest F1 among tested methods. On SWE-bench Verified, AR also shows improvements over the baselines on repository-level coding tasks. Together, AR demonstrates that cooperative code review does not require many agents or complex communication structures: it requires that disagreement be minimal, structured, and evidence-grounded.

1. Introduction

The introduction frames reliable multi-agent organization for coding tasks as an open design problem between large role-separated teams and lightweight main-agent-with-subagents systems. It introduces Adversarial Review as a minimal reviewer–critic protocol intended to preserve useful interaction while addressing agreement-driven errors.

  • Motivation: The paper targets a middle ground between large role-separated teams and lightweight subagent cooperation, constraining communication while preserving agent interaction.This design responds to diminishing or negative returns associated with scaling peer agents on complex repository-level coding tasks.
  • Protocol: The protocol freezes each artifact while a reviewer and critic iteratively evaluate the review before the main agent edits the artifact.If the review converges immediately without identifying flaws, the artifact is accepted; otherwise, the main agent produces an updated version.
  • Contributions: Adversarial Review uses a minimal reviewer–critic loop to improve agentic coding and program repair within a main-agent-with-subagents paradigm.The protocol is evaluated across LiveCodeBench, SWE-PRBench, and SWE-bench Verified.
  • Contributions: Adversarial Review addresses a core cooperative-agent failure mode: agents may optimize for agreement rather than correctness.The paper argues that structured disagreement can fix false consensus and improve code generation and review quality across three benchmarks.

2. Related Work

This section positions Adversarial Review (AR) between single-model refinement, debate-heavy multi-agent systems, and specialized coding-agent teams. It motivates AR as a minimal, evidence-grounded reviewer–critic protocol within main-agent workflows, evaluated across complementary coding benchmarks.

  • Self-refinement and external verification for code: AR extends test-time refinement by adding external, structured review rather than relying solely on a model’s self-correction loop (Madaan et al., 2023; Huang et al., 2024).Self-Refine keeps generation, feedback, and revision within one model, while subsequent work questions reliable self-correction without external feedback.
  • Multi-agent debate and structured disagreement: AR takes a middle position between open-ended multi-agent debate and cheaper aggregation schemes by preserving interaction through one communication channel with explicit disagreement types (Du et al., 2024; Liang et al., 2024).Debate systems can depend on prompt and protocol choices, whereas MARS uses independent reviewers whose comments are aggregated by a meta-reviewer.
  • Multi-agent systems for code generation and review: Rather than adding specialized coding roles, AR asks whether a minimal reviewer–critic pair can provide reliable, evidence-grounded review within a main-agent-with-subagents workflow.Prior systems divide coding among roles for recall, planning, generation, debugging, implementation, testing, or execution, including MapCoder and AgentCoder.
  • Agentic coding systems and portable protocols: AR is motivated by production agent architectures that use main agents with tools, subagents, or portable Skills, and is evaluated as both a Python orchestrator and a pure-text Skill.Subagents are separate instances for focused subtasks, while Skills are reusable instruction bundles activated through SKILL.md files (Anthropic, 2024; 2026; 2025).

3. Construction of AR on LiveCodeBench

The section constructs Adversarial Review (AR) as a three-role protocol in which a reviewer and critic interact before the main agent edits. On LiveCodeBench, AR achieves the highest reported performance with 87% pass-rate and 43/57 pass-on-hard, outperforming MARS by 5 percentage points with fewer reviewing roles.

  • Construction: Self-Refine matches Zero-shot at 77% pass-rate because the same model generates and critiques code, motivating a separate reviewer call.Single-reviewer also reaches 77% pass-rate and 36/57 pass-on-hard, remaining in the same performance cluster.
  • Construction: Two independent reviewers reach 75% pass-rate and 34/57 pass-on-hard, staying within the same cluster and motivating interaction rather than additional independent samples.MARS breaks out of that cluster at 85% pass-rate and 43/57 pass-on-hard, but uses five agents per task and incurs higher token cost.
  • AR protocol: AR uses main agent M, reviewer R, and critic C: R reviews an unedited artifact, C challenges the review, and the exchange repeats until convergence or five inner rounds.After convergence, a flaw-free first review accepts the artifact; otherwise M edits, updates the change log, and starts another outer review round.
  • Results: 87% pass-rate and 43/57 pass-on-hard make AR the highest-performing of six LCB methods; it gains +5pp over MARS using two reviewing roles versus four.Table 1 summarizes the six-method comparison, while the reported role counts are R and C for AR versus three reviewers and one meta-reviewer for MARS.
  • AR protocol: AR is the smallest interactive configuration, retaining three roles—M, R, and C—while interaction occurs between R and C.Removing C yields Single-reviewer, whereas adding reviewers or rounds produces MARS-style or MAD-style structures.

4. Review quality on SWE-PRBench

On SWE-PRBench, naive Adversarial Review underperforms other review methods because its interactive reviewer–critic loop encourages speculative additions and false agreement. Adding explicit, evidence-grounded pushback through prompt changes raises AR to the highest F1 among the compared methods.

  • 4.1 Setup: Naive AR scores F1 = 0.457, the lowest of four methods, while the other subset methods reach approximately 0.50.The comparison includes AR, MARS, Two-reviewers, and Single-reviewer; Zero-shot and Self-Refine are excluded because they do not apply to review-only evaluation.
  • 4.3 Failure modes: The reviewer–critic loop can over-decompose reviews by confirming hedged concerns and adding speculative bugs, producing fabricated comments.In Case A, the judge marks 3 of 5 comments as fabricated because they are too thin or speculative.
  • 4.3 Failure modes: The loop can also produce false agreement: a critic raises a verified concern, but a weak rebuttal makes the critic yield and the final review drops the real bug.For the same bug, AR achieves F1 = 0.286 while MARS achieves F1 = 0.667.
  • 4.4 Prompt iteration: The fix explicitly distinguishes evidence-based disagreement from epistemic concern and requires code-grounded responses, preventing agents from agreeing without sufficient evidence.The text constraint gives the critic AGREE, DISAGREE EVIDENCE, or DISAGREE CONCERN; the reviewer must revise, confirm, or drop flags based on cited code.
  • 4.4 Prompt iteration: F1 = 0.533 makes AR with text constraint the highest-performing method among the four-method SWE-PRBench subset.The protocol retains one reviewer and one critic; only the prompts change.

5. Real-world coding on SWE-bench Verified

On SWE-bench Verified, AR achieves the highest pass rate among the three evaluated methods, outperforming Zero-shot and MARS while using substantially more tokens. Case studies show that structured disagreement can either identify root causes or amplify speculative, out-of-scope edits.

  • Overall results: 75.2% pass-rate: AR outperforms Zero-shot at 71.6% and MARS at 72.6% on SWE-bench Verified.The benchmark contains 500 real GitHub issues, and evaluation uses pass@1 through the official Docker harness.
  • Overall results: AR uses about 4.5× Zero-shot’s tokens, trading token efficiency for higher performance.
  • Case studies: In matplotlib-20826, AR patches the state-destroying implementation-layer operation, producing an approximately 20-line patch that alone passes hidden tests.Zero-shot and MARS patch the caller with 45- and 50-line symptom fixes that both fail hidden tests; AR’s critic catches the reviewer’s initial mistake.
  • Case studies: In astropy-14182, AR expands a required 24-line fix into a 32-line speculative patch that fails hidden tests, whereas Zero-shot stays in scope and passes.The critic fails to reject the reviewer’s out-of-scope concern about a separate read() call, leading the main agent to add unnecessary changes.
  • Failure mode: AR helps when disagreement redirects the agent toward the true bug but hurts when it turns plausible concerns into unnecessary edits, motivating explicit scope discipline.This is the main failure mode identified for AR on SWE-bench Verified.

6. Cost-quality position across benchmarks

Across all three benchmarks, AR lies on the cost-quality Pareto frontier: although it uses more tokens than Zero-shot, no tested method is both cheaper and better.

  • Cost-quality position across benchmarks: AR is on the cost-quality Pareto frontier across all three evaluated benchmarks, as shown in Figure 2.The figure plots per-task median tokens against performance for each benchmark.
  • Cost-quality position across benchmarks: AR uses more tokens than Zero-shot on all three benchmarks, trading higher token cost for performance.
  • Cost-quality position across benchmarks: No tested method is both cheaper and better than AR on these benchmarks.By definition, the Pareto frontier contains methods for which no other method is simultaneously cheaper and higher-performing.

7. Discussion and limitations

AR’s benefit comes from narrow, role-separated cooperation and evidence-grounded disagreement rather than simply adding agents. The discussion also identifies false consensus, token costs, benchmark limitations, and the need for adaptive, auditable invocation.

  • Contribution: AR separates editing from review and criticism, preventing agents from jointly rewriting the solution while disagreeing about what is wrong.
  • Contribution: AR occupies a middle ground between passive subagent tools and larger agent teams: the main agent edits, while reviewer and critic interact before editing.
  • Failure mode: False consensus is a structural reliability failure in which conversational pressure can make weak or challenged findings appear independently validated.
  • Design principle: Evidence-grounded disagreement improves auditability by requiring the critic to classify disagreement and the reviewer to answer concerns with code evidence.
  • Limitations: The claims are empirical and benchmark-dependent: controlled Python workflows and looser SWE-bench Verified instructions answer different questions, while judge design may penalize valid review comments.
  • Future work: Future work should make cooperation adaptive and auditable through scope checks, confidence gates, and training signals derived from disagreement traces.

8. Conclusion

AR achieves the highest measured score among the tested methods on all three benchmarks while preserving a compact main-agent-with-subagents structure. Its central finding is that structured, explicit, evidence-grounded disagreement can improve code generation and code review more effectively than naively increasing agent count.

  • 8. Conclusion: AR achieves the highest measured score among the tested methods on all three benchmarks while retaining a compact interactive structure.The progression compares Zero-shot, Self-Refine, Single-reviewer, Two-reviewers, MARS, and AR on LCB before extending the finding across all three benchmarks.
  • 8. Conclusion: Structured, explicit, evidence-grounded disagreement may improve code generation and code review more effectively than naively scaling agent count.This finding motivates compact cooperation within the main-agent-with-subagents paradigm rather than larger agent teams.

A. Prompt templates

The appendix provides every method prompt, grouped by benchmark, and explains runtime placeholder substitution and planned release with the implementation.

  • A. Prompt templates: The appendix dumps every prompt used by every method, organized by benchmark.This provides the complete prompt set for the methods covered in the appendix.
  • A. Prompt templates: Python-style format-string placeholders are substituted at runtime by the orchestrator.Examples include {specification}, {entry point}, {frozen ask}, {diff}, {problem statement}, {reviewer latest}, and {prior exchange}.
  • A. Prompt templates: The prompts will be released alongside the implementation upon acceptance.

A.1. LiveCodeBench (LCB) – end-to-end code generation … A.1.6. SINGLE-REVIEWER, TWO-REVIEWERS, AND AR: REVIEWER PROMPT (LCB)

The LCB setup standardizes initial code generation and review-based editing across methods, while distinguishing zero-shot, self-refinement, and reviewer-based prompts. Reviewer prompts constrain feedback to evidence-grounded correctness issues and require structured verdicts.

  • A.1.1. ALL METHODS: MAIN-AGENT FIRST-VERSION WRITE (LCB): All methods first generate complete Python code from the specification, preserving the mandatory entry-point name and emitting only executable code.The initial prompt also requires necessary imports and inline implementations for undefined helper utilities.
  • A.1.2. ALL METHODS: MAIN-AGENT EDIT-FROM-REVIEW (LCB): Review-having methods revise the previous artifact using a consistent review, rewriting the complete updated Python code without explanatory prose or markdown fences.The edit prompt supplies both the prior code and the reviewer-critic loop’s consistent review.
  • A.1.3. ZERO-SHOT (LCB): Zero-shot uses only the shared initial-write prompt and includes no reviewer or critic prompts.This defines zero-shot as the no-review baseline within the LCB setup.
  • A.1.4. SELF-REFINE: SELF-CRITIQUE (LCB): Self-refine first asks the model to critique its own code for correctness, listing specific bugs, edge-case misses, and logical errors without rewriting.The critique must end with a standalone NEEDS_REVISION verdict line.
  • A.1.5. SELF-REFINE: REVISE-FROM-CRITIQUE (LCB): Self-refine then provides the task specification, original code, and self-critique before requiring a complete rewrite that addresses every flagged issue.The revised output must contain only code, with no prose or markdown fences.
  • A.1.6. SINGLE-REVIEWER, TWO-REVIEWERS, AND AR: REVIEWER PROMPT (LCB): Reviewer-based prompts provide the frozen original ask, artifact code, and any prior inner-loop exchange as context for correctness review.The reviewer is framed as a senior software engineer reviewing a code artifact.
  • A.1.6. SINGLE-REVIEWER, TWO-REVIEWERS, AND AR: REVIEWER PROMPT (LCB): Reviewer feedback is restricted to specific correctness problems—bugs, edge cases, and logic errors—while excluding style comments and requiring a standalone NEEDS_CHANGES verdict.The format rejects any other final line.

A.1.7. AR: CRITIC PROMPT (LCB) … A.1.10. MARS: META-REVIEWER PROMPT (LCB)

The LCB appendix specifies AR’s critic prompt and MARS’s author, reviewer, and meta-reviewer prompts. Together, these prompts enforce independent, evidence-based review, structured disagreement, and consolidated revision guidance.

  • A.1.7. AR: CRITIC PROMPT (LCB): AR’s critic evaluates whether each reviewer-flagged issue is a real bug and identifies any real bugs the reviewer missed.The critic must end with a valid verdict, including DISAGREE when disputing or extending the review.
  • A.1.8. MARS: AUTHOR PROMPT (LCB): MARS reuses its main-agent writing prompt for round one and its edit-from-review prompt thereafter, inserting meta-reviewer suggestions into the consistent-review slot.
  • A.1.9. MARS: REVIEWER PROMPT (LCB): MARS dispatches three reviewer subagents in parallel, independently evaluating a generated Python program without seeing one another’s outputs.The reviewer schema contains Decision, Confidence, Justification, and Errors fields.
  • A.1.9. MARS: REVIEWER PROMPT (LCB): Each MARS reviewer first derives the correct behavior from the original ask, then checks the candidate for compliance, edge cases, logic errors, and scope.Reviews must provide evidence-based reasoning and cite line numbers or code snippets when applicable.
  • A.1.10. MARS: META-REVIEWER PROMPT (LCB): Once per round, MARS’s meta-reviewer reads the candidate and three independent reviews, issuing one consolidated decision and concrete revision suggestions when rejecting.Its four-field schema is Decision, Justification, Suggestions, and Recommended-answer.
  • A.1.10. MARS: META-REVIEWER PROMPT (LCB): The MARS meta-reviewer must weigh reviewers’ reasoning and evidence rather than count votes or treat confidence scores as decisive.When reviewers disagree, it should prefer the position grounded in concrete evidence about the candidate program.

A.2. SWE-PRBench – review-only inner loop

On SWE-PRBench’s review-only task, Adversarial Review compares a baseline inner loop with a text-constrained variant, while omitting any main-agent edit step. The text constraint improves AR’s F1, and the protocol explicitly grounds reviewer–critic disagreement in code evidence.

  • Results: F1 rises from 0.457 for baseline AR to 0.533 for AR with text constraint on SWE-PRBench.The benchmark is review-only, so the reviewer and critic produce a review of a real PR diff without a main-agent edit step.
  • Protocol: AR uses one reviewer call inside a reviewer–critic inner loop, whereas Single-reviewer and Two-reviewers invoke the shared reviewer prompt differently.Single-reviewer uses one call, Two-reviewers uses two independent calls, and AR uses one call as part of an R–C inner loop.
  • Text constraint: The text-constrained AR variant adds only a response-rule block that makes the reviewer preserve, revise, or drop findings according to the critic’s verdict and evidence.It preserves legitimate findings after acceptance, revises or drops flags contradicted by cited code, and requires evidence-based confirmation or refutation for unsupported concerns; missed bugs are incorporated.
  • Critic protocol: The critic distinguishes code-grounded disagreement from epistemic concern, using DISAGREE_EVIDENCE for contradictions in the diff and DISAGREE_CONCERN for objections that require further evidence.DISAGREE_CONCERN is intended to request evidence rather than suppress a reviewer finding.
  • MARS baseline: MARS independently dispatches three reviewers and then uses one meta-reviewer to consolidate their evidence into the submitted review, without counting votes.Because SWE-PRBench has no author revision step, the meta-reviewer’s Justification and Errors-style content is scored, while Suggestions and Recommended-answer fields are not.

A.3. SWE-bench Verified – repository-level coding tasks

On SWE-bench Verified, methods use explicit, self-contained prompts built from a shared wrapper, method-specific protocol body, and shared overrides. The evaluated AR row uses baseline two-verdict AR, while MARS uses three reviewers, one meta-reviewer, and up to two rounds.

  • Prompt construction: Each SWE-bench Verified prompt combines a shared repository-and-issue wrapper, a method-specific body, and shared model-matching overrides.Prompts are inlined to make every protocol explicit and reproducible; the wrapper supplies repository state, issue text, scope rules, and budget.
  • Compared protocols: The SWE-bench Verified Zero-shot prompt contains only the shared wrapper, whereas the evaluated AR row uses baseline two-verdict AR rather than the later text-constraint upgrade.The MARS row retains its three-reviewer, one-meta-reviewer, two-round protocol.
  • MARS protocol: MARS has one author, three independent reviewers, and one meta-reviewer who consolidates evidence and provides revision suggestions when rejecting a candidate.Reviewers do not communicate with one another, and the author revises the existing fix for at most two rounds.
  • MARS protocol: MARS makes the meta-reviewer the decision authority: it weighs reviewers’ reasoning and concrete evidence instead of counting votes or relying on confidence scores.Structured labelled outputs allow the author to apply decisions and suggestions deterministically.
Loading 2608.18167v1…