PaperlayerView exact revision on arXiv

Source-linked AI summary

Reviewer Capability Governs Rejection Targeting, Not Repair Skill: Evidence from LLM Execute-Review-Revise Pipelines

Faizan Tanveer

arXiv:2609.04270v1cs.SEcs.CLcs.MA

TL;DR

Multi-agent pipelines delegate roles to cheaper models, but prior work largely holds reviewer capability fixed relative to the executor. This paper varies reviewer capability and measures each rejection’s outcome; a cross-family mid-tier reviewer improves accuracy by 12 percentage points with zero damaged answers, while self-review’s high detection does not yield significant gain.

  • Problem

    Prior work largely holds reviewer capability fixed relative to the executor, leaving unclear which pipeline roles can be delegated to weaker models without degrading outcomes.

    Source: §1 · p. 1

  • Method

    The study holds the executor fixed across 100 olympiad mathematics problems, varies reviewer capability, and classifies every rejection by its outcome.

    Source: §1 · p. 1

  • Results

    A cross-family mid-tier reviewer improves accuracy by 12 percentage points with zero damaged answers, while same-model self-review detects most errors but yields no significant gain.

    Source: §1 · p. 1

  • Takeaways & Limitations

    Review failure depends on reviewer capability, and self-review’s apparent safety can arise from revision inertia rather than reviewer quality.

    Source: §1 · p. 1

    The findings describe 100 problems from one benchmark, one difficulty band, one executor, one revision round, and one reviewer prompt template, not review stages generally.

    Source: §6 · p. 5

Abstract

from arXiv · show

Multi-agent LLM pipelines increasingly assign roles, including execution and verification, to models of different capability tiers. This is done because running a flagship model at every stage is expensive. Previous literature has established that verification stages are not always beneficial, but holds reviewer capability roughly fixed relative to the executor. We vary it. We replace the reviewer with models spanning a capability range down to one that cannot solve the problems at all, and measure the outcome of every individual rejection. This is done across a constant set of 100 olympiad mathematics problems. A cross-family mid-tier reviewer improves final accuracy by 12 percentage points, from 52 to 64 percent (p = 0.0005), with zero damaged answers. Same-model self-review attains the highest error-detection rate of any condition (0.85 recall) yet yields no significant gain: it rejects 2.1 times as often for a third the repair rate (15 against 43 percent, p = 0.0074) and falsely rejects 35 percent of its own correct answers against 2 percent for the cross-family reviewer (paired p = 0.000015). The low damage rate of self-review proves to be an artifact of revision inertia rather than reviewer quality: of 18 falsely rejected correct answers, the three where the executor complied all became wrong, while the fifteen it ignored survived unchanged. Below a capability floor the role becomes inert: our weakest reviewer changed zero of 100 final answers while doubling token cost. These findings describe a single executor-reviewer configuration on 100 problems and should be read as a controlled pilot rather than a general claim about verification stages.

1 Introduction

The paper varies reviewer capability while holding the executor fixed, measuring final accuracy and the outcome of every rejection. A cross-family mid-tier reviewer improves accuracy, while self-review and a weak reviewer expose distinct limits of detection and revision.

1 Introduction

  • The study holds a capable executor constant across 100 olympiad mathematics problems and varies reviewers down to a model unable to solve the problems itself.It classifies each rejection as repair, damage, or no change.Source: §1 · p. 1
  • Prior studies largely held reviewer capability fixed relative to the executor, leaving its effect on review outcomes untested.The study varies reviewer capability to test that gap.Source: §1 · p. 1
  • A cross-family mid-tier reviewer improves accuracy by 12 percentage points with zero damaged answers, while self-review detects most errors but produces no significant gain.Self-review rejects 2.1 times as often for one-third the repair rate and falsely rejects 35 percent of its own correct answers.Source: §1 · p. 1
  • The weakest reviewer changes none of the 100 final answers while doubling token cost.The paper identifies this as a capability floor below which review is effectively inert.Source: §1 · p. 1

2 Related Work

Related work studies team role assignment, critique uptake, verification errors, and self-correction, but generally does not vary reviewer capability relative to the executor. This paper instead sweeps reviewer capability and decomposes rejection outcomes.

Role assignment and cost

  • AgentCARD finds heterogeneous teams can achieve up to 44 percent higher accuracy at equivalent cost, while role bottlenecks depend on domain.It treats roles as assignment slots rather than reviewer capability as a variable to sweep.Source: §2 · pp. 1–2

Reviewer precision versus critique uptake

  • Yang et al. report higher reviewer precision for a planner–executor–reviewer pipeline than broadcast discussion, but lower performance when useful critiques less often change the carried answer.Their reviewers remain at the same capability tier as the other actors.Source: §2 · p. 2

Verification harm as a mechanism

  • Detection Without Correction reports conditional miscorrection rates of 53 to 94 percent across model families and benchmarks, but does not systematically vary reviewer capability.Its mechanism is established using near-peer models.Source: §2 · p. 2

Self-correction and mistake finding

  • Self-review studies disagree on gains, while this paper reports 0.85 recall and only 15 percent repair for self-review, locating poor outcomes in imprecise targeting.Tyen et al. attribute self-correction failure to detection; Self-Refine reports gains, while Huang et al. find no improvement and frequent degradation.Source: §2 · p. 2

This work

  • Prior work holds reviewer capability approximately fixed, whereas this study holds the executor fixed and sweeps reviewer capability while classifying every rejection outcome.The sweep includes a reviewer that solves approximately none of the problems unaided.Source: §2 · p. 2

3 Method

The method uses a paired two-stage executor–reviewer pipeline, varying reviewer conditions on a filtered 100-problem set. It applies exact numeric grading and rejection-outcome measures, with a prompt change for reviewer input and paired statistical tests.

3.1 Pipeline

  • The paired design gives each condition the same initial executor solution, so condition differences are attributable to the reviewer rather than executor sampling variation.On rejection, the executor revises once; the final answer is the revision, otherwise the initial answer.Source: §3.1 · p. 2

3.2 Conditions

  • Four conditions keep Gemini 3.1 Flash-Lite fixed and vary review: none, self-review, cross-family gpt-oss-20b, or weak Llama-3.1-8B.The weak reviewer solves approximately none of the problems unaided.Source: §3.2 · p. 2

3.3 Tasks

  • The 100 tasks are shuffled from Omni-MATH difficulty band 5.0 to 6.0 after numeric-answer and malformed-problem filters.Numeric answers permit exact automated grading, but the restriction favors number theory and combinatorics and excludes proof- and expression-valued problems.Source: §3.3 · pp. 2–3

3.4 Grading

  • Answers are scored by exact match to the reference value after canonicalization, with no human or model judgment in the scoring path.The same grader scores initial and final answers.Source: §3.4 · p. 3

3.5 Reviewer input

  • Reviewer input was shortened after rate-limit truncation affected 29 percent of mid-tier reviews; the weak reviewer’s results therefore describe it under the shortened prompt.The shortened input removes executor answer-format instructions and truncates solutions to their final 2,000 characters; under it, the weak reviewer produces fluent, hedged prose while still accepting almost everything.Source: §3.5 · p. 3

3.6 Metrics

  • The method reports final accuracy, recall, and false-rejection rate, and classifies each rejection as repair, damage, no change, or miscorrection.It also reports exact token cost and additional tokens per additional correct answer relative to C0; Table 1 presents final accuracy by condition.Source: §3.6 · p. 3, Table 1 · p. 3

3.7 Statistics

  • Accuracy comparisons use exact McNemar tests on paired outcomes, while rejection proportions use Fisher’s exact test; tests with fewer than five rejections are omitted.Pairwise tests are uncorrected for multiplicity, so borderline comparisons warrant caution.Source: §3.7 · p. 3

4 Results

Across 100 olympiad mathematics problems, a cross-family reviewer improved final accuracy, while the weakest reviewer made no changes under the tested prompt. Detection alone did not predict benefit: rejection outcomes, revision inertia, targeting, and cost distinguished the conditions.

4.1 Final accuracy

  • 12 percentage points: the cross-family reviewer improved final accuracy, with the gain entirely one-directional across the 100 problems.C3 changed no final answers under the shortened prompt; its measured inertness was prompt-conditional because the same model behaved differently under the original prompt.Source: §4.1 · p. 3

4.2 Detection does not predict benefit

  • Self-review had the highest error-detection rate but no significant accuracy gain, and it falsely rejected 35 percent of answers that were already correct.The cross-family reviewer detected fewer errors but produced the only significant gain.Source: §4.2 · p. 4

4.3 Rejection outcomes

  • 15 percent versus 43 percent: self-review and cross-family review differed in repair rate; their no-change rejection rates were 41 percent versus 14 percent.Self-review required 6.6 rejections per useful repair, compared with 2.3 for the cross-family reviewer.Source: §4.3 · p. 4

4.4 Revision inertia conceals reviewer error

  • Of self-review’s 18 false rejections, all 3 revisions the executor made became wrong, while the other 15 correct answers remained unchanged.Had the executor complied with every false rejection, its accuracy would have fallen below the no-review baseline; this is a bounded counterfactual, not a prediction of compliance.Source: §4.4 · p. 4

4.5 Targeting and repair skill are separable

  • Among 52 initially correct answers, self-review uniquely falsely rejected 17 problems; repair skill did not significantly differ, at 26 percent versus 50 percent (p = 0.096).The repair comparison was underpowered, with 35 and 24 attempted changes; the authors report absence of evidence, not evidence of absence.Source: §4.5 · p. 4

4.6 Cost

  • Roughly 24,000 versus 26,000 tokens per additional correct answer: cross-family review and self-review were similar in cost, but the former delivered twice the accuracy improvement with no damaged answers.The weak reviewer roughly doubled baseline token cost while producing no additional correct answer, making its cost per additional correct answer undefined.Source: §4.6 · p. 4

5 Discussion

The discussion argues that reviewer utility depends on more than error detection: which answers are rejected and whether critiques are acted upon matter. It connects this pattern to revision inertia and recommends evaluating reviewers by their rejection outcomes, while treating qualitative examples as illustrations rather than frequency evidence.

5 Discussion

  • The reviewers that rejected most were not those producing the largest gains: self-review had the highest detection rate without significant accuracy gain, while cross-family review produced the largest gain with no observed damaging changes.The final-accuracy comparison between these conditions was directionally consistent but did not reach significance (p = 0.238).Source: §5 · p. 4
  • Self-review left 41 percent of rejections unchanged versus 14 percent for cross-family review, while miscorrection rates after attempted changes were similar, at 39 percent versus 43 percent.The discussion locates the difference in rejection frequency, targeting, and critique uptake rather than correction quality once a change is attempted.Source: §5 · p. 4
  • Of 18 false self-review rejections, the executor acted on only 3 and all three revisions became wrong; the other 15 returned the original correct answer.Low observed damage can therefore reflect revision inertia, not reliable critiques; full compliance would have pushed accuracy below the no-review baseline.Source: §5 · pp. 4–5
  • Reviewer selection should consider precision and rejection outcomes, not simply which model detects the most errors.In these experiments, cross-family review matched or exceeded self-review on recorded rejection-outcome measures, but their final-accuracy comparison was underpowered.Source: §5 · p. 5

5.1 Qualitative failure modes

  • Three illustrative failure modes were fabricated justifications, confidently wrong critiques, and fluent but vacuous endorsements; the examples were selected for clarity, not systematic sampling.The transcripts show how articulate, specific critiques can lack genuine verification, complementing rather than measuring the frequency of the quantitative failure modes.Source: §5.1 · p. 5

6 Limitations

The reported findings are bounded by a single experimental configuration and several specific measurement and sampling constraints. In particular, the accuracy comparison between the cross-family reviewer and self-review is underpowered, and the cross-family recall estimate is a lower bound.

6 Limitations

  • The study covers 100 problems from one benchmark and difficulty band, with one executor, one revision round, and one reviewer prompt template.The authors caution that the findings describe this configuration rather than review stages generally.Source: §6 · p. 5
  • The cross-family reviewer versus self-review final-accuracy comparison is underpowered at this sample size (p = 0.238), so the authors do not claim it.Their claims about differences between these conditions concern rejection behavior instead.Source: §6 · pp. 5–6
  • The cross-family reviewer’s recall is a lower bound because 16 percent of reviews hit a provider rate limit and were counted as accepts.This conservative treatment can only reduce measured recall.Source: §6 · p. 6
  • Several proportions rely on small denominators, particularly the 3 self-review damage cases and repair counts of 9 and 12.Denominators are shown so readers can judge the weight of each figure directly.Source: §6 · p. 6
  • Restricting the sample to numeric-answer problems favors number theory and combinatorics and excludes proof- and expression-valued problems.This selects a non-random subset of the benchmark.Source: §6 · p. 6
  • The weak reviewer’s inertness result is limited by prompt sensitivity: under the original longer prompt, the model returned mostly empty text.The reported C3 results use a shortened reviewer prompt.Source: §6 · p. 6
  • Free-tier provider limits constrained both the choice of models and the sample size.Source: §6 · p. 6

7 Reproducibility

The authors document how calibration led them to rescope the study before running the reported experiment. They retain the project records and describe the materials and logging that support reproducibility.

7 Reproducibility

  • Calibration probes undermined the original error-diversity premise, prompting a documented rescope before the reported experiment was run.The original plan asked whether cross-model error diversity predicts multi-agent benefit, but available models did not err comparably across the tested task families and difficulty bands.Source: §7 · p. 6
  • The revised study carried over the pipeline mechanics, paired design, exact-match numeric grading, models, and temperature unchanged.Both the original plan and amendment are retained, and all prompts are frozen in the repository.Source: §7 · p. 6
  • The harness caches model calls and resumes after interruption; model identifiers, temperature, seeds, and per-call token usage are recorded.Code and logs are available in the repository.Source: §7 · p. 6
Loading 2609.04270v1…