Source-linked AI summary
Unanimity Without Persuasion: A Single Round of Debate Erases the Disagreement That Verification Needs
Yang Shu
TL;DR
The paper examines whether debate preserves disagreement needed for downstream verification and intervention. It tracks a heterogeneous judge panel across debate rounds and finds that one round sharply increases unanimity while eliminating the tested ballot’s later corrective effect.
Problem
Debate is commonly evaluated by final accuracy, while whether it preserves disagreement for exposing and correcting suspect decisions is less directly measured.
Source: §1 · pp. 1–2Method
Results
On the fixed n=291 cohort, unanimity rose from 39.5% to 95.2% in round 1, while execution-ballot substitution changed 8 of 2,037 pre-debate instances and zero in each later round.
Source: §1 · p. 2Takeaways & Limitations
The authors recommend verifying before a second-pass evaluation or peer exposure and not treating post-debate unanimity as independent evidence of reliability.
Source: §6 · p. 7The fixed cohort over-represents easy candidates, while heavily contested candidates remain unobservable in debate because judges’ outputs on them cannot be parsed.
Source: §4.1 · p. 4, §6 · p. 7
Abstract
from arXiv · showhide
A debate panel can become unanimous without becoming more correct. This is dangerous for downstream safeguards: a substituted verification ballot can change only narrow-margin votes, while richer arbiters lose disagreement as a natural targeting signal. We show that one debate round can erase that resource without requiring persuasion. Tracking a heterogeneous 7-judge panel through a blind round and three debate rounds on 600 code-correctness candidates, unanimity on a fixed cohort jumps from 39.5\% to 95.2\% in round 1 (93.1\% of the total collapse), while accuracy moves by less than one point and 96.3\% of verdict flips follow the displayed peer majority. An execution-based verification ballot corrects 8 of 2,037 pre-debate candidate-substitution instances but changes zero in every later round; by round 3 every wrong decision is unanimous, erasing dissent that had flagged two-thirds of the panel's errors. Identical-cohort controls explain why: no-peer reconsideration reproduces 79.6\% of the collapse, real labels without reasoning reproduce 91.5\%, and random labels steer flips toward whatever they display; the full-debate condition adds 4.3 percentage points over labels only (clustered 95\% CI 0.7--8.1). The one-round collapse reproduces in two additional real runs and two fake-label seeds, remains under panel sizes 3--7, and appears in MATH-500. Parse failures concentrate on contested candidates ($p<0.001$), making attrition non-ignorable. The design implication is operational: verify before any second-pass evaluation or peer exposure, and never treat post-debate unanimity as independent evidence of reliability.
1 Introduction
The paper asks whether debate preserves the disagreement that downstream verification and intervention need. It finds that one round sharply reduces disagreement and leaves a tested substituted verification ballot unable to change later decisions.
1 Introduction
- 39.5% to 95.2%: unanimity rose after one debate round on the fixed n=291 cohort, accounting for 93.1% of the total collapse.Unanimity then reached 98.6% and 99.3% in rounds 2–3.Source: Figure 1 · p. 1, §1 · p. 2
- Debate’s disagreement loss matters because richer overrides can target likely errors, and panel disagreement is one natural targeting signal.The paper distinguishes such overrides from single-ballot substitution, which can change only pivotal decisions.Source: §1 · pp. 1–2
- 8 of 2,037 pre-debate substitutions corrected a decision, while the same execution-based ballot changed zero decisions in each later round.Substitution effects are confined to pivotal candidates, and only two remained after round 1.Source: §1 · pp. 1–2
- 79.6% of the unanimity jump came from no-peer reconsideration, real labels without reasoning reproduced 91.5%, and full debate added 4.3 percentage points over labels only.The full-debate increment had a clustered 95% CI of 0.7–8.1 percentage points.Source: §1 · p. 2
- Parse failures concentrate on contested candidates, so attrition is non-random and naive cross-sample comparisons overstate the collapse by about 13 points.Dropped candidates were pivotal at round 0 eight times as often as retained candidates: 31.1% versus 3.7%.Source: §1 · p. 2
2 Related Work
Prior work studies debate’s accuracy, majority pressure, and post-debate intervention, but the paper focuses on how debate reshapes disagreement and the resulting opportunity for verification.
Multi-agent debate
- The paper complements debate audits by measuring vote-margin trajectories, their control-based decomposition, and their consequences for single-ballot verification.The authors report that they are not aware of prior work making these measurements.Source: §2 · p. 2
Intervening on debate outcomes
- He et al. [2026] study post-debate overrides, whereas this paper measures how debate reshapes disagreement before intervention.Their baseline arbiter flips more decisions wrongly than correctly, while a conservative trained gate flips safely.Source: §2 · pp. 2–3
Correlated judges and where single ballots matter
- Because LLM judge errors are correlated, a panel has fewer effective votes than its nominal size, motivating non-LLM evidence and attention to pivotal cases.The pivotal-voter arithmetic is classical; the paper’s empirical contribution is measuring the margin distribution under debate.Source: §2 · p. 3
Disagreement as an uncertainty signal
- Panel vote margins can serve as an uncertainty signal, but in this study dissent disappears even from wrong decisions after debate.At round 0, accuracy was 65.3% on pivotal cases versus 85.5% elsewhere.Source: §2 · p. 3
Conformity, anchoring, sycophancy
- Matched controls attribute 79.6% of the unanimity jump to a no-peer second look, 91.5% to second look plus real labels, and a 4.3-point residual to full debate.The paper connects label-following to conformity and anchoring [Asch, 1956, Tversky and Kahneman, 1974] and LLM sycophancy [Sharma et al., 2024].Source: §2 · p. 3
3 Setup: Panels, Margins, and Where a Ballot Can Matter
The study uses code correctness to measure panel decisions against executable ground truth, then tracks how panel margins affect the potential for one-ballot substitution.
Candidates and ground truth
- The study samples 600 candidate solutions from HumanEval+ and MBPP+ and uses full-suite execution as ground truth, while the verification signal tests a fixed 20% subset.The partial signal can miss bugs on untested inputs, unlike the full-suite reference.Source: §3 · p. 3
Judges and rounds
- Seven judges make blind independent round-0 decisions, then reconsider after seeing anonymized peer verdicts and reasoning excerpts of up to 400 characters.The excerpts distinguish full debate from labels-only controls but do not isolate semantic persuasion from context length or presentation.Source: §3 · p. 3
Three intervention rules, kept distinct
- The paper distinguishes substitution, addition, and override; its measurements use substitution, where one external signal replaces a judge’s ballot.Addition creates an eighth ballot requiring a tie-breaking convention, while override can replace the panel decision outright.Source: §3 · pp. 3–4
Margins and the pivotal region
- Under one-ballot substitution, a majority decision can change only when the panel is pivotal, at margin m_i=1.The pivotal share bounds the possible effect; the signal’s quality on pivotal cases determines whether it helps or harms.Source: §3 · p. 4
A motivating stratification
- At round 0, majority accuracy was 65.3% on 98 pivotal candidates versus 85.5% elsewhere; substitution’s pivotal-region gain was directional, not established.The +5.83-point gain had a bootstrap 95% CI of −1.1 to +12.6 points.Source: §3 · p. 4
4 Debate Destroys Disagreement Within One Round
On a fixed cohort, debate erased most panel disagreement in the first round while accuracy barely changed. Verdict flips overwhelmingly followed the displayed peer majority, and execution-ballot substitution had no effect after round 0.
4.1 Attrition Is Not Random – So All Comparisons Are Paired
- 31.1% of candidates dropped at round 1 were pivotal at round 0, versus 3.7% of retained candidates, so attrition was non-random.The fixed cohort consequently over-represents easier candidates: its round-0 unanimity was 39.5%, versus 25.0% in the full batch.Source: §4.1 · p. 4, Table 1 · p. 4
- The front-loaded collapse reproduced in a six-judge panel, with unanimity rising from 55.4% to 95.3% after one round.The fixed cohort contained 298 candidates, and the panel excluded the judge most associated with parse failures.Source: §4.1 · p. 4
4.2 The Collapse Is Front-Loaded
- 55.7 percentage points—93.1% of the total unanimity increase—occurred in round 1, as unanimity rose from 39.5% to 95.2%.Pivotal candidates fell from 10 to 2 to zero; the fixed cohort included 291 candidates usable in every round.Source: §4.2 · p. 4, Figure 2 · p. 5
4.3 The Flips Are a One-Way Ratchet
- 96.3% of verdict flips (260/270) moved toward the displayed peer majority, with six tied cases excluded.Per-judge round-0-to-1 flip rates ranged from 0.7% to 26.1%; their ranking under real and fabricated peer opinions had Spearman ρ=0.99, an exploratory result from seven judges.Source: §4.3 · p. 5, Table 3 · p. 5
4.4 Confidence Explodes; Correctness Does Not
- Majority-vote accuracy changed from 91.8% to 92.4%, while by round 3 all 22 majority errors were unanimous.At round 0, 16 of 24 majority errors occurred on split votes; the count of unanimously wrong verdicts rose from 8 to 22.Source: §4.4 · pp. 5–6
- Ten of twelve initially pivotal candidates reached margin ≥5 within one round, eliminating their status as a distinct contested group.The majority outcome improved on two candidates and degraded on none.Source: §4.4 · p. 6
4.5 The Verifier’s Realizable Effect, Round by Round
- Eight of 2,037 round-0 substitutions corrected decisions, but substitution changed zero decisions in each later round.The eight changes involved two pivotal candidates; each was corrected by four majority-side replacements.Source: Table 2 · p. 5, §4.5 · p. 6
5 What Drives the Collapse? Label, Reasoning, and Second-Look Controls
Matched controls show that most unanimity growth comes from second-pass reconsideration and displayed labels, not peer reasoning alone. Random labels also steer verdict changes toward the majority they display.
5 What Drives the Collapse? Label, Reasoning, and Second-Look Controls
- The controls compare real peer labels with reasoning withheld, a no-peer second look, and candidate-independent fake labels with template justifications.The no-peer prompt asks judges to take a second careful look rather than repeating the identical prompt.Source: §5 · p. 6, §5.1 · p. 6
5.2 Matched Controls: Real Labels Only, and No Peers
- On the identical n=280 cohort, no-peer reconsideration reproduces 79.6% and real labels without reasoning 91.5% of full debate’s unanimity jump; reasoning adds 4.3pp.Flip volume is nearly identical across the four intervention conditions (10.4–11.8%).Source: Figure 3 · p. 6, §5.2 · p. 7
5.1 Random Labels Alone Reproduce Most of It
- Fake labels reproduce 67.9% of the real unanimity jump, and 89.5% of induced flips move toward the displayed random majority.The fake labels are candidate-independent, so their convergence effect does not require case-specific information.Source: §5.1 · p. 6
- The fake-label reproduction fraction is not a causal decomposition because that condition changes the label configuration as well as the reasoning.The matched real-label and no-peer controls separate those components more directly.Source: §5.1 · p. 6, §5.2 · p. 7
5.2 Matched Controls: Real Labels Only, and No Peers
- Seven contested candidates become unanimously wrong under the matched controls, including a dissenting judge who capitulates by appealing to the majority count.The worked example’s judge had correctly identified a bug before changing its verdict.Source: §5.2 · p. 7
6 Discussion
Debate removes disagreement that substitution and disagreement-based overrides can use to identify suspect decisions. The authors recommend verification before any second-pass evaluation or peer exposure, while noting scope and control limitations.
6 Discussion
- By round 3 every wrong decision is unanimous; tested single-ballot substitution has no effect after round 0, and override policies lose disagreement as a targeting signal.A margin-gated override is corrective before debate (+0.69pp), while the ungated policy turns net negative on post-debate votes (−0.34pp).Source: §6 · p. 7
- Rounds 2–3 contribute only 7% of the collapse, while even a peer-free second pass removes most disagreement, so verification belongs before reconsideration or peer exposure.Post-debate unanimity should not be treated as independent evidence of reliability.Source: §6 · p. 7
- The fixed cohort still over-represents easy candidates, and the primary results concern code correctness under one protocol at temperature 0.The smaller MATH-500 study does not repeat every control, and domain breadth remains limited.Source: §6 · p. 7
- The controls isolate interface-level increments rather than semantic persuasion, because full debate also changes context length, formatting, and attention allocation.Rationales are limited to 400 characters, and length-matched and shuffled-rationale controls remain future work.Source: §6 · p. 7
- The authors release stored records, prompts, and deterministic analysis scripts, with reported statistics recomputed from those records.Stored debate records contain verdicts and displayed response excerpts rather than unrestricted full API responses.Source: §6 · pp. 7–8
7 Conclusion
Debate erases disagreement that downstream interventions rely on, while the tested verification ballot loses its corrective effect after debate.
7 Conclusion
- Debate destroys disagreement within one round: by round 3 every wrong decision is unanimous, and the substituted verification ballot has no effect after debate.The conclusion attributes the loss to debate reaching the disagreement before the verifier does.Source: §7 · p. 8