PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Collapse and Correction in Homogeneous-Panel LLM Debate

Xin Li, Mengbing Liu, Chau Yuen

arXiv:2609.35279v1cs.CL

TL;DR

Final-accuracy evaluation misses that debate can either correct initially wrong majorities or collapse initially correct ones. The paper audits these transitions and replays interventions, finding that a probe-gated freeze can prevent collapses while losing more corrections under equal weights.

  • Problem

    Final-accuracy evaluation conflates debate correcting an initially wrong majority with debate collapsing an initially correct one.

    Source: §1 · p. 1

  • Method

    The protocol records preserved, collapse, correction, and unrepaired transitions, collapse onset, and signed intervention utility for homogeneous multiple-choice debate.

    Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5

  • Results

    29 collapses prevented but 108 corrections lost under equal weights, yielding net utility −79 for a leave-one-model-out probe-gated freeze.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The transition ledger is an auditable evaluation target for comparing interventions under declared utility weights, not a deployable debate controller.

    Source: §8 · p. 10

    The evidence is narrow: homogeneous same-model, closed-book MCQ debate, with a realized matched cohort of 14 rows across 7 families and no established benchmark transfer.

    Source: §7 · p. 10

Abstract

from arXiv · show

Multi-agent large language model (LLM) debate is often evaluated by whether final answers improve, but movement is not necessarily improvement: the same discussion can rescue an initially wrong majority or destroy an initially correct one. Standard final-accuracy evaluations conflate these opposing mechanisms. We introduce an auditable protocol for homogeneous debate on multiple-choice questions (MCQs) that records each run as a transition ledger over collapse, correction, onset, and signed intervention utility. On 6,925 MMLU-Pro debates, the protocol identifies 253 collapses and a parallel correction ledger that changes how interventions should be judged. Replay experiments reveal the central tradeoff: a leave-one-model-out probe-gated freeze prevents 29 collapses but loses 108 corrections under equal weights, so collapse prevention alone can recommend the wrong policy. A compact pre-debate 8-probe screen is a triage signal: its unadjusted family-level association with conditional-collapse risk is high (G=7, Spearman rho=0.893, exact two-sided p=0.0123), but initial-majority accuracy is a close comparator (rho=0.821; family partial rho=0.767, p=0.0877), so we do not treat it as calibrated or capability-adjusted prediction. Round-level traces localize many collapses to the first debate round, where early disagreement can precede both harmful cascades and useful recovery. We release replayable schemas, coders, audits, cost cards, and zero-API rebuild scripts so future model-scaffold rows can be compared under the same denominators and signed utility ledger.

1 Introduction

Debate can either correct an initially wrong majority or collapse an initially correct one, so final accuracy alone misses the direction of movement. The paper introduces a transition audit and signed replay to measure both outcomes and assess interventions.

1 Introduction

  • 253 collapses across 6,925 MMLU-Pro debates show that homogeneous, closed-book debate can destroy initially correct majorities without an external adversary, tool output, or new evidence.For the most susceptible models, more than one in ten initially correct majorities are lost.Source: §1 · p. 1
  • The 8-probe compliance screen is a protocol-bound triage signal, not a calibrated per-question oracle or capability-free property; initial-majority accuracy is a close comparator.Its capability-adjusted family-level association is no longer confirmatory.Source: §1 · p. 2
  • 149/253 tracked collapses originate in Round 1, where trajectory features raise pooled out-of-fold AUC from 0.669 to 0.768, but effects vary by model and do not establish causation.Probe-ablation effects are uneven, with Qwen-3 rows near zero.Source: §1 · p. 2
  • 29 collapses prevented but 108 corrections lost under equal weights, leaving the probe-gated freeze at net utility −79.The result shows why judging an intervention by prevented collapses alone can mislead.Source: §1 · p. 2
  • Reusable schemas, coders, audits, cost cards, release flags, and zero-API rebuilds let new model–scaffold rows recover denominators and signed-replay cells.Open-tier artifacts rebuild headline transition and signed-replay summaries; gated fields affect transcript-derived diagnostics and safety-sensitive prompt phrasings.Source: §1 · p. 2
  • The audit records preserved, collapse, correction, and unrepaired transitions, plus collapse onset and weighted signed utility for what interventions save or discard.It decomposes each debate by initial and final majority correctness and scores interventions by prevented and lost outcomes.Source: §1 · p. 2

2 Related Work

Related work spans debate for reasoning, evaluation, and oversight; sycophancy and compliance measurement; and scalable oversight and aggregation. This paper positions its contribution as an evaluation target for debate interventions, not a new scaffold or deployable controller.

Multi-agent debate and collapse

  • The paper contributes a transition-table evaluation of when debate destroys initially correct majorities, rescues initially wrong ones, and how interventions trade these outcomes.It builds on work on debate for reasoning, evaluation, and oversight [3] [5] [7] [29], as well as debate failure and mitigation [9] [11] [21] [22] [27] [28] [32].Source: §2 · p. 3

Sycophancy, conformity, and compliance

  • The 4×2 argument/social factorial is a behavioral screen, not a novelty claim about revisability or a measure of activation-cache circuit scores.The paper draws on research about sycophancy, conformity, misleading cues, and informational versus normative pressure [15] [16] [20] [24] [26] [31].Source: §2 · p. 3

Scalable oversight and aggregation

  • Homogeneous traces cannot forecast heterogeneous overseer/worker gaps; αtot is a pre-debate risk measure, Round 1 features are runtime diagnostics, and replay scores signed utility.Unlike work proposing scaffolds, mitigation methods, or aggregation rules, this paper provides an evaluation target; its negative intervention rows are calibration baselines, not competing controllers.Source: §2 · p. 3

3 Evaluation Protocol: Collapse, Correction, and Revisability

The protocol audits homogeneous debate through transition accounting, pre-debate probes, trace diagnostics, and held-out replay, keeping measurement distinct from controller certification.

3 Evaluation Protocol: Collapse, Correction, and Revisability

  • Four reporting layers separate row selection, transition accounting, collapse localization, and signed replay; the reusable audit surface supports new rows without changing measurement definitions.Third-party controllers can emit row-level actions for replay under the same saved denominators.Source: §3 · p. 3, Table 1 · p. 3

3.1 Debate and Probe Setup

  • Three agents answer independently, then discuss and update for Rounds 1–3; collapse, correction, and onset are coded from majority transitions.The protocol also uses an eight-probe 4 × 2 design and rule-based parsing and voting, without an LLM judge or human coder.Source: §3.1 · p. 4

3.2 Revisability Metrics

  • The protocol distinguishes total answer-change rate αtot from adversarial αadv and corrective αcor flips, and summarizes social versus argument sensitivity with S/A.The S/A statistic uses outcome-blind, per-model high-FR questions; sensitivity checks include shared-pool and full-pool variants.Source: §3.1 · p. 4, §3.2 · p. 5

3.3 Signed Replay Evaluation

  • Signed replay scores a gate by collapses prevented minus corrections lost, with weights declared so intervention utility reflects both outcomes.A fired freeze returns the initial rather than saved final majority; leave-one-model-out probe-gated freezes use a threshold tuned on held-in models and evaluated on a held-out model.Source: §3.3 · p. 5

4 MMLU-Pro Results: Transition Accounting, Triage, and Transfer Checks

The MMLU-Pro results use family-level pre-debate screening as the primary association, then bound that signal with reliability, capability and per-question diagnostics, and transfer checks. Across these checks, the screen supports triage rather than capability-adjusted or per-question prediction, while transfer evidence concerns transition accounting, not screen transfer.

4.2 Primary Result: A Pre-Debate Screen Gives a Family-Level Triage Signal

  • ρ=+0.8929 over G=7 families (exact two-sided p=0.0123) for the unadjusted αtot–conditional-collapse association, the primary pre-debate screening result.Family means average related model variants; Qwen contributes 6/14 rows, and Meta is the visible upper-tail inversion.Source: Table 3 · p. 6, §4.2 · p. 6

4.1 Probe Reliability

  • Split-half probe reliability is 0.873, test–retest reliability is 0.760, and inter-agent ICC is 0.630–0.633.The outcome-blind high-FR item selection’s shared-pool sensitivity keeps S/A positive in 13/15 models, while full-pool estimates remain within per-model bootstrap intervals.Source: §4.1 · p. 6

4.2 Primary Result: A Pre-Debate Screen Gives a Family-Level Triage Signal

  • Qwen’s individual conditional-collapse rates span 5.11% to 43.90%, so its realized-cohort variants are a dependence cluster, not an exchangeability claim.The appendix reports model-row results, leave-one-Qwen-row checks, and taxonomy splits.Source: §4.2 · p. 6

4.3 Sensitivity: Capability and Per-Question Boundaries

  • Initial-majority accuracy is a close comparator (ρ=+0.821); the family partial is ρ=+0.767 (p=0.0877), supporting triage rather than capability-adjusted prediction.Sensitivity checks retain positive rank correlations: N=14 model-row ρ=+0.8295, worst leave-one-family-out ρ=+0.741, and sealed-vintage ρ=+0.78.Source: §4.3 · p. 7, Table 4 · p. 7
  • After initial answers are available, disagreement is the stronger per-question diagnostic; probe FR and S/A are not row-level collapse detectors.Adding S/A changes AUC by at most 0.004, bounding the screen as model-level triage rather than a row-level oracle.Source: §4.3 · p. 7, Table 5 · p. 7

4.4 Transfer Check: Accounting, Not Screen Transfer

  • GPQA rows support transition accounting, not screen transfer: Mistral Small 4’s collapses and corrections nearly cancel, while DeepSeek V4 Flash gains as corrections dominate.Four Mistral questions lack a valid initial majority; excluding them changes signed net from +1 to −1, and Llama and DeepSeek per-question α–collapse associations are near zero.Source: §4.4 · p. 8

5 Trace Localization: Where Collapse Appears in Debate

Trace analyses localize many collapses to early debate rounds and find behavioral separation between argument and social prompts. These are diagnostic findings, not evidence of causal mediation or uniform channel separation.

5.1 Round 1 Onset: Collapse Is Often an Early Event

  • 149/253 collapses (58.9%) began in Round 1; adding Round 1 features raised pooled OOF AUC from 0.669 to 0.768, and Round 2/3 features raised it to 0.846.Round 1 was the largest onset bucket, but a Round 1-only gate leaves a substantial late-collapse tail.Source: Table 6 · p. 8, §5.1 · p. 8

5.2 Channel Manipulation: Compliance Is Behaviorally Separable

  • Anti-argument and anti-social prompts produced different flip distributions in 16/16 tested models, with grand mean Cohen’s d=0.53, but two Qwen models showed near-zero separability.Removing the strongest corrective variant preserved the model-level S/A ordering and changed matched collapse-prediction AUC by at most 0.004.Source: §5.2 · pp. 8–9
  • The checks rule out the simplest authority-cue-only explanation but do not prove perfect factorization; αtot remains an aggregate revisability readout, not evidence of clean channel separation within Qwen.The authors treat αadv as a behavioral diagnostic rather than a standalone sycophancy construct.Source: §5.2 · p. 9

6 Intervention Evaluation: Collapses Prevented vs. Corrections Lost

Freeze-style interventions can prevent collapses while discarding more useful corrections, so the paper scores policies with signed replay rather than collapse prevention alone. Follow-up comparisons also show that private revision does not settle the comparison with compute-matched solo reasoning.

6 Intervention Evaluation: Collapses Prevented vs. Corrections Lost

  • All three freeze-style gates lost more corrections than they prevented collapses; the closest Round 1 gate prevented 56 and lost 69.A prompt-only independence ablation also failed to improve held-out accuracy.Source: §6 · p. 9
  • On 6,525 debates, the probe-gated freeze prevented 29/251 collapses but lost 108/804 corrections; freeze-style rows were net-negative at equal weights.The replay cohorts differ, so these rows are not a matched comparison.Source: §6 · p. 9, Table 7 · p. 9
  • The probe-gated and Round 1 replay rows use different cohorts, and equal weights are the accuracy-equivalent default; alternative utility ratios must be declared before policy selection.At collapse weight r=2, the probe-gated freeze remained net-negative (−0.77pp), while the Round 1 majority-change rule was positive (+3.43pp).Source: §6 · p. 9
  • Across 12 repeated MCQ settings, private revision was 1.44–4.81 points less accurate than peer debate when ties counted as wrong; expected-value tie credit narrowed the difference to −1.62 to +0.38 points.Private revision reduced wrong-answer collapses but left more ties.Source: §6 · p. 9
  • In eleven reasoning-mode rows, collapses and corrections both persisted, and private revision had a negative point difference in all eleven; the comparison with compute-matched solo reasoning remains unsettled.Intervals were below zero in seven rows.Source: §6 · p. 9, Table 8 · p. 9

7 Limitations

The evidence is bounded by a small, uneven realized cohort and a narrow debate setting, and the authors distinguish diagnostic associations from calibrated prediction or causal explanation.

7 Limitations

  • The primary association uses a realized matched cohort of N=14 rows across G=7 families, not the preregistered N=18; the exact family tests and sensitivity checks bound rather than remove this limitation.The study focuses on homogeneous, same-model, closed-book MCQ debate and does not establish transfer to heterogeneous systems, tools, retrieval, or free-form generation.Source: §7 · p. 10
  • The pre-debate screen is protocol-bound triage, not a capability-free property, per-question oracle, or controller; initial-majority accuracy is a close comparator.Round 1 trajectories localize risk without causal mediation, and policies require correction-preserving uncertainty, prespecified weights, and held-out signed replay.Source: §7 · p. 10
  • The primary cohort does not isolate reasoning mode or effort, and the follow-up does not establish that debate outperforms a compute-matched solo-reasoning baseline.One primary-cohort row ran with provider reasoning enabled.Source: §7 · p. 10
  • The released artifact rebuilds aggregate transition and signed-replay summaries without API calls, but transcript-derived diagnostics and safety-sensitive prompts are release-tiered.Exact per-debate Round 1 and strict matched-τ DisagreementGate matrices remain gated, and provider reruns may drift.Source: §7 · p. 10

8 Conclusion

The conclusion calls for auditable signed transition utility alongside final accuracy, so interventions can be compared by both collapses prevented and corrections lost. The accompanying materials support reuse while preserving boundaries on cohort inference and transcript-derived data.

8 Conclusion

  • Debate evaluation should report signed transition utility rather than final accuracy alone, showing denominators, action rules, prevented collapses, and useful corrections discarded.A scaffold is safer only when harmful cascades fall without erasing recoveries; the ledger allows user-stated utility weights to affect controller selection.Source: §8 · p. 10

A.1 Symbol and Cohort Glossary

  • Table 9 provides a compact glossary of notation, cohorts, and short labels used across the paper.The passage identifies the table’s purpose but does not enumerate its entries.Source: Table 9 · p. 13

A.2 Probe Templates

  • The probe templates combine argument-strength counterarguments with an optional social-majority suffix; very-strong probes use the correct answer when the initial response is wrong.The social suffix appears in 4/8 probes.Source: §A.2 · pp. 13–14

A.3.1 Realized-Cohort Statistical Audit

  • The realized matched cohort fell from preregistered N=18 to N=14, so the family-level exact test is a design-aware correction for the realized cohort, not completion of the original target unchanged.The supplementary repository includes zero-API derived audit summaries and scripts for sensitivity analyses and parser checks.Source: §A.3.1 · pp. 14–15
  • The Round 1 multilevel-model input remains gated because it derives from full debate transcripts; open-tier aggregate traces support the primary family-level test and related audit checks.The multilevel model is an anatomy check rather than the primary family-level inference.Source: §A.3.1 · p. 15

A.3.2 Operational Reuse Cards

The supplementary cards make debate evaluations reusable through documented schemas, release rules, and zero-API rebuilds. The accompanying audits bound parser and measurement sensitivities while describing the probe and debate protocols and their scope.

A.3.2 Operational Reuse Cards

  • The reuse card records model, benchmark, measurement, transition, utility-weight, and gate fields so new model–scaffold rows can be compared on named held-out cohorts.The artifacts include trace schemas, audits, cost accounting, release flags, and zero-API rebuild scripts.Source: §A.3.2 · pp. 15–17

A.3.3 Answer Parser and Missing-Parse Sensitivity

  • Rule-based extraction has no LLM-judge fallback, and audits expose fallback sensitivity: strict parsing changed transition membership in 51 of 966 labeled rows.The strict audit found 20 collapse-label and 29 correction-label changes; its high unlabeled rate was concentrated in long Qwen traces.Source: §A.3.3 · p. 17, Table 13 · p. 17

A.3.4 Protocol and Provenance Card

  • The debate protocol uses three agents and three rounds, with specified temperatures and alphabetic tie-breaking after unparsed answers are filtered.The supplement also records that provenance is partial for some older sealed lanes and more complete in newer expansion files.Source: §A.3.4 · pp. 17–18, Table 14 · p. 18

A.3.5 Sealed Pre-A6 Vintage Tables (Cross-Check)

  • The sealed N=9 association is retained as a cohort-revision cross-check; it agrees directionally with the realized N=14 result (ρ=+0.78 vs. ρ=+0.829).The sealed vintage is not the realized-cohort headline result.Source: §A.3.5 · p. 18, Table 15 · p. 18, Table 16 · p. 18

A.4 Ethics and Dual-Use Statement

  • The convince-wrong probes create dual-use risk because per-model αadv rankings could be misread as a misleadingness leaderboard, so the paper pairs αadv with αcor and discourages standalone safety comparisons.The probe templates and tuned phrasings are gated, while aggregate statistics and full code are openly released.Source: §A.4 · p. 19

B.1 High-FR Question Pool: Selection and Sensitivity

  • The high-FR pool is selected using each model’s probe behavior, not debate outcomes; its rankings remain stable on a shared pool and full-pool estimates fall within bootstrap intervals.On the shared pool, S/A rank correlation with the per-model-pool rankings is ρ=0.95 for four overlap models; the full-pool estimates fall within their per-model bootstrap 95% CIs.Source: §B.1 · pp. 19–20, §B.2 · p. 20, Table 17 · p. 20

B.3 Shared-Pool S/A: 195-Question Intersection

  • On a shared 195-question pool, S/A remains positive in 13 of 15 models, although Sonnet 4.6 and Opus 4.5 become mildly negative.The per-model-pool to shared-pool rank correlation across the 15 models is ρ=0.62.Source: §B.3 · p. 20, Table 18 · p. 21

B.4 Alpha Decomposition: Per-Model αadv and αcor

  • The probe flip-rate decomposition separates right-to-wrong revision after initially correct answers (αadv) from wrong-to-right revision after initial errors (αcor).The three conditional rates sum to the total flip rate within rounding under the non-social condition.Source: §B.4 · p. 21, Table 19 · p. 21

B.5 Probe-Control Robustness

Probe-control and channel-manipulation checks preserve several behavioral rankings, while family-level analyses qualify how strongly the pre-debate screen should be interpreted. The section also reports limits on prediction, comparator strength, and post-hoc extensions.

B.5 Probe-Control Robustness

  • Removing the very-strong probe preserves model-level S/A ordering exactly across four models, and probe simplifications change the combined predictor’s AUC by at most 0.005.Question-level maximum flip rates remain correlated with the full 8-probe estimate under both controls; these results do not establish perfectly orthogonal prompt factors.Source: §B.5 · pp. 21–22

B.6 Full 16-Model Channel Manipulation Results

  • Anti-argument prompts elicit more revision than anti-social prompts in all 16 tested models; 12 survive Holm–Bonferroni correction, with grand mean Cohen’s d=0.53.The table marks the manipulation conditions and reports valid agent-condition counts.Source: Figure 3 · p. 22, Table 20 · p. 22

B.7 Family-Level Channel Patterns

  • Channel-manipulation effects vary by family: OpenAI has mean d=0.82, Llama-3.1-8B has d=2.39, and Qwen models have d=0.06–0.09.These are functional descriptions under the probing protocol, not claims about internal architecture or training causes.Source: §B.7 · p. 23

B.8 Post-Hoc OpenRouter Alpha-Channel Extension

  • The post-hoc OpenRouter extension is an external channel-separability and parser/throughput check, not part of the headline association; its six near-complete lanes all preserve AA>AS.Rows with low coverage, forced hidden reasoning, or high parsed-None rates are treated as diagnostics rather than substantive counterevidence.Source: §B.8 · p. 23, Table 21 · p. 23

C.2 Family-Level Headline Aggregation

  • Seven-family aggregation gives the screen–conditional-collapse association ρ=+0.8929 (exact two-sided p=0.0123), while the family-level partial controlling initial accuracy is positive but non-confirmatory (ρ=+0.767, p=0.0877).The family correction addresses the realized imbalance from Qwen’s six rows but does not establish benchmark-general robustness.Source: §C.2 · pp. 23–24, Table 23 · p. 24, Table 24 · p. 24

C.3 Family-Level Partial Rank Sensitivity

  • The family-level partial association remains positive but is not confirmatory: ρα|init acc=+0.767, with exact two-sided p=0.0877 over seven families.The authors treat this as a capability-pressure sensitivity rather than a separate confirmatory test.Source: §C.3 · p. 25

C.4 Held-Out-Family Predictive Check

  • The held-out-family diagnostic supports a ranking interpretation, not calibrated forecasting: predicted-versus-observed Spearman is +0.821, and the 80% intervals cover 5/7 families.The uncertainty-propagated intervals cover 5/7 families at 80% and 7/7 at 95%, but are also described as diagnostic rather than calibrated.Source: §C.4 · p. 25, Table 25 · p. 25, Table 26 · p. 25

C.5 N14 Baseline and Confound Bakeoff

  • Initial accuracy is a strong capability comparator, while trace-only initial diversity and Round 1 majority change are runtime diagnostics limited to six rows across three families.The initial-disagreement measures each correlate with conditional collapse at ρ=+0.928; the partial association controlling initial disagreement is underpowered (ρ=+0.196, p=0.752).Source: §C.5 · p. 26, Table 27 · p. 26

C.6 Pre-Registered Robustness Checks

The registered checks qualify the screen’s association and its measurement, while held-out and post-hoc debate rows illustrate transition trade-offs. The evidence supports a ranking and audit interpretation rather than a calibrated forecast or a sharp threshold.

C.6 Pre-Registered Robustness Checks

  • The realized N=14 measurement-error rerun retains a positive latent rank association (median +0.8022; 95% interval [+0.7099, +0.8725]), while the channel-only αadv partial is weaker and non-confirmatory.The paper keeps αtot as a triage measurement and treats αadv/αcor as descriptive decomposition rather than a standalone leaderboard.Source: §C.6 · pp. 26–27

C.7 Specification Curve and Errors-in-Variables

  • All 27 specification-curve cells are positive on the sealed N=9 cohort, and the realized N=14 errors-in-variables rerun has median latent rank correlation +0.8022.Across the 27 N=9 cells, the median is +0.62; the N=14 interval is [+0.7099, +0.8725].Source: §C.7 · p. 27

C.8 Sealed N9 Hierarchical Decomposition Sensitivity

  • In the sealed N=9 sensitivity, total α has leave-one-model-out predictive R²=0.26 versus 0.12 for αadv, but the GEE coefficients are indistinguishable from zero.The authors characterize the rank result as a marginal directional signal, not a robust covariate-adjusted partial effect.Source: Table 28 · p. 26, §C.8 · p. 28

D.1 Preregistered Held-Out Sanity Check

  • In the preregistered GPT-5.4-mini check, 200 debates produced 1 collapse and 10 corrections, with 9 of 10 corrections in Round 1 and the sole collapse in Round 3.The authors treat the low-S/A model rows as sanity checks, not strong evidence for a sharp S/A threshold.Source: §D.1 · p. 28

D.2 Post-Hoc Matched Holdout Stress Tests

  • In post-hoc matched holdouts, Mistral Small 4 had 7/140 R4 collapses and 13/60 R4 corrections; the panel is not a powered independent validation of monotone ranking.The authors emphasize that interventions should be judged on both harmful collapses and useful corrections.Source: §D.2 · p. 28, Table 29 · p. 29

D.3 Cross-Benchmark Boundary and Stress Checks

The cross-benchmark checks show that transition patterns vary by setting, while trace analyses place many collapses early without establishing a causal mechanism. In the four-model trace analysis, question-level heterogeneity exceeds model-level heterogeneity, and the residual model-level probe effect remains uncertain.

D.3 Cross-Benchmark Boundary and Stress Checks

  • On ARC-Challenge, Haiku and GPT-4o-mini have near-zero collapse, while wrong questions have higher flip rates than correct ones.Conditional collapse is 0.5% for Haiku and 0.0% for GPT-4o-mini; the passage interprets the higher wrong-question flip rates as consistent with revisability tracking uncertainty rather than random compliance.Source: §D.3 · p. 29
  • The 100-debate Gemini GPQA/TruthfulQA check is underpowered for transfer; the larger GPQA Diamond lane still has sparse collapse and nonzero corrections.The expanded lane reports 2/147 collapses and 9/51 corrections; its null is lack of collapse variation, not lack of measured revisability.Source: §D.3 · p. 29
  • Post-hoc GPQA rows show that signed debate outcomes vary: Mistral Small 4 is nearly neutral, while Llama 3.3 70B and DeepSeek V4 Flash are net positive.Across GPQA Diamond rows, Mistral has 20/120 collapses and 21/78 corrections (+1 net); Llama and DeepSeek are net positive by +7 and +37 debates. These rows are stress checks, not broad benchmark-generalization evidence.Source: §D.3 · p. 29, Table 30 · p. 29

D.4 Superseded Mistral Small 4 Smoke Slice

  • The 20-question Mistral Small 4 smoke slice is provenance only; substantive evidence comes from the later 200-question matched holdout.The small slice showed the targeted failure pattern but was too small to carry substantive weight.Source: §D.4 · p. 30

D.5 Post-Hoc Local-vLLM Holdout: Qwen3-32B

  • In the post-hoc Qwen3-32B holdout, 8/73 at-risk majorities collapsed and 39/127 initially wrong majorities were corrected, making the run net accuracy-positive.The run is excluded from the sealed N=14 headline table; high revisability coincided with both collapses and productive corrections.Source: §D.5 · p. 30, Table 31 · p. 30

E.1 Round 1 AUC: Bootstrap Audit

  • Round 1 features raise pooled out-of-fold AUC from 0.669 to 0.768 over pre-debate features, a +0.099 lift.The paired debate-index bootstrap gives a 95% CI of [+0.080, +0.118]; the open tier cannot recover the pre-registered clustered confidence interval.Source: §E.1 · pp. 30–31

E.2 Cascade-Onset Round Distribution

  • 58.9% of collapses begin in Round 1, but later-round onsets mean an R1-only intervention would miss four in ten cascades.The full trajectory reaches pooled out-of-fold AUC 0.846, compared with 0.669 for the pre-debate baseline.Source: §E.2 · p. 31

E.2.1 Post-A6 R1 Replication

  • In the post-A6 four-round replication, R1 accounts for 40/75 collapses (53.3%), with the remaining onsets spread across R2–R4.This 1,200-debate replication uses a separate trace schema and parser, rather than supplying the three-round headline table.Source: §E.2.1 · p. 31

E.5 Independence Ablation Details

The intervention analyses find that preventing collapse can also suppress useful corrections, and the tested frozen policies do not transfer reliably to held-out questions. Development-set comparisons are more favorable to the probe-based policy than the held-out results.

E.5 Independence Ablation Details

  • The independence prompt reduces Haiku’s probe flip rate by 43%, but does not reduce collapse and slightly worsens debate accuracy on both tested models.The collapse tests are nonsignificant for Haiku and GPT-4o-mini.Source: §E.5 · pp. 32–33, Table 34 · p. 32

E.6 Frozen Deployable Policy Evaluation

  • The frozen ACC policy does not transfer reliably: its positive Phi-4-mini trend failed to replicate, and tests on Haiku and Qwen3-4B were also null or negative.In a fresh 1,500-question Phi-4-mini test, ACC reduced accuracy by 1.93pp and increased collapse by 0.40pp; the authors restrict evidence to proof-of-concept on development data.Source: §E.6 · p. 33, Table 36 · p. 33

E.7 Correction-Preserving Policy Replay

  • On the 1,255-trace replay cohort, the R1-majority-changed gate prevents 56 collapses but loses 69 corrections, for a −1.04pp accuracy change.Always freezing is also net-negative, and strict leave-one-model-out R1 stumps prevent only 2 collapses while losing 31 corrections.Source: §E.7 · p. 34

E.8 Pilot-Gated Intervention: LOMO Evaluation

  • The probe-gated LOMO freeze is pooled net-negative (−1.21pp), including a Qwen3-4B transfer failure that loses 87 corrections.The authors treat the probe as risk screening rather than a standalone deployable gate; converting runtime signals into net accuracy gains remains unresolved.Source: §E.8 · p. 34, Table 37 · p. 34

E.9 Cheap Baselines: DisagreementGate and DRS

  • On development sets, ACC beats simple disagreement and DRS baselines on collapse-prone models, but the available same-trace DisagreementGate replay is net-negative.That replay prevents 63 collapses but loses 104 corrections (net −41); the authors do not claim a stricter matched LOMO comparison.Source: §E.9 · pp. 34–35

F.1 B Monotonicity Pilot: Registered Negative Entry

The registered monotonicity ladder fails its prespecified criteria, but the fallback F3 floor is usable for two of the three pilot models. The appendix identifies these response-period analyses as outside the primary family analysis.

F.1 B Monotonicity Pilot: Registered Negative Entry

  • The nine-rung ladder fails both registered brightlines: median ρ is 0.253, and Qwen3.5-9B has ρm=−0.295.The decision rule made either condition independently sufficient for failure.Source: §F.1 · p. 35, Table 39 · p. 35
  • The F3 fallback is usable in 2/3 models, with floor magnitudes of +20 pp for Qwen3.5-4B and +10 pp for Qwen3.6-27B-FP8.Both usable models pass the prespecified floor and McNemar gates; the authors designate F3, conditional on the ladder failure, as the B-contribution headline.Source: §F.1 · p. 36, Table 40 · p. 36
  • Qwen3.5-9B is framed as harness-detected anti-monotonicity but statistically indistinguishable from null (permutation p=0.78).Its low, flat response across ladder rungs is retained as a limitation case rather than interpreted as a reliable wrong-direction effect.Source: §F.1 · p. 36

G Response-Period and Follow-Up Analyses

G.1 Screen Decision Value and Family-Count Planning

The response-period analyses test the screen against alternative orderings, correct a probe-temperature record, and examine further settings and reasoning modes. Their conclusions remain bounded by post-hoc designs, unresolved measurement issues, and release-tier differences.

G.1 Screen Decision Value and Family-Count Planning

  • The 26 post-hoc row pairs do not show that the screen is a better selector than initial-majority accuracy; it remains a distinct secondary signal for trace prioritization.The motivating Llama/Qwen reordering does not survive correction, and the authors characterize the pairwise tests as exploratory.Source: §G.1 · pp. 36–37, Table 41 · p. 37

G.2 Probe Temperature Correction

  • Initial probe answers were sampled at debate temperatures, not temperature 0, and robustness to reply temperature remains unresolved.Temperature fields are missing from several later saved probe records.Source: §G.2 · p. 37

G.3 Scope Checks Beyond Homogeneous MMLU-Pro Debate

  • Beyond the primary homogeneous MMLU-Pro setting, the checks are feasibility or stress cases, not broad transfer evidence; tool use and other regimes remain untested.Several mixed-panel results depend on the tie rule, and the Mistral GPQA row changes signed net from +1 to −1 when four invalid initial majorities are excluded.Source: §G.3 · p. 38, Table 42 · p. 38

G.4 Reasoning Models: Response-Period Checks

  • The response-period reasoning checks include a quasi-paired Qwen3-8B null that is not evidence of equivalence, alongside a later shared-initial-answer follow-up.The earlier check pairs only questions whose independently sampled initial-majority labels match.Source: §G.4 · p. 38

G.5 Follow-Up: Shared Initial Answers, Private Revision, and Reasoning Modes

  • Across twelve shared-Round-0 settings, private revision is 1.44–4.81 points less accurate than peer debate under binary scoring, while corrections decrease in every setting.Expected tie scoring narrows differences to −1.62 to +0.38 points; ties counted as wrong account for much of the binary loss.Source: §G.5 · pp. 38–39, Table 43 · p. 39
  • Private revision reduces wrong-answer collapses but also raises correct-to-tied transitions and lowers corrections; the study does not compare debate with longer single-agent reasoning budgets.Across reasoning-mode rows, both collapses and corrections occur, and private revision has a negative point difference in all eleven rows.Source: §G.5 · pp. 39–40, Table 44 · p. 39

G.6 Reproducibility Tiers

  • The release separates zero-API rebuildable results from aggregate-only, provider-dependent, and access-gated artifacts.The probe-gated LOMO ledger is available as a frozen aggregate, while its row-level decision matrix is not released.Source: §G.6 · p. 40

Related papers

Loading 2609.35279v1…