Debating with More Persuasive LLMs Leads to More Truthful Answers

Akbir Khan; John Hughes; Dan Valentine; Laura Ruis; Kshitij Sachan; Ansh Radhakrishnan; Edward Grefenstette; Samuel R. Bowman; Tim Rocktäschel; Ethan Perez

Source-linked AI summary

Debating with More Persuasive LLMs Leads to More Truthful Answers

Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, Ethan Perez

arXiv:2402.06782v4Feb 9, 2024cs.AIcs.CL

TL;DR

The paper asks whether weaker judges can assess stronger models without ground-truth labels and evaluates debate, in which two experts argue opposing answers for a non-expert judge. Debate yields 76% accuracy for non-expert LLM judges and 88% for humans, while optimising debaters for persuasiveness improves judges’ ability to identify truth.

Problem
As LLMs surpass available expertise, alignment needs scalable oversight that does not depend on ground-truth labels.
Method
The paper evaluates debate, using two expert LLMs arguing opposing answers for non-expert judges, and optimises debaters for judge approval without ground-truth labels.
Results
76% and 88% accuracy are achieved by non-expert LLM and human judges with debate, versus 48% and 60% for naive baselines.
Takeaways & Limitations
Debate provides empirical evidence for scalable oversight and for improving truth identification by optimising debaters for persuasiveness.
Takeaways & Limitations
The evaluation assumes stronger models differ from weaker ones only in information access and uses RLHF-tuned models with a propensity for honesty.

Abstract

from arXiv · show

Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.

1. Introduction

The paper investigates debate as scalable oversight when ground-truth labels become unavailable, using information-asymmetric reading-comprehension debates between stronger expert models and weaker judges. It finds that debate improves truth identification for both LLM and human judges, including when debaters are optimized only for persuasiveness.

Motivation: As LLMs surpass human expertise, labelled-data alignment may become unusable, motivating scalable oversight methods that scale with model capability.The paper frames weaker-model evaluation of stronger models as a response to the loss of reliable ground truth.
Main findings: 88% human and 76% LLM judge accuracy with debate exceed naive baselines of 60% and 48%, while also exceeding consultancy at 78% and 54%, respectively.Across protocols, adversarial debate outperforms non-adversarial baselines, while human judges outperform LLM judges.
Experimental setup: In information-asymmetric reading-comprehension debates, experts access the source text and quote tool, whereas non-expert judges do not and select answers from competing arguments.The study compares two-expert debate with interactive debate and single-expert consultancy, using both human and LLM judges.
Persuasiveness: Optimizing debaters for judge approval through best-of-N and critique-and-refinement increases judge accuracy, generalizes to unseen judges, and contrasts with persuasive consultants reducing accuracy.Persuasiveness is measured by judge approval and requires no ground-truth labels.
Calibration and implications: Human judges are better calibrated with debate than consultancy, and debate maintains higher accuracy across all rejection thresholds.The results provide empirical evidence in one domain that debate can enable scalable oversight by human and model judges as model capabilities increase.

2. Methods

The methods evaluate protocols for enabling information-poor judges to elicit truth from information-rich experts, using debate variants, consultancy, controlled reading-comprehension experiments, and label-free debater metrics. They also introduce tournament-based comparisons and judge-accuracy measures to assess persuasiveness and truthfulness.

2.1. Protocols: Debate uses two experts arguing opposing answers over N rounds, while interactive debate additionally lets the judge ask questions and request clarifications; consultancy provides a single-expert baseline.All protocols retain a transcript, and consultancy is run for both correct and incorrect answers to match debate’s 50/50 prior.
2.2. Experimental Details: Experiments give experts access to the source information while withholding it from non-expert judges, using QuALITY’s HARD subset with correct answers and best distractors across separate LLM and human splits.The texts come from Project Gutenberg science fiction stories with a median length of approximately 7000 tokens; LLM splits are TL (400) and DL (291), while human splits are TH (153) and DH (47).
2.2. Experimental Details: Protocols run for three rounds with a 900-word transcript limit, use verified-quote tools, and vary debater capability through best-of-N sampling and critique-and-refinement selected by GPT-4-Turbo log-probability preferences.LLM judges swap debater order across repeated judgments to mitigate positional bias and apply strict word limits to mitigate verbosity bias.
2.3. Measuring Debater and Judge Performance: The study measures debater persuasiveness without ground-truth labels using assignment-corrected win rates and Elo ratings, while judge accuracy is computed from debates between copies of the same model.A cross-play Swiss-style tournament compares many debaters in O(n log n) matches, and Figure 3 relates aggregate ratings to correct-versus-incorrect advantages and judge accuracy.

3. Results with Language Model Judges

Debate improves non-expert accuracy over consultancy and naive judging, with stronger debaters and judges further improving truth identification. However, optimizing consultants for persuasiveness can reduce accuracy, whereas human debate judgments are accurate and well-calibrated.

Insight 1: 75.6% judge accuracy shows debate outperforms consultancy and narrows the gap with the 92.5% expert-judge accuracy.All evaluated protocols outperform the naive judge baseline on test sets TH and TL.
Insight 2: Correct-answer debaters achieve higher Elo, with this persuasiveness advantage increasing as aggregate Elo rises across judges.GPT-4-Turbo is the strongest base LLM, while best-of-N is more persuasive than critique-and-refinement.
Insight 3: Higher aggregate Elo strongly correlates with judge accuracy, suggesting that optimizing debaters for win rate can produce more truthful debate outcomes.This provides weak evidence for truthful information under optimal play, consistent with Irving et al. (2018).
Insight 4: Optimizing consultants disproportionately increases the persuasiveness of incorrect answers, causing consultancy accuracy to degrade as consultant skill rises.The effect may arise because optimization enables exploration of deceptive approaches, whereas models already advocate correct answers relatively well after RLHF.
Human judge protocols: Human judges using debate outperform their corresponding baselines, achieve higher selective accuracy than consultancy, and exhibit better calibration.Across debate protocols, human judges are under-confident in their answers.
Insight 5: More capable judges identify correct arguments more accurately across the full range of debater strengths and produce a wider range of debater ratings.GPT-4-Turbo achieves higher accuracy for all debates than the other evaluated judges.

4. Results with Human Judges

With human judges, debate outperforms consultancy in both static and interactive protocols, while human judges outperform LLM judges overall. Interactive debate does not improve accuracy, but highly confident debate judgements are reliable and remaining errors often reflect weak debater arguments.

Experimental design: The test phase collected 953 human judgements from batches of 47 questions per judge over four weeks, following five weeks of iteration.Answer positions and assignments were randomised, and judges received no feedback after initial training.
Protocol comparison: Debate achieves higher human-judge accuracy than consultancy in both static and interactive protocols, and humans outperform LLM judges across protocols.The debate advantage is statistically significant in both settings (p=0.001 and p=0.048).
Protocol comparison: Static and interactive debate produce identical judge accuracy, indicating that additional non-expert interaction does not improve information-asymmetric debates.Static protocols provide one final judge interaction, whereas interactive protocols allow judges to ask clarifying questions throughout.
Judge reliability: At a 75% confidence-rejection threshold, static debate retains 65% of questions while achieving 94% judge accuracy.Across rejection thresholds, debate has higher accuracy and better calibration than consultancy.
Error analysis: The main debate error is weak argumentation from the correct debater, often caused by poor quote selection, making LLM debaters the bottleneck for higher accuracy.In consultancy, the common error is judges failing to elicit key quotes from the incorrect consultant.

5. Related Work

This work builds on methods for supervising stronger models, human-AI decision support, and debate-based oversight. It distinguishes its LLM-debate protocol from prior studies through information asymmetry, controllable debater skill, self-play, and human judging.

Prior supervision methods exploit easier solution identification, reasoning decomposition, or inductive biases enabling weaker models to supervise stronger ones (Christiano et al., 2017; Saunders et al., 2022; Burns et al., 2023).
Unlike human-AI teams studied for specific tasks, this work uses general LLMs across tasks and finds debate judges consistently underconfident rather than overly confident (Bansal et al., 2021).
Although the debate game was proposed for safe-AI oversight, prior studies used human debaters, whereas LLM debaters enable controlled skill comparisons and self-play debates (Irving et al., 2018; Barnes, 2020; Parrish et al., 2022b;a; Michael et al., 2023).
Compared with prior LLM-debate studies, this work differs from source-text extraction, unsuccessful QuALITY LLM debates, and single-turn reinforcement-learning protocols without human judges (Perez et al., 2019; Michael et al., 2023; Radhakrishnan, 2023; Du et al., 2023).

6. Discussion, Limitations & Conclusion

The study finds that debate enables non-experts, including automated LLM and human judges, to identify correct answers from expert transcripts, supporting scalable oversight. It also finds that greater debater persuasiveness improves truth-seeking, while highlighting limitations involving model capabilities, deception, evidence access, and generalisation.

Contributions: Debate lets non-experts judge full transcripts between experts to identify correct answers, and this oversight can be automated with LLM judges.The approach produces expert labels using only non-experts and untrustworthy experts.
Robustness: Debate generalises across expert and judge base LLMs and human judges, indicating robustness to variation in judge skill as models advance.The result differs from the stricter protocol proposed by Irving et al. (2018), because full-transcript judging is already useful.
Limitations: The setup distinguishes strong and weak models only by information access, so debate’s suitability for differences in reasoning ability or for deceptive models remains unclear (Greenblatt et al., 2023; Hubinger et al., 2024).The evaluated models were fine-tuned with RLHF and therefore had a propensity for honesty.
Limitations: The results depend on debaters providing verified evidence; without it, an incorrect debater could construct an alternative narrative that the judge cannot check.The study used short-story reading comprehension with approximately 8000 tokens, leaving generalisation to larger evidence sets and difficult-to-verify parametric knowledge open.
Future directions: Useful debate implementations are expected to ground arguments in verifiable sources or domain-specific tools, such as simulators for mathematics or experimental data for physics.Such tool use may help judges decide debates more quickly and accurately.
Persuasiveness: Greater model persuasiveness makes debate more truth-seeking because optimisation improves arguments for the correct answer relatively more than for the incorrect answer.This supports future fine-tuning of LLMs via debate and scalable oversight.

Impact Statement

The work addresses how humans might supervise superhuman models, but cautions that incorrectly applied supervision could make malicious behaviour harder to detect (Hubinger et al., 2024).

Impact Statement: The authors present the work as contributing to supervision of superhuman models while recommending caution because incorrect application may hinder detection of malicious behaviour (Hubinger et al., 2024).The focus is on methods that allow humans to supervise models, particularly in the open problem of supervising superhuman systems.

Contributions · A. Further Results with Language Model Judges · A.1. Consultancy and debate best-of-N persuasiveness

Best-of-N augmentation links greater debater persuasiveness to higher judge accuracy in debate, while increased persuasiveness in consultancy has the opposite effect. The comparison uses Claude 2.1 and GPT-4-Turbo experts and excludes critique-and-refinement and GPT-3.5-Turbo results.

Contributions: AK led the project and participated in all its components, while JH led LLM experiments and infrastructure.The contributions passage identifies these project and experiment leadership roles.
Contributions: DV led human experiments and developed infrastructure, with KS also contributing infrastructure development.The passage distinguishes human-experiment leadership from additional infrastructure work.
A. Further Results with Language Model Judges: The comparison covers judge accuracy and expert win rate for consultancy and debate protocols.The results are presented side by side in Figure 7 using outcomes from the cross-play tournament and consultancy findings.
A.1. Consultancy and debate best-of-N persuasiveness: The reported comparison includes only Claude 2.1 and GPT-4-Turbo experts with best-of-N augmentation.This scope is specified for the Figure 7 analysis.
A.1. Consultancy and debate best-of-N persuasiveness: The critique-and-refinement protocol and GPT-3.5-Turbo are excluded from the comparison for simplicity.These exclusions limit the displayed comparison to the two specified expert models and protocols.
A.1. Consultancy and debate best-of-N persuasiveness: Greater best-of-N persuasiveness improves judge accuracy in debate, whereas increased persuasiveness in consultancy reduces judge accuracy.The analysis reports these opposite relationships across the two protocols.

A.2. Comparing protocol accuracy with more samples · A.3. Best-of-N versus critique-and-refinement

Across the larger question set, debate remains stronger than consultancy, while interactive judge–debater interaction does not improve LLM-judge accuracy. Best-of-N produces increasingly persuasive debaters, whose skill improves debate oversight but harms consultancy performance as consultant skill rises.

A.2. Comparing protocol accuracy with more samples: Interactive debate does not improve LLM-judge accuracy, whereas humans achieve the same accuracy with interactive and static debate.This pattern is consistent across each LLM judge.
A.2. Comparing protocol accuracy with more samples: Debate achieves significantly higher judge accuracy than consultancy and the naive baseline on 400 questions, while stronger judge base models improve debate more than consultancy.The comparison uses identical GPT-4-Turbo-bo16 expert transcripts and a GPT-4-Turbo interactive judge.
A.2. Comparing protocol accuracy with more samples: Debate recovers the most performance, with a 60% performance gap recovered, and its advantage over other protocols increases as judge capability rises.Consultancy’s performance-gap recovery declines as the judge base model becomes more capable.
A.3. Best-of-N versus critique-and-refinement: Debate judge accuracy improves with debater skill, whereas consultancy judge accuracy diminishes as consultant proficiency increases.Consultancy aggregate win rate and debate aggregate Elo measure expert persuasion skill in their respective protocols.
A.3. Best-of-N versus critique-and-refinement: Debater skill plateaus beyond Bo16 for GPT-4-Turbo; Bo32’s higher temperature of 1.0 versus 0.8 may have made arguments too diverse for further gains.The cited temperature difference is offered as a possible explanation for the plateau.
A.3. Best-of-N versus critique-and-refinement: Combining best-of-N with critique-and-refinement is competitive but less effective than best-of-N alone at equal expert-model inference calls; Bo4 Co8 uses 16 calls, matching Bo16.Applying best-of-N to both initial arguments and refinements yields 2*boN + cN inference calls.
A.3. Best-of-N versus critique-and-refinement: Critique-and-refinement improves expert skill over Bo1, but best-of-N produces more skilled debaters and continues improving as N increases.Figure 10 compares these optimisation strategies across debate and consultancy.
A.3. Best-of-N versus critique-and-refinement: Stronger debaters use longer, more verified quotes, while weaker debaters more often use fake, low-matching, or duplicate quotes.GPT-4-Turbo shows a notably lower proportion of unverified quotes, and critique-and-refinement tends to produce longer arguments.

A.4. Analysis of debater argument statistics

Debater argument statistics link stronger performance to more authentic, verified quoting and longer arguments, while a strict word filter concentrates GPT-4-Turbo arguments near 100 words. Majority voting further improves debate judge accuracy, with three votes at 0.8 temperature reducing error by 19%.

Quote usage and authenticity: GPT-4-Turbo uses predominantly verified quotes and fewer fake quotes than Claude 2.1 and GPT-3.5-Turbo, while best-of-N and critique-and-refinement increase verified-quote use across base models.Claude models use more quotes overall, but their quotes are shorter; fake quotes are identified as over-7-word quotes with zero normalized n-gram similarity to the surrounding text.
Argument length: The strict 70–150-word filter mitigates judge verbosity bias, and GPT-4-Turbo arguments cluster more tightly around the 100-word target than other base models.The word-count distribution is shown in Figure 11-Right.
Majority voting: 19% error reduction results from using debate with 0.8 temperature and three votes, although increasing votes improves debate accuracy while reducing consultancy accuracy.The result is based on 400 questions, with error bars showing the standard error of the mean.

A.5. Improving LLM judge performance · A.6. Impact on using and not using the quote system

Judge accuracy improves through majority voting, gold-label supervision, and limited few-shot prompting, whereas human-reasoning imitation and excessive votes or examples can hurt. Quote access substantially improves debate judging, while arguments-only settings permit incorrect experts to persuade judges more easily.

A.5. Improving LLM judge performance: 4% absolute gain: majority voting improves GPT-4-Turbo judge accuracy, with stable performance from three votes at temperature 0.8 using Chain-of-Thought prompting.Using more than three votes does not further improve performance.
A.5. Improving LLM judge performance: Up to 0.5% degradation: majority voting slightly reduces judge accuracy with consultancy as the number of votes increases.This supports the finding that stronger judges can reduce consultancy performance.
A.5. Improving LLM judge performance: Gold-label fine-tuning improves judge accuracy, but imitating human chain-of-thought reasoning degrades it.Fine-tuning GPT-3.5-Turbo on gold labels achieves parity with Claude 2.1 zero-shot, whereas human-reasoning fine-tuning does not help.
A.5. Improving LLM judge performance: 6% absolute boost: GPT-4-Turbo judges improve with 2–4 few-shot examples, while additional examples plateau or degrade performance.GPT-3.5-Turbo does not benefit from few-shot samples, and performance using debates that human judges got wrong degrades beyond four examples.
A.6. Impact on using and not using the quote system: Arguments-only settings lower judge accuracy across debate and consultancy because incorrect experts can lie more easily, whereas quote-only settings strongly benefit correct experts.The study nevertheless uses quotes plus arguments because this is more natural for human readability.
A.6. Impact on using and not using the quote system: Longer arguments increase aggregate Elo among LLM judges, partly because of judge verbosity bias.A Swiss tournament over 291 QuALITY questions demonstrates this effect, while GPT-4-Turbo follows word limits with higher precision and lower variance than Claude 2.1.
A.6. Impact on using and not using the quote system: LLM judges exhibit positional bias, unlike human judges who show minimal bias; GPT-3.5-Turbo has the largest bias across protocols.Positional bias is also notably higher for human judges in the naive baseline condition.

A.7. Understanding LLM judge biases · A.8. Impact of number of rounds on accuracy · A.9. Improving the debaters with fine-tuning and few-shot learning

The appendix identifies several systematic LLM judge and debater biases, and shows that additional rounds help consultancy but can reduce debate accuracy for weaker judges. Fine-tuning provides limited gains under context constraints, while few-shot learning offers little benefit and underperforms best-of-N selection.

A.7. Understanding LLM judge biases: The judges exhibit verbosity, positional, overconfidence, and sycophancy biases, including preference for longer arguments, position-dependent choices, miscalibrated confidence, and belief in unverified quotes.GPT-4-Turbo’s aggregate Elo rises with argument length, and positional swaps can substantially change accuracy; humans also show positional bias, especially in the naive case.
A.7. Understanding LLM judge biases: Self-enhancement bias does not affect self-play debates, although cross-play tournaments alter some debater rankings without clearly favoring debaters from the same base model.The self-play setup uses the same model instance for correct and incorrect debaters, while different judges are used for cross-play evaluation.
A.7. Understanding LLM judge biases: The incorrect debater often concedes, so selecting a next-best distractor answer reduces self-defeating behavior when the story is ambiguous.This intervention addresses cases where models tend to refuse to argue for the wrong side.
A.8. Impact of number of rounds on accuracy: Additional rounds decrease debate accuracy for weaker judges but increase consultancy accuracy, with three rounds chosen as a practical interaction minimum.Debate accuracy is best after one or two rounds for weaker judges, while three rounds allow critique and defense and support two human judge interactions.
A.7. Understanding LLM judge biases: Positional bias can produce high-confidence reversals: 25% of cases remained above 98% confidence even when swapping answer positions changed the decision.Confidence generally falls when outcomes change after swapping answers, but some cases retain very high confidence, showing that the bias is not reliably reflected in confidence.
A.7. Understanding LLM judge biases: LLM judges are consistently overconfident, while interactive debate becomes more calibrated, likely because judges can read the human’s interactive messages.Selective accuracy also shows debate outperforming consultancy across coverage rates despite LLM overconfidence, especially in consultancy.
A.8. Impact of number of rounds on accuracy: As debate rounds increase, incorrect debaters become more persuasive to judges, partly because correct-answer quote selection is poor and best-quote recall could mitigate declining accuracy.In consultancy, both incorrect and correct consultants become less persuasive over rounds, consistent with interactive judges becoming less sycophantic toward consultants.
A.9. Improving the debaters with fine-tuning and few-shot learning: Fine-tuning improves GPT-3.5-Turbo’s debater skill but remains below Claude 2.0, while 32-shot learning degrades performance and both few-shot settings trail best-of-N.The fine-tuned model’s Elo falls between GPT-3.5-Turbo and Claude 2.0; 8-shot learning slightly helps, whereas 32-shot learning reduces Elo relative to zero-shot.

A.10. Analysis of different preference models · B. Further Results with Human Judges

Matching the preference model to the judge improves debate accuracy and best-of-N scaling, while mismatched models weaken the relationship between debater skill and judge accuracy. Preference-model behavior also differs across rounds and models, with GPT-4-Base showing stronger positional bias.

A.10. Analysis of different preference models: Matching the preference model to the judge produces higher judge accuracy and better best-of-N scaling than using GPT-4-Base.Figure 23 reports less pronounced correlation between debater skill and accuracy when preference and judge models differ.
A.10. Analysis of different preference models: The preference-model analysis assumes consistent debater Elos despite possible skill changes when the preference model is altered.This assumption limits interpretation of Figure 23’s comparisons.
A.10. Analysis of different preference models: Using a disjoint preference model does not invalidate the finding that judge accuracy increases with debater strength.Figure 5 shows increasing linear trends for three different judges.
A.10. Analysis of different preference models: The incorrect debater’s selection probability decreases across rounds, while the correct debater becomes harder to displace.The incorrect debater also shows greater probability variance, which best-of-N can reduce by discarding unpersuasive answers.
A.10. Analysis of different preference models: GPT-4-Base assigns consistently lower selection probabilities to the incorrect debater than GPT-4-Turbo, suggesting a substantial positional bias.The two models exhibit markedly different correct-versus-incorrect selection distributions.
A.10. Analysis of different preference models: GPT-4-Turbo shows a wider probability range and a pronounced tail across the 16 best-of-N samples, unlike GPT-4-Base’s shifted distributions.The broader GPT-4-Turbo spread may be influenced by RLHF training.

B.1. Error Analysis … C.1. Task setup variations

The analyses identify debate’s main human-judging errors, show that debate improves consistency and accuracy across judges and setups, and define three task regimes for applying debate. They also find that debater quality and ensemble aggregation materially affect performance, while human feedback clarifies judging strategies and protocol preferences.

B.1. Error Analysis: Human judges made 161 errors among 953 annotations, with errors in 22% of consultancies versus 12% of debates; debate failures mostly reflected correct debaters’ poor quote selection.Consultancy errors commonly involved judges failing to extract key evidence while incorrect consultants strategically presented plausible supporting quotes.
B.2. Judge Variation; B.3. Question Variation: Debate has lower variance in judge accuracy and calibration than consultancy, while correct consultancy is similar to debate and incorrect consultancy is much more variable.Judges were also slightly more confident when judging debates; question-level variance differences were less pronounced than judge-level differences.
B.5. Human Judge Iteration: Across human iteration experiments, static debate accuracy ranges from 86% to 92% and consistently exceeds consultancy across variations in models, prompts, optimization, preference models, and word limits.The experiments varied debater models, prompting, optimization, argument length, transcript-length controls, and interaction protocols.
B.4. Human Judge Ensembles: Confidence-weighted ensembles boost debate accuracy from 88% to 95–98%, with all ensemble methods improving accuracy across protocols.The ensemble analysis pools multiple judgments per question; confidence-weighted methods perform best.
B.6. Low Elo debaters: Low-Elo debaters are less persuasive to humans and produce significantly worse debate performance, suggesting human and LLM judges are persuaded by different features.The low-Elo comparison used Claude-2.1-bo4 and GPT-3.5-bo16 against GPT-4-Turbo-bo16.
B.7. Human judge feedback: Human feedback favored debate over consultancy: 75% found debates easier to judge, 56% preferred longer arguments, and 81% mostly relied on quotes.No judges said they usually decided after the first round, while 44% wanted more than three rounds.
C. Recommendations for working on debate; C.1. Task setup variations: The recommendations describe capability-asymmetric, information-asymmetric, and symmetric debate regimes, each paired with a corresponding few-shot, chain-of-thought, or consultancy baseline.In capability asymmetry debaters are stronger than the judge; in information asymmetry debaters access privileged information; in symmetry agents share capability and information.
C.1. Task setup variations: Inference-time protocols use untuned models with augmentation such as few-shot learning, chain of thought, or best-of-N, but current models perform poorly for symmetric-regime self-improvement at inference time.Self-improvement can use multiple copies of one model, as in self-refine (Madaan et al., 2023), where models critique and refine completions.

C.2. Task setup findings … D.8. Judge Prompt Iteration

This section examines the conditions and methods that shape effective debate judging, including information asymmetry, debater optimization and ranking, bias control, and prompt refinement. It also considers implementation choices such as task selection, Elo-based optimization, best-of-N, critique-and-refinement, and explicit quote verification.

C.2. Task setup findings: 76% and 88% accuracy are achieved by non-expert models and humans with debate, while symmetric-regime debate generally underperforms single-model baselines except on TRUTHFULQA.Information asymmetry is crucial because judges must rely on debaters; in QuALITY, judges lack the story, whereas capability asymmetry with Claude 2 judging GPT-4 debates did not improve performance.
C.3. Dataset recommendations: QuALITY is recommended for debate because long stories create information asymmetry and provide abundant, plausible evidence for both correct and distractor answers.Its best distractor reduces self-defeating behaviour, whereas datasets such as MMLU and AQuA make persuasive incorrect arguments difficult; prompt engineering and GPT-3.5-Turbo fine-tuning did not solve this problem.
C.4. Iteration recommendations: Optimize debaters for Elo rather than judge accuracy, because aggregate win rates across both answer assignments reward persuasion for correct and incorrect answers without labels.Direct accuracy optimization can inflate scores when incorrect debaters concede, allowing the judge to identify the correct answer trivially.
C.4. Iteration recommendations: Test prompt changes across model families and inspect debates interactively, since small wording changes can alter concessions and expose failure modes such as repetition, disengagement, and incorrect quote tagging.A consultant-prompt change affected an extra 3% of questions, and some prompt changes transferred poorly between Claude and GPT models.
C.5. Implementation recommendations: Implementation should mitigate positional and verbosity biases, use at least GPT-4-Turbo judging capability, and keep correct and incorrect debaters on identical prompts.Identical prompts reduce self-defeating behaviour and information leakage, while evaluating both answer permutations and enforcing word limits address judge biases.
D.1. Filtering Questions: Questions are filtered for clear, difficult comprehension judgments, then reduced from four choices to the correct answer and best distractor; human experiments use 200 unseen questions.The selection uses project Gutenberg science-fiction stories and requires unanimous untimed correctness, low timed accuracy, answerability, sufficient context, and agreement with the writer label.
D.2. Quote Verification System: Debaters support assigned answers with story quotations wrapped in XML tags, which a normalized direct-string-match verifier marks as verified or unverified.Verified quotes receive <v_quote> tags, failed matches receive <u_quote> tags, and untagged quotations are not processed.
D.3. Optimising Debaters: Best-of-N selects the most persuasive candidate using judge-model preferences, while critique-and-refinement selects helpful feedback and revises arguments; using the same preference model as the judge produced the largest Elo and accuracy gains.Critique-and-refinement targets repeated or unverified quotes, opposing evidence, and overlooked support, but can increase concessions and therefore requires iterative safeguards.
D.3. Optimising Debaters: Debater optimisation uses critique feedback and rejection sampling, but Claude 2.1 sometimes refuses refinement and the incorrect debater more often mentions the critique.When refinement is refused, the original argument is retained; preventing critique disclosure is handled through prompting and rejection sampling.
D.4. Cross-Play Tournament: A Swiss-style cross-play tournament ranks 20 debater variants using both correct and incorrect permutations, with four rounds chosen as representative while conserving API credits.The tournament uses 291 quality-development questions for cross-play and reports comprehensive win rates across all 40 matches with three judges.
D.5. Calculating Elo Ranking: Elo estimates relative debater persuasiveness from observed win rates using BFGS optimisation and 1000-seed bootstrapping for confidence intervals.The reported fit statistics are 0.997 for Claude-2.1, 0.989 for GPT-3.5-turbo, 0.998 for GPT-4-turbo, and 0.991 overall.
D.6. Mitigating LLM bias: Strict word limits, rejection sampling, and position-swappable transcripts mitigate verbosity and ordering biases when judging LLM debates.Prompts avoid position identifiers for model debaters, while human trials run debates in both positions because interactive prompts include names and answer-choice letters.
D.7. Debater Prompt Iteration: Debater prompt optimisation targets aggregate Elo rather than accuracy because incorrect-side concessions can artificially improve accuracy while revealing self-defeating behaviour.Identical prompts are used for both sides to avoid label leakage, and prompt changes sometimes failed to transfer across base models, motivating simultaneous iteration over models.
D.8. Judge Prompt Iteration: Explicit quote-verification instructions made judges down-weight unverified arguments, penalising fake quotes and poor use of the quote system.Earlier prompts treated verified and unverified quotes similarly; after revision, accuracy became high when incorrect arguments were made unverified and low when correct arguments were made unverified.

D.9. Fine-tuning debaters methodology … E.2. Human Judge Recruitment and Onboarding

The paper fine-tunes debaters on truncated human-debate contexts, evaluates them on shorter Slate questions, and describes dated LLM runs. Human judging used controlled transcript assignment, structured confidence judgments, and staged recruitment, filtering, and calibration.

D.9. Fine-tuning debaters methodology: Stories were truncated around human debaters’ quotes, reducing average length from 7k to 2.8k tokens for a 4k-token context window.The pipeline normalized quotes, scanned overlapping windows, retained quote-containing windows, and combined them into truncated stories.
D.9. Fine-tuning debaters methodology: 1063 fine-tuning samples used winning debaters’ next arguments, with 4 epochs producing the best win rate against Claude 2.0 on Slate questions.Testing used Slate rather than Gutenberg because context-window limitations made the longer Gutenberg stories impractical.
D.10. LLM experiment dates: LLM experiments used gpt-3.5-turbo, gpt-4-1106-preview, claude-2.1, and claude-v1.3 across dates specified separately for each figure and appendix run.Figure-specific periods covered January 10–26, 2024 for human-dataset results, December 18, 2023–January 5, 2024 for optimisation runs, and December 30, 2023–January 2, 2024 for judge comparisons.
E.1. Annotation Platform: The annotation platform varied question–protocol assignments with Latin squares, randomized question and answer order, and generated independent transcripts for each judge.Consultancy accuracy was computed as 1/2(mean acc correct + mean acc incorrect) to address assignment imbalance.
E.1. Annotation Platform: Judges submitted probabilities for answers A and B in 5% increments, selected the higher-confidence answer, and explained each decision without seeing correctness feedback.Static transcripts were pre-generated, whereas interactive protocols generated subsequent rounds after judges entered interactions; extra rounds typically took 30–60 seconds.
E.2. Human Judge Recruitment and Onboarding: 30 fluent English-speaking, high-reasoning workers were recruited through Pareto at $35/hr, trained on 24 labeled Easy Slate questions, and filtered on 24 unlabeled Hard Gutenberg questions.Training covered static and interactive debate and consultancy equally, supporting generalization from easy labeled data to hard unlabeled data.
E.2. Human Judge Recruitment and Onboarding: The filtering phase reduced the worker pool from 30 to 21 using inter-rater agreement and explanation quality rather than Hard-subset accuracy.Accuracy and inter-rater agreement produced very similar worker rankings after filtering.
E.2. Human Judge Recruitment and Onboarding: Quadratic-scoring feedback confused judges, so it was removed and replaced with calibration instructions plus 1–2 hours of Metaculus training before experiments.The original score led many judges to believe maximal confidence was optimal.

F. Comprehensive Tournament Results · G. Prompts · H. Samples

The supplementary sections report cross-play tournament preferences, provide complete expert and judge prompts, define additional baselines and preference-model prompts, and include human debate transcripts. The tournament shows model-dependent preferences, including stronger support for GPT-4-Turbo with critique and weaker support for Claude 2.1 with best-of-N.

F. Comprehensive Tournament Results: GPT-4-Turbo with critique is favoured by Claude 2.1 and GPT-3.5-Turbo, whereas Claude 2.1 with best-of-N is less favoured, likely because its preference model mismatches judge preferences.The rankings come from the final cross-play tournament and reflect differing preferences for persuasiveness across judge models.
F. Comprehensive Tournament Results: The cross-play tournament comprised 40 matches played over four rounds, with GPT-4-Turbo judging during play and Claude 2.1 and GPT-3.5-Turbo evaluating the same transcripts after completion.Win rates are measured relative to Debater 1, who wins when the rate exceeds 0.5; best-of-N is denoted boN and critique-and-refinement cN.
G.1. Expert Prompts: Expert prompts assign debaters opposing answers and instruct them to persuade an information-limited judge through assertive, evidence-based arguments within a word limit.Debaters structure responses with separate thinking and argument tags, while round-specific advice and new-argument requests are inserted into the prompt.
G.1. Expert Prompts: Expert prompts require direct story quotes, external verification, nonrepetition, and explicit treatment of unsupported or misleading opponent quotes when relevant.The judging criteria reward verified evidence, logical integration, new arguments, effective counterarguments, and contextual opening statements.
G.2. Judge Prompts: Judge prompts ask information-limited judges to select the likely correct answer by weighing argument strengths, weaknesses, and verified versus unverified quotes.Interactive variants instead ask targeted follow-up questions that probe suspicious reasoning, expose inconsistencies, and draw attention to unverified evidence.
G.3. Additional Baselines: Additional baselines directly show the story to a super-intelligent model or provide only the question and answer choices while requiring a binary answer.The prompts require the model to choose an answer even when uncertain and, in one variant, to reason step by step before formatting the final choice.
G.4. Best-of-N Prompts Preference Model (Debate): The debate best-of-N preference model uses distinct prompt endings for GPT-4-Turbo and GPT-4-Base because chat and completion models behave differently.The supplied prompt examples ask which answer is most likely correct, with GPT-4-Turbo requesting only A or B and GPT-4-Base continuing a completion.
H. Samples: The human-experiment samples include transcripts with verified quotes highlighted in green and unverified quotes highlighted in yellow.The entire human dataset is released alongside the work.

H.1. Successful debate … H.5. Failed consultancy - cannot extract key evidence

These examples examine when debate and consultancy help judges identify correct answers and when they fail due to poorly selected or unavailable evidence, ambiguous attribution, persuasive inferences, or incorrect consultants. In the failed consultancy, uncertainty persisted because the evidence supported a fake name but did not establish an interstellar visitor.

H.1. Successful debate: Two debaters made the correct answer clear, and the human judge selected A with 85% confidence by distinguishing prospecting during the hike from a later isotope discussion.The example contrasts Teena accompanying Eddie on a prospecting hike with isotope instruction occurring after the hike.
H.2. Successful correct static consultancy: A correct consultant supported Answer A with multiple quoted crime-prevention methods, leading the human judge to assign 95% confidence despite the alternative claim about brain implants.The cited methods include surveillance, subliminal messaging, public shaming, and psychological treatments including lobotomy.
H.3. Successful incorrect interactive consultancy: Interactive questioning exposed that the consultant could not produce evidence that a Senator had actually chewed out Conners, so the judge rejected the inference and chose A with 85% confidence.The consultant acknowledged that the argument relied on implication rather than a direct confrontation or communication.
H.4. Failed debate - suboptimal evidence use: The debate failed because the correct debater repeatedly used poor quote selection, while ambiguous speaker attribution led the human judge to choose B with 55% confidence.The correct answer concerned Paul researching protein, but the judge trusted the already-highlighted thrill-tourism quote attributed to Paul.
H.5. Failed consultancy - cannot extract key evidence: Consultancy can fail when the incorrect consultant withholds quotes supporting the correct answer, leaving the judge unable to extract key evidence.The passage notes that although many quotes could support Answer A, an incorrect consultant may simply avoid selecting them.
H.5. Failed consultancy - cannot extract key evidence: The consultant argued that widespread ignorance among residents and authorities showed Haron Gorka was using a fake name rather than being an unidentified interstellar visitor.The argument relied on repeated failures by the post office, police, and other local institutions to recognize the name.
H.5. Failed consultancy - cannot extract key evidence: The consultant rejected the recluse explanation, reasoning that even a reclusive real resident would likely be known through mail, legal matters, or official interactions.This reasoning again treated the post office and police’s lack of recognition as evidence that the identity was fabricated.
H.5. Failed consultancy - cannot extract key evidence: 55% confidence in Answer B: the human judge leaned toward the fake-name explanation while finding insufficient evidence for an interstellar visitor.The judge noted that no one in town, including the post office, knew Haron Gorka, but considered an alien presence unlikely rather than impossible.