PaperlayerView exact revision on arXiv

Source-linked AI summary

Debating with More Persuasive LLMs Leads to More Truthful Answers

Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R. Bowman, Tim Rocktäschel, Ethan Perez

arXiv:2402.06782v4cs.AIcs.CL

TL;DR

The paper asks whether weaker judges can assess stronger models without ground-truth labels. It tests information-asymmetric debate, where text-accessing experts argue opposing answers and judges without the text choose between them. Debate improves accuracy for both LLM and human judges, and optimising debaters for persuasiveness improves truth identification, though the setup relies on verified evidence and does not test differences in reasoning ability or deceptive models.

  • Problem

    The paper asks whether weaker models can oversee stronger models as increasing capability makes expert-labelled ground truth difficult to obtain.

  • Method

    The study evaluates information-asymmetric debate, giving experts the source text and requiring non-expert judges without it to select between opposing arguments.

  • Results

    Debate enables 88% human and 76% LLM non-expert judge accuracy, versus naive baselines of 60% and 48%, respectively.

  • Takeaways & Limitations

    The results provide empirical evidence in one domain that debate can support scalable oversight by human and model judges without ground-truth labels.

  • Takeaways & Limitations

    The setup makes strong and weak models differ only in information access and evaluates RLHF models with a propensity for honesty, leaving deceptive models unclear.

Abstract

from arXiv · show

Common methods for aligning large language models (LLMs) with desired behaviour heavily rely on human-labelled data. However, as models grow increasingly sophisticated, they will surpass human expertise, and the role of human evaluation will evolve into non-experts overseeing experts. In anticipation of this, we ask: can weaker models assess the correctness of stronger models? We investigate this question in an analogous setting, where stronger models (experts) possess the necessary information to answer questions and weaker models (non-experts) lack this information. The method we evaluate is debate, where two LLM experts each argue for a different answer, and a non-expert selects the answer. We find that debate consistently helps both non-expert models and humans answer questions, achieving 76% and 88% accuracy respectively (naive baselines obtain 48% and 60%). Furthermore, optimising expert debaters for persuasiveness in an unsupervised manner improves non-expert ability to identify the truth in debates. Our results provide encouraging empirical evidence for the viability of aligning models with debate in the absence of ground truth.

1. Introduction

The paper asks whether weaker judges can oversee stronger models without ground-truth labels, evaluating adversarial debate in an information-asymmetric reading-comprehension setting. Debate improves truth identification for both LLM and human judges, while unsupervised persuasiveness optimisation further improves debate outcomes.

  • Motivation: The paper targets scalable oversight when increasingly capable models surpass available human expertise and ground-truth labels become unavailable.It motivates using weaker systems to oversee stronger ones and asks whether debate can help weaker judges evaluate stronger models.
  • Approach: Experts access the underlying text while non-expert judges do not, creating an information-asymmetric setting for testing oversight.Two experts argue for opposing answers, and the weaker judge selects an answer from their arguments.
  • Findings: 88% human and 76% LLM non-expert judge accuracy with debate exceeded naive baselines of 60% and 48%, respectively.Debate also exceeded consultancy, where human and LLM judges achieved 78% and 54%, respectively.
  • Findings: Persuasiveness optimisation made debaters better at arguing for correct answers relative to incorrect answers and increased judge accuracy in debate.Best-of-N and critique-and-refinement optimise outputs for judge approval without ground-truth labels; the effect generalised to unseen judges.
  • Findings: Human judges were better calibrated with debate than consultancy and maintained higher accuracy across rejection thresholds.Across protocols, human judges outperformed LLM judges, indicating limitations for automated scalable oversight.
  • Implications: The results provide empirical evidence in one domain that debate can support scalable oversight by both human and model judges as models become more capable.The authors frame this as evidence for further research on adversarial oversight methods.

2. Methods

The study uses information-asymmetric QuALITY reading-comprehension experiments in which text-accessing experts argue for answers and text-inaccessible judges evaluate them. It compares debate, interactive debate, and consultancy, and measures performance using judge accuracy and unsupervised debater metrics.

  • Protocols: In debate, two experts argue opposing answers simultaneously for N rounds, after which the judge reads the transcript and chooses an answer.Debaters see previous arguments and generate the next round's arguments.
  • Protocols: Interactive debate adds a judge statement after each round for clarifications, sub-problems, or directed questions.The statement remains visible to all players for the rest of the debate.
  • Protocols: Consultancy uses one expert assigned an answer who answers probing judge questions, with evaluations run for both correct and incorrect assignments.This preserves the same 50/50 prior as debate.
  • Task setting: The QuALITY task withholds approximately 7000-token Project Gutenberg science-fiction texts from judges while experts retain access.The evaluation uses the HARD subset of the dataset.
  • Implementation: Expert models use verified quotation tools, three rounds, and transcript word limits to constrain evidence and information quantity.Verified quotes are marked separately from unverified quotes, and judges are instructed to trust only verified quotes.
  • Optimisation and evaluation: Best-of-N sampling selects persuasive arguments using a preference model, while critique-and-refinement supplies additional inference-time optimisation.The study evaluates five LLMs as debaters or consultants and mitigates LLM judge biases by swapping transcript order and limiting verbosity.
  • Performance measurement: Cross-play tournaments compare debaters, using Swiss-style matches to produce rankings in O(n log n) rather than O(n^2) matches.Human evaluation uses trained annotators filtered by inter-annotator agreement and explanation quality rather than ground-truth labels.
  • Performance measurement: Win rate measures how often a judge selects a debater's assigned answer, with answer assignments flipped and averaged to reduce assignment bias.The metric is defined for a debate match involving two models and a judge.

3. Results with Language Model Judges

Debate accuracy improves as debaters become more persuasive, whereas consultancy accuracy degrades under optimisation. Stronger judges perform better across debates, and debate outperforms consultancy for human judges.

  • 75.6% vs 92.5%: Debate reduces the gap between judge and expert accuracy as judges become stronger.The comparison evaluates protocol performance against an expert judge.
  • Correct-answer assignments give debaters a persuasiveness advantage that increases with aggregate Elo.The pattern holds across different judges; GPT-4-Turbo is the strongest base LLM, while critique-and-refinement is less persuasive than best-of-N.
  • Higher aggregate Elo correlates strongly with higher judge accuracy in self-play debates.The experiments use GPT-4-Turbo to judge self-play debates on questions separate from cross-play matches.
  • Higher consultant skill leads to worse judge accuracy as optimisation makes incorrect consultants relatively more persuasive.The consultancy protocol degrades as consultant win rate rises, because judges do not know which answer is correct.
  • Different judges produce different ratings and accuracies, but stronger judges achieve higher accuracy across the full range of debater strengths.GPT-4-Turbo is reported as the strongest judge, with Claude2.1 used as the zero reference policy.

4. Results with Human Judges

Human judges outperform consultancy with debate, while additional interaction does not improve accuracy. Highly confident debate judgments are reliable, but weak LLM arguments remain the main bottleneck.

  • Protocol comparisons: Debate significantly outperforms consultancy in both static and interactive protocols for human judges.The reported significance levels are p=0.001 and p=0.048, respectively.
  • Protocol comparisons: Human judges achieve higher accuracy than LLM judges across all evaluated protocols.
  • Interaction: Static and interactive debate produce identical judge accuracy, indicating that more non-expert interaction does not improve performance.Interaction could let judges direct debates toward key uncertainties, but no accuracy gain was observed.
  • Confidence and reliability: Rejecting static-debate judgments below 75% confidence retains 65% of questions at 94% judge accuracy.Across rejection thresholds, debate also has higher accuracy and better calibration than consultancy.
  • Error analysis: Weak arguments from correct debaters, often caused by poor quote selection, are the most common debate error.This error analysis suggests that LLM debaters remain far from optimal.

5. Related Work

Related work spans scalable oversight, reasoning decomposition, human-AI decision support, and debate. This paper differs by using LLM debaters in information-asymmetric debates and examining controllable debater skill.

  • Scalable oversight: Prior scalable-oversight methods exploit the premise that identifying a correct solution is easier than generating one.
  • Scalable oversight: Reasoning-decomposition approaches encourage models to break problems into smaller reasoning steps, paralleling debate’s structured elicitation.
  • Human-AI decision support: Human-AI teams can improve credit-risk reasoning but may become overconfident, whereas debate judges were consistently underconfident.
  • Debate research: Earlier debate studies mainly used human debaters, while this work uses LLM debaters to control debater skill and investigate self-play.
  • Debate research: In prior comprehension debates, debaters extracted source statements rather than generating arguments, and one information-asymmetric LLM-debate study found no positive results.

6. Discussion, Limitations & Conclusion

The paper presents debate as a way for non-experts to identify correct answers and generate labels, with results extending across judges and base models. Its scope is limited by assumptions about information access, honesty, verifiable evidence, and domain scale.

  • Discussion and conclusion: Non-experts can judge transcripts between experts to identify correct answers, and LLM judges can automate this oversight.
  • Discussion and conclusion: Full-transcript judging is useful for producing expert labels using non-experts and untrustworthy experts, with findings generalising across base LLMs and human judges.
  • Limitations: The setup distinguishes strong and weak models only through information access, not reasoning ability or other skills.
  • Limitations: Debate requires debaters to provide verified evidence; without it, an incorrect debater could construct an alternative narrative that an inaccessible judge cannot challenge.
  • Limitations: The experiments use reading-comprehension stories of approximately 8000 tokens, leaving generalisation to domains requiring much larger evidence sets open.
  • Discussion and conclusion: In this domain, greater persuasiveness makes debate more truth-seeking and supports future research on fine-tuning LLMs via debate.
  • Implementation: The code used for the work is publicly available.The paper provides a GitHub repository link.

Impact Statement

The work targets human supervision of superhuman models, but applying supervision incorrectly may make malicious behaviour harder to detect. The authors therefore recommend caution.

  • Impact and caution: Incorrect application of supervision can make detecting malicious behaviour harder, so the authors recommend caution.

Contributions

The analysis compares debate and consultancy using best-of-N-augmented experts, finding that greater debater persuasiveness improves judge accuracy while greater consultant persuasiveness has the opposite effect.

  • The comparison includes Claude 2.1 and GPT-4-Turbo experts using best-of-N augmentation, excluding critique-and-refinement and GPT-3.5-Turbo results.
  • Best-of-N augmentation makes debaters more persuasive, and greater debater persuasiveness correlates with improved judge accuracy.
  • In consultancy, increasing consultant persuasiveness has a reciprocal effect on judge accuracy.

A.2. Comparing protocol accuracy with more samples

Across a larger question set, debate remains stronger than consultancy for non-expert judges. Its performance advantage grows with judge capability, while interactive debate does not improve LLM judge accuracy.

  • Debate remains a stronger oversight protocol than consultancy for every non-expert judge on the larger TL question set.
  • Interactive debate does not improve LLM judge accuracy when judges engage with debaters, whereas human interactive debate matches static debate accuracy.
  • Performance Gap Recovered measures the percentage of accuracy an oversight protocol recovers relative to an expert baseline with text access.
  • Debate recovers the most performance, and its advantage over other protocols increases as judge capability improves.
  • Judge accuracy in debate improves with debater skill, while consultancy accuracy diminishes as consultant proficiency increases.
  • Best-of-N produces more skilled debaters than critique-and-refinement or the unaugmented Bo1 model, with skill increasing as N increases.
  • GPT-4-Turbo debater skill plateaus beyond Bo16, possibly because Bo32 uses temperature 1.0 instead of 0.8.
  • Combining best-of-N with critique-and-refinement is less effective than best-of-N alone when expert inference calls are held constant.

A.4. Analysis of debater argument statistics

Stronger debaters use more authentic, longer verified quotes, while word filtering constrains argument length and majority voting improves debate accuracy but harms consultancy accuracy.

  • GPT-4-Turbo debaters predominantly use verified quotes, and best-of-N or critique-and-refinement increases verified-quote use across base models.
  • The word filter allows arguments between 70 and 150 words to mitigate judge verbosity bias.
  • A higher temperature of 0.8 with three votes yields a 19% error reduction for debate across 400 questions.

A.5. Improving LLM judge performance

Judge performance improves through majority voting, gold-label fine-tuning, and limited few-shot examples, but consultancy, human-chain-of-thought fine-tuning, and several biases constrain these gains.

  • Majority voting adds a 4% absolute accuracy gain in debate, equivalent to a 19% relative error reduction, with no improvement beyond three votes.
  • Majority voting degrades consultancy accuracy by up to 0.5% as the number of votes increases.
  • Fine-tuning on gold labels improves judge accuracy, but fine-tuning to imitate human chain-of-thought reasoning degrades it.
  • Few-shot performance improves for stronger judges with 2–4 examples, producing a 6% absolute boost for GPT-4-Turbo.
  • Using samples that human judges answered incorrectly matches ordinary few-shot performance through four shots but degrades GPT-4-Turbo accuracy beyond that.
  • Arguments-only settings lower judge accuracy because incorrect experts can lie more easily, while quotes-only settings strongly benefit correct experts.
  • Longer arguments increase aggregate Elo, influenced by the judge’s verbosity bias.
  • LLM judges exhibit positional bias, with GPT-3.5-Turbo showing the largest bias across protocols while humans show minimal bias.

A.7. Understanding LLM judge biases

LLM judges exhibit several biases, including positional bias, overconfidence, verbosity preference, and sycophancy. Debate performance also depends on round count and debater behavior, with incorrect arguments sometimes becoming more persuasive over time.

  • Judge biases: LLM judges show verbosity, positional, overconfidence, and sycophancy biases.These biases can favor longer arguments, particular answer positions, miscalibrated confidence, or unsupported quotes.
  • Judge biases: Higher word counts increase aggregate Elo, indicating GPT-4-Turbo favors longer arguments over shorter, more truthful responses.The authors investigated this verbosity bias using Swiss tournaments with 100-, 200-, and 300-word arguments.
  • Judge biases: Positional bias can substantially change accuracy when answer positions are swapped, especially for less capable judges.GPT-3.5-Turbo exhibits stronger positional bias, while some GPT-4-Turbo decisions remain highly confident after reversal.
  • Round effects: More rounds reduce LLM judge accuracy in debate but improve consultancy accuracy, as incorrect debaters can become more persuasive over time.For consultancy, the authors hypothesize that additional judge questions reduce sycophancy toward consultants.
  • Judge biases: LLM judges are consistently overconfident, although interactive debate produces better calibration than consultancy.The improved calibration in interactive debate may reflect the model reading the human judge’s messages.
  • Debater behavior: Debater skill varies across optimisation strategies: best-of-8 outperforms few-shot learning, while 32-shot sampling degrades skill relative to vanilla.Fine-tuning improves Elo but does not raise debater skill as much as hoped.

A.10. Analysis of different preference models

The preference model used to optimise debaters affects debate accuracy because different models rank arguments differently. With GPT-4-Turbo as judge, matching the preference and judge models improves scaling with best-of-N.

  • Preference-model alignment: Using GPT-4-Base instead of GPT-4-Turbo as the preference model decreases self-play accuracy with a GPT-4-Turbo judge.The models’ different preferences cause best-of-N arguments to persuade them differently.
  • Preference-model alignment: Matching the preference model to the judge maximises debate accuracy as best-of-N optimisation increases.Figure 23 reports higher judge accuracy when the preference model aligns with GPT-4-Turbo.
  • Round effects: With GPT-4-Turbo preference models, incorrect debaters become less likely to be selected as rounds continue, while correct debaters become more persuasive.Incorrect-debater selection probabilities also show greater variance, suggesting best-of-N discards unpersuasive outputs.
  • Model differences: GPT-4-Base shows a large shift between correct- and incorrect-debater selection distributions, partly explained by positional bias.The positional bias differs substantially between GPT-4-Turbo and GPT-4-Base.

B.1. Error Analysis

Human-juged error analysis identifies evidence extraction and debater quote selection as key sources of incorrect judgments. Debate also yields lower performance variation and benefits substantially from confidence-based ensembling.

  • Error rates: Human judges selected the wrong answer in 22% of consultancies and 12% of debates.The test set contained 161 incorrect judgments among 953 annotations.
  • Debate errors: Most debate errors stem from correct debaters selecting poor or repetitive quotes rather than presenting decisive evidence.Some questions are difficult because the correct answer is hard to demonstrate with quotes or is implicit beyond the text.
  • Consultancy errors: Consultancy errors commonly occur when judges fail to extract key evidence from strategically selective incorrect consultants.Judges may be unable to distinguish honest absence of contrary evidence from consultant omission.
  • Variation across judges and questions: Debate has lower variance in human judge accuracy and calibration than consultancy, driven by high variance among incorrect consultancies.Judges were also slightly more confident when judging debates.
  • Robustness: Human debate accuracy remains 86%–92% across varied debater models and experimental setups.The experiments vary prompting, optimisation, models, preference models, and word limits, with different question sets.
  • Ensembling: Confidence-weighted ensembles boost debate accuracy from 88% to 95–98%.All ensemble methods improve accuracy across protocols, with confidence-weighted methods performing best.

C.2. Task setup findings

Debate’s effectiveness depends strongly on task setup: it fails in symmetric settings but improves judging when debaters have information unavailable to the judge. QuALITY is recommended for testing debate, while dataset choice, prompt design, and debater optimisation remain important constraints.

  • Symmetric regimes: Inference-time debate performed worse than the Single-Model baseline across 10 open-source benchmarks except TRUTHFULQA when judges and debaters shared information and capability.The Single-Model baseline selects the best zero-shot, 5-shot, chain-of-thought, or plan&solve result.
  • Capability asymmetry: Debate did not improve Claude 2’s performance over zero-shot in a capability-asymmetric sandwiching setup with GPT-4 debaters.The authors attribute this result to the weaker judge’s deliberation ability being a bottleneck and suggest revisiting the setup with newer models on harder domains.
  • Information asymmetry: Information asymmetry increased judge accuracy because judges without the story had to rely on debaters, unlike knowledge-symmetric judges who could ignore the transcript.In QuALITY, debaters access the story while judges do not; the setup also permits using highly capable models as judges.
  • Dataset recommendations: QuALITY is recommended because its long stories support information asymmetry, strategic quoting, and arguments for either answer with relatively little self-defeating behaviour.Stories average 7k tokens, and the best distractor provides a plausible incorrect answer supported by selectable quotes.
  • Dataset recommendations: MMLU and AQuA are less suitable for current debate experiments because debaters may struggle to construct persuasive incorrect arguments and concede instead.The authors report unsuccessful prompt-engineering attempts to mitigate this self-defeating behaviour and limited success fine-tuning GPT-3.5-Turbo.
  • Iteration recommendations: Optimising for Elo rather than judge accuracy avoids rewarding concession-driven accuracy and measures persuasion across both correct and incorrect assigned answers.Concessions occur more often on the incorrect side, which can inflate judge accuracy without producing a useful label-free protocol.
  • Implementation recommendations: Prompt changes require testing across model families because small wording changes can alter concession rates and may not transfer between Claude and GPT models.One prompt revision affected an additional 3% of questions through concessions.
  • Debater optimisation: Using the same preference model as the judge produced the largest gains in aggregate Elo and judge accuracy.The finding is reported in connection with Figure 23’s comparison of optimisation choices.

D.9. Fine-tuning debaters methodology

The methodology addresses context-window limits and standardizes annotation and judge procedures for fine-tuning and evaluating debaters. It also documents experiment organization, worker selection, and calibration training.

  • Fine-tuning debaters methodology: Stories were truncated around human debaters’ quotes because Gutenberg QuALITY stories exceeded GPT-3.5-Turbo’s fine-tuning context window.The resulting procedure preserved context around quoted passages while shortening stories for the available context window.
  • Fine-tuning debaters methodology: 1063 samples were produced for fine-tuning by using transcript history and the next argument from winning human debaters.Each sample corresponded to a debate round and selectively used arguments from winning debaters.
  • Fine-tuning debaters methodology: Fine-tuned debaters were evaluated on Slate rather than Gutenberg questions because of context-window limitations, with 4 epochs yielding the best win rate against Claude 2.0.Slate articles were significantly shorter than Gutenberg stories.
  • Human judge recruitment and onboarding: Workers judged assigned protocols through an annotation platform, choosing the higher-confidence answer and submitting explanations with each judgment.Confidence was recorded in 5% increments up to 95%, with 50% unavailable.
  • Human judge recruitment and onboarding: 30 workers were recruited, trained, and filtered to 21 using inter-rater agreement and explanation quality rather than accuracy during selection.Filtering removed workers in the bottom agreement quartile or rated low for explanation quality.
  • Human judge recruitment and onboarding: The overall quadratic-scoring display confused judges, so it was removed and replaced with calibration instructions and Metaculus training.The additional calibration training took 1–2 hours.

F. Comprehensive Tournament Results

The tournament compares debater models and prompting strategies through standardized debate transcripts and judge evaluations. The reported tables describe model preferences, win-rate conventions, and prompt structure.

  • Comprehensive Tournament Results: Table 8 ranks 20 debaters in the cross-play tournament and reports model-specific preferences for critique and best-of-N variants.Claude 2.1 with best-of-N was less favored by Claude 2.1 and GPT-3.5-Turbo, likely because the preference model and judge preferences differed.
  • Comprehensive Tournament Results: Table 9 reports debater win rates over 40 matches played across 4 cross-play rounds.GPT-4-Turbo judged the tournament initially, while Claude 2.1 and GPT-3.5-Turbo later evaluated the same transcripts.
  • Comprehensive Tournament Results: Debater prompts require arguments to fit a word limit, use direct quotes, and present reasoning and arguments in separate tags.The judge sees only the content inside the argument tags.
  • Comprehensive Tournament Results: The quote system marks exact string matches as verified and inaccurate quotations as unverified for later debate rounds.Prompts instruct debaters to avoid repeating quotes and to use quotations as evidence against opposing arguments.

H.1. Successful debate

The examples illustrate how debate can clarify the correct answer when two debaters present competing interpretations, while consultancy can also persuade judges through extensive direct evidence. They also show that evidence selection and attribution remain important failure points.

  • Successful debate: Two debaters make the correct interpretation clearer than a single consultant in the Teena example.The example contrasts a prospecting-and-lunch account with an unsupported claim that Eddie teaches Teena isotopes during the hike.
  • Successful debate: The human judge assigned 85% confidence to the prospecting-hike answer, distinguishing the hike from the later isotope discussion.The judge identified the hike as involving uranium prospecting and placed the isotope conversation at dinner.
  • Successful debate: In the consultancy example, verified quotations supporting surveillance, subliminal messaging, public shaming, and psychological treatments led the judge toward Answer A.The judge gave Answer A 95% confidence despite acknowledging that brain implants might also be used.
  • Successful debate: A debate judge can be misled when the correct debater selects weak quotations and repeats them instead of presenting the most relevant evidence.The short word limit constrains how much of the relevant story can be quoted, making strategic selection important.
  • Successful debate: Unclear quote attribution can shift a judge toward the debater whose quotation is fully highlighted as verified.One judge assigned 55% confidence to B when attribution between Paul and the narrator remained uncertain.
Loading 2402.06782v4…