PaperlayerView exact revision on arXiv

Source-linked AI summary

On scalable oversight with weak LLMs judging strong LLMs

Zachary Kenton, Noah Y. Siegel, János Kramár, Jonah Brown-Cohen, Samuel Albanie, Jannis Bulian, Rishabh Agarwal, David Lindner, Yunhao Tang, Noah D. Goodman, Rohin Shah

arXiv:2407.04622v2cs.LG

TL;DR

The paper asks whether weaker judges can accurately supervise stronger AI agents across capability gaps beyond extractive QA. It evaluates debate, consultancy, and direct question answering across multiple task types, finding debate consistently beats consultancy while its advantage over direct QA varies by task.

  • Problem

    Empirical debate studies had largely focused on reading comprehension with information asymmetry, leaving its performance across broader capability gaps insufficiently tested.

    Source: §1 · p. 2

  • Method

    The authors evaluate debate, consultancy, and direct question answering across extractive, closed, and multimodal tasks using LLM agents and weaker LLM judges.

    Source: §1 · p. 2, §3.2.1 · p. 7

  • Results

    Debate consistently outperforms consultancy, but its advantage over direct question answering depends on task type and is small or absent on tasks without information asymmetry.

    Source: §1 · p. 4

  • Takeaways & Limitations

    Open debate is less likely than open consultancy to amplify an agent's incorrect answer, while stronger debaters tend to produce marginally higher judge accuracy.

    Source: §1 · p. 4, §4.3 · p. 12

    The study evaluates inference-only prompting and does not directly establish debate's properties as a training protocol.

    Source: §5 · p. 15

Abstract

from arXiv · show

Scalable oversight protocols aim to enable humans to accurately supervise superhuman AI. In this paper we study debate, where two AI's compete to convince a judge; consultancy, where a single AI tries to convince a judge that asks questions; and compare to a baseline of direct question-answering, where the judge just answers outright without the AI. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models. We benchmark on a diverse range of asymmetries between judges and agents, extending previous work on a single extractive QA task with information asymmetry, to also include mathematics, coding, logic and multimodal reasoning asymmetries. We find that debate outperforms consultancy across all tasks when the consultant is randomly assigned to argue for the correct/incorrect answer. Comparing debate to direct question answering, the results depend on the type of task: in extractive QA tasks with information asymmetry debate outperforms direct question answering, but in other tasks without information asymmetry the results are mixed. Previous work assigned debaters/consultants an answer to argue for. When we allow them to instead choose which answer to argue for, we find judges are less frequently convinced by the wrong answer in debate than in consultancy. Further, we find that stronger debater models increase judge accuracy, though more modestly than in previous studies.

1. Introduction

The paper tests whether debate lets weaker judges supervise stronger AI agents across a broader range of capability gaps than prior extractive-QA studies. Debate beats assigned-role consultancy, but its advantage over direct question answering depends on the task.

Motivation

  • Prior empirical debate studies largely focused on reading comprehension with information asymmetry, leaving its performance across broader capability gaps an open question.The paper notes that long-run judge-agent differences may include mathematical and logical reasoning, creativity, and factual knowledge.Source: §1 · p. 2

Study scope

  • The study evaluates extractive, closed, and multimodal tasks, varying oversight protocols and the capability gap between judges and agents.The main experiments use judges equal to or weaker than the models they judge.Source: §1 · pp. 2–3

Protocol setup

  • Assigned-role consultancy tests judges with consultants arguing correct and incorrect answers, while debate has two agents argue opposing answers before the judge.The assigned-answer comparison is designed to model a worst-case variant of RLHF in which human feedback cannot distinguish correct from incorrect answers.Source: §1 · p. 3
  • Open consultancy and open debate let agents choose an answer, testing whether weaker judges follow an agent's potentially mistaken choice.In open debate, the protagonist's answer is set to the answer that the same debater model chooses under direct QA.Source: §1 · p. 3

Findings

  • Debate consistently outperforms consultancy, but its advantage over direct question answering depends on task type and is small or absent outside extractive QA with information asymmetry.On extractive QA, debate beats QA without article but not QA with article; for other tasks, it has small or no advantage over QA without article when judges are weaker but not too weak.Source: §1 · p. 4
  • Across open-role protocols, judges are less often convinced by an incorrect choice in debate than in consultancy, and stronger debaters tend to raise judge accuracy.The increase with debater strength is more modest than in previous studies.Source: §1 · p. 4

2. Related Work

Prior work on debate and scalable oversight spans empirical evaluations, theory, and other LLM applications. This paper broadens empirical evaluation across tasks and protocols while focusing on weaker judges supervising stronger agents.

Empirical evaluations of debate protocols

  • Earlier debate evaluations reported mixed results, while the closest prior study primarily used QuALITY and found no advantage over standard QA on limited non-information-asymmetric datasets when judges matched debater strength.Other work found promising results with LLM debaters and judges on QuALITY, including with training.Source: §2 · pp. 4–5
  • This study expands on prior work with broader task and model coverage, open-role protocols, and more extensive ablations.The reported ablations found little effect from best-of-n sampling, few-shot prompting, chain-of-thought reasoning for judges, or answer order.Source: §2 · p. 5

LLM debaters and judges

  • Other LLM-debate research includes self-improvement and reasoning applications, whereas this paper evaluates weak-judge accuracy for scalable oversight.The related work also notes evidence that debate efficacy for reasoning tasks is no better than majority voting when using the same number of model responses.Source: §2 · p. 5

Scalable oversight evaluations

  • The task categories differ in information access: extractive tasks have information asymmetry, while closed and multimodal tasks do not.This distinction provides a basis for comparing oversight across task settings.Source: Table 1 · p. 6

3. Methods

The methods evaluate judge accuracy across tasks, protocols, and judge models, using binary-choice QA datasets and both direct-QA and agent-assisted conditions. The study also varies whether agents are assigned an answer or choose one.

3. Methods

  • The evaluation varies scalable oversight protocol, QA task, and judge model, measuring judge answers against ground-truth dataset labels.The tasks wrap binary-choice question answering datasets in prompts for the different protocols.Source: §3 · p. 6

3.1. Tasks

  • Extractive QA gives source articles to agents but not judges, closed QA supplies no article, and multimodal QA includes image and text inputs.The task set includes extractive, closed, and multimodal categories.Source: §3.1 · p. 6

3.2. Protocols

  • The protocols include direct-QA baselines, assigned-role consultancy and debate, and open-role variants in which an agent chooses an answer.The experiments are inference-only and measure judge accuracy and scaling trends as capability gaps increase.Source: §3.2 · p. 7

3.2.1. QA Protocols

  • Direct QA prompts the judge with a question and two answer options, with the article omitted or supplied for extractive tasks.The article-supplied protocol is restricted to extractive QA and gives the judge the full source information.Source: §3.2.1 · p. 7

3.2.2. Assigned-Role Protocols

  • In consultancy, one agent argues an assigned answer and answers judge questions; in debate, two agents argue opposing assigned answers over multiple rounds.The default debate uses three rounds, and the study does not evaluate interactive debate.Source: §3.2.2 · pp. 7–8

3.2.3. Open-Role Protocols

  • Open consultancy lets the consultant choose its answer, while open debate lets a protagonist choose and assigns the antagonist the opposing answer.The chosen answer is identified using the model's answer under direct QA.Source: §3.2.3 · p. 8

3.3. Models

  • The study uses Gemma7B, GPT-3.5, Gemini Pro 1.0, and Gemini Pro 1.5, with Pro 1.5 as agents in the first experiments and varied agents later.The first experiments use all listed models as judges; the later debater-strength analysis uses Pro 1.0 and Pro 1.5 as judges.Source: §3.3 · p. 8

4. Experiments

Across assigned-role tasks, debate consistently beats consultancy, but its advantage over direct QA depends on the task and judge. In open-role settings, debate is less vulnerable to an incorrect advocate, while stronger debaters yield only modest accuracy gains; most tested inference-time changes have little effect.

4.1. Assigned-Role Results

  • Debate beats consultancy across tasks, but versus QA without article its advantage varies: it is strongest on extractive tasks, mixed on closed tasks, and nonsignificant on multimodal tasks.On extractive tasks, QA with article remains best, and the weakest judge does no better with debate than QA without article.Source: §4.1 · p. 9
  • Reducing protocols from three turns to one had no significant accuracy effect, while BoN=4 increased Elo without significantly increasing judge accuracy.The authors conclude that their BoN implementation was not helpful for judge accuracy.Source: §4.1 · pp. 9–10
  • Six-shot prompting did not significantly improve judge accuracy, and chain-of-thought prompting tended to harm performance or have no significant effect.The proposed explanation is that weaker judges may disregard stronger agents’ verbalized reasoning when using their own chain of thought.Source: §4.1 · p. 10
  • Simultaneous and sequential debate showed no significant accuracy difference, and evaluating both answer orders reduced variance without changing mean accuracy.The authors suggest randomizing answer position is sufficient instead of evaluating both orders.Source: §4.1 · p. 10
  • These findings apply to inference-only protocols; evaluations of debate used for training may produce different results.The authors identify improvement over direct-QA baselines as a key challenge for future debate evaluations.Source: §4.1 · pp. 10–11

4.2. Open-Role Results

  • When advocates choose their answer, open debate more reliably identifies an incorrect choice than open consultancy, trading off some accuracy when the protagonist is correct.Open consultancy has greater or equal advocate win rates across tasks, but that does not always produce higher judge accuracy.Source: §4.2 · p. 12
  • Open consultants arguing for incorrect answers have lower judge accuracy than assigned-role consultants, plausibly because their questions are harder on average.The authors contrast self-selection with assigned answers independent of question difficulty.Source: §4.2 · p. 12

4.3. Persuasiveness in Debate

  • Stronger debater Elo is associated with marginally higher judge accuracy, especially on extractive tasks; closed tasks show less differentiation and the gain is narrower than earlier findings.The authors suggest judges depend more on debater statements in extractive tasks because they lack access to the information themselves.Source: §4.3 · pp. 12, 14

5. Conclusion

The study finds weakly promising signs for debate: stronger debaters are associated with higher judge accuracy, but the experiments are inference-only proxies for training. The authors identify important limits in evaluating training protocols and in approximating human oversight of superhuman AI.

5. Conclusion

  • Across tasks, stronger debaters’ Elo scores are associated with higher judge accuracy, but the effect is relatively weak.The authors characterize this as a weakly promising sign for debate as a scalable oversight protocol, based on inference-only experiments.Source: §5 · p. 14
  • The experiments use prompted models in an inference-only setting and provide limited evidence about the effectiveness of debate or consultancy as training protocols.Training would produce optima or equilibria under incentives that these experiments do not evaluate directly.Source: §5 · p. 15
  • The consultants and debaters were RLHF fine-tuned for qualities including honesty, which may hamper dishonest agents.The authors suggest studying the helpful-only setting.Source: §5 · p. 15
  • The study uses neither human judges nor AI that is superhuman on many capabilities, leaving uncertainty about judging superhuman-AI arguments.The authors specifically flag whether humans can follow and accurately judge such arguments.Source: §5 · p. 15
  • Future work should train debaters and consultants using the judge’s signal on the studied tasks.The authors describe this as the full setup needed to test scalable oversight methods.Source: §5 · p. 15

Appendices

The appendices provide supplementary experimental results, task descriptions, and materials for inspecting and reproducing the analyses.

Appendices

  • Appendix A presents additional results divided by extractive, closed, and multimodal task types.It also includes ablation plots and a comparison of open with assigned-role consultancy.Source: Appendices · p. 18
  • Appendix B breaks down results across nine individual tasks, including main results, ablations, open-role protocols, and invalid judge responses.The main results appear in Figure B.1; additional figures cover the other analyses.Source: Appendices · p. 18
  • The appendices document error analysis, statistical tests, Elo calculations, task details, passage verification, prompts, and debate and consultancy transcripts.These materials appear in Appendices C through I.Source: Appendices · pp. 18–19

A. Results by task type

The appendix reports statistical testing for the main task-type results and points readers to task-type ablations and an open-versus-assigned consultancy comparison.

A. Results by task type

  • Debate and consultancy judge accuracy differ with low p-values, except in the multimodal case, which contains only one task.The statistical tests concern the settings used for Figure 2 in the main text.Source: A. Results by task type · p. 19
  • Ablation plots from Section 4.1 are split by extractive, closed, and multimodal task type in Figures A.2–A.8.The appendix identifies these as task-type-specific visualizations of the Section 4.1 ablations.Source: A. Results by task type · p. 19
  • Figure A.9 compares open consultancy with assigned-role consultancy.Source: A. Results by task type · p. 19

B. Results by individual task

The appendix maps the individual-task result visualizations to the main results, ablations, open-role protocols, and invalid judge responses.

B. Results by individual task

C. Transcript Error Analysis

The transcript analysis examines mistakes by the strongest judge, Gemini 1.5 Pro, in 24 debates from closed QA tasks. Failures differed by task: judge logic errors predominated in ProntoQA, while TruthfulQA errors had several sources.

C. Transcript Error Analysis

  • 11 of 12 ProntoQA failures stemmed directly from judge logical errors, while the remaining failure was attributed to poor debating by the honest debater.The losing debater often identified flaws in the dishonest debater’s arguments, but the judge did not capitalize on that information.Source: C. Transcript Error Analysis · p. 19
  • 6 of 12 TruthfulQA errors involved natural question ambiguity that let the dishonest debater interpret the question in support of its answer.Other errors involved suboptimal honest debating, judge logic errors, unavailable verified information, or judge inattention.Source: C. Transcript Error Analysis · p. 19
  • TruthfulQA’s remaining failures included two cases where the dishonest debater encouraged the judge to focus on an incomplete part of the question.The other listed cases included two suboptimal honest debates, one judge logical error, and one requiring verified external information.Source: C. Transcript Error Analysis · p. 19

D. Error bars

The error-bar procedure first combines paired samples with switched answer order, then estimates 95% confidence intervals under an IID, approximately normal assumption within each model, protocol, and task split.

D. Error bars

  • For questions sampled twice with binary answer order switched, the procedure first averages the two identical questions.The paired samples may be correlated.Source: D. Error bars · pp. 19–20
  • Within each model, protocol, and dataset or task split, the procedure assumes the resulting data are IID and approximately normally distributed around the mean.Source: D. Error bars · p. 20
  • 95% confidence intervals are calculated from the sample data using the normal distribution.Source: D. Error bars · p. 20

E. Elo Calculation

The Elo calculation derives debater ratings from judged cross-play outcomes, then estimates uncertainty by bootstrapping and separates ratings by whether debaters were assigned correct or incorrect answers.

E. Elo Calculation

  • 512 cross-play debates across five debaters were judged by Pro 1.5 to form a win-rate matrix, from which aggregate Elo scores were calculated.The pairings were sampled across datasets equally, using one-turn simultaneous debates; the debaters included Gemini Pro 1.5 with BoN=1 and BoN=4.Source: E. Elo Calculation · p. 20
  • The expected win-rate represents the expected probability that one debater beats another, and aggregate Elo is fitted by optimizing negative log-likelihood against actual win-rates.The optimization uses BFGS.Source: E. Elo Calculation · p. 20
  • Confidence intervals use statistical bootstrapping with 500 seeds, and correct-Elo and incorrect-Elo scores treat each debater as two players assigned opposite answer correctness.The resulting ten-player setup yields corresponding Elo scores.Source: E. Elo Calculation · p. 20

F. Task details

This section details task-specific dataset construction and prompts across extractive, closed-answer, and multimodal QA, including how debate, consultancy, direct-answer prompts, and the best-of-N response scorer are adapted. It also describes passage and quote verification used to help judges assess evidence.

F.1. Extractive QA

  • Extractive QA covers QuALITY, BooLQ, and GPQA-extractive, with differing information availability and difficulty; prior QuALITY results also suggest verification may matter more than debate argumentation.QuALITY uses 400 binary questions from a filtered Gutenberg sci-fi subset; BooLQ offers shorter documents, and GPQA-extractive supplies expert explanations as evidence.Source: §F.1 · pp. 20–21

F.2. Closed QA

  • Closed QA includes binary-choice versions of MMLU, GSM8K, PrOntoQA, and GPQA, plus TruthfulQA questions designed to elicit imitative falsehoods.GSM8KQA pairs generated Python solutions with correct and incorrect answers by whether execution matches the dataset answer; PrOntoQA creates incorrect traces by negating Boolean statements.Source: §F.2 · pp. 21–22

F.3. Multimodal

  • MMMU contributes 2,035 college-level multimodal questions with text-and-image inputs and golden explanations.The subset enables a controlled comparison if an extractive MMMU version is added later.Source: §F.3 · p. 22

G. Passage verification tool

  • Extractive debaters provide article passages in tags, which a regex check marks verified or unverified in subsequent transcripts.Participants are told what the verification labels mean; the approach follows Khan et al. (2024) and Radhakrishnan (2023).Source: G. Passage verification tool · p. 22

H. Prompt Templates

  • Prompts adapted from Khan et al. (2024) replace “quote” with “passage”; closed tasks omit passage/quote references, multimodal prompts include the image, and consultants and debaters have a 150-word per-turn limit.Source: H. Prompt Templates · p. 22

H.1.1. Extractive Tasks

  • Extractive consultant prompts make the judge article-blind and instruct consultants to support their assigned answer with passages, answer judge questions, and avoid repeating evidence.The prompt treats verified passages and their logical integration as important to judging.Source: §H.1.1 · pp. 23–24

H.1.2. Closed Tasks

  • Closed-task consultant prompts remove article and passage references but retain assigned-answer advocacy, judge-question responses, and evidence-based argument guidance.For MMMU, the prompt adds the attached image to the question.Source: §H.1.2 · pp. 24–26

H.2. Debater Prompt

  • Debate prompts assign Alice to the first answer and Bob to the second, with extractive debaters instructed to use verified passages and critique opponents’ arguments.The extractive debate judge cannot see the article.Source: §H.2 · p. 26, §H.2.1 · pp. 26–27

H.2.2. Closed Tasks

  • Debaters receive opposing positions and argue for their assigned answer; their prompts guide opening arguments and later critiques or counterarguments, with extractive tasks also using article quotes.The prompts ask debaters to supply context for the judge and develop evidence-based arguments; the extractive rebuttal instructions ask them to identify critiques, counter them, and select disconfirming article quotes.Source: §H.2.1 · p. 27, §H.2.2 · pp. 28–29

H.3.1. Extractive Tasks

  • In extractive consultancy, the judge has not read the article, questions the consultant, and treats only verified article quotes as trustworthy when choosing an answer.The judge is instructed to ask an impactful question based on the prior exchange and assess the consultant’s evidence before giving a final answer.Source: §H.3.1 · pp. 29–30

H.3.2. Closed Tasks

  • For closed-task consultancy, the judge evaluates the consultant’s claims through questions and the prior transcript; the MMMU prompt also supplies an image.The judge prompt is adapted from Khan et al. (2024).Source: §H.3 · p. 29, §H.3.2 · p. 30

H.4.1. Extractive Tasks

  • In extractive debate, judges who have not read the article compare the debaters’ arguments and rely on verified rather than unverified quotes when choosing an answer.The judge prompt is adapted from Khan et al. (2024).Source: §H.4 · p. 30, §H.4.1 · p. 31

H.4.2. Closed Tasks

  • For closed-task debate, the judge is instructed to decide which debater is correct from the arguments and evidence, without the article-based framing.The prompt replaces the opening paragraphs used for extractive debate judging.Source: §H.4.2 · p. 31

H.5. Judge Prompt: QA without article

  • The direct QA prompt without an article asks the judge to select the more likely correct answer and consider each answer’s strengths and weaknesses.The prompt supports both answer-only and reasoning-plus-answer formats.Source: §H.5 · pp. 31–32

H.6. Judge Prompt: QA with article

  • The direct QA prompt with an article is limited to extractive tasks and asks the judge to assess the two choices using the supplied article.It also supports answer-only and reasoning-plus-answer formats.Source: §H.6 · p. 32

H.7. Best-of-N scorer prompt

  • The best-of-N scorer reuses debate judge prompts but selects the most persuasive response from a list rather than choosing an answer.The shown prompt presents four responses and requests the selected option in a fixed answer format.Source: §H.7 · p. 33

I.1. Transcript pre-processing

The appendix illustrates how debate transcripts are processed before presentation, including scratchpad removal and quote verification.

I.1. Transcript pre-processing

  • Example BoolQ debate arguments are shown before and after scratchpad removal and passage-quote verification.Verified and unverified quoted passages receive distinct tags after processing.Source: §I.1 · p. 33

I.2. Sample debates

The appendix provides sample transcripts and ablations of debate and consultancy settings, reporting little benefit from several setup changes and persistent positional bias despite reversing answer order.

I.2. Sample debates

  • Increasing the number of debate or consultancy turns shows no observed benefit.The figure compares performance across protocols, with lighter colors indicating fewer rounds.Source: Figure A.2 · p. 35
  • Best-of-N sampling yields no consistent debate-performance benefit.The ablation concerns best-of-N sampling on debater responses.Source: Figure A.3 · p. 35
  • Few-shot prompting and judge chain-of-thought show no consistent benefit, and chain-of-thought sometimes degrades performance.The default judge prompt uses zero shots and prediction without chain-of-thought.Source: Figure A.4 · p. 36, Figure A.5 · p. 36
  • Sequential and simultaneous turns show no significant performance difference.Simultaneous turns are the default; the comparison uses an effective Best-of-N setting of N=1.Source: Figure A.6 · p. 37
  • Using both answer orderings does not significantly change performance or resolve judges’ typical positional bias.A value of 1.5 in the positional-bias analysis indicates no bias.Source: Figure A.7 · p. 37, Figure A.8 · p. 38
  • Open consultancy chooses the correct answer in 88% of extractive, 84% of closed, and 71% of multimodal questions.The figure contrasts open and assigned-role consultancy, splitting assigned roles by whether the argued answer is correct.Source: Figure A.9 · p. 38
  • The open-protocol comparison plots judge accuracy against protagonist or consultant win rate, split by task and judge model.A separate figure splits judge accuracy according to whether the protagonist or consultant chose the correct or incorrect answer.Source: Figure B.9 · p. 47, Figure B.10 · p. 48
  • Example transcripts show a judge identifying an incorrect consultancy argument and selecting the appropriate answer in debate.The debate example also shows a dishonest debater attacking the question’s premises despite being assigned a flawed answer.Source: Figure I.3 · p. 53, Figure I.4 · p. 54
Loading 2407.04622v2…