Source-linked AI summary
AI Safety via Debate is Compromised by Cognitive Biases
Gefei Liu, Sonya Rashkovan, Sophia Lloyd George, Isaac Sheidlower, Serena Booth
TL;DR
Human evaluations can favor persuasive features over correctness, motivating debate as a way to scrutinize competing claims. This paper tests rhetorical bias interventions in detective-story debates and finds that, pooled across intervention types, they shift human judgments toward the manipulated side.
Problem
Debate relies on human adjudicators recognizing correctness, but features unrelated to correctness may systematically influence their judgments.
Source: §1 · p. 1Method
The authors construct 68 detective-story debates with known solutions and apply four rhetorical interventions to either the true-culprit or innocent-suspect side.
Source: §1 · p. 2, §3.1 · p. 4Results
Across the four intervention types, judgments shifted toward the manipulated side (p = .002) in a study with 369 participants.
Source: §1 · p. 2Takeaways & Limitations
Human adjudication remains sensitive to targeted argumentative manipulations even within debate, a vulnerability for judgments used in debate-based supervision.
Source: §1 · p. 2Because the LLMs were explicitly instructed to use the strategies, the study does not establish whether they emerge through training or how often they arise outside laboratory conditions.
Source: §7.1 · p. 10
Abstract
from arXiv · showhide
Reinforcement learning from human feedback (RLHF) has played a central role in making large language models responsive to human instructions. However, human evaluators often favor flattering or persuasive responses over truthful ones, creating incentives for models to appeal to evaluators at the expense of accuracy. AI safety via debate has been proposed as a way to improve the supervision of language models: in this paradigm, two agents argue opposing positions and challenge each other's claims, potentially exposing falsehoods to the adjudicator. A central premise of AI safety via debate is that truthful arguments are easier to defend than false ones under adversarial scrutiny. In this work, we investigate whether this advantage persists when debaters use rhetorical strategies that exploit biases in human judgment. Inspired by competitive debate, we construct 68 LLM-generated dialogues about detective mysteries with known culprits, spanning four interventions: anchoring, fallacy oversight, pro-jargon, and verbosity. We apply each intervention to either the side advocating for the true culprit or the side advocating for an innocent suspect, allowing us to distinguish influence on adjudication from correctness. In a study with 369 participants, we find that, pooled across bias types, these interventions significantly shift judgments toward the manipulated side. These findings expose a vulnerability in debate-based supervision: human adjudication is sensitive to manipulative rhetorical strategies.
1 Introduction
The paper asks whether rhetorical strategies that exploit cognitive biases can sway human judgments even when debate gives opposing arguments a chance to challenge one another. In 369 participants, the tested interventions shifted judgments toward the manipulated side, exposing a vulnerability in debate-based supervision.
1 Introduction
- Debate aims to improve evaluation by having competing agents challenge claims, but its effectiveness depends on human adjudicators determining correctness despite adversarial structure.The premise that truthful arguments are easier to defend may fail if correctness-unrelated features systematically sway adjudication.Source: §1 · p. 1
- The study tests whether targeted argumentative interventions influence human adjudication using 68 LLM-generated detective-puzzle debates with known solutions.Each intervention was applied to the side advocating either the true culprit or an innocent suspect, separating intervention influence from position correctness.Source: §1 · p. 2
- The interventions target verbosity, anchoring, fallacy oversight, and pro-jargon through controlled rubrics applied to AI-generated arguments.The study also examines intervention-type effects and qualitatively analyzes participant responses.Source: §1 · p. 2
- Across four intervention types, judgments shifted toward the manipulated side (p = .002) in a study with 369 participants.The authors identify this sensitivity as a vulnerability in judgments used as training signals for debate-based supervision.Source: §1 · p. 2
2 Related Work
Prior work motivates debate as a way to help human judges scrutinize competing claims, while also documenting biases in both human and LLM evaluation. This study tests whether deliberate rhetorical strategies advantage one side in debates with known correct answers.
2 Related Work
- Debate may help expose failures in outputs when opponents identify counterexamples, but that is useful only if judges recognize the flaw.The approach is proposed as a way to improve human judgments used to train AI systems.Source: §2 · p. 2
- Prior research finds human judgment susceptible to biases, including anchoring, hindsight bias, and errors in statistical reasoning, with checklists and explicit criteria proposed as remedies.Related work also studies AI-assisted evaluation, confirmation bias in oversight, and jury adjudication.Source: §2 · p. 2
- This study builds on calls to test human judgment under adversarial argumentation by examining rhetorical strategies in debates with known correct answers.Source: §2 · p. 3
- LLM evaluators can favor longer responses even when added text does not improve quality, and this study uses an LLM adjudicator to screen candidate debates for intervention effects.The study focuses on human adjudicators, providing a comparison with work on LLM evaluation biases.Source: §2 · p. 3
3 The Debate Setting
The paper constructs controlled debates from detective stories with known solutions, then manipulates argumentative style on either side to test its influence separately from correctness. Dialogues use six turns, and the corpus includes a shared baseline and manipulated variants.
3 The Debate Setting
- Each dialogue is a multi-turn exchange in which two LLM debaters argue for different suspects and an adjudicator identifies the culprit.The source stories provide a fixed evidence pool shared by both debaters, competing interpretations, and a known solution.Source: §3 · p. 3
- Three debate stages comprise scene reconstruction, arguments, and rebuttals, with one speech per side at each stage and no external evidence or fabricated facts.The openings emphasize different story facts; arguments support accusations and challenge explanations, while rebuttals respond without adding evidence.Source: §3 · pp. 3–4
3.1 Bias Interventions
- Four interventions target anchoring, verbosity, fallacy oversight, and pro-jargon; the study defines bias as an argumentative style intended to exploit a cognitive bias.Fallacy oversight includes seven selected fallacy subtypes; pro-jargon concerns credibility attributed to technical-sounding language without added substance.Source: §3.1 · p. 4
3.2 Experimental Conditions
- The corpus comprises 4 baseline and 64 manipulated dialogues, with each story’s unmanipulated dialogue shared across bias types and two variants per condition.Interventions are assigned to either side, while speaker position is treated as a design factor rather than a bias intervention.Source: §3.2 · p. 4
4 Dataset Construction
The dataset is built from detective stories and generated using separate plans for argument content and rhetorical style. Dialogues are checked for debate quality and whether LLM adjudicators shift toward the targeted intervention.
4 Dataset Construction
- Stories involving serious crime were excluded, and each story was segmented into uniquely identified sentences to ground generated content.The innocent suspect with the highest net reader guilt vote was selected as the culprit’s rival; reasons and interpretations were generated for each suspect.Source: §4 · p. 5
4.1 Dialogue Generation
- Content plans specify arguments and evidence, while style plans guide expression through definitions, assessment criteria, examples, and contrasts with baseline.Most interventions retain the baseline content plan; anchoring changes where information appears, and verbosity adds a supporting reason. Only the manipulated side’s argument and rebuttal turns are regenerated; opening turns remain shared.Source: §4.1 · p. 5
4.2 Dialogue Validation
- An LLM judge evaluates debate quality, and failed dialogues are regenerated; bias effectiveness is screened using six LLMs as simulated participants.A biased dialogue passes if at least one model shifts toward the bias and none shifts away; otherwise, it is regenerated.Source: §4.2 · pp. 5–6
5 Human Evaluation
The human evaluation tests whether bias-targeted arguments shift adjudicators’ choices, using debates with known solutions and a preregistered set of hypotheses. Participants adjudicate one debate, then complete measures that support an analysis of overall and bias-specific shifts.
5 Human Evaluation
- The study tests whether bias shifts judgments toward the bias-active side and whether that shift differs across anchoring, pro-jargon, fallacy oversight, and verbosity.These are the preregistered H1, H2a, and H2b hypotheses.Source: §5 · p. 6
5.1 Experimental Design
- Each participant adjudicates one debate, with assignments scheduled to balance five participants per manipulated dialogue and 20 per baseline dialogue.The planned allocation was N = 400.Source: §5.1 · p. 6
5.2 Participants
- Of 400 completed submissions, 31 were excluded for failing the story check or reading faster than 400 words per minute, leaving 369 participants for analysis.Participants were recruited through Prolific and had to meet English-fluency and approval-rate criteria.Source: §5.2 · p. 6
5.3 Procedure
- Participants practiced with feedback, read debate turns sequentially, and answered bias questions only after adjudicating to avoid priming.Adjudication was available after every turn had been marked read; participants then completed a story-comprehension check.Source: §5.3 · pp. 6–7
5.4 Measures
- The primary outcome is adjudication choice, with derived correctness and bias-side selection; additional measures include confidence, explanations, bias perceptions, background, and task checks.Ratings use seven-point scales.Source: §5.4 · p. 7
5.5 Analysis
- The analysis models selection of Side A against a signed predictor for which side receives the bias, testing its effect and estimating the probability-scale shift with a 95% bootstrap confidence interval.The predictor is +1 for bias on Side A, −1 for bias on Side B, and 0 in baseline debates; variations test individual biases and differences among them.Source: §5.5 · p. 7
6 Results
Across intervention types, adjudication shifts toward the side receiving the manipulation, while individual-bias and between-bias results remain inconclusive. Additional analyses find associations involving bias recognition, speaking order, and attention to physical evidence.
6.1 Primary Analysis
- 8.6 percentage points (95% bootstrap CI [+2.9, +14.2] pp): interventions shifted judgments toward the manipulated side, with improved model fit (χ2(1) = 9.31, p = .002).Because interventions targeted either the true-culprit or rival side, the shift favored manipulation rather than correctness.Source: §6.1 · p. 7
- 7.1–11.1 percentage points: all four bias-specific estimates favored the manipulated side, but none was statistically significant before or after Holm correction.The individual estimates are less precise because each uses only the judges who saw that bias.Source: §6.1 · p. 8
- 4.0 pp was the largest observed difference between bias effects; the omnibus test found no evidence that their magnitudes differed (χ2(3) = 0.32, p = .956).The results are inconclusive about whether effects meaningfully differ across bias types.Source: §6.1 · p. 8
6.2 Additional Findings
- 30% identified the biased side and 11% identified the bias type; participants who identified the side selected it less often (48% vs. 63%, p = .014).Correctly identifying the specific bias was also associated with less selection of the biased side (46% vs. 60%), but that difference was not statistically significant (p = .108).Source: §6.2 · p. 9
- Accuracy was higher when the ground-truth side spoke first (63% vs. 41%, p < .001), and the biased side was selected more often when it spoke first (67% vs. 50%, p = .004).Source: §6.2 · p. 9
- Participants who focused on physical evidence adjudicated correctly 77% of the time, compared with 46% for others.The authors describe directing attention toward evidence bearing on correctness as a potential way to assist adjudicators.Source: §6.2 · p. 9
7 Discussion
The discussion identifies cognitive-bias interventions as a vulnerability for debate-based supervision and considers procedural safeguards and risks in evaluating AI-generated content. The study’s detective-story setting and highly controlled debates constrain how far its findings extend.
7 Discussion
- Bias-inducing interventions can shift adjudications toward the side using them, potentially weakening the connection between winning a debate and correctness.Debate’s safety premise is that false claims are harder to defend than to refute, but systematic imperfections in human judgment may reward strategies that exploit them.Source: §7 · p. 9
- Judges could be asked to assess evidentiary support, logical validity, and responses to counterarguments, though these procedural changes warrant further evaluation.The authors suggest adapting checklists and structured evaluation criteria used in other settings, alongside approaches that model systematic biases.Source: §7 · pp. 9–10
- Rhetorical presentation can influence judgments even when an opposing argument can respond, raising concerns about using debate to evaluate AI-generated content.The interventions were not personalized; the authors note that personalization could create still more opportunity to manipulate content consumers.Source: §7 · p. 10
- Participants whose explanations focused on physical evidence adjudicated correctly more often, but this association does not establish that directing attention to such evidence improves accuracy.The authors suggest further testing evidence checks and retrieval-based cross-checking as possible supports for distinguishing persuasive presentation from sound reasoning.Source: §7 · p. 10
7.1 Limitations
- Because the study uses detective stories, its observed effects may differ in other domains, including scientific discourse.Scientific judgments may require assessing explanations and novel hypotheses rather than checking only a final answer.Source: §7.1 · p. 10
- Highly controlled debates isolated individual biases; the study does not establish whether such strategies emerge through model training or how often they arise outside laboratory conditions.Less constrained settings may involve interacting biases and subtler strategies.Source: §7.1 · p. 10
Ethics Statement
The study used an online Prolific sample, obtained informed consent, compensated participants, and analyzed de-identified responses; the task was assessed as minimal risk and not subject to formal IRB review.
Ethics Statement
- Participants gave informed consent, were compensated, and had their responses analyzed in de-identified form; the fictional detective task posed no more than minimal risk and was not subject to formal IRB review.The study was conducted online through Prolific, and the specific bias was introduced only after adjudication to avoid influencing the primary outcome.Source: Ethics Statement · p. 11
Data and Code Availability
This section documents the released dialogues, supplementary materials, analysis code, and de-identified participant responses, as well as the mystery records and procedures for constructing evidence, generating debates, and evaluating debate quality.
Appendix Table of Contents
- The appendices document human-evaluation procedures and analyses, including protocol deviations, participant flow, questionnaires, qualitative coding, primary analyses, and exploratory analyses.Appendix B.1 describes changes from the preregistered plan and the reasons for those changes.Source: Data and Code Availability · p. 11, Appendix Table of Contents · p. 17
A.2 Source Data and Preprocessing
- The data-construction documentation covers source stories, preprocessing, generation prompts and bias specifications, and validation procedures.Appendix A also records source-story records, culprit–rival scores, and evidence-unit construction choices.Source: Data and Code Availability · p. 11, §A.2 · p. 19
Data and Code Availability
- The authors release the final 68 dialogues, study materials, analysis code, and de-identified participant responses, and provide the code and supplementary materials online.They note that LLM outputs cannot be regenerated exactly and that the study was preregistered on OSF.Source: Data and Code Availability · p. 11
A.1 Experimental Conditions and Dialogue Inventory
- Table 2 maps all 68 dialogue IDs to their stories, bias types, conditions, and variants, with boldface indicating the culprit’s side.Conditions are listed in Side A–Side B order.Source: §A.1 · p. 18, Table 2 · p. 18
A.2.1 Source Story Records
- Tables 3–6 reproduce source records for the four stories used in the study.The records include story outcomes and clues used to construct the mystery dialogues.Source: §A.2.1 · p. 19, Table 3 · p. 19, Table 4 · p. 21
- The source-story records include a mystery in which Tony is revealed as the trophy thief after the stolen trophy is found in his car.The story’s clues include the thief’s opportunity to reach the lobby and motive to obtain money.Source: §A.2.1 · p. 28
A.2.2 Reader Judgments and Culprit–Rival Construction
- Reader-judgment records identify each story’s culprit and rival; for The Mystery of the Leprechaun’s Trophy, Tony is listed as culprit and Casey as rival.The table’s values are reader counts for guilty and innocent judgments and their difference.Source: §A.2.2 · pp. 28–29, Table 7 · p. 28
A.2.3 Evidence-Unit Construction
- Mysteries were segmented into sentence-level evidence units with pySBD [43] and dialogue-aware post-processing, then indexed in original order as generation references.The units are internal references and are not separate records in the released corpus.Source: §A.2.3 · p. 29
A.2.4 Suspect-Level Reason Generation
- Suspect-level reasons were generated with gpt-4.1 at temperature 0 and a 1,500-token maximum, using hidden outcome context to guide selection and interpretation of public evidence.Each reason required two distinct interpretations supported by its evidence cluster; hidden inputs were not cited or stated directly in generated information.Source: §A.2.4 · pp. 29–31
A.3.1 Content Plan Templates
- Content-plan templates specify each turn’s argumentative role and evidence allocation across variants.The template marks shared baseline structure and additions or replacements for anchoring- and verbosity-biased dialogues.Source: §A.3.1 · p. 34, Table 15 · p. 34
A.3.2 Style Plan and Bias Specifications
- Style plans specify anchoring, pro-jargon, verbosity, and fallacy-oversight biases; the seven fallacy subtypes were chosen from 12 common fallacies for clear expression in textual debate.The appendix provides definitions, evaluation metrics, examples, and contrasts to baseline for the style plans.Source: §A.3.2 · p. 35, Table 16 · p. 36, Table 17 · p. 36
A.3.3 Dialogue Rollout Instructions
- The dialogue-rollout prompt distinguishes shared instructions from opening-, argument-, and rebuttal-specific instructions.It directs turns to follow content and style plans while preserving specified evidence and claims.Source: §A.3.3 · pp. 39–40
A.4.1 Adapted Public Forum Debate Validation Rubric
- The adapted Public Forum rubric scores each side on six dimensions from 1 to 5 in 0.5-point increments, with both sides needing at least 27/30 to pass.The dimensions are clarity, organization, evidence, argumentation, responsiveness, and conduct; scores are summed to a maximum of 30.Source: §A.4.1 · p. 40
A.4.2 LLM-Participant Results
The appendix reports simulated LLM adjudications, protocol deviations, and the participant-to-dialogue assignment schedule. The supplied passages describe the tables and analysis changes but do not provide the LLM decision outcomes.
A.4.2 LLM-Participant Results
- Six LLMs simulated participants, and Table 31 reports their decisions across four stories, with accuracy counted out of six.Table 30 provides the adjudication prompt; Table 31 marks decision shifts and whether accuracy increased or decreased relative to baseline.Source: §A.4.2 · p. 42, Table 30 · p. 42, Table 31 · p. 43
B.1 Deviations from the Preregistered Protocol
- The executed protocol excluded story-check failures and readers above 400 words per minute; comprehension failures instead stopped before study entry.The preregistration did not specify its comprehension or attention checks; the executed study also changed the incentive to $0.50 and used four rollout checkpoints over 14 days.Source: §B.1 · p. 45
- The authors expanded H2 into individual-bias and between-bias questions, dropped H3, and did not analyze the available human–LLM comparison.The executed analysis reports a probability-scale average shift with a bootstrap confidence interval rather than the preregistered coefficients and odds ratios.Source: §B.1 · p. 45
B.2 Dialogue Assignment Schedule
- Figure 5 depicts assignments for 400 participants, made in the order participants completed the comprehension check.Dialogue IDs correspond to the inventory in Appendix A.1.Source: §B.2 · p. 45, Figure 5 · p. 46
B.3 Participant Flow and Sample Characteristics
The appendix details participant flow, the study interface and questionnaires, response coding, and supplementary primary-analysis tables. The final analysis included 369 participants after 31 exclusions.
B.3 Participant Flow and Sample Characteristics
- Of 400 completed submissions, 31 were excluded for failing the story check or reading above 400 words per minute, leaving 369 participants for analysis.Demographic summaries instead describe 400 Prolific-approved participants before study-specific exclusions; they covered 35 countries and ages 18–73 years.Source: §B.3 · p. 46
B.4 Participant-Facing Study Procedure and Questionnaire
- Participants completed background and comprehension questions, practiced with feedback, then read one assigned debate under timed, sequentially opened turns before choosing a side.They saw only the generated debate, received no correctness feedback on the main response, and could report confidence and reasoning; the questionnaire also asked about bias noticeability and perceived influence.Source: §B.4 · pp. 48, 50, 52, 54
B.5.1 Coding Procedure
- Two researchers independently coded free-text responses, then discussed disagreements; initial primary-reasoning agreement was 80%, and consensus resolved disagreements in that category.After consensus, task-frame and hedge agreement were 84% and 88%, respectively.Source: §B.5.1 · p. 57
B.5.2 Codebook
- Table 35 lists the free-text reasoning codebook and counts after coder consensus for 242 responses.Three non-English responses were excluded from the coded-response count.Source: Table 35 · p. 57, §B.5.2 · p. 57
C.1 Primary Analyses
- Tables 36–39 provide supplementary H1 and H2 quantities, including selection rates, regression coefficients, per-bias cell counts, and omnibus-comparison standard errors.The passages identify the tables’ contents but do not provide their numerical values.Source: §C.1 · p. 58, Table 36 · p. 58, Table 37 · p. 58
C.2 Bias Detectability
Participants had limited sensitivity to the manipulated biases, and detecting bias was not associated with greater culprit accuracy. Correctly identifying the bias-active side was associated with less bias-following.
C.2 Bias Detectability
- Bias detection ranged from 49.3–61.1% in manipulated debates versus a 48.6% baseline false-alarm rate, and specificity declined; verbosity was correctly identified by type by 20.8%.Table 40 summarizes the detection cascade across manipulation types.Source: §C.2 · pp. 58–59, Table 40 · p. 59
- At all three detection levels, judges who met the detection criterion were no more accurate about the culprit than those who did not.Separate logistic regressions found no significant association between any detection level and adjudication accuracy.Source: Table 41 · p. 59, §C.2 · p. 59
- Judges identifying the bias-active side selected it less often than other judges (47.7% vs. 63.2%, p = .014); the bias-type comparison was not significant.For the 33 judges correctly identifying the bias type, bias-following was 45.5% versus 60.2% (p = .108).Source: Table 42 · p. 59, §C.2 · p. 59
C.3 Position Bias
Speaking order was associated with both adjudication accuracy and bias-following. Confidence did not meaningfully track accuracy, and self-reported expertise offered no consistent protection against the manipulation.
C.3 Position Bias
- Accuracy was 63% when the ground-truth side spoke first versus 41% when it spoke second; bias-following was 50% when the biased side spoke second versus 67% when it spoke first.Without a bias manipulation, accuracy was 75% with the ground-truth side first and 32% second; speaking order was balanced to estimate position effects separately from ground truth and bias placement.Source: §C.3 · pp. 59–60, Table 43 · p. 60
C.4 Confidence
- Accuracy did not differ meaningfully by confidence, and judges at the highest confidence levels were less than 50% accurate.The estimated odds of choosing the biased side increased by 19% per confidence point (OR = 1.19, 95% CI 1.00–1.42, p = .048, n = 297).Source: Figure 7 · p. 60, §C.4 · p. 60
C.5 Expertise
- Adjudication accuracy did not increase with any self-reported expertise measure; high detective-story familiarity corresponded to 42% accuracy versus 58% for low familiarity.The comparison was OR = 0.53, 95% CI 0.28–1.00, p = .048.Source: Figure 8 · p. 61, §C.5 · p. 61
- None of the four expertise measures consistently protected against bias-following, and familiarity with the specific bias had no effect (OR = 1.00, 95% CI 0.87–1.14, p = .946).Moderate debate-judging experience showed suggestive higher bias-following than little or extensive experience (quadratic p = .077).Source: Figure 8 · p. 61, §C.5 · p. 61
C.6 Free-Text Adjudication Rationales
Free-text rationales suggest that reasoning basis sometimes tracked adjudication accuracy, but generally did not shield judges from bias-following. Judges who focused on logical inconsistencies were a notable exception.
Adjudication accuracy
- Physical or material evidence was the primary reasoning basis in cases adjudicated correctly 77% (43/56) of the time, versus 46% (86/186) for all other codes.The authors describe this as a trend and caution that the small sample across primary adjudication codes limits the inference.Source: §C.6 · p. 62
- Inconsistency and character codes clustered in the highest-accuracy story, complicating comparisons between reasoning codes and accuracy.Physical evidence appeared across all four stories, but was rare in the two stories with the lowest accuracy.Source: §C.6 · p. 62
- In ‘Our Quarterback Is Missing,’ judges invoking physical evidence were correct 63% (12/19), versus 76% (31/41) citing other evidence; in ‘The Diamond Necklace,’ physical-evidence judges were correct 85% (29/34), versus 25% (4/16) without it.Varying story difficulty limits interpretation of the physical-evidence code’s relationship to judge accuracy.Source: §C.6 · p. 62
Bias-following
- Judges sided with the biased debater in 62% (121/196) of coded responses from biased debates, and primary reasoning codes generally did not protect them.Figure 9 reports adjudication accuracy and bias-following by primary adjudication code.Source: Figure 9 · p. 62, §C.6 · p. 63
- Inconsistency-focused judges chose the biased debater less often (25%, 2/8) than other judges (63%, 119/188) and more often identified the biased debater afterward (50%, 4/8 versus 29%, 55/188).The authors connect this result to judges’ use of “logical inconsistency” in their explanations.Source: §C.6 · p. 63
Free-text responses
- Only 13 of 194 coded responses from biased debates directly alluded to bias; judges who did so followed the biased side at a similar rate to those who did not.The coding covered primary reasoning, bias allusion, task frame, and hedging.Source: §C.6 · pp. 61, 63