Source-linked AI summary
Single-Turn Debate Does Not Help Humans Answer Hard Reading-Comprehension Questions
Alicia Parrish, Harsh Trivedi, Ethan Perez, Angelica Chen, Nikita Nangia, Jason Phang, Samuel R. Bowman
TL;DR
The paper examines whether debate-style explanations can help humans judge difficult QA answers when ground truth is hard to verify. It builds evidence-supported arguments for correct and incorrect options and tests them in long-context multiple-choice QA. The explanations do not improve accuracy, whereas human-selected snippets do, motivating alternative debate setups.
Problem
QA systems often produce believable but false answers, and humans may be unable to verify them when ground truth is difficult or time-consuming to obtain.
Method
The authors collect evidence-supported arguments for both correct and incorrect options from long-context QuALITY questions and test human judgments under a 90-second limit.
Results
The debate-style explanations do not improve human accuracy, while the passage-plus-snippet baseline produces the highest accuracy across conditions.
Takeaways & Limitations
The findings motivate future debate data collection that changes the task setup and better supports human evaluation of QA explanations.
Takeaways & Limitations
The study is a limited case study and is not strong evidence against the broader potential usefulness of debate-style explanations.
Abstract
from arXiv · showhide
Current QA systems can generate reasonable-sounding yet false answers without explanation or evidence for the generated answer, which is especially problematic when humans cannot readily check the model's answers. This presents a challenge for building trust in machine learning systems. We take inspiration from real-world situations where difficult questions are answered by considering opposing sides (see Irving et al., 2018). For multiple-choice QA examples, we build a dataset of single arguments for both a correct and incorrect answer option in a debate-style set-up as an initial step in training models to produce explanations for two candidate answers. We use long contexts -- humans familiar with the context write convincing explanations for pre-selected correct and incorrect answers, and we test if those explanations allow humans who have not read the full context to more accurately determine the correct answer. We do not find that explanations in our set-up improve human accuracy, but a baseline condition shows that providing human-selected text snippets does improve accuracy. We use these findings to suggest ways of improving the debate set up for future data collection efforts.
1 Introduction
The paper asks whether debate-style explanations with evidence can help humans judge difficult QA answers, especially when ground truth is hard to verify. It constructs opposing explanations for correct and incorrect options and finds no accuracy improvement in this setup.
- 1 Introduction: Current QA systems can produce believable but false answers, leaving humans unable to verify them when ground truth is inaccessible.The problem is especially relevant for dense technical text and long passages whose answers require substantial reading time.
- 1 Introduction: The paper creates debate-style explanations by collecting evidence-supported arguments for both predetermined correct and incorrect answer options.The dataset targets long-context multiple-choice questions from QuALITY and tests whether these explanations improve human judgments.
- 1 Introduction: The explanations do not improve human accuracy compared with baseline conditions without explanations.The authors release the dataset and suggest modifying the task setup in future work.
- 1 Introduction: The longer-term goal is to train models that explain why an answer is correct and cite understandable, helpful supporting evidence.
2 Related Work
Prior work studies model-generated explanations and their effects on human-AI collaboration, but less work examines explanations as tools for verifying predictions humans cannot independently check. This dataset addresses that gap by pairing evidence and explanations with both correct and incorrect options.
- 2 Related Work: Earlier studies generate explanations, but relatively little work uses them to verify predictions when humans cannot perform the task directly.Related applications include medical diagnosis, data annotation, and deception detection.
- 2 Related Work: Explanation helpfulness depends partly on complexity, while users also want explanations of why models mark answers incorrect.The dataset responds by providing evidence and explanations for both correct and incorrect options.
3 Argument Writing Protocol
The authors build a long-context QA dataset by having experienced writers argue for paired answer options and select supporting passage snippets. The released collection contains balanced correct and incorrect arguments from QuALITY passages.
- Argument Writing Protocol: The protocol presents each argument as a human-written, evidence-supported case for one of two answer options.This creates the single-turn debate-style format evaluated by the paper.
- Argument Writing Protocol: Writers select incorrect options that humans often choose under time limits or regard as strong distractors after reading the full passage.The source passages are primarily science-fiction stories of about 5,000 words with four-option questions.
- Argument Writing Protocol: Fourteen experienced writers construct arguments of up to 500 characters and select one to three supporting snippets of up to 250 characters.Correct and incorrect options are assigned to writers at roughly equal rates.
- Argument Writing Protocol: The released dataset contains 2,944 arguments, split evenly between correct and incorrect arguments, from 112 unique passages.It also includes selected snippets and judgment-experiment results, while excluding arguments from non-public test passages.
4 Judging Protocol
Human judges compare two answer options under a 90-second limit across passage, snippet, and argument conditions. The experiment finds that snippets improve accuracy, while adding arguments provides no clear benefit and may reduce performance relative to snippets alone.
- Judging Task Protocol: Judges select which of two answer options is correct after viewing the question, passage, arguments, and linked snippets within 90 seconds.Clicking a snippet scrolls to and highlights its source passage section.
- Baselines: The main protocol compares passage-plus-snippet-plus-argument, passage-plus-snippet, and passage-only conditions using counterbalanced rounds.Workers do not judge the same question in multiple conditions.
- Comparison Across Conditions: Workers are most accurate with passages and snippets, while arguments show no clear benefit and the passage-only condition performs worst.Both unanimous and majority-vote agreement produce the same ordering across conditions.
5 Results
Workers were most accurate when given text snippets, while adding arguments produced no clear accuracy benefit; accuracy rose only slightly over time and was unrelated to average task time.
- Comparison Across Conditions: Workers were most accurate with text snippets alone (PS), while adding arguments produced no clear accuracy benefit.Both unanimous and majority-vote agreement showed the same ordering: PS highest and P lowest.
- Effects of Time: Accuracy improved by about 4 percentage points between workers’ first and final 10 tasks in every condition.The small increase may reflect familiarity with the task format or a moderately effective strategy.
- Effects of Time: Most workers answered before 90 seconds, but average task time did not correlate with average accuracy.Median completion times were 69 seconds in P, 54 seconds in PS, and 57 seconds in PSA.
- Follow-up Survey: Among surveyed workers, those who reported reading snippets had significantly higher accuracy, whereas reported argument reading was not associated with PSA accuracy.A quarter of workers reported mistrusting the arguments, but mistrust did not correlate with performance.
6 Discussion
The authors interpret the null accuracy result as limited evidence about debate-style explanations, citing independent arguments, artificial timing, and possible harms from misleading information.
- Scope: The study’s limited scope means its null result is not strong evidence against the broader usefulness of debate-style explanations.The authors consider explanations potentially beneficial in some tasks and conditions.
- Risks: Because QA models already provide false and misleading information, misleading explanations may be more harmful when users are not told they can be wrong.A quarter of workers mistrusted the arguments, raising a concern about this setup.
- Future Debate Set-ups: Independent correct and incorrect arguments may have been more misleading than helpful because effective debate could require explicit counterarguments.The authors leave whether multi-turn debate is needed as a question for future research.
- Task Design: The 90-second limit made the task more artificial, although pilot variations from 60 to 120 seconds produced virtually no performance difference.The authors note that 120 seconds may still have been too short and that accuracy was slightly lower at 120 than 90 seconds.
7 Conclusion
The paper tested whether opposing answer arguments could improve human accuracy without full-passage reading and found little to no setup effect, while releasing supporting data and identifying future directions.
- 7 Conclusion: The task setup had little to no effect on human accuracy when users judged opposing arguments for multiple-choice answers without reading the passage.The conclusion frames this result as raising questions about when such explanations may be useful.
- 7 Conclusion: The study released a dataset of correct and incorrect arguments, selected snippets, and judgment results for future research.The data collection used writer incentives, feedback, and a structured interface for selecting snippets and writing arguments.
- 7 Conclusion: Questions judged ambiguous in QuALITY metadata were removed after writers constructed arguments for 11–15 questions per passage.The released passages therefore contain 7–15 two-option questions, averaging 13.3.
C Judging Task Crowdworker Recruitment
The study recruited and screened writers and judges, then evaluated three timed judging conditions with catch trials and varying time limits.
- Judging Task Protocol: Clicking a snippet revealed the relevant passage section with matching color-coded highlights, while timer expiration removed the information and left 30 seconds to answer.If the final timer also expired, the task auto-submitted without a selection and was marked incorrect.
- Catch Trials: Catch trials were mixed into about 10% of tasks and were designed to have an obvious answer under the strict time limit.Among workers completing at least five catch trials, median accuracy was 88.9%, and 6.2% lost qualification for low catch-trial accuracy.
- Results with Different Time Limits: Longer time limits produced only a small accuracy increase overall, with the clearest improvement in P rather than PSA.The authors therefore conclude that PSA performance was likely not strongly driven by the time available to read arguments.
G Effect of Question Selection Method
The analysis examines whether the method used to select incorrect options explains condition differences, finding no compelling evidence that writer argument quality accounts for the observed pattern.
- G Effect of Question Selection Method: The study compared incorrect options chosen as strong distractors in QuALITY’s time-limited validation with those selected by untimed validators.About half of the examples met both selection criteria.
- G Effect of Question Selection Method: Workers were slightly less accurate on questions whose incorrect option was selected as the best distractor by untimed validators, but this does not explain the PSA comparison.The authors consider it unlikely that the difference arose because writers constructed better arguments for those questions.
- G Effect of Question Selection Method: The observed P pattern was opposite the expected effect relative to QuALITY’s comparable time-limited task, and the authors have no compelling explanation.They suggest that more time, fewer answer options, or easier searching might contribute, but this does not explain why QuALITY time-limited questions had the highest accuracy.
H Per-Worker Results
Worker performance varied substantially, and performance in one condition significantly predicted performance in the others, indicating that differences were more individual than question-driven.
- H Per-Worker Results: Individual worker performance varied substantially across the judging task.The variation may reflect differences in identifying useful search terms or recognizing answer-relevant patterns.
- H Per-Worker Results: Performance in each condition significantly predicted the same worker’s performance in the other conditions, rather than being explained mainly by question difficulty.The authors interpret this relationship as evidence that individual differences contributed more than noise from easier versus harder questions.
I Additional Results
Additional analyses show that workers improve modestly with task familiarity, but spending more time does not clearly improve accuracy. Workers favor longer arguments and arguments with more supporting snippets, while argument-word statistics are not informative and some workers mistrust arguments enough to ignore them.
- Improvements Over Time: Workers’ accuracy increases by about four percentage points as they complete more tasks, with most gains after the first 20 and similar effects across conditions.The analysis includes workers who completed at least 50 tasks in a condition.
- Accuracy by Time Spent on Tasks: There is no clear accuracy advantage to spending more time on tasks, although PSA shows a very slight, nonsignificant positive correlation.The figure plots each worker’s average accuracy against average time spent per task.
- Length of Arguments and Snippets: In PSA, each additional argument word raises answer-selection rates by about 1.2 percentage points relative to P, while each additional snippet raises them by 4.2 points; the corresponding PS snippet effect is 2.8 points.The argument-length effect is significant at p = 0.001, and the snippet effects differ significantly from analogous rates in P (p < 0.001 and p = 0.01).
- Effective Argument Words: Common words and bigrams in correct arguments do not distinguish arguments by how often judges choose them.The most common words recur across arguments chosen 0, 1, 2, or 3 times, while common bigrams are not frequent enough to be informative.
- Survey Results Discussion: Mistrust: Nearly a quarter of surveyed workers reported mistrusting and ignoring the arguments, an unintended consequence of presenting convincing explanations known to be false.Although mistrust may be a logical strategy, ignoring the arguments prevented them from serving as intended evidence.