PaperlayerView exact revision on arXiv

Source-linked AI summary

What Question Answering can Learn from Trivia Nerds

Jordan Boyd-Graber, Benjamin Börschinger

arXiv:1910.14464v3cs.CLcs.AIcs.LG

TL;DR

Question-answering research relies heavily on evaluation, but trivia tournaments have spent decades learning how to create fair and effective competitions. This paper applies trivia practices to QA datasets and argues that ambiguity and poor discrimination can weaken comparisons between systems.

  • Problem

    QA research emphasizes evaluation and leaderboards, while the trivia community’s decades of experience running competitions have largely been overlooked.

    Source: §1 · p. 1

  • Method

    The paper analyzes QA datasets through the lens of trivia tournaments and reviews how Quizbowl practices could inform QA.

    Source: §1 · p. 1

  • Results

    The paper argues that ambiguous questions and questions that fail to distinguish systems can undermine leaderboard comparisons.

    Source: §2.3 · p. 3, §3.1 · p. 4

  • Takeaways & Limitations

    Playtesting can expose dataset artifacts and scoring problems, while reporting secondary metrics can reveal trade-offs hidden by a primary leaderboard score.

    Source: §2.1 · p. 2, §2.2 · p. 3

    Quizbowl’s answer-timing requirement and complex pyramidal structure can hurt popularity and make some questions difficult to ask.

    Source: §4 · p. 8

Abstract

from arXiv · show

In addition to the traditional task of getting machines to answer questions, a major research question in question answering is to create interesting, challenging questions that can help systems learn how to answer questions and also reveal which systems are the best at answering questions. We argue that creating a question answering dataset -- and the ubiquitous leaderboard that goes with it -- closely resembles running a trivia tournament: you write questions, have agents (either humans or machines) answer the questions, and declare a winner. However, the research community has ignored the decades of hard-learned lessons from decades of the trivia community creating vibrant, fair, and effective question answering competitions. After detailing problems with existing QA datasets, we outline the key lessons -- removing ambiguity, discriminating skill, and adjudicating disputes -- that can transfer to QA research and how they might be implemented for the QA community.

1 Introduction

The paper treats QA datasets and their leaderboards as analogous to trivia tournaments, which have developed practices for identifying strong answerers. It proposes adapting those practices to QA research without replacing QA’s distinct goals.

1 Introduction

  • QA datasets and leaderboards resemble trivia tournaments: questions are written, participants answer, and a winner is declared.Trivia organizers have learned from past mistakes and built a community that reliably identifies strong question answerers.Source: §1 · p. 1
  • Tournament safeguards can help QA datasets make questions clear, unambiguous, and rewarding of knowledge.The paper presents these safeguards as lessons for QA, not as a reason for academic QA to surrender to trivia questions or its community.Source: §1 · p. 1
  • The paper also reviews Quizbowl as a format that may inform traditional QA.It closes by discussing how the trivia community’s expertise, pedantry, and passion could benefit research using fun, fair, and good questions.Source: §1 · p. 1

2 Surprise, this is a Trivia Tournament!

The paper maps trivia-tournament practices onto QA research, emphasizing playtesting, evaluation that reflects multiple abilities, discriminative questions, and the effects of annotation error. These practices offer ways to make QA datasets and leaderboards more useful.

2 Surprise, this is a Trivia Tournament!

  • Trivia tournaments and QA research share a sequence of creating questions, presenting them, collecting answers, and determining a winner.The paper argues that tournament practices can inform QA, particularly in making competitions fun, well-calibrated, and discriminative.Source: §2 · pp. 1–2

2.1 Are we Having Fun?

  • Playtesting lets dataset creators find artifacts, scoring problems, dull questions, and shortcuts before they undermine an evaluation.The authors cite SQuAD questions that can be frustrating or invite Ctrl+F shortcuts, and a SearchQA answer key that rejects “Samuel Clemens” in favor of “Mark Twain.”Source: §2.1 · p. 2

2.2 Am I Measuring what I Care About?

  • A single score can conceal trade-offs among QA abilities; reporting separate measures makes those trade-offs visible.FEVER’s 2018 shared task prioritized correct final answers while also reporting justification precision and recall; its most precise system ranked fourth on the primary leaderboard.Source: §2.2 · p. 3

2.3 Do my Questions Separate the Best?

  • Questions answered correctly or incorrectly by nearly every system do little to distinguish participants; questions with answer probability near 0.5 are more discriminative.The paper calls these “Goldilocks” questions and argues they matter both for learning and for leaderboard rankings.Source: §2.3 · p. 3

2.4 Why so few Goldilocks Questions?

  • Annotation error concentrated on discriminative questions can make a dataset less useful for deciding leaderboard winners.The effective dataset proportion ρ captures the share of questions that are useful for determining the winner; Figure 1 contrasts datasets with the same 0.16 annotation error but different discrimination and error distributions.Source: §2.4 · pp. 3–4, Figure 1 · p. 3
  • Pyramidality is offered as a way to make individual questions more discriminative when a dataset needs many questions.The paper points to Quizbowl’s long questions as the format in which trivia writers use this technique.Source: §2.4 · p. 4

3 The Craft of Question Writing

Question writing is a craft, yet QA datasets often rely on wild data or untrained crowdworkers, which can introduce ambiguity and shortcuts. The section recommends clearer questions, human adjudication, and focused attention to questions most likely to distinguish systems.

3 The Craft of Question Writing

  • QA datasets often use wild or crowdworker-written questions, whose quality variation and idiosyncrasies can create machine-learning shortcuts.The section argues that question writing requires craft and that question ambiguity can frustrate answerers and undermine assessment.Source: §3 · p. 4, §3.1 · p. 4

3.1 Avoiding Ambiguity and Assumptions

  • Natural Questions’ hockey query admits multiple interpretations, while its gold answers assume men’s ice hockey and omit women’s and field hockey.Unacknowledged assumptions can add noise and bias leaderboard rankings, potentially equating dataset assumptions with correctness; computers may find ambiguity where humans do not.Source: §3.1 · pp. 4–5
  • When all questions discriminate, Figure 2 indicates 2,500 test questions suffice; when three-quarters are too easy, too hard, or erroneous, 15,000 are needed.The required test size also depends on the accuracy difference and average accuracy of the systems.Source: Figure 2 · p. 5

3.2 Avoiding superficial evaluations

  • Trivia writers clarify acceptable answers and requested answer types, while QA datasets often rely on exact strings or token-overlap F1.Because no answer set is perfect, the section describes resolving protests through tournament rules and mutually agreed experts when staff cannot settle a dispute.Source: §3.2 · pp. 5–6
  • In a manual inspection of 100 sampled XLNet-123 wins, around two-thirds were judged “spurious”: 56% essentially identical, 7% omitted a correct alternative, and 5% were bad questions.This was despite an almost four absolute F1 development-data margin (94 vs 98).Source: §3.2 · p. 6
  • After systems submit simultaneously and questions and answers are revealed, authors can protest rulings before metrics are computed; a new, smaller annual test set would gradually refine annotations and process.The authors present this as an alternative for MRQA while acknowledging the limitations of automatic evaluation.Source: §3.2 · p. 6

3.3 Focus on the Bubble

  • Dataset creators should spend extra effort on questions likely to distinguish top systems, without removing all easy or hard questions and introducing sampling bias.The section notes that QA dataset authors do not appear to give these questions special attention.Source: §3.3 · pp. 6–7

4 Why Quizbowl is the Gold Standard

The paper presents Quizbowl as a source of practices that QA research can borrow, not a format it expects mainstream QA to adopt universally. Its interrupted, pyramidal questions reward deciding when to answer, while their complexity can make them harder to use across tasks.

4 Why Quizbowl is the Gold Standard

  • The authors argue that QA can borrow fair-evaluation practices from Quizbowl without expecting mainstream QA to adopt the format universally.Source: §4 · p. 7
  • Quizbowl uses interruptible questions: players signal when ready, gain points for a correct answer, and lose points for a wrong one as the question continues.Its buzzers discriminate knowledge, unlike Jeopardy!’s buzzer, which is available only after the question ends.Source: §4 · p. 7
  • Pyramidal questions move from difficult clues to accessible ones, requiring answerers to judge after each word whether they know enough to respond.This places discrimination within each question and is described as yielding a nearly 1.0 effective-question proportion.Source: §4 · p. 7
  • Quizbowl questions are created in phases: authors build answer clues, subject editors remove ambiguity and tune discrimination, and packetizers ensure diversity and uniform difficulty.Source: §4 · p. 7
  • Trivia expertise can clarify ambiguous clues, while the paper maintains that QA systems must still cope with real-world questions’ inherent noise and ambiguity.The authors distinguish coping with those complexities from glossing over them.Source: §4 · pp. 7–8
  • Complicated Quizbowl requires choosing when to answer, improving discrimination but potentially reducing popularity because its code cannot be copied directly from other QA tasks.Its pyramidal structure can also make some questions difficult to ask, although the underlying mechanisms share properties with other tasks.Source: §4 · p. 8

5 A Call to Action

The authors recommend borrowing trivia practices to improve QA tasks and leaderboards, from question design and review to community participation.

Talk to Trivia Nerds

  • Partnering with trivia experts can pool question-writing resources, develop computer-challenging questions, and acknowledge contributors’ role in dataset creation.The authors also point to tools that could help trivia writers and to expertise available through public platforms.Source: §5 · p. 8

Eat Your Own Dog Food

  • Human play-testing helps expose hidden assumptions and unfair metrics, while human shortcuts can also reveal routes computers may exploit.The authors recommend that coauthors play the task and compare scores; they caution that human and computer skills differ.Source: §5 · p. 8

Won’t Somebody Look at the Data?

  • Discuss individual questions after dataset release and enable downloading development predictions so researchers can inspect question quality and system behavior.The authors frame question discussion as a way to reinforce norms for precise, well-written questions.Source: §5 · p. 9

Make Questions Discriminative

  • Make questions challenging but not impossible, using progressively easier views or added metadata to improve discrimination and reveal answerability.Examples include increasing image resolution or adding coreference, disambiguation, or knowledge-base alignment.Source: §5 · p. 9

Embrace Multiple Answers or Specify Specificity

  • Specify required answer specificity or acceptable alternatives, and make unavoidable ambiguity explicit rather than silently overlooking it.The authors suggest clarifying questions or enumerating acceptable answers when multiple answers are possible.Source: §5 · p. 9

Revel in Spectacle

  • For complex evaluations, recurring human-judged events may help test sets reflect iterative system development and make failures visible to researchers and the public.The authors cite yearly TRECQA evaluations and the 2019 FEVER iteration as examples.Source: §5 · p. 9

Be Honest in Crowning QA Champions

  • Recognize distinct model strengths and resource constraints, and compare systems with appropriate experts rather than distracted crowdworkers.Possible distinctions include training data, runtime, or builder background; expert players can also advise how to fix a dataset.Source: §5 · p. 10

Appendix

Appendix figure, footnote, and table numbering continues from the main article.

Appendix

  • Appendix figure, footnote, and table numbers continue the numbering used in the main article.Source: Appendix · p. 14

A An Abridged History of Modern Trivia

Modern trivia grew from postwar popular culture into organized competitions with formalized practices aimed at making outcomes more rigorous.

A Abridged History of Modern Trivia

  • Modern US trivia expanded after World War II through game shows, including College Bowl, which preceded Quizbowl.Source: A An Abridged History of Modern Trivia · p. 14
  • Cheating scandals and trivia’s entry into intercollegiate competition prompted the community to develop more formal structures.The passage highlights the Van Doren scandal as a prominent controversy.Source: A An Abridged History of Modern Trivia · p. 14
  • As experienced players entered adulthood, trivia organizations created routes for top players to guide competitions toward more rigorous outcomes.By 2019, these organizations had institutionalized the practices described in the paper.Source: A An Abridged History of Modern Trivia · p. 14

B Simulating the Test Set Needed

The appendix simulates head-to-head systems to estimate the test-set size needed to identify a better performer, emphasizing how discriminative-question prevalence affects that size.

B Simulating the Test Set Needed

  • The simulation models systems with accuracies separated by delta and samples their results from binomial distributions over a test set scaled by the discriminative-question proportion.The setup compares System A and System B, with test-set size N and effective dataset proportion rho.Source: B Simulating the Test Set Needed · p. 14
  • The simulation seeks the minimum test-set size that ranks the better system higher in 95% of experiments, using an experiment size of 5000.Source: B Simulating the Test Set Needed · p. 14
  • A smaller proportion of discriminative questions requires a larger test set to detect the better system.Source: B Simulating the Test Set Needed · p. 14

C Qualitative Analysis Examples

The appendix introduces examples of classes of XLNet-123 wins over XLNet and explains how answer spans are marked.

C Qualitative Analysis Examples

C.1 Insignificant and significant span differences

Two answer-span examples show that Token-F1 can score a contextually sufficient answer below an answer omitting essential context. The section also cautions that test-set size is only one consideration.

C.1 Insignificant and significant span differences

  • “A majority of all MEPs” is contextually equivalent to “majority,” yet receives Exact Match 0 and Token-F1 0.4.Source: §C.1 · p. 14
  • “Significant resources” omits relevant context and should not receive full, or possibly any, credit despite its Token-F1 of 0.57.Source: §C.1 · p. 14
  • Test-set size is only one consideration; demographic balance, linguistic variation, and task-specific phenomena may also matter.Source: §C.1 · p. 14

C.2 Missing Gold Answers

The examples identify missing gold answers as a problem for static answer lists and argue that disputed cases need adjudication.

C.2 Missing Gold Answers

  • The authors observed 7 missing gold answers among 100 cases.Source: §C.2 · p. 15
  • “Jury nullification” is a plausible answer absent from the two annotated gold spans for the civil-disobedience question.Source: §C.2 · p. 15
  • Because individual cases may turn on subtle wording, static answer strings without adjudication do not capture QA’s complexity in official comparisons.Source: §C.2 · p. 15

C.3 Bad Questions

The examples distinguish annotation errors from questions whose wording or evidence makes them problematic. The authors argue that such cases warrant discussion rather than silent scoring.

C.3 Bad Questions

  • The authors observed 5 genuinely bad questions.Source: §C.3 · p. 15
  • The Selmur catalogue question’s correct answer, the Programming Library, is absent from the paragraph, making this an annotation error.Source: §C.3 · p. 15
  • Meaning-changing typos, such as “kind” for “king,” raise whether systems should charitably reinterpret question text.Source: §C.3 · p. 15
  • Cases like these warrant discussion instead of being silently glossed over when scores are computed.Source: §C.3 · p. 15

Related papers

Loading 1910.14464v3…