PaperlayerView exact revision on arXiv

Source-linked AI summary

BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, Kristina Toutanova

arXiv:1905.10044v1cs.CL

TL;DR

The paper asks whether naturally occurring yes/no questions can provide a challenging reading-comprehension test of inference. It builds BoolQ and evaluates transfer-learning baselines, finding that MultiNLI transfer remains beneficial with BERT and yields 80.43% accuracy, below 90% human accuracy.

  • Problem

    Naturally occurring yes/no questions are understudied as a test of complex inference, despite requiring facts and entailment-like conclusions beyond textually explicit information.

  • Method

    The paper constructs BoolQ from 16,000 naturally occurring yes/no questions paired with Wikipedia paragraphs and evaluates transfer learning followed by fine-tuning on BoolQ.

  • Results

    80.43% accuracy was achieved by the best model, compared with 62.31% for the majority baseline and 90% for human annotators.

  • Takeaways & Limitations

    MultiNLI entailment transfer and BERT pre-training are complementary, and entailment transfer remains beneficial on top of language-model pre-training.

  • Takeaways & Limitations

    The paper leaves sequential or multi-task pre-training across several suggested datasets to future work.

Abstract

from arXiv · show

In this paper we study yes/no questions that are naturally occurring --- meaning that they are generated in unprompted and unconstrained settings. We build a reading comprehension dataset, BoolQ, of such questions, and show that they are unexpectedly challenging. They often query for complex, non-factoid information, and require difficult entailment-like inference to solve. We also explore the effectiveness of a range of transfer learning baselines. We find that transferring from entailment data is more effective than transferring from paraphrase or extractive QA data, and that it, surprisingly, continues to be very beneficial even when starting from massive pre-trained language models such as BERT. Our best method trains BERT on MultiNLI and then re-trains it on our train set. It achieves 80.4% accuracy compared to 90% accuracy of human annotators (and 62% majority-baseline), leaving a significant gap for future work.

1 Introduction

The paper introduces BoolQ, a dataset of naturally occurring yes/no questions designed to test difficult inference from text. It finds that these questions require broad inferential abilities, and that MultiNLI transfer combined with BERT pre-training gives the strongest reported results.

  • Motivation: Natural language understanding requires inferring facts that extend beyond what text states directly.The paper illustrates this with implications drawn from a sentence about an athlete winning an Olympic medal.
  • Dataset motivation: Naturally occurring yes/no questions provide an alternative way to test complex inferential abilities without prompting authors to write particular question types.Question authors were not required to write yes/no questions and did not know the answers.
  • Dataset motivation: These questions often seek non-factoid information and require a wide range of inferences, making BoolQ a highly inferential reading-comprehension dataset.The dataset is also directly related to the practical task of answering user yes/no questions.
  • Dataset construction: BoolQ contains 16,000 naturally occurring yes/no questions, each paired with a Wikipedia paragraph marked by an independent annotator as containing the answer.The task maps a question and passage to a “yes” or “no” output.
  • Results: 80.43% accuracy was achieved by the best model, versus 62.31% for the majority baseline and 90% for humans.The best model transfers from MultiNLI and uses unsupervised BERT pre-training; the paper describes the approaches as complementary.

2 Related Work

Prior datasets contain yes/no questions, but they were generally designed for other goals or used prompting and class imbalance. BoolQ instead targets naturally occurring questions, while the paper compares transfer from entailment and other related data sources.

  • Existing QA datasets: CoQA, QuAC, HotPotQA, and ShARC include yes/no questions, but their primary goals are conversational question answering or multi-step reasoning.These design goals complicate using them specifically to test inferential abilities in yes/no question answering.
  • Existing QA datasets: QuAC is the only one of those four datasets whose question authors could not view the answering text, but it still heavily prompts users and has 80% “yes” answers.Its questions are limited to pre-selected Wikipedia articles.
  • Existing QA datasets: MS Marco contains some yes/no questions, but heuristic identification produces potentially noisy annotations and an 80% “yes” class imbalance.The resulting annotation quality is unknown.
  • Scope: The paper focuses on natural-language text rather than templated bAbI stories or visual question-answering datasets.This distinguishes BoolQ’s setting from other contexts in which yes/no QA has been studied.
  • Dataset design: Existing advanced-reasoning QA datasets often engineer difficulty by prompting multi-step questions or filtering out easy questions.BoolQ is motivated as a naturally occurring alternative to these construction strategies.
  • Entailment datasets: Naturally occurring yes/no questions provide greater independence between questions and source passages than entailment datasets whose statements were written with premise knowledge.The paper contrasts this with sources such as SciTail, RTE-6, and RTE-7.
  • Transfer learning: The paper reports evidence that crowd-sourced entailment examples transfer better to natural yes/no questions than converted short-answer or multiple-choice examples.This comparison concerns transfer-learning sources for BoolQ.

3 The BoolQ Dataset

BoolQ is a 16,000-question dataset of naturally occurring yes/no questions paired with Wikipedia passages and annotated answers. Its questions span varied topics and information types, often require inference beyond paraphrase, and support high-quality but challenging reading-comprehension evaluation.

  • Data Collection: Each BoolQ example pairs a yes/no question with a Wikipedia article passage selected by an annotator and an answer label.The dataset uses the selected passage rather than the entire document to reduce ambiguity and keep inputs small.
  • Data Collection: 16,000 questions are split into 9.4k training, 3.2k development, and 3.2k test examples, with “yes” answers comprising 62.31% of training data.The questions average 8.9 tokens, while passages average 108 tokens.
  • Annotation Quality: 90% answer-annotation accuracy against gold-standard labels on 110 examples supported using singly annotated examples for the larger dataset.Disagreements included six ambiguous cases and five annotator errors.
  • Question Types: Questions commonly concern entertainment media and sports, while also covering historical events, the natural world, entities, and general factual information.About one-sixth each concern existence, event occurrence, and definitional questions.
  • Types of Inference: Less than 40% of examples can be solved through paraphrase detection; many require factual reasoning, examples, implicit information, or missing-mention inference.Question-only models perform poorly, indicating that the questions alone do not provide enough information to predict answers.
  • Discussion: The authors hypothesize that people use yes/no questions for more complex information because simpler factual queries are usually phrased as short-answer questions.Rare negation also means that “no” answers often require understanding exclusion or implausibility rather than matching wording.

4 Training Yes/No QA Models

The training study evaluates transfer learning for BoolQ by adapting models from entailment, multiple-choice QA, extractive QA, paraphrasing, heuristic yes/no data, and unsupervised language-model pre-training. It emphasizes constructing related training objectives and pre-training before fine-tuning on BoolQ.

  • Transfer Learning: Training on BoolQ alone reaches 69.6% accuracy, only 8% above the majority baseline, motivating transfer learning from larger related datasets.Models are pre-trained on related tasks and then fine-tuned on BoolQ training data.
  • Entailment: Entailment transfer uses MultiNLI and SNLI, with the entailment probability treated during fine-tuning as the probability of a “yes” answer.The datasets were chosen because they are widely used and large enough for pre-training.
  • Multiple-Choice QA: Multiple-choice QA transfer uses RACE by converting questions and answer options into statement-like inputs.This follows an entailment-style formulation for reading-comprehension data.
  • Extractive QA: Extractive QA transfer was ineffective when only lower-level model weights were transferred, so the study constructs entailment-like data instead.Methods use QNLI, SQuAD 2.0 answer and distractor candidates, and NQ long-answer paragraphs.
  • Other Sources: The study also transfers from QQP paraphrase pairs, unsupervised ELMo, BERT, and GPT pre-training, and a 38k-example heuristic yes/no corpus from MS Marco.The MS Marco corpus is formed by pairing questions with related snippets when free-form answers begin with “yes” or “no”.

5 Results

Transfer learning substantially improves BoolQ performance, with entailment data—especially MultiNLI—and BERT pre-training providing the strongest results. The gains persist despite limited usefulness from several QA and paraphrase sources, and MultiNLI benefits BERT even further.

  • QA transfer from RACE and SQuAD 2.0 was unsuccessful, while Y/N MS Marco and QQP produced only small amounts of transfer.The authors associate these results with domain mismatch, adversarial distractors, small size, class imbalance, or passage-domain differences.
  • MultiNLI outperformed all other supervised transfer methods by a large margin, while QNLI and NQ provided only modest gains.The NQ result indicates that selecting question-relevant text is partially transferable to yes/no QA.
  • BERTL was the strongest unsupervised method, surpassing the other evaluated language-model pre-training approaches.The compared models included ELMo, BERT, and OpenAI GPT.
  • MultiNLI added 3.5 points beyond BERTL, and its benefit was at least 5–6 points across training-data sizes, reaching nearly 10 points for BERTL with 1,000 examples.With small datasets, the recurrent model using MultiNLI pre-training outperformed BERTL.
  • Entailment datasets transferred better than question-and-passage datasets, indicating that adapting from sentence pairs was not a major obstacle.Converting BoolQ yes/no questions into declarative statements did not improve MultiNLI transfer in preliminary experiments.
  • Removing contradiction examples reduced the two-step model to 78.43% dev accuracy, close to BERTL with unsupervised pre-training alone.The authors hypothesize that contradiction examples are a key advantage of MultiNLI.

6 Conclusion

The paper introduces BoolQ as a challenging reading-comprehension dataset of naturally occurring yes/no questions. It finds that broad inference abilities and entailment-based transfer learning are useful for this task.

  • BoolQ is a reading-comprehension dataset of naturally occurring yes/no questions that require a wide range of inference abilities.The paper also studies transfer learning and finds that crowd-sourced entailment datasets can improve performance beyond language-model pre-training.

A.1 Randomly Selected Examples

The appendix presents randomly selected BoolQ training examples in a consistent question-answer-passage format.

  • Each example displays the question in bold, the answer in parentheses, and the supporting passage below.

A.2 Recurrent Model

The recurrent model treats BoolQ as text-pair classification, encoding question and passage representations, exchanging information through co-attention, and predicting a binary output. Its pipeline consists of embedding, recurrent encoding, attention, pooling, and classification stages.

  • Embed: The model embeds premise and hypothesis text with fastText word vectors and learned character vectors, then applies a shared bidirectional LSTM.The resulting context-aware embeddings represent both text components before cross-attention.
  • Co-Attention: It computes a learned co-attention matrix between hypothesis and premise embeddings using individual and elementwise-interaction terms.The matrix combines representations through learned weights and elementwise multiplication.
  • Attend: Row-wise softmax attention produces attended hypothesis and premise vectors, allowing each text component to incorporate information from the other.The transpose of the attention matrix is used to compute attended premise vectors analogously.
  • Pool: A second bidirectional LSTM processes attended representations, after which attention-weighted pooling produces fixed-size vectors for both text components.The pooled vectors are formed from softmax-normalized attention scores over recurrent embeddings.
  • Classify: The pooled premise and hypothesis vectors pass through a fully connected layer and softmax layer to predict the output class.The model uses dropout between layers, Adam optimization, 200-dimensional LSTMs, and a 100-dimensional fully connected layer.
Loading 1905.10044v1…