PaperlayerView exact revision on arXiv

Source-linked AI summary

NewsQA: A Machine Comprehension Dataset

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, Kaheer Suleman

arXiv:1611.09830v3cs.CLcs.AI

TL;DR

Machine-comprehension datasets need both sufficient scale and questions that demand more than surface matching. NewsQA addresses this gap with a large CNN-based, crowd-generated dataset collected through reasoning-oriented stages, and humans outperform neural models by 0.198 F1. The resulting corpus is intended to support further progress in machine comprehension.

  • Problem

    Existing comprehension datasets often lack either the scale needed for deep learning or questions that require reasoning beyond word and context matching.

  • Method

    NewsQA collects 119,633 crowdworker-generated questions on 12,744 CNN articles through a four-stage process with answer spans, null answers, and validation.

  • Results

    0.198 F1 separates humans from deep neural models on NewsQA.

  • Takeaways & Limitations

    NewsQA provides a large and complex benchmark intended to spur further advances in machine comprehension.

  • Takeaways & Limitations

    The reported experiments use only the subset with answer agreements and leave identifying unanswerable questions for future work.

Abstract

from arXiv · show

We present NewsQA, a challenging machine comprehension dataset of over 100,000 human-generated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting of spans of text from the corresponding articles. We collect this dataset through a four-stage process designed to solicit exploratory questions that require reasoning. A thorough analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (0.198 in F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available at https://datasets.maluuba.com/NewsQA.

1 INTRODUCTION

NewsQA is introduced as a large, challenging machine-comprehension dataset designed to test reasoning beyond word matching. Its collection and evaluation show that existing models remain substantially behind humans.

  • Dataset contribution: NewsQA contains 119,633 questions from 12,744 CNN news articles, with crowdworker-selected answer spans.The dataset uses news articles as source material and answers are spans within the corresponding articles.
  • Dataset contribution: The collection process targets exploratory questions requiring synthesis and reasoning across different parts of an article.The authors explicitly designed the methodology to encourage curiosity-based questions rather than straightforward textual reformulations.
  • Dataset characteristics: NewsQA allows arbitrary-length answer spans, null answers, and no candidate options, while encouraging lexical and syntactic divergence between questions and answers.These characteristics distinguish the task from many previous comprehension datasets.
  • Dataset characteristics: A significant proportion of NewsQA questions requires reasoning beyond simple word- and context-matching.This property is presented as a central challenge of the dataset.
  • Evaluation and significance: NewsQA offers a greater challenge to existing models than SQuAD, as shown through several metrics.The comparison identifies NewsQA as a more difficult alternative among related comprehension tasks.
  • Evaluation and significance: Humans significantly outperform two strong neural-network baselines on NewsQA, leaving room for further machine-comprehension research.The paper measures human performance and compares it with two neural models.

2 RELATED DATASETS

Earlier comprehension datasets trade off scale, task difficulty, and the kind of reasoning they measure. NewsQA is positioned most closely against SQuAD while addressing limitations in existing large-scale benchmarks.

  • MCTest: MCTest contains 660 children’s stories with questions requiring rudimentary reasoning and synthesis, but its candidate answers provide a useful crutch.Its small size also limits training of expressive statistical models.
  • CNN/Daily Mail: CNN/Daily Mail contains about 1.4 million synthetically generated cloze question-answer pairs, enabling large-scale deep-learning training.Its questions are formed by deleting entities from abstractive summary points.
  • CNN/Daily Mail: CNN/Daily Mail mainly tests textual entailment, and strong models nearly match human performance after requiring only limited reasoning.The automatic cloze construction makes the dataset scalable but less demanding in reasoning.
  • Children’s Book Test: The Children’s Book Test evaluates prediction of a missing word from context, so comprehension may be insufficient for solving the task.Questions delete a word in the next sentence of a 20-sentence passage.
  • BookTest: BookTest enlarges CBT’s named-entity and common-noun strata by over 60 times and produces a model matching human CBT performance.The authors nevertheless question the relevance of story prediction as a comprehension task.
  • SQuAD: SQuAD provides 107,785 span-answer questions from 536 Wikipedia articles without candidate answers, making it the dataset most closely related to NewsQA.Its questions are naturally written by crowdworkers on Wikipedia paragraphs.
  • SQuAD: 0.778 F1 was achieved by the strongest published SQuAD model, compared with human measurements of 0.905 and 0.807 F1 under different methodologies.The authors use this narrowing gap to motivate more difficult alternatives such as NewsQA.

3 COLLECTION METHODOLOGY

NewsQA was collected through staged crowdsourcing that separates question generation from answer selection and adds validation and cleanup. The process is designed to elicit difficult, sometimes unanswerable questions while improving answer quality.

  • Collection pipeline: NewsQA uses four stages—article curation, question sourcing, answer sourcing, and validation—followed by answer-agreement consolidation and span merging.These stages are presented as the dataset’s overall collection methodology.
  • Article curation: The dataset samples 12,744 CNN articles from 90,266 retrieved articles and randomly assigns them to training, development, and test sets.The split proportions are 90% training, 5% development, and 5% test.
  • Question sourcing: Questioners see only article headlines and summary points, which encourages curiosity and reduces questions that simply reformulate article sentences.This incomplete view also increases the likelihood of questions whose answers are absent from the full article.
  • Answer sourcing: Separating questioners from answerers is intended to encourage more complex questions, while answerers can reject nonsensical questions or select a null answer.Answerers receive the full article together with the crowdsourced question.
  • Validation: Validation presents the full article, question, and unique answers to additional workers, who select the best answer or reject all answers.Each article-question pair was validated by an average of 2.48 workers for 43.2% of questions lacking prior answer agreement.
  • Validation: 86.0% of NewsQA questions have answers agreed upon by at least two crowdworkers after validation.Questions without agreement remain in the corpus but are specially marked and may be treated as null-answer examples.
  • Cleanup: 5.68% of answers contain multiple spans, and 71.3% of those multi-spans fall within the three-word merging threshold.The multi-span answers often represent lists, creating an additional modeling challenge.

4 DATA ANALYSIS

NewsQA analysis characterizes answer diversity and the reasoning demands that distinguish it from SQuAD. Human evaluation indicates that synthesis and inference occur more often in NewsQA, while some questions have null or unvalidated answers.

  • Answer Types: 22.2% of answer spans are common noun phrases, while clause phrases, person, numeric, and other answers comprise 18.3%, 14.8%, 9.8%, and 11.2%.These categories show that NewsQA answers are linguistically diverse.
  • Answer Types: 9.5% of the full corpus has an agreed null answer, and 4.5% has no agreement after validation.The answer-type proportions exclude these cases.
  • Reasoning Types: NewsQA stratifies questions into word matching, paraphrasing, inference, synthesis, and ambiguous or insufficient reasoning types.The categories are ordered from lower to higher difficulty, with ambiguous or insufficient questions representing cases lacking an answer or unique answer.
  • Reasoning Types: 32.7% of NewsQA examples involve word matching, compared with 39.8% for SQuAD in manually labelled 1,000-example development subsets.Some examples receive multiple labels, but are assigned the more challenging type.

5 BASELINE MODELS

The paper evaluates match-LSTM and a cheaper model, BARB, as neural baselines for NewsQA. BARB combines contextual encoding, bilinear annotation, re-encoding, and convolutional answer-boundary prediction, enabling faster training than match-LSTM.

  • Baseline models: Three systems are evaluated on NewsQA: human data analysts, match-LSTM, and the authors’ computationally cheaper BARB model.The match-LSTM was selected for straightforward implementation and strong, though not state-of-the-art, SQuAD performance.
  • BARB architecture: BARB uses four stages: word encoding, bilinear annotation, re-encoding, and convolutional boundary pointing.The document and question are encoded with bidirectional GRUs; annotations compare document and question representations before re-encoding.
  • Training cost: 3.9k seconds versus 8.1k seconds per NewsQA training epoch are reported for BARB and match-LSTM, respectively.The configurations correspond to the results reported in Section 6.2.
  • BARB architecture: The re-encoding stage combines document encodings, annotation vectors, and question-overlap features before producing boundary-pointing representations.Each document word is represented by the concatenation of h_i, g_i, and q_i, which is processed by another bidirectional GRU.
  • BARB architecture: The boundary-pointing network applies convolutional filters and ReLU nonlinearities to predict answer-span boundaries.The model uses two convolutional layers and conditions end-boundary prediction on the start-boundary probabilities.

6 EXPERIMENTS4

The experiments evaluate human and model performance on NewsQA, compare it with SQuAD, and analyze answer and reasoning types. NewsQA produces substantially larger human–machine gaps and lower sentence-level accuracy than SQuAD.

  • 6.1 HUMAN EVALUATION: Four English speakers answered 1,000 NewsQA development questions using F1, EM, BLEU, and CIDEr evaluation.The study supplements SQuAD’s F1 and EM with BLEU and CIDEr for human-annotated questions.
  • 6.1 HUMAN EVALUATION: 0.694 F1 was the average human score on NewsQA, while human EM was 0.465.The lower EM reflects semantically equivalent answers that differ in span selection.
  • 6.2 MODEL PERFORMANCE: 0.198 points F1 separated humans and machines on NewsQA, compared with 0.098 on SQuAD under the same evaluation scheme.The reported gaps indicate substantial room for improvement in machine comprehension.
  • 6.2 MODEL PERFORMANCE: BARB performs better on named-entity answers, whereas inference and synthesis questions are more difficult, especially on NewsQA.The NewsQA–SQuAD difference is smallest for word matching and largest for synthesis; the authors relate this to longer-term dependencies in longer stories.
  • 6.3 SENTENCE-LEVEL SCORING: 79.4% sentence-level accuracy on SQuAD fell to 35.4% on NewsQA for the isf method.Lengthening SQuAD articles reduces accuracy, but it remains higher than NewsQA at comparable or greater lengths.

7 CONCLUSION

NewsQA is introduced as a large, challenging comprehension dataset whose collection process yields diverse answers and many reasoning-dependent questions. Its difficulty is reflected in a large human–neural-model performance gap.

  • 7 CONCLUSION: NewsQA contains over 100,000 examples collected by crowdworkers from CNN articles and highlights.The collection produces diverse answer types and a significant proportion of questions requiring reasoning.
  • 7 CONCLUSION: 0.198 F1, 0.479 BLEU, and 1.165 CIDEr measure the performance gap between humans and deep neural models.The authors present NewsQA’s size and complexity as an extension to existing comprehension datasets.

A IMPLEMENTATION DETAILS

The implementation details specify the training and optimization setup used for the mLSTM and BARB models.

  • A IMPLEMENTATION DETAILS: Both models use GloVe embeddings initialized from the 840-billion-token Common Crawl corpus without updating embeddings during training.Out-of-vocabulary embeddings are initialized with zero.
  • A IMPLEMENTATION DETAILS: Both models maximize boundary-pointer log likelihood using stochastic gradient descent with ADAM and gradient clipping.Learning rates are decayed when validation loss fails to decrease at the end of an epoch.
  • A IMPLEMENTATION DETAILS: Hyperparameters for both models are tuned with hyperopt, selecting configurations with the best observed performance.The appendix reports the resulting configurations separately for each model.

mLSTM

The model architecture uses bidirectional recurrent networks and specified initialization schemes for its components.

  • mLSTM: The preprocessing and answer-pointing layers use bidirectional RNNs with hidden size 192.These settings follow those used by Wang and Jiang.
  • mLSTM: LSTM and Match-LSTM parameters use orthogonal or normal initialization according to the component-specific scheme.The listed matrices and biases receive different initialization distributions or fixed values.
  • mLSTM: The answer-pointing layer initializes V and W_a orthogonally, selected biases normally, and c to 1.This specifies the initialization scheme for the answer-pointing parameters.

B DATA COLLECTION USER INTERFACE

The paper presents interfaces supporting three data-collection activities: question sourcing, answer sourcing, and validation.

  • The interfaces cover question sourcing, answer sourcing, and question/answer validation.These activities are part of NewsQA’s data-collection workflow.
Loading 1611.09830v3…