Source-linked AI summary
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task
Danqi Chen, Jason Bolton, Christopher D. Manning
TL;DR
The paper asks how much language understanding is needed for CNN/Daily Mail reading comprehension, a task created to address limited annotated data. It combines hand-analysis with carefully designed simple systems and finds strong accuracies, concluding that the datasets are noisy, relatively simple, and near a performance ceiling. The paper points toward using them to help address more complex reasoning tasks.
Problem
Limited human-annotated reading-comprehension data leaves unclear what language understanding models need to learn on the CNN/Daily Mail task.
Method
The paper combines hand-analysis of sampled problems with carefully designed conventional and neural systems to examine task difficulty and learned behavior.
Results
73.6% and 76.6% accuracy are obtained on CNN and Daily Mail, respectively, with results approaching the task’s performance ceiling.
Takeaways & Limitations
The datasets are valuable for training statistical reading-comprehension models, but the required reasoning is simple and current neural networks have nearly reached their performance ceiling.
Takeaways & Limitations
The datasets remain noisy because of their automatic creation and coreference errors, and future work must address more complex reasoning tasks with less annotated data.
Abstract
from arXiv · showhide
Enabling a computer to understand a document so that it can answer comprehension questions is a central, yet unsolved goal of NLP. A key factor impeding its solution by machine learned systems is the limited availability of human-annotated data. Hermann et al. (2015) seek to solve this problem by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and show that a neural network can then be trained to give good performance on this task. In this paper, we conduct a thorough examination of this new reading comprehension task. Our primary aim is to understand what depth of language understanding is required to do well on this task. We approach this from one side by doing a careful hand-analysis of a small subset of the problems and from the other by showing that simple, carefully designed systems can obtain accuracies of 73.6% and 76.6% on these two datasets, exceeding current state-of-the-art results by 7-10% and approaching what we believe is the ceiling for performance on this task.
1 Introduction
The paper examines what level of language understanding the CNN/Daily Mail reading-comprehension task requires, motivated by limited human-labeled data and uncertainty about what models learn. It finds that carefully designed simple systems achieve strong performance, suggesting the task is easier than previously believed.
- Motivation: Limited human-annotated datasets make it difficult to train powerful statistical models for complex textual reasoning.Existing datasets contain only hundreds of documents because annotation requires considerable expertise and careful design.
- Dataset idea: CNN and Daily Mail bullet-point summaries provide a cheap source of large-scale supervised training data by hiding entities from article summaries.The task assumes that understanding the article allows a system to infer the missing entity in a summarized bullet point.
- Paper aim: The paper investigates what depth of natural language understanding is needed to perform well on this automatically created task.It combines hand-analysis of a small problem subset with diagnosis of the behavior learned by its systems.
- Conclusions: The analysis concludes that the dataset is easier than previously realized and that current systems resemble single-sentence relation extractors more than discourse-understanding systems.The paper also reports that its systems are close to the ceiling for single-sentence, unambiguous cases.
2 The Reading Comprehension Task
The CNN/Daily Mail task uses news articles and bullet summaries to create large-scale reading-comprehension data, with named entities anonymized by coreference-chain markers. The paper then uses straightforward systems and data analysis to assess performance and task difficulty.
- Dataset construction: The datasets pair CNN and Daily Mail articles with their bullet-point summaries to form reading-comprehension examples.The datasets exploit the abundance of online news articles.
- Entity anonymization: Named entities in coreference chains are replaced with indexed @entity markers so systems must rely on the passage rather than world knowledge.This makes the task more artificial and can make some anonymized items difficult or impossible for humans.
- Dataset scale: 380k and 879k training examples are provided for CNN and Daily Mail, respectively.Passages average about 30 sentences and 800 tokens, while questions average 12–14 tokens.
- Evaluation strategy: The paper builds straightforward systems to estimate a lower bound for current NLP performance and analyzes sampled items to estimate an upper bound.The analysis examines how many questions are trivial, noisy, or not answerable.
3 Our Systems
The paper compares an entity-centric feature classifier with an end-to-end attentive neural network for selecting the passage entity corresponding to a placeholder. The neural model uses question-guided attention and several simplifications relative to the original AttentiveReader.
- 3.1 Entity-Centric Classifier: The task ranks candidate entities e using a feature vector f_p,q(e) and weight vector θ, with the correct answer expected to rank highest.The classifier is designed around passage, question, and answer triples, where the answer entity appears in the passage.
- 3.1 Entity-Centric Classifier: The entity-centric classifier combines occurrence, frequency, position, matching, distance, co-occurrence, and dependency-parse features.Its templates include n-gram exact matches around the placeholder and candidate entity, question-word distance, sentence co-occurrence, and dependency relations.
- 3.2 End-to-end Neural Network: The neural system encodes passage and question tokens with bidirectional recurrent networks, using GRUs for computational efficiency.Passage contextual embeddings concatenate forward and backward hidden states, while a second bidirectional RNN produces the question embedding.
- 3.2 End-to-end Neural Network: Attention computes relevance probabilities between the question and contextual passage embeddings, then forms output vector o as their weighted combination.The resulting vector summarizes passage information selected according to question relevance.
- 3.2 End-to-end Neural Network: The model predicts among entities appearing in the passage and trains with a softmax over W_a^⊺o plus a negative log-likelihood objective.It uses o directly for prediction rather than combining it with the question embedding through another nonlinear layer.
4 Experiments
The experiments compare a conventional entity-centric classifier and a neural network, evaluate training and model-selection procedures, and report strong test accuracy on CNN and Daily Mail.
- Systems and training: The classifier uses LambdaMART to rank candidate entities, with features designed for the ranking formulation and dependency-parsed documents.The system scores candidate entities using a feature vector and learned weights, while restricting LambdaMART features to first-ranked accuracy.
- Systems and training: The neural network keeps the 50k most frequent words, uses 100-dimensional GloVe initialization, and applies dropout and gradient clipping during training.Hidden sizes are 128 for CNN and 256 for Daily Mail; optimization uses SGD with learning rate 0.1 and mini-batches of 32.
- Experimental protocol: The models are trained across multiple epochs, selected by development accuracy, and evaluated using averages over five random seeds and five-model ensembles.Training uses one GPU, with approximately 3 hours per CNN epoch and 12 hours per Daily Mail epoch.
- Main results: The conventional feature-based classifier obtains 67.9% accuracy on the CNN test set and outperforms the reported symbolic and neural baselines.Feature ablations identify n-gram matching and entity frequency as the two most important feature classes.
- Main results: Relabeling entity markers by their first occurrence improves results by 0.6% and 0.9% on the two datasets.The authors report results both with and without relabeling; the relabeling is also found to make training converge faster.
- Main results: 73.6% and 76.6% are the state-of-the-art accuracies on CNN and Daily Mail respectively after relabeling.These results exceed previous results by over 5%, while five-model ensembles provide further 2−4% gains.
5 Data Analysis
A manual analysis of 100 CNN examples separates answerable cases from noisy or difficult ones and examines what these categories reveal about system performance. Most answerable cases appear to require finding one relevant sentence rather than multi-sentence inference.
- Analysis setup: The analysis randomly sampled 100 CNN development examples to assess trivial, noisy, and unanswerable questions.The examples were classified by difficulty and answerability.
- Example categories: Exact-match cases can be answered directly from nearby passage words, while sentence-level paraphrases preserve the answer through semantic rephrasing.These categories represent progressively less literal matches between question and passage.
- Example categories: Partial-clue cases require inferring an answer from incomplete semantic matches such as word or concept overlap.The question and sentence need not have a complete semantic match.
- Inference demands: Only 2 examples required multiple sentences for inference, fewer than expected, suggesting that most answerable cases involve one relevant sentence.The authors hypothesize that the answer is then inferred from that sentence.
- Noisy and difficult cases: 25% of the sample consisted of coreference errors or ambiguous/hard cases, creating a barrier to accuracy much above 75%.Coreference-error examples were treated as not answerable, while ambiguous or hard examples were judged difficult for humans to answer confidently.
- Per-category performance: Both systems achieved 100% on exact-match cases, performed poorly on ambiguous or entity-linking-error cases, and differed mainly on paraphrasing and partial-clue cases.The authors report that the neural network handled semantic matches involving paraphrasing or lexical variation better.
6 Related Tasks
Related reading-comprehension tasks vary in reasoning demands, question formats, dataset size, and linguistic realism. These differences shape both the capabilities they test and the difficulty of training statistical models.
- MCTest: MCTest uses fictional short stories with multiple-choice questions and targets a 7-year-old reading-comprehension level.It is an open-domain task created through crowdsourcing.
- MCTest: Over 50% of MCTest questions require multiple sentences, and questions span categories including what, why, how, whose, and which.The full dataset contains only 660 paragraphs, limiting statistical-model training.
- MCTest: MCTest solutions rely heavily on manually curated syntactic and semantic features supplemented by word embeddings and lexical or paragraph databases.The cited systems use additional knowledge resources alongside hand-designed features.
- Children Book Test: The Children Book Test uses 20-sentence passages from children’s books to predict a missing word in the 21st sentence.Questions are categorized by missing-word type: named entity, common noun, preposition, or verb.
- bAbI: bAbI comprises 20 artificial reasoning datasets designed to test chaining, induction, and deduction across sentence sequences.Its limited 100–200-word vocabulary and simple language variations leave a gap from real-world datasets.
7 Conclusion
The paper’s systems achieved state-of-the-art results, while hand analysis examined the dataset’s characteristics. The authors conclude that the datasets are useful but noisy, relatively simple, and near a neural-network performance ceiling.
- The systems demonstrated state-of-the-art results, alongside a careful hand analysis of the dataset.
- The datasets are valuable because they provide a promising avenue for training effective statistical models for reading comprehension.
- The datasets remain noisy because of their creation method and coreference errors.
- Current neural networks have almost reached a performance ceiling on this dataset, whose required reasoning and inference remain quite simple.
- Future work should use these datasets and trained models to address more complex reading-comprehension reasoning tasks with less annotated data.
A Samples and Labeled Categories from the CNN Dataset
The analysis sampled 100 CNN development-set examples and documented their labeled categories and complete sample index.
- The analysis uniformly sampled 100 examples from the CNN development set.
- Table 7 presents the labeled categories assigned to the 100 samples.
- Table 8 provides the full index list of the sampled examples.