Source-linked AI summary
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi
TL;DR
Existing reading-comprehension datasets often emphasize literal text understanding, leaving contextual commonsense reasoning over everyday narratives less studied. The paper introduces COSMOS QA, a multiple-choice dataset built around unstated causes, effects, facts, and counterfactuals, and evaluates neural baselines alongside a new architecture variant. The best machine performance reaches 68.4% accuracy versus 94.0% for humans, leaving substantial headroom for further research.
Problem
Reading-comprehension systems need to reason about unstated causes, effects, likely facts, and counterfactuals in context, an aspect relatively less studied in prior machine-reading literature.
Method
The paper introduces COSMOS QA, a 35,588-problem multiple-choice dataset from 21,886 everyday personal-narrative contexts, and evaluates neural reading-comprehension models plus a new architecture variant.
Results
68.4% accuracy for machines versus 94.0% for humans demonstrates a significant performance gap on COSMOS QA.
Takeaways & Limitations
COSMOS QA provides a benchmark for contextual commonsense machine comprehension and leaves substantial headroom for future research.
Takeaways & Limitations
In 33% of model errors, the selected choice is inconsistent with human common sense, while 14% arise from unanswerable questions.
Abstract
from arXiv · showhide
Understanding narratives requires reading between the lines, which in turn, requires interpreting the likely causes and effects of events, even when they are not mentioned explicitly. In this paper, we introduce Cosmos QA, a large-scale dataset of 35,600 problems that require commonsense-based reading comprehension, formulated as multiple-choice questions. In stark contrast to most existing reading comprehension datasets where the questions focus on factual and literal understanding of the context paragraph, our dataset focuses on reading between the lines over a diverse collection of people's everyday narratives, asking such questions as "what might be the possible reason of ...?", or "what would have happened if ..." that require reasoning beyond the exact text spans in the context. To establish baseline performances on Cosmos QA, we experiment with several state-of-the-art neural architectures for reading comprehension, and also propose a new architecture that improves over the competitive baselines. Experimental results demonstrate a significant gap between machine (68.4%) and human performance (94%), pointing to avenues for future research on commonsense machine comprehension. Dataset, code and leaderboard is publicly available at https://wilburone.github.io/cosmos.
1 Introduction
COSMOS QA targets reading comprehension that combines contextual commonsense reasoning with understanding the supplied narrative. It introduces a large dataset designed around unstated answers, diverse everyday contexts, and questions about causes, effects, and hypotheticals.
- Motivation: Reading comprehension in COSMOS QA requires inferring unstated answers through commonsense while using the context paragraph to disambiguate plausible choices.The examples emphasize that correct answers are absent as explicit text spans, but context remains necessary for selecting among alternatives.
- Dataset contribution: COSMOS QA contains 35,588 reading comprehension problems requiring reasoning about event causes and effects, likely facts about entities, and hypotheticals or counterfactuals.The problems are drawn from 21,886 distinct contexts covering everyday situations in personal narratives.
- Dataset contribution: 93.8% of COSMOS QA questions require contextual commonsense reasoning, compared with 27.4% in MCScript.The dataset formulates questions as multiple choice because the correct answer cannot be found as a text span, while also permitting generative evaluation.
- Approach: The paper evaluates state-of-the-art reading-comprehension models and proposes an architecture variant intended for commonsense-driven reading comprehension.The model comparison establishes baselines and motivates a design better suited to the dataset’s reasoning demands.
- Results: 68.4% accuracy for machines versus 94.0% for humans reveals a substantial performance gap on COSMOS QA.The authors analyze this gap to identify potentially promising directions for contextual commonsense reasoning research.
2 Dataset Design
COSMOS QA is constructed from everyday personal narratives through crowdsourced commonsense questions, answer choices, and multi-stage validation. Its design spans causes, effects, entity facts, counterfactuals, and question types that differ from conventional reading-comprehension benchmarks.
- Data collection: The dataset draws diverse everyday situations from personal narratives in the Spinn3r Blog Dataset.These narratives provide the paragraph contexts used for question construction.
- Question construction: AMT workers create up to two context-related questions per paragraph, covering causes, effects, entity facts, and counterfactuals.Workers also produce up to two correct answers and three incorrect answers for each question.
- Validation: Validation checks paragraph quality, question relevance, answerability, commonsense dependence, and whether the answer can be found without the paragraph.A question set is valid when at least two of three workers select the intended answer and all workers judge the materials satisfactory.
- Unanswerable questions: 2,369 questions judged answerable without context or commonsense are converted by replacing their correct choice with “None of the above.”This creates an explicit unanswerable-question category for evaluating whether systems rely on the intended evidence.
- Dataset split: COSMOS QA contains 35,588 question sets, with 7K test examples and 3K development examples sampled from highly rated workers.The remaining question sets form the training data, and the paper reports dataset statistics in Table 1.
3 Model
The model encodes each paragraph, question, and candidate answer with pre-trained BERT, then applies multiway bidirectional attention to strengthen their contextual interactions. It fuses these representations into a joint vector for classifying candidate answers.
- Motivation: Multiway attention is motivated by prior results showing that interactions among the paragraph, question, and candidate answers improve context interpretation.The paper also notes BERT fine-tuning's ability in commonsense inference and uses attention to further enhance context understanding.
- Encoding with Pre-trained BERT: Pre-trained BERT encodes the paragraph, question, and each candidate answer as a single token sequence.The sequence uses [CLS] and [SEP] delimiters, with token, segment, and position embeddings producing hidden states.
- Multiway Attention: Multiway attention computes paragraph representations attentive to the question, the answer, and both together.The same attention approach is applied analogously to the question and candidate answer representations.
- Multiway Attention: The architecture fuses attended representations with the original BERT encoding before constructing paragraph, question, and answer features.Learnable parameters perform the fusion, and column-wise max pooling produces the new representations.
- Classification: For each candidate answer, the model forms a joint vector F = [FP : FQ : FA] used for classification.This vector represents the paragraph, question, and a particular candidate answer together.
4 Experiments
Experiments compare reading-comprehension and pre-trained language-model baselines on COSMOS QA, then evaluate BERT with multiway attention and human performance.
- Baseline Methods: The experiments compare reading-comprehension approaches with pre-trained language-model approaches, including GPT-FT and BERT-FT.
- Baseline Methods: BERT-FT applies a pre-trained bidirectional transformer language model followed by fine-tuning on COSMOS QA.
- Results and Analysis: Table 3 reports characteristics and accuracy for the evaluated approaches and human performance.
- Results and Analysis: More than 83% of correct COSMOS QA answers are unstated in the paragraphs, limiting approaches based only on semantic relatedness.
- Results and Analysis: Pre-trained language models with fine-tuning achieve more than 20% improvement over reading-comprehension approaches.
- Baseline Methods: BERT with multiway attention performs attention over BERT-FT to enhance interactive context, question, and answer comprehension.
- Results and Analysis: Attention over BERT-FT further improves performance, and the multiway-attention approach selects the correct answer in an example where BERT-FT selects an entailed alternative.
5 Discussion
The discussion examines dataset bias, knowledge transfer, model behavior, generative evaluation, and recurring error patterns in contextual commonsense reasoning.
- 5.1 Ablation Study: Ablating questions does not cause significant performance drop, while ablating other problem components causes more significant drops.The authors attribute this to limited diversity in high-level question types, which can often be guessed from context and answer choices.
- 5.2 Knowledge Transfer: RACE provides greater direct knowledge-transfer benefit than SWAG for COSMOS, while sequential SWAG fine-tuning improves performance.The authors associate SWAG transfer with improved commonsense inference ability and RACE transfer with stronger overall benefit.
- 5.4 Generative Evaluation: 84%+ of generated answers judged correct by humans are semantically consistent with COSMOS gold answers, supporting its use for generative commonsense reasoning.Fine-tuning GPT2 on COSMOS produces more accurate answers than the untuned model.
6 Related Work
COSMOS extends reading-comprehension and commonsense benchmarks by combining contextual narrative understanding with commonsense inference.
- Generative Evaluation: Table 6 reports generative performance for pretrained GPT2 and GPT2-FT on COSMOS QA, with automatic metrics averaged over 10 sampled-output sets.The table concerns the paper’s generative evaluation rather than its multiple-choice model comparison.
- Existing Datasets: Most prior reading-comprehension datasets emphasize explicit context understanding, with a relatively small or unknown commonsense-reasoning component.The paper positions COSMOS against datasets including SQuAD, NEWSQA, NarrativeQA, RACE, and MCScript.
- Comparison with ReCoRD: COSMOS differs from ReCoRD by using everyday weblog narratives and by making more than 83% of answers unstated in the context.ReCoRD uses news contexts and answers contained in paragraphs, whereas COSMOS targets contextual commonsense reasoning in personal events.
- Commonsense Reasoning: COSMOS combines reading comprehension with commonsense reasoning, unlike datasets focused primarily on commonsense question answering or other inference types.The paper also investigates attention mechanisms and their design choices for contextual commonsense reasoning.
7 Conclusion
The paper introduces COSMOS QA and compares neural architectures for contextual commonsense reading comprehension. Its best model still leaves substantial room relative to human performance.
- 7 Conclusion: COSMOS QA is a large-scale dataset for machine comprehension with contextual commonsense reasoning.The dataset targets reading between the lines in everyday narratives.
- 7 Conclusion: The authors report extensive comparisons of state-of-the-art neural architectures and a new model variant that achieves the best result.The conclusion frames the model comparison as empirical evidence about architecture design for this task.
- 7 Conclusion: 25.6% separates the best model performance from human performance, encouraging further research on contextual commonsense reasoning.The conclusion identifies this headroom as the central remaining opportunity.
A Context Paragraph Extraction
The dataset extracts short context paragraphs from a large corpus of personal weblog narratives and uses worker guidance to make questions challenging for AI systems.
- Source Data: The source corpus contains over 1.6 million non-spam weblog entries describing everyday personal events.The paper uses personal narratives from the ICWSM 2009 Spinn3r Blog Dataset.
- Context Extraction: Pre-trained BERT predicts confidence scores for sentence pairs to help extract short substories as context.The passage describes sentence segmentation and tokenization before context extraction.
- Question Construction: Workers are instructed to minimize wording copied from paragraphs in correct answers while making incorrect answers appealing through paragraph word overlap.They are also encouraged to keep candidate answers similar in length and style.
- Worker Qualification: Workers are restricted to English-speaking countries and substantial prior HIT experience, with a 99% acceptance-rate requirement and qualification testing.The restriction covers the United States, Canada, and United Kingdom.
C Implementation Details
Baseline implementations are retrained on the dataset using shared preprocessing, while BERT denotes the pre-trained BERT large uncased model.
- Baseline methods use released open-source implementations and are retrained on the dataset.
- All approaches segment paragraphs into sentences and tokenize sentences, questions, and candidate answers with spaCy.
- BERT refers throughout the paper to the pre-trained BERT large uncased model.
D Impact of Training Data Size
The study evaluates BERT-FT across progressively larger training subsets. Performance increases with more data but shows no significant improvement beyond 15K questions.
- Performance increases as successive 10% portions are added to the training data.The full training set is divided into 10 folds, and portions are added successively using BERT-FT.
- No significant improvement is observed when the training data increases beyond 15K questions.
- Figure 8 presents COSMOS performance for varying amounts of training data.
E Details for Generative Evaluation
Generative evaluation fine-tunes GPT2 to produce candidate answers conditioned on paragraph–question–answer inputs, then assesses them automatically and with human judgments.
- GPT2, a 117M-parameter OpenAI pre-trained transformer, is fine-tuned on the COSMOS QA training set.
- GPT2-FT generates 10 candidate answers conditioned on each paragraph, question, and correct answer using top-k sampling.Sampling uses k ∈{3, 10, 50, 100, 1000}.
- Questions whose correct answer is “None of the above” are omitted from training, development, and test sets.
- Automatic evaluation compares generated candidates with the original human-authored correct choice and averages metric scores across 10 candidate-answer sets.
- Human evaluation samples 200 paragraph-question pairs, presents four generated answers per question, and uses three workers to select plausible correct answers.
- An automatically generated answer counts as correct only when all three workers judge it correct.