PaperlayerView exact revision on arXiv

Source-linked AI summary

DREAM: A Challenge Dataset and Models for Dialogue-Based Reading Comprehension

Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, Claire Cardie

arXiv:1902.00164v1cs.CL

TL;DR

Existing reading-comprehension benchmarks provide limited coverage of in-depth multi-turn, multi-party dialogue understanding, motivating DREAM. The paper constructs and evaluates this examination-based dataset with dialogue structure and general world knowledge, finding that both improve performance while substantial gaps remain.

  • Problem

    Existing reading-comprehension datasets do not focus on in-depth multi-turn, multi-party dialogue understanding, where many questions require multiple sentences and commonsense knowledge.

  • Method

    The paper constructs DREAM from expert-designed English examinations and evaluates rule-based and machine-learning methods augmented with dialogue structure and general world knowledge.

  • Results

    General world knowledge and speaker-focused information improve DSW accuracy by 5.5%, while FTLM++ significantly outperforms FTLM (p-value = 0.03).

  • Takeaways & Limitations

    DREAM’s results show the effectiveness of incorporating dialogue structure and general world knowledge for dialogue-based reading comprehension.

  • Takeaways & Limitations

    The study does not investigate adding ELMo representations or applying newer extractive models, because these models target span localization and tend to perform less well with abstractive options.

Abstract

from arXiv · show

We present DREAM, the first dialogue-based multiple-choice reading comprehension dataset. Collected from English-as-a-foreign-language examinations designed by human experts to evaluate the comprehension level of Chinese learners of English, our dataset contains 10,197 multiple-choice questions for 6,444 dialogues. In contrast to existing reading comprehension datasets, DREAM is the first to focus on in-depth multi-turn multi-party dialogue understanding. DREAM is likely to present significant challenges for existing reading comprehension systems: 84% of answers are non-extractive, 85% of questions require reasoning beyond a single sentence, and 34% of questions also involve commonsense knowledge. We apply several popular neural reading comprehension models that primarily exploit surface information within the text and find them to, at best, just barely outperform a rule-based approach. We next investigate the effects of incorporating dialogue structure and different kinds of general world knowledge into both rule-based and (neural and non-neural) machine learning-based reading comprehension models. Experimental results on the DREAM dataset show the effectiveness of dialogue structure and general world knowledge. DREAM will be available at https://dataset.org/dream/.

1 Introduction

DREAM introduces a large multiple-choice dataset centered on multi-turn, multi-party dialogue comprehension. Its questions frequently require multi-sentence reasoning, commonsense knowledge, and dialogue-aware methods, while existing models remain far below human performance.

  • Dataset: 10,197 questions cover 6,444 multi-turn, multi-party dialogues collected from English examinations designed by educational experts.Each question has three answer options, exactly one of which is correct.
  • Challenges: 34% of questions require unspoken commonsense knowledge, including scene information, social implications, or speaker intentions.The dialogue alone can be insufficient to identify an utterance’s intended meaning.
  • Challenges: 85% of DREAM questions require information from multiple sentences and multiple dialogue turns involving multiple speakers.Answering them also requires aligning utterances with their corresponding speakers.
  • Methods: Only 16% of correct answers are text spans from the source documents, motivating evaluation of abstractive multiple-choice methods.The authors therefore examine rule-based and neural reading-comprehension approaches.
  • Results: 45.5% is the highest accuracy achieved by neural models designed for non-dialogue reading comprehension, versus 44.6% for a simple lexical baseline.These models primarily exploit surface-level information from the source documents.
  • Results: 59.5% is the best automated performance, compared with 98.6% human ceiling performance on DREAM.The gap indicates the difficulty of dialogue-based reading comprehension.

2 Related Work

Prior reading-comprehension datasets emphasize extractive spans, while DREAM contributes an examination-based dialogue dataset with predominantly abstractive answers and substantial inference demands. Its design combines expert-created questions with multiple-sentence and commonsense reasoning challenges.

  • Extractive and Abstractive Datasets: Extractive datasets typically use short answer spans from source documents, whereas abstractive datasets use human-generated answers based on documents or summaries.Conversational datasets such as CoQA still contain mostly extractive answers because annotators tend to copy spans.
  • Extractive and Abstractive Datasets: 83.7% of correct answer options in DREAM are free-form text, distinguishing it from datasets dominated by extractive answers.DREAM is discussed as a multiple-choice dataset whose options are not restricted to document spans.
  • Multiple-Choice Datasets: Multiple-choice datasets require extensive human involvement to generate questions, correct options, and distractors, and often demand multi-sentence reasoning or external knowledge.These datasets are not limited to surface matching.
  • Examination Datasets: Examination datasets designed by educational experts support objective accuracy evaluation and are generally clean and high-quality.Reported human performance ceilings include 94.5% on RACE and 98.6% on DREAM.
  • DREAM: DREAM is the first large multiple-choice dialogue-based examination dataset with many multi-sentence inference questions.The authors also describe it as containing the largest number of commonsense-reasoning questions among examination datasets.

3 Data

DREAM collects dialogue-based comprehension problems from expert-designed English exams and analyzes the dataset by dialogue characteristics and required reasoning types. The analysis distinguishes matching from reasoning questions and examines dialogue quality against Friends.

  • Collection Methodology: DREAM problems come from English-language exams designed by experts to assess Chinese learners’ listening or reading comprehension.The source exams include national and college-level tests, and the problems are publicly accessible.
  • Question Types: Questions are categorized into matching and reasoning according to the knowledge required for answering.Matching questions are entailed or paraphrased by one sentence, whereas reasoning questions require more than a single sentence’s surface meaning.
  • Reasoning Subcategories: Summary questions require understanding the dialogue’s overall topic and speaker relations, while logic questions require relations implied across multiple sentences.The distinction separates whole-dialogue interpretation from multi-sentence reasoning that does not involve the entire dialogue.
  • Reasoning Subcategories: Arithmetic questions require numerical inference, and commonsense questions require knowledge not stated in the dialogue.Some questions combine logic and commonsense requirements, including cases involving properties of recognizable entities or relationships between speakers.
  • Data Analysis: DREAM dialogues have fewer spelling mistakes and a slightly higher overall quality score than equally sized Friends dialogues, but both receive the same simple readability classification.DREAM contains 10 article errors and 8 verb-form errors among 23 grammar errors, and its language uses simpler but less precise vocabulary.

4 Approaches

The paper formulates dialogue-based multiple-choice comprehension and augments rule-based, feature-based, and neural approaches with dialogue structure and general world knowledge.

  • 4.1 Problem Formulation and Notations: The task chooses one correct option from three candidates given a multi-turn dialogue and question, treating each instance as three-class classification.
  • 4.2 Rule-Based Approaches: Speaker-focused scoring matches question-and-option words against turns spoken by the target speaker, complementing speaker-independent dialogue matching.
  • 4.2 Rule-Based Approaches: The rule-based method combines sliding-window matching with distance-based scores and semantic similarity derived from ConceptNet embeddings.
  • 4.3 Feature-Based Classifier: The feature-based classifier uses bag-of-words, ConceptNet relations and embeddings, rule-based scores, matching position, and PMI features.PMI measures answer-option and target-speaker-context relatedness through word co-occurrences in external Reddit data.
  • 4.3 Feature-Based Classifier: Matching position assumes facts or opinions near the dialogue’s end tend to be more critical for answering questions.
  • 4.4 End-To-End Neural Network: The neural model adapts a generative pre-trained transformer to multiple-choice comprehension and adds trainable speaker embeddings for target and other dialogue speakers.Input sequences concatenate dialogue, question, answer option, and special tokens; speaker embeddings are added only to dialogue tokens.

5 Experiment

Experiments show that surface-level matching is insufficient for DREAM, while adding general world knowledge, dialogue structure, and complementary model ensembles substantially improves accuracy.

  • Results and Analysis: 45.5% was the highest accuracy among surface-focused neural models, close to the 44.6% achieved by the rule-based DSW baseline.These models primarily exploit surface-level information from the dialogue.
  • Results and Analysis: 5.5% accuracy gain resulted from enhancing DSW with ConceptNet embeddings and speaker-focused information.FTLM reached 55.5%, indicating the value of external linguistic knowledge already encoded in its pretraining.
  • Results and Analysis: 11.9% absolute accuracy gain over Co-Matching came from FTLM++, which combines pretrained knowledge with dialogue structure.The best single model FTLM++ significantly outperformed FTLM, with p-value = 0.03.
  • Results and Analysis: 59.5% accuracy was achieved by ensembling one GBDT++ model with three FTLM++ models.The ensemble improved over the individual models by combining diverse knowledge sources and architectures.
  • Ablation Tests: 1.4% accuracy loss occurred when DSW++ ignored the target speaker, supporting the usefulness of dialogue structure.Removing speaker-focused features from GBDT++ produced a similar performance drop.
  • Ablation Tests: 4.7% and 6.2% accuracy decreases followed removal of ConceptNet embeddings from DSW++ and all general-world-knowledge features from GBDT++, respectively.Randomly initialized FTLM++ dropped to 36.2%, only slightly above a random baseline.
  • Results by Dialogue Length: For 10–48 dialogue turns, DSW++ (56.8%) and GBDT++ (57.4%) outperformed FTLM++ (55.7%).FTLM++ peaked at 0–10 turns and suffered severe drops on longer dialogues.
  • Results by Question Type: GBDT++ outperformed FTLM on matching questions (68.1% vs 57.0%), whereas FTLM better handled implicit questions requiring multi-sentence aggregation or world knowledge.Both models performed worse than random guessing on math problems.

6 Conclusion and Future Work

The paper introduces DREAM as a challenging dialogue-based reading comprehension benchmark and finds that surface information alone is insufficient. Its results support incorporating dialogue structure and general world knowledge, while identifying dialogue problem generation as future work.

  • Conclusion: DREAM is the first multiple-choice dialogue-based reading comprehension dataset from English language examinations.Its questions require multi-turn, multi-party dialogue understanding.
  • Conclusion: 85% of DREAM questions require multiple-sentence reasoning, and 34% require commonsense knowledge, making the task very challenging.The dataset therefore tests reasoning beyond direct surface matching.
  • Conclusion: Experiments show that surface-level information is insufficient, while general world knowledge and dialogue structure improve rule-based and machine-learning methods.The authors identify these factors as a promising direction for dialogue-based reading comprehension.
  • Future Work: Future work will investigate problem generation for dialogues and whether it improves dialogue-based reading comprehension.
Loading 1902.00164v1…