Source-linked AI summary
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension
Alon Talmor, Jonathan Berant
TL;DR
RC research has created many datasets, but their cross-dataset generalization and usefulness for improving new datasets have received limited analysis. The paper studies generalization and transfer across 10 datasets, finding that multi-dataset training improves both and supports state-of-the-art performance, while focusing on single-turn extractive RC.
Problem
The paper addresses limited analysis of how RC datasets generalize to one another and whether existing datasets can improve performance on new datasets.
Method
The authors train models on one or more source RC datasets, evaluate generalization and transfer on target datasets, and develop MULTIQA using multiple datasets.
Results
Multi-dataset training improves generalization and transfer, and MULTIQA achieves new state-of-the-art results on five datasets.
Takeaways & Limitations
Pre-training on multiple source datasets can reduce the examples needed for a target dataset and supports strong performance without dataset-specific fine-tuning.
Takeaways & Limitations
The evaluation focuses on single-turn extractive RC and excludes abstractive and conversational datasets.
Abstract
from arXiv · showhide
A large number of reading comprehension (RC) datasets has been created recently, but little analysis has been done on whether they generalize to one another, and the extent to which existing datasets can be leveraged for improving performance on new ones. In this paper, we conduct such an investigation over ten RC datasets, training on one or more source RC datasets, and evaluating generalization, as well as transfer to a target RC dataset. We analyze the factors that contribute to generalization, and show that training on a source RC dataset and transferring to a target dataset substantially improves performance, even in the presence of powerful contextual representations from BERT (Devlin et al., 2019). We also find that training on multiple source RC datasets leads to robust generalization and transfer, and can reduce the cost of example collection for a new RC dataset. Following our analysis, we propose MultiQA, a BERT-based model, trained on multiple RC datasets, which leads to state-of-the-art performance on five RC datasets. We share our infrastructure for the benefit of the research community.
1 Introduction
This section identifies limited evidence about how RC datasets relate and whether source datasets improve transfer, then presents a 10-dataset analysis and MULTIQA. The study finds that multi-dataset training improves generalization and transfer, yielding strong results and open infrastructure.
- Motivation: RC datasets overlap in required abilities, but relatively little work has examined whether models trained on one dataset generalize to another.The paper frames this gap within growing interest in evaluating language-understanding models on new setups.
- Approach: The study trains DOCQA and BERT-based models on one or more source datasets, evaluating generalization without target examples and transfer with additional target examples.Experiments span 10 RC benchmarks.
- Findings: BERT substantially improves generalization, especially for Wikipedia and Newswire documents, but its contribution is moderate for web snippets.The differing effect is associated with the document sources used by the datasets.
- Findings: Generalization worsens as differences accumulate in document language, question language, and the linguistic phenomenon examined.The paper analyzes these three factors as main causes of poor generalization.
- Findings: Pre-training on one or more source RC datasets substantially improves target-dataset performance during fine-tuning, including in the BERT-based model.The transfer experiments report consistent improvements despite BERT's contextual representations.
- Contributions: Training on multiple datasets supports generalization and transfer, while MULTIQA achieves new state-of-the-art results on five datasets.The authors also release infrastructure for mixing datasets, fine-tuning, and evaluating multiple models and datasets.
2 Datasets
The investigation covers ten reading-comprehension datasets that vary in size, context source, question-generation process, and reasoning demands. The analysis focuses on single-turn extractive reading comprehension, excluding abstractive and conversational datasets.
- Each dataset provides question-context-answer triples, with models mapping unseen question-context pairs to answer spans in the context.The study focuses on the single-turn extractive setting.
- The dataset collection includes large and small benchmarks, with 75K examples used to control size effects for large datasets.Large datasets contain more than 75K examples, while small datasets contain fewer than 75K.
- Dataset contexts come from sources including Wikipedia, CNN articles, and web snippets retrieved from search engines.The datasets also differ in how questions are generated, including crowdsourcing, trivia sources, and web queries.
- Some datasets are designed to test multi-hop reasoning, while others emphasize local matching, quantitative reasoning, or questions derived from knowledge-base queries.HOTPOTQA requires multi-hop reasoning over related paragraphs, and DROP focuses on quantitative reasoning.
- The study uses only 33,573 extractive DROP examples for training while evaluating on the entire development set.
3 Models
The study compares DOCQA with a BERT-based model using a shared extractive preprocessing pipeline across datasets. Both models predict answer spans from question-context chunks, but they differ in architecture and training implementation.
- All experiments use the same training procedure across datasets with minimal hyper-parameter tuning.
- Pre-processing: Preprocessing splits long paragraphs, ranks chunks by tf-idf similarity, greedily merges them up to length L, and marks matching answer spans.
- DOCQA: DOCQA encodes questions and documents with bidirectional RNNs, applies question-document attention and document self-attention, then predicts span boundaries.It processes chunks of up to L=400 tokens using GloVe embeddings and selects the highest-probability span across chunks.
- DOCQA: DOCQA defines the gold span as the first occurrence of the gold answer in the context.
- BERTQA: BERTQA encodes each question-chunk pair as 512 wordpiece tokens and predicts start and end positions from top-layer BERT outputs.Special tokens separate the question and chunk, and a linear softmax layer produces span-position distributions.
- BERTQA: BERTQA trains separately on chunks containing the answer and selects the span with the maximal logit at test time.
4 Controlled Experiments
The controlled experiments test cross-dataset generalization and transfer across RC datasets, identifying dataset similarities, context effects, and the benefits of multi-dataset training. Results show that pre-training improves transfer, reduces data needs, and supports robust generalization even with BERT.
- Models generalize poorly in the zero-shot setup, with performance decreasing 31.5% on average relative to training and testing on the target dataset.
- BERTQA improves zero-shot generalization by 21.2% on average over DOCQA, while its gains are larger for Wikipedia and Newswire than for web snippets.
- Increasing training data from 37K to 375K examples improves generalization by 26% on average, while multi-dataset pre-training reaches 95% of final performance with 30% of examples.
- Pre-training on source datasets improves transfer by 21% for DOCQA and 7% for BERTQA on average, improving 8 and 10 of 11 datasets respectively.
- Multi-dataset training achieves nearly the best source-dataset performance, falling 3% short for DOCQA and 0.3% for BERTQA on average.
- Fine-tuning can overcome severe distribution mismatch: WIKIHOP rises from 12.6 EM to 50.5 EM after target-data training.
- Context type strongly affects generalization, with cross-context performance dropping from 60.7 to 48.4 for TQA-G, 53.1 to 44.6 for TQA-U, and 50.1 to 43.3 for TQA-W.
5 MULTIQA
MULTIQA is a BERT-based model trained on multiple RC datasets, evaluated against BERT-large and prior state of the art. It improves state-of-the-art performance on five datasets and shows strong robustness on adversarial SQuAD.
- MULTIQA is a BERT-based model trained on multiple RC datasets to obtain state-of-the-art results across several datasets.
- MULTIQA improves state-of-the-art performance on five datasets without training on all examples from the large datasets.
- MULTIQA outperforms BERT-large in all evaluated cases, with especially notable gains on the small datasets COMQA, CWQ, and CQ.
- The authors conclude that dataset size may be the main bottleneck rather than model capacity in many NLU tasks.
- On adversarial SQuAD, MULTIQA obtains 66.7 EM and 73.1 F1, exceeding BERT-large’s 60.4 EM and 66.3 F1.
6 Related Work
Related work situates this study among earlier transfer experiments, emerging work on general-purpose language representations, and evaluations across RC datasets. The paper focuses on reading comprehension as a broad language-understanding task.
- Earlier studies examined transfer from large RC datasets to smaller ones, but at smaller scale or mainly for multi-choice questions.
- Recent work increasingly studies general-purpose language representations through unsupervised, multi-task, and transfer learning.
- The evaluation resources distinguish datasets using EM and token F1 from those evaluated with average recall, precision, and F1.
- The paper positions reading comprehension as an important and broad language-understanding task.
7 Conclusions
The study finds that multi-dataset training improves transfer and generalization across RC benchmarks, while also reducing the data needed for new targets. It releases infrastructure to support further multi-dataset experiments.
- The authors release infrastructure for experiments spanning multiple RC datasets.
- Multi-dataset pre-training consistently improves performance on target RC datasets, even with BERT representations.
- Training on multiple RC datasets substantially reduces the number of target examples needed to reach a fixed performance level.
- BERT-large trained across multiple RC datasets performs well on all trained datasets without separate dataset-specific fine-tuning.
- BERT improves generalization, but its effect is more moderate for web-snippet contexts than for Wikipedia and newswire.