Source-linked AI summary
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, Danqi Chen
TL;DR
The paper addresses whether reading-comprehension systems generalize beyond their training distributions. It introduces a unified, cross-domain extractive QA shared task with hidden evaluation domains and analyzes submitted approaches. The best system achieved 72.5 average F1 on hidden test data, 10.7 points above the BERT baseline.
Problem
MRQA systems were primarily evaluated in-domain, leaving generalization to new test distributions as a central challenge.
Method
The shared task unifies QA datasets into an English extractive format and evaluates systems trained on six datasets across 12 held-out domains, including hidden domains.
Results
72.5 average F1 on hidden test data was 10.7 absolute points higher than the best BERT-based baseline.
Takeaways & Limitations
The shared task and released datasets provide evidence about approaches to generalizing QA systems beyond their training distribution.
Takeaways & Limitations
The task excludes unanswerable, multi-turn, and open-domain question types, and unanswerable questions were difficult to synthesize reliably without substantial noise.
Abstract
from arXiv · showhide
We present the results of the Machine Reading for Question Answering (MRQA) 2019 shared task on evaluating the generalization capabilities of reading comprehension systems. In this task, we adapted and unified 18 distinct question answering datasets into the same format. Among them, six datasets were made available for training, six datasets were made available for development, and the final six were hidden for final evaluation. Ten teams submitted systems, which explored various ideas including data sampling, multi-task learning, adversarial training and ensembling. The best system achieved an average F1 score of 72.5 on the 12 held-out datasets, 10.7 absolute points higher than our initial baseline based on BERT.
1 Introduction
MRQA is used to evaluate language understanding, but systems remain challenged by distribution shifts. The MRQA 2019 Shared Task directly tests extractive QA generalization across domains.
- MRQA has become an important testbed for evaluating whether systems understand human language.
- Systems are still primarily evaluated on in-domain accuracy, while generalization to new test distributions remains challenging.
- The shared task evaluates extractive question-answering models on distributions different from their training distribution.
- Ten teams submitted systems exploring data sampling, multitask learning, adversarial training, and ensembling.
- 72.5 average F1 on hidden test data was 10.7 absolute points above the best baseline.
2 Task Description
The task evaluates English extractive QA under cross-domain conditions using unified datasets and hidden test domains. Systems are ranked primarily by macro-averaged test F1 under string-match evaluation.
- Participants trained on six QA datasets and were evaluated on 12 held-out test datasets.
- Six evaluation datasets provided development data, while six others were completely hidden from participants.
- The task restricts systems to English extractive QA, requiring an answer segment to be found in the context passage.
- Passage sources, question styles, and passage-question relationships vary across datasets, including multi-hop reasoning requirements.
- Systems are evaluated with exact match and word-level F1, and ranked by macro-averaged test F1 using answer-string matching.
- The task does not test unanswerable, multi-turn, or open-domain question types.
3 Dataset Curation
The shared-task dataset unifies 18 sub-domains into an extractive format and separates training, development, and hidden evaluation domains. Curation includes truncation, filtering, and answer-span normalization choices.
- The MRQA dataset combines many sub-domains and requires each answer to appear as a token span in its passage.
- Passages are concatenated and truncated to the first 800 tokens to reduce computational requirements.
- Extractive formatting was chosen because it supports stable evaluation and is easier to obtain from multiple-choice datasets.
- Unanswerable questions were not synthesized because automated distant supervision would introduce significant noise.
- Sub-domain Splits: The 18 sub-domains are partitioned into training, development-visible, and completely hidden evaluation splits.
- Sub-domain Splits: Testing portions of Splits II and III are balanced to 1,500 examples per sub-domain without sharing contexts across development and testing.
- Dataset Adaptation: Datasets are adapted through document and paragraph concatenation, answer-string occurrence labeling, and filtering of incompatible multiple-choice questions.
4 Baseline Model
The baseline is a multitask BERT extractive QA model that predicts answer-span boundaries over chunked question-passage inputs. Training mixes examples from all training datasets and selects answer-containing chunks.
- The baseline is a simple multitask model based on BERT and following the MultiQA approach.
- BERT encodes the concatenated question and passage, while separate MLPs predict start and end indices for the answer span.
- The final span is decoded by maximizing the product of predicted start and end probabilities.
- Examples longer than BERT’s limit are processed as overlapping 512-token chunks with a stride of 128 tokens.
- Training uses only chunks containing answers, and inference selects the span with the maximal logit across chunks.
- Multitask training samples up to 75K examples from each training dataset and combines them into mixed batches.
5 Shared Task Submissions
Ten teams submitted systems using varied approaches to out-of-domain MRQA, including pretrained-language-model combinations, data strategies, multitask learning, adversarial training, and ensembling.
- 10 teams submitted systems for the final evaluation, with 6 teams also submitting system description papers.The shared task ran from May to August 2019 through CodaLab.
- Baidu combined multiple pretrained language models and ultimately ensembled XLNet- and ERNIE-based models.Its experiments also included extra raw text and auxiliary multitask training, but these were omitted from the final system.
- Apple evaluated pretrained models, data sampling, and back-translation, but its final XLNet system used sampled negative examples and no data augmentation.Data augmentation did not improve development performance, so it was excluded from the final submission.
- HKUST compared data-feeding schemes and built its submission on XLNet with an MLP span-prediction layer.Replacing the MLP with an attention-over-attention layer was not helpful.
- Fuji Xerox combined a BERT-based mixture-of-experts architecture with NLI multitask learning and ensembled three random-seed models.
- 42Maru and Samsung Research used adversarial training to learn domain-invariant hidden representations for unseen domains.A domain discriminator predicted dataset labels while the QA model learned representations that thwarted the discriminator.
6 Results
Across 12 held-out datasets, the best system substantially outperformed the BERT-Large baseline, with gains associated with model choice, data handling, and ensembling. Performance improvements varied by dataset characteristics, while in-domain and out-of-domain rankings were strongly correlated.
- Main Results: 72.5 F1 was achieved by D-Net on the hidden test datasets, a 10.7-point absolute improvement over the baseline.The improvement was 11.5 points on Split II and 10.0 points on Split III.
- Main Results: Top out-of-domain submissions also obtained the highest scores on the six provided training datasets, indicating a strong correlation between in-domain and out-of-domain performance.The authors report a very strong correlation between the two evaluation settings.
- Main Results: DROP showed the largest dataset-level improvement, rising from 43.5 F1 to 61.5 F1, whereas RelationExtraction improved from 84.9 F1 to 89.0 F1.Submission rankings were mostly preserved across individual datasets.
- Improvements per data types: Datasets with naturally collected questions received large improvements, and Wikipedia or educational materials gained more than Web snippets or transcriptions.Datasets with passage-dependent questions improved by 11.9 points versus 9.2 points for independently collected questions.
- Pre-trained language models: XLNet-based systems generally outperformed BERT-Large systems, with the top three submissions using XLNet and the isolated model change yielding significant overall gains.The authors note that XLNet gains were not equally strong for every dataset, including DROP and RACE.
- Data sampling: Shuffling examples from all six training datasets was more effective than sequential dataset-by-dataset feeding, while downsampling SearchQA and TriviaQA examples increased overall performance.Sampling negative examples produced gains of up to +1.5 F1 on Split II and up to +4 F1 on Split I.
- Ensembling: Ensembling produced modest performance gains but reduced computational efficiency during both training and inference.The winning system ensembled two different pretrained language models.
7 Conclusions
The MRQA 2019 Shared Task tested whether reading comprehension systems generalize beyond their training domains. Its submissions improved over the baseline and explored diverse techniques, while the released datasets support further study of cross-domain generalization.
- The shared task focused on testing whether reading comprehension systems generalize to examples outside their training domain.
- Many submissions improved significantly over the baseline and investigated a wide range of techniques.
- The authors argue that deployed NLP systems must handle diverse inputs that may differ from those seen during training.
Appendix
The appendix reports per-dataset results for shared-task submissions and baselines across the Split II and Split III evaluation datasets. The accompanying tables identify the evaluation metrics and dataset splits.
- The appendix presents per-dataset performances for shared-task submissions and baselines.
- Table 6 reports performance on the six datasets in the Split II test portion.
- Table 7 reports results on the six datasets in Split III.