PaperlayerView exact revision on arXiv

Source-linked AI summary

UnifiedQA: Crossing Format Boundaries With a Single QA System

Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, Hannaneh Hajishirzi

arXiv:2005.00700v3cs.CLcs.AI

TL;DR

QA research has largely treated answer formats as separate, leaving unclear whether linguistic reasoning can transfer across them. The paper builds UnifiedQA, a single pre-trained text-to-text QA system trained across four formats, and reports strong performance across datasets, transfer to unseen datasets, and state-of-the-art results after fine-tuning.

  • Problem

    Prior QA efforts were largely constrained by narrow formats, and no single system was known to target all four formats despite potentially shared reasoning abilities.

  • Method

    UnifiedQA is a single pre-trained text-to-text QA system trained on seed datasets from multiple formats using natural text without format-specific prefixes.

  • Results

    UnifiedQA performs on par with dataset-specific expert models, generalizes to unseen datasets of seen formats, and achieves state-of-the-art performance after fine-tuning.

  • Takeaways & Limitations

    Crossing QA format boundaries is quantitatively beneficial and makes UnifiedQA a strong starting point for building specialized QA systems.

  • Takeaways & Limitations

    The factors affecting generalization are not well-understood, and further curriculum, distillation, or teacher-annealing methods remain future work.

Abstract

from arXiv · show

Question answering (QA) tasks have been posed using a variety of formats, such as extractive span selection, multiple choice, etc. This has led to format-specialized models, and even to an implicit division in the QA community. We argue that such boundaries are artificial and perhaps unnecessary, given the reasoning abilities we seek to teach are not governed by the format. As evidence, we use the latest advances in language modeling to build a single pre-trained QA model, UnifiedQA, that performs surprisingly well across 17 QA datasets spanning 4 diverse formats. UnifiedQA performs on par with 9 different models that were trained on individual datasets themselves. Even when faced with 12 unseen datasets of observed formats, UnifiedQA performs surprisingly well, showing strong generalization from its out-of-format training data. Finally, simply fine-tuning this pre-trained QA model into specialized models results in a new state of the art on 6 datasets, establishing UnifiedQA as a strong starting point for building QA systems.

1 Introduction

QA research has been divided by answer format, despite the possibility that underlying linguistic reasoning generalizes across formats. UnifiedQA tests this idea with one format-agnostic, multi-format QA system.

  • Motivation: QA datasets commonly use extractive, abstractive, multiple-choice, and yes/no formats, encouraging format-specific model designs.These formats differ in whether answers are spans, generated text, candidate choices, or binary responses.
  • Motivation: Prior multi-dataset efforts largely remained within a single format, and no single QA system was known to target all four formats.The cited examples include multiple-choice and extractive-only generalization studies.
  • Approach: UnifiedQA exploits information across four QA formats to perform strongly across 20 factoid and commonsense QA datasets.The datasets and formats are summarized in the study’s dataset overview.
  • Approach: UnifiedQA is a single pre-trained QA system trained on seed datasets from multiple formats using natural text without format-specific prefixes.Its text-to-text approach keeps the input representation unified across QA tasks.
  • Results: UnifiedQA works across different formats, generalizes to unseen datasets, and reaches state-of-the-art results after fine-tuning on target tasks.The contribution summary reports performance on par with dedicated systems and new state-of-the-art results after specialization.

2 Related Work

Prior work studied transfer across QA datasets, but generally constrained evaluation to one format or transformed diverse tasks into a common QA format. UnifiedQA instead focuses on unifying multiple QA formats directly within a text-to-text framework.

  • Format-specific transfer: MultiQA studied generalization and transfer across extractive span-selection datasets, while its single system underperformed dataset-specific models.Its leave-one-out results were strong, but not competitive with models tuned separately for each dataset.
  • Format-specific transfer: ORB evaluated multiple datasets spanning extractive and abstractive formats, but its broader cross-format scope differed from UnifiedQA’s objective.The supplied passage introduces ORB’s benchmark but does not state its complete result pattern.
  • Unified text-to-text framing: T5-style text-to-text transfer mixes diverse NLP tasks with textual prefixes, whereas UnifiedQA narrows the task space to QA and avoids format-specific prefixes.UnifiedQA’s models infer the required format from the question content.
  • Unified text-to-text framing: Prior work attributed weaker generalization to studying an overly broad set of NLP tasks, motivating a more coherent collection of QA tasks and formats.The supplied related-work passage presents this as a presumed explanation.

3 UNIFIEDQA: Multi-format Training

UNIFIEDQA trains one text-to-text QA system across multiple formats by mixing datasets without format-specific prefixes. The resulting model supports unseen datasets and can be fine-tuned into specialized systems.

  • 3 UNIFIEDQA: Multi-format Training: UNIFIEDQA targets k QA formats using datasets grouped by format, including training and evaluation examples and optionally unseen evaluation-only datasets.The framework allows a dataset’s training set to be empty when testing generalization to an unseen dataset.
  • 3 UNIFIEDQA: Multi-format Training: Training questions are converted into plain-text inputs and combined into a mixed pool containing all available training instances.Batches are drawn from this pooled data for multi-format training.
  • 3 UNIFIEDQA: Multi-format Training: Each training set contributes batches with probability proportional to 1/|T_i^j|, giving each set the same average number of instances regardless of size.This sampling strategy is intended to balance datasets of different sizes.
  • 3.2 UNIFIEDQA: The Pre-Trained Model: The pre-trained model is built from text-to-text frameworks and trained on eight seed datasets spanning the four formats.The seed datasets include SQuAD 1.1 and 2.0, NarrativeQA, RACE, ARC, OBQA, MCTest, and BoolQ.
  • 3.1 Text-to-Text Encoding: The system uses natural question-first text-to-text encoding with context paragraphs, candidate answers, or both separated by newline markers.The encoding is designed to remain humanlike without being overly specific to one format.
  • 3.1 Text-to-Text Encoding: The four supported formats are extractive, abstractive, multiple-choice, and yes/no question answering.Extractive answers are spans from context, abstractive answers may not be spans, multiple-choice questions provide candidates, and yes/no questions require one of two responses.
  • 3.1 Text-to-Text Encoding: Unlike prior work, UNIFIEDQA uses no task-, dataset-, or format-specific prefixes in its input representation.The output format is inferred from the natural-text instance rather than explicitly specified.
  • 3.2 UNIFIEDQA: The Pre-Trained Model: UNIFIEDQA can be extended with other format combinations and used as a starting point for fine-tuning on additional QA datasets.The reported system uses an 11B-parameter T5 model unless otherwise noted.

4 Formats and Datasets

The evaluation covers 20 QA datasets spanning extractive, abstractive, multiple-choice, and yes/no formats, plus contrast-set variants. Datasets differ in context, candidate-answer, and answer-location properties, so each format uses an appropriate evaluation metric.

  • 4.1 Datasets: 20 existing datasets are evaluated across different QA formats and complex linguistic phenomena.The datasets are grouped into format categories and summarized by their structural properties.
  • 4.1 Datasets: Extractive QA includes SQuAD 1.1, SQuAD 2, NewsQA, Quoref, and ROPES.These datasets use the extractive format.
  • 4.1 Datasets: Abstractive QA includes NarrativeQA, open-domain NaturalQuestions, and DROP.The listed datasets require answers that may not be direct context substrings.
  • 4.1 Datasets: Multiple-choice QA includes MCTest, RACE, OBQA, ARC, QASC, CommonsenseQA, PIQA, SIQA, and Winogrande.Several multiple-choice datasets lack accompanying paragraphs, including ARC, QASC, and OBQA.
  • 4.1 Datasets: The approach is evaluated on datasets with different numbers of answer candidates, including candidate counts not seen during training.The evaluation discusses generalization to such datasets in Section 6.2.
  • 4.1 Datasets: Yes/no QA includes BoolQ, BoolQ-NP, and the binary subset of MultiRC.BoolQ-NP is a naturally perturbed version of BoolQ.
  • 4.1 Datasets: Contrast sets provide expert-generated perturbations for BoolQ, ROPES, Quoref, and DROP that deviate from patterns in the original datasets.These variants are marked with “CS.”
  • 4.2 Evaluation Metrics for Textual Output: Evaluation uses prior-work metrics: span F1 for extractive QA, ROUGE-L or exact match for abstractive QA, candidate-match accuracy for multiple choice, and generated-output accuracy for yes/no QA.The metric is selected according to the format and dataset conventions.

5 Pilot Study: Can Out-of-Format Training Help?

The pilot study tests whether adding out-of-format QA data improves performance on matching and unseen datasets. Results show that such transfer helps broadly, motivating the dataset selection and unified training strategy used later.

  • Pilot findings: 51% to 59%: adding SQuAD2 training data to BoolQ boosts performance on the NP-BoolQ evaluation set.Both BoolQ and NP-BoolQ use the yes/no format, while SQuAD2 supplies out-of-format training data.
  • Pilot findings: At least one out-of-format training set improves performance across the anchor and evaluation datasets examined.The pilot establishes the existence of helpful cross-format transfer rather than a single unified model.
  • Pilot design: The pilot compares anchor-only training with joint training that adds an out-of-format dataset, evaluating both matching and unseen datasets of the anchor format.The study uses small T5 models and identifies the most helpful out-of-format dataset for each evaluation set.
  • Dataset contributions: Figure 3 represents training datasets on the left and evaluation datasets on the right; wider edges indicate greater contribution from out-of-format training.Negative edges and edges between datasets sharing a format are omitted.
  • Dataset contributions: NarrativeQA is the most helpful dataset overall, while large size alone does not guarantee transfer and BoolQ is least helpful in the mix.RACE is large but contributes relatively little, whereas NarrativeQA helps multiple formats.
  • Dataset contributions: Beneficiary datasets include all four QA formats, indicating that cross-format gains are not confined to one evaluation format.The graph also shows strong connections between answer-bearing and extractive datasets because their definitions are similar.

6 Experimental Results

The experiments evaluate UnifiedQA against dedicated systems, on unseen datasets, and as a fine-tuning initialization. UnifiedQA remains competitive across formats, generalizes broadly, and provides a strong starting point for specialized QA models.

  • 6.1 UNIFIEDQA vs. Dedicated Models: UnifiedQA performs almost as well as individual T5 models and sometimes outperforms single-dataset experts such as OBQA and NQA.The comparison uses one multi-dataset system against separate T5-based systems tailored to individual datasets.
  • 6.1 UNIFIEDQA vs. Dedicated Models: UnifiedQA clearly outperforms the ensemble of dataset- and format-specific systems on average while compromising almost nothing in flexibility.The system handles multiple QA formats without the usual large gap from dataset-specific experts.
  • 6.2 Generalization: UnifiedQA shows stronger average generalization across unseen datasets than models trained only on other datasets of a single format.Table 4 reports this pattern across all formats, including cases where UnifiedQA never saw the target datasets during training.
  • 6.2 Generalization: 9 of 12 unseen datasets: UnifiedQA generalizes better than any single-format expert, including tasks with more candidate answers than seen during training.QASC has 8 candidate answers and CommonsenseQA has 5, despite multi-choice training examples using 4 candidates.
  • 6.3 Fine-tuning: Fine-tuning UnifiedQA consistently outperforms fine-tuned T5 and BART and surpasses the best previous scores on the evaluated datasets.Table 5 reports gains over fine-tuned T5 ranging from 1% on CommonsenseQA to 13% on ARC-challenge.
  • 6.4 Ablation: A relatively small training set can suffice when it includes representatives from each QA format.The leave-one-out analysis identifies BoolQ, SQuAD 2.0, OBQA, and NarrativeQA as the top four contributing datasets, each with a different format.

7 Discussion

The discussion argues that successful unification depends on coherent QA task selection, while dataset size and domain only inconsistently predict generalization.

  • Factors affecting generalization: Larger datasets often improve generalization, but RACE and OBQA show similar benefits despite RACE being much larger.
  • Factors affecting generalization: Similar domains can help transfer, yet BoolQ and SQuAD barely benefit each other despite both using Wiki paragraphs.
  • Factors affecting generalization: The factors affecting generalization remain poorly understood, leaving room for future investigation.
  • Unifying QA formats and text-to-text models: A coherent choice of QA tasks and formats is more likely to succeed in multi-task learning than mixing very different tasks.
  • Unifying QA formats and text-to-text models: Restricting the task space to coherent QA formats allows inputs to omit explicit prefixes identifying tasks or formats.

8 Conclusion

The paper argues that QA formats impose artificial boundaries because desired reasoning abilities are not format-specific. UNIFIEDQA unifies four formats, performs competitively across datasets, generalizes to unseen datasets, and supports state-of-the-art fine-tuning, while extensions remain future work.

  • 8 Conclusion: QA format boundaries are artificial because the reasoning abilities being taught and tested are not tied to specific formats.
  • 8 Conclusion: UNIFIEDQA is a single text-to-text pre-trained QA system spanning four common QA formats.
  • 8 Conclusion: UNIFIEDQA performs on par with 8 dataset-specific expert models and generalizes to many unseen datasets with seen formats.
  • 8 Conclusion: Fine-tuning UNIFIEDQA on target datasets achieves state-of-the-art performance on many QA tasks.
  • 8 Conclusion: Extensions to direct-answer questions are identified as a promising avenue for future work.

A.1 Datasets: Details

The appendix organizes the evaluation into extractive, abstractive, multiple-choice, yes/no, and contrast-set datasets, describing their answer formats, contexts, and evaluation variations.

  • Dataset organization: The evaluation covers 19 existing datasets spanning varied QA formats and complex linguistic phenomena.Table 2 summarizes properties such as paragraph availability, answer containment, and candidate answers.
  • Extractive QA (EX): Extractive QA requires answers to be substrings of a context paragraph, with SQuAD 2 adding unanswerable questions.
  • Abstractive QA (AB): Abstractive QA permits answers that are not mere context substrings and includes narrative understanding, rudimentary arithmetic, multi-part inference, and retrieved open-domain context.
  • Multiple-choice QA (MC): Multiple-choice QA supplies candidate answers, while datasets differ in whether they include context paragraphs and how many candidates they provide.
  • Yes/No QA (YN) and contrast sets: Yes/no QA contains binary-answer questions, and contrast sets use expert-generated perturbations that depart from patterns in the original datasets.

A.2 Details on the experiments:

The experiments use T5 and BART text-to-text models with specified model sizes, sequence limits, optimization settings, hardware, and training schedules for pretraining and fine-tuning.

  • Models: The experiments use T5 and BART text-to-text frameworks.
  • Model sizes: Most experiments use T5(11B) with 11 billion parameters, while BART large has 440 million parameters.
  • Input/output size: Input and output token limits are 512 and 100, respectively.
  • Pretraining: Pretraining on seed datasets runs for 100k steps.
  • Optimization: Learning rates are 1e-3 for T5 and 1e-5 for BART, with batch sizes of 8 and 120, respectively.
  • Infrastructure: T5 experiments use v3-8 TPUs, whereas BART experiments use eight 32GB GPUs.
  • Training time: Pretraining UNIFIEDQA takes approximately 36 hours on T5(11B) and 55 hours on BART.
  • Fine-tuning on datasets: Fine-tuning runs for 60k steps with checkpoints every 2k steps, selecting the model with the highest development-set score.

A.3 UNIFIEDQA: Different Sizes

UNIFIEDQA’s cross-format training remains effective across model sizes and datasets. It performs nearly as well as dataset-specific experts, benefits from mixed-format training, and gains an advantage when fine-tuning with limited Winogrande data.

  • Different sizes: The different-size evaluation reports one UNIFIEDQA system per dataset row.The accompanying table presents UNIFIEDQA variants of different sizes across the datasets.
  • Cross-dataset performance: UNIFIEDQA performs almost as well as the best single-dataset experts and sometimes exceeds them, including on OBQA and NQA.The comparison’s average performance is summarized in the final column.
  • Pairwise mixing: Mixing datasets with different QA formats improves within-format generalization across several anchor datasets.Extractive and abstractive datasets help NQA generalize to other abstractive datasets, while cross-format training also improves RACE/OBQA and yes/no models.
  • Pairwise mixing: Extractive and abstractive datasets help an NQA model generalize better to other abstractive datasets.This is the result reported for the NQA anchor subtable.
  • Winogrande: Fine-tuning UNIFIEDQA produces stronger Winogrande results when training data is limited than fine-tuning the corresponding vanilla language models.Across the overall AUC metric, UNIFIEDQA has a slight edge over fine-tuned vanilla T5 and BART.
Loading 2005.00700v3…