PaperlayerView exact revision on arXiv

Source-linked AI summary

An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering

Shayne Longpre, Yi Lu, Zhucheng Tu, Chris DuBois

arXiv:1912.02145v1cs.CL

TL;DR

Extractive question answering remains difficult to generalize to target domains without similar training data or knowledge of their distributions. The paper evaluates pre-trained models, sampling strategies, and back-translation augmentation, finding that simple negative sampling is particularly effective and that the combined approach ranked second in the MRQA-19 Shared Task.

  • Problem

    Extractive question answering generalization is especially challenging for target domains without similar training data or knowledge of their distributions.

    Source: §1 · p. 1

  • Method

    The authors evaluate pre-trained models alongside domain and data sampling strategies and query and context back-translation augmentations.

    Source: §1 · p. 1, §5 · p. 6

  • Results

    The XLNet-based submission ranked second in the MRQA-19 Shared Task, and No Answer sampling provided the most significant benefits, particularly on long sequences.

    Source: §5 · p. 6

  • Takeaways & Limitations

    No Answer sampling is a particularly important simple technique for training extractive models on long sequences.

    Source: §5 · p. 6

Abstract

from arXiv · show

To produce a domain-agnostic question answering model for the Machine Reading Question Answering (MRQA) 2019 Shared Task, we investigate the relative benefits of large pre-trained language models, various data sampling strategies, as well as query and context paraphrases generated by back-translation. We find a simple negative sampling technique to be particularly effective, even though it is typically used for datasets that include unanswerable questions, such as SQuAD 2.0. When applied in conjunction with per-domain sampling, our XLNet (Yang et al., 2019)-based submission achieved the second best Exact Match and F1 in the MRQA leaderboard competition.

1 Introduction

The paper addresses the challenge of generalizing extractive question answering to domains with little or no matching training data, and adapts its data and learning regime to reduce domain bias. It finds that simple negative sampling helps out-domain performance, while back-translated augmentations yield no noticeable improvement.

1 Introduction

  • Generalizing to a target domain is especially difficult when similar training data or knowledge of its distribution is unavailable.The authors identify this as the setting of the MRQA Shared Task.Source: §1 · p. 1
  • The authors curate input data and the learning regime to encourage representations not biased by any one domain or distribution.They argue that more training-domain data does not necessarily improve target-domain performance.Source: §1 · p. 1
  • The study compares BERT and XLNet and explores domain sampling, data sampling, and query and context augmentation strategies.The models are large pre-trained language models; the augmentations include back-translation.Source: §1 · p. 1
  • Negative samples designed to teach abstention prove highly effective out-domain, while back-translated augmentations yield no noticeable improvement.The authors also report no noticeable improvement from back-translated augmentations sampled with active-learning strategies.Source: §1 · p. 1

2 Data

The section describes six separately collected training QA datasets and six out-domain development sets from other QA datasets. It explains how long contexts are segmented and how segments without answer spans are treated.

2 Data

  • The training data comprises six separately collected QA datasets, with six associated in-domain development sets and six out-domain development sets from other QA datasets.Table 1 reports examples, segments, and the percentage of no-answer segments by dataset.Source: §2 · p. 1, Table 1 · p. 2
  • Table 1 reports each dataset’s number of question-context pairs, number of question-context chunks, and percentage of no-answer chunks.The caption identifies the table’s measures; it does not state their values.Source: Table 1 · p. 2
  • Long contexts are split into overlapping segments to satisfy memory constraints, and segments lacking the gold answer span are termed “no-answer.”The sliding window is parameterized by document stride D and maximum sequence length M.Source: §2 · p. 2

3 System Overview

The system combines pretrained models with domain-aware sampling and back-translation augmentation to encourage broad generalization. Experiments find benefits from multi-domain pre-fine-tuning and negative sampling, while testing several ways to generate and sample paraphrases.

3.1 XLNet

  • XLNet Large served as the final-submission model, with training-time end logits conditioned on the gold start and inference using beam search over end logits.The implementation also added per-dataset sampling, subepoch validation checkpoints, and postprocessing to remove special tokens and unusual spacing.Source: §3.1 · p. 2

3.2 Domain Sampling

  • Multi-domain pre-fine-tuning raised Exact Match almost universally across in-domain and out-domain sets, while later domain-specific fine-tuning did not significantly erase those gains.SearchQA- and TriviaQA-fine-tuned models performed relatively poorly outside their own datasets; the authors consequently reduced their sampling to avoid oversampling them.Source: §3.2 · pp. 2–3, Figure 1 · p. 3

3.3 Negative Sampling

  • Including No-Answer segments yields non-trivial MRQA improvements, magnified when contexts are much longer than the maximum sequence length.Table 2 compares performance with and without No-Answer segments across maximum sequence lengths; the authors hypothesize that irrelevant spans in negative segments can attract probability mass.Source: Table 2 · p. 3, §3.3 · p. 4

3.4 Paraphrasing by Back-Translation

  • The authors generated query and context paraphrases through English-German-English back-translation, translating contexts sentence by sentence and attempting to recover answer spans in paraphrased text.They used string matching first, then a character-level 2-gram Jaccard-overlap heuristic; examples without recoverable answer spans were discarded.Source: §3.4 · p. 4

3.4.1 Augmentation Strategy

  • For each query-context pair, query and context paraphrases were independently selected over the originals according to separate probabilities, and an example was added if either paraphrase was selected.This allowed the authors to vary how often query or context augmentations entered the training data.Source: §3.4.1 · pp. 4–5

3.4.2 Active Learning

  • Active-learning augmentation replaced uniform sampling with example probabilities derived from the best model’s F1 scores, using hard, moderate, or soft scoring distributions.The hard distribution gives examples with F1 = 1 negligible probability; moderate and soft distributions penalize correct predictions less aggressively, and ε = 0.01 prevents zero probability.Source: §3.4.2 · p. 5

4 Experiments and Discussion

Experiments compared sampling and augmentation strategies across BERT and XLNet configurations. No-answer sampling improved out-domain performance, while back-translated query and context augmentations did not help XLNet Large; the final submission used domain and negative sampling.

4 Experiments and Discussion

  • BERT Base Cased supported the most expensive sampling explorations, XLNet Base Cased checked their transfer, and XLNet Large Cased formed the initial basis for final-submission contenders.For XLNet Large Cased, the authors found gradient accumulation and a lower learning rate critical for training stability.Source: §4 · p. 5
  • Models were fine-tuned for two epochs, and the checkpoint with the highest Out-Domain Macro-Average on a 2,000-example dev subsample was selected for each run.Training data was capped at 120k examples per dataset, except SearchQA at 100k, with one detected answer per example.Source: §4 · p. 5
  • At max sequence length 200, including no-answer segments raised XBC Out-Domain EM from 43.78 to 50.04, with a particularly noticeable effect on longer sequences.The authors report that including no-answer segments drastically outperformed the typical practice of excluding them.Source: §4 · p. 5
  • Back-translated query and context augmentations did not help in-domain or out-domain performance on XLC, despite small BBC metric boosts.The authors suggest XLC may already capture paraphrase variation and that the training data is already diverse; SearchQA and TriviaQA gained more than one F1 point from some augmentation.Source: §4 · p. 6, Table 3 · p. 6
  • The final submission used fine-tuned XLNet Large with domain and negative sampling, omitting augmentation and active sampling techniques that had not aided out-domain performance.Its hidden development-set breakdown and held-out leaderboard test results are reported in Tables 4 and 5.Source: §4 · p. 6, Table 4 · p. 6, Table 5 · p. 6

5 Conclusion

The paper evaluates pre-trained models, domain and negative sampling, back-translation, and active learning for multi-domain question answering. No-answer sampling was particularly beneficial for long sequences, and the combined findings yielded the second-ranked MRQA-19 Shared Task submission.

5 Conclusion

  • The combined findings culminated in the second-ranked submission on the MRQA-19 Shared Task.The submission used XLNet with per-domain and negative sampling, as stated in the paper abstract.Source: §5 · p. 6
  • No-answer sampling provided the most significant benefits and was particularly important for training extractive models on long sequences.Source: §5 · p. 6
  • The experiments assessed BERT and XLNet, domain sampling, negative sampling, back-translation augmentation, and active learning.Source: §5 · p. 6

Related papers

Loading 1912.02145v1…