PaperlayerView exact revision on arXiv

Source-linked AI summary

Frustratingly Easy Natural Question Answering

Lin Pan, Rishav Chakravarti, Anthony Ferritto, Michael Glass, Alfio Gliozzo, Salim Roukos, Radu Florian, Avirup Sil

arXiv:1909.05286v1cs.CL

TL;DR

The paper addresses limited reproducibility and uncertain evidence about what drives strong QA performance on Natural Questions. It combines BERT transfer learning with algorithmic improvements, data augmentation, and ensembling, finding that a simple approach can outperform a system trained on 4 million more examples and that ensembling improves results further.

  • Problem

    QA leaderboard systems often emphasize complex models, synthetic data, or undocumented ensembles, leaving reproducibility and the value of simple transfer learning insufficiently established on Natural Questions.

  • Method

    The paper fine-tunes BERT for NQ and evaluates Attention-over-Attention, BERT-layer combinations, pre-training, data augmentation, and ensemble strategies.

  • Results

    The simple transfer-learning approach achieves new SOTA on NQ while using 307,373 training examples, outperforming a system trained on 4 million additional examples.

  • Takeaways & Limitations

    Simple BERT transfer learning can be highly competitive on NQ, while documented ensembling and lightweight augmentations provide additional performance gains.

  • Takeaways & Limitations

    Small learning-rate changes can affect final F1 scores by a couple of points, so learning-rate selection remains insufficiently robust.

Abstract

from arXiv · show

Existing literature on Question Answering (QA) mostly focuses on algorithmic novelty, data augmentation, or increasingly large pre-trained language models like XLNet and RoBERTa. Additionally, a lot of systems on the QA leaderboards do not have associated research documentation in order to successfully replicate their experiments. In this paper, we outline these algorithmic components such as Attention-over-Attention, coupled with data augmentation and ensembling strategies that have shown to yield state-of-the-art results on benchmark datasets like SQuAD, even achieving super-human performance. Contrary to these prior results, when we evaluate on the recently proposed Natural Questions benchmark dataset, we find that an incredibly simple approach of transfer learning from BERT outperforms the previous state-of-the-art system trained on 4 million more examples than ours by 1.9 F1 points. Adding ensembling strategies further improves that number by 2.3 F1 points.

Introduction

The paper targets reproducible, high-performing machine reading comprehension on Natural Questions, a benchmark designed to reduce observation bias. It contributes BERT-based algorithmic improvements, data augmentation, and ensemble strategies.

  • Background: MRC is framed as reading text and answering questions from it, with applications in natural language understanding, conversational agents, and customer support.
  • Motivation: Natural Questions samples search-log queries intended to be answerable from Wikipedia, avoiding the observation bias of questions written after seeing answer-containing passages.The task requires both short-answer extraction and long-answer selection, with reported human upper bounds of 76% and 87%, respectively.
  • Contributions: The paper adds an Attention-over-Attention layer and BERT-layer combinations, with AoA producing the best single-model performance on NQ.A simple BERT transfer-learning strategy nearly matches the more complex AoA model.
  • Contributions: Shuffling training data improves performance enough to outperform a prior system trained on 4 million additional synthetic QA examples.The paper presents this as a simple but effective data augmentation strategy.
  • Contributions: The paper documents ensemble methods that combine outputs from single MRC systems to improve leaderboard performance.This addresses the limited documentation of ensemble techniques in prior leaderboard systems.

Related Work

Prior MRC systems commonly rely on BERT-like models, synthetic data augmentation, and undocumented ensembles. The paper responds by describing its ensemble recipe alongside algorithmic and augmentation improvements for NQ.

  • BERT-based systems: Recent MRC leaderboard systems are predominantly BERT-based and often add components such as syntax layers or Attention-over-Attention to achieve super-human results.
  • Data augmentation: Synthetic question generation gave a prior NQ system 4 million additional training examples and state-of-the-art performance.The cited approach filters generated examples using round-trip consistency between question generation and answer extraction.
  • Ensembling: Leaderboard submissions are often ensembles, but their underlying combination strategies are rarely documented, motivating this paper’s reproducible ensemble recipe.The paper specifically notes that even a prominent BERT ensemble does not provide its ensemble strategies.

Model Architecture

The model starts from BERT-for-QA, which predicts answer spans and answer types, then adds Attention-over-Attention and optional combinations of BERT layers. AoA fuses question and document attention to support answer extraction.

  • Attention-over-Attention: Attention-over-Attention combines query-to-document and document-to-query attentions into document attention weighted by query-word importance.The mechanism uses question and context representations and produces a final document-level attention vector.
  • BERT-for-QA: BERT uses Transformer layers with special [CLS] and [SEP] tokens to represent question-and-context inputs as one sequence.The supplied passages distinguish BERTLARGE’s 24 layers and 16 heads from BERTBASE’s 12 layers and 12 heads.
  • BERT-for-QA: BERT-for-QA predicts token-level answer beginnings and ends while classifying short, long, yes/no, or null answer types from the [CLS] representation.For NQ long answers, the model identifies the containing HTML span after optimizing short-answer spans.
  • Attention-over-Attention: The two-headed AoA layer is trained with the answer-extraction layer after separating BERT’s question and context representations.Its outputs are combined with the beginning and ending answer distributions for extraction.
  • BERT Layer Combination: Instead of using only BERT’s final layer, the model can learn a softmax-weighted linear combination of all encoded layers for downstream answer extraction.The trainable layer weights are jointly optimized with BERT-for-QA parameters.

Model Training

Training begins with pre-trained BERT followed by task-specific NQ fine-tuning, while additional pre-training and augmentation strategies are explored to improve performance.

  • Training Pipeline: The training pipeline fine-tunes pre-trained BERT on NQ with an added QA sequence-prediction layer, preceded when useful by task-specific pre-training.The paper evaluates different pre-training and data augmentation strategies before target-dataset fine-tuning.
  • Pre-training: The paper explores Whole Word Masking and Span Selection Pre-Training as BERT parameter pre-training options before NQ fine-tuning.Whole Word Masking masks complete words, while Span Selection Pre-Training adds an unsupervised QA-specific auxiliary task.

3. BERT-for-QA

The paper builds on BERT-for-QA with task-specific pre-training and data augmentation from prior MRC datasets, including SQuAD 2.0, NewsQA, and TriviaQA. Similarity-based sampling favors SQuAD 2.0 examples because its task aligns most closely with NQ.

  • SQuAD 2.0 pre-training performs better than SQuAD 1.1 for NQ, likely because both datasets include unanswerable questions.
  • The authors augment NQ training with human-annotated examples from SQuAD 2.0, NewsQA, and TriviaQA.
  • Augmentation can dramatically increase training time and overshadow target-domain NQ examples, motivating random and similarity-based sampling strategies.
  • Similarity-based sampling ranks augmentation examples by a classifier’s predicted likelihood of belonging to NQ.
  • Approximately 80% of similarity-sampled examples come from SQuAD 2.0, while TriviaQA examples are least similar to NQ.

Experiments

The experiments evaluate BERT-based NQ models on a dataset with short- and long-answer annotations, confidence-based abstention, and separate Short Answer and Long Answer F1 metrics. NQ is positioned as a challenging benchmark with partial unanswerability and substantial headroom over automated systems.

  • NQ provides 307,373 training queries, 7,830 development queries, and 7,842 test queries.
  • Each question includes annotated short-answer offsets when available and a containing long-answer span when available.
  • Models must output confidence scores so evaluation can select an answer threshold, with Short Answer and Long Answer F1 as headline metrics.
  • NQ combines partial un-answerability and natural question generation, making it useful for evaluating architecture and training choices.
  • Training uses the Huggingface PyTorch implementation of BERT, starting from either BERTBASE or BERTLARGE.

1. Stride

The model processes long Wikipedia articles by splitting them into overlapping spans because BERT accepts at most 512 tokens. The authors vary stride length to balance computational cost and F1 performance.

  • 1. Stride: Splitting articles with a sliding window accommodates BERT’s 512-token input limit while allowing stride-length experiments.

2. Negative Instance Sub-Sampling

The section addresses span imbalance, answerability confidence, and ensemble construction for NQ. It evaluates multiple model-combination strategies under a fixed four-model decoding budget and reports design choices for candidate aggregation and search.

  • 2. Negative Instance Sub-Sampling: Only 65% of questions are answerable by a short span, and 90% of those contain one correct span averaging four words, creating severe positive-negative span imbalance.
  • 2. Negative Instance Sub-Sampling: The authors vary answerable and unanswerable sampling rates because preliminary BERTBASE models were overconfident on unanswerable questions.
  • Multiple Model Ensembling Experiments: Four-model ensembles are used because four models can be decoded within the 24-hour NQ leaderboard limit.
  • Multiple Model Ensembling Experiments: Ensembling compares different random seeds with different architectures and pre-training data to reduce variance or combine complementary systems.
  • Multiple Model Ensembling Experiments: The ensemble procedure averages scores across each system’s top-20 long- and short-answer candidates, imputing zero for spans absent from a model.
  • Multiple Model Ensembling Experiments: Arithmetic mean outperforms median, geometric mean, and harmonic mean for combining model scores.
  • Multiple Model Ensembling Experiments: Greedy search builds short- and long-answer ensembles from 41 BERTLARGE models, optimizing the corresponding F1 metric.

Results

The experiments show that simple pre-training, targeted sampling, and ensemble design improve Natural Questions performance, with gains depending on architecture, aggregation, and search strategy.

  • Stride: 0.9% gain from increasing stride length to 192, while reducing spans and training time.Further increases deteriorate performance, possibly because of relevant-context size in Wikipedia articles.
  • Negative Instance Sub-Sampling: 0.8% SA F1 improvement comes from sampling 1% of negative instances and 4% of unanswerable-question examples instead of the prior setting.Higher negative-instance rates eventually deteriorate performance.
  • Pre-Training and Model Architecture: The best single-model performance comes from WWM BERT pre-trained on SQuAD 2.0 and fine-tuned with an AoA layer on NQ.Layer combinations slightly improve short-answer F1 but reduce long-answer F1 by the same amount relative to BERT-for-QA.
  • Data Augmentation: BERT+WWM with sentence-order shuffling outperforms the prior SOTA trained on 4 million synthetic QA pairs, despite using 307,373 augmentation examples.Naively mixing augmentation examples into early mini-batches does not significantly improve single-model performance over BERT+WWM.
  • Ensembling: Ensembling four random-seed models gains roughly 2.5% in both SA and LA F1, while diverse models add 1% SA F1 and 1.2% LA F1.Greedy search optimizing long-answer performance generalizes best to dev-test, and the final ensemble includes configurations that are individually weaker but complementary.
  • Duplicate Answer Span Aggregation: Max aggregation gives the best short-answer F1, whereas normalized noisy-or gives the best long-answer F1 for duplicate answer spans.The final submission combines these aggregation choices with greedy long-answer search.

Conclusion

The paper shows that computationally inexpensive transfer learning can match state-of-the-art performance on benchmark MRC tasks and achieve new state-of-the-art results on Natural Questions.

  • Conclusion: Transfer learning reaches state-of-the-art benchmark performance while being computationally inexpensive.The approach uses the original NQ training set rather than relying on millions of additional examples.
Loading 1909.05286v1…