Source-linked AI summary
A BERT Baseline for the Natural Questions
Chris Alberti, Kenton Lee, Michael Collins
TL;DR
Natural Questions may be a harder question-answering challenge because its questions are posed before the relevant document is seen and its documents are longer. This note presents BERTjoint, which predicts short and long answers in one model; it reports reducing the gap to the human upper bound by 30% for long answers and 50% for short answers.
Problem
The authors hypothesize that NQ may be harder than SQuAD 2.0 and CoQA because people ask questions before seeing the document and NQ documents are longer.
Source: §1 · p. 1Method
Results
30% for long answers and 50% for short answers are the reported relative reductions in the gap between original dataset-paper model F1 scores and the human upper bound.
Source: §1 · p. 1Takeaways & Limitations
The authors present the BERT-based model as a new baseline for the newly released Natural Questions dataset.
Source: §5 · p. 4The baseline always predicts a single short answer, leaving yes/no and multiple disjoint-span predictions as potential extensions.
Source: §3 · p. 3
Abstract
from arXiv · showhide
This technical note describes a new baseline for the Natural Questions. Our model is based on BERT and reduces the gap between the model F1 scores reported in the original dataset paper and the human upper bound by 30% and 50% relative for the long and short answer tasks respectively. This baseline has been submitted to the official NQ leaderboard at ai.google.com/research/NaturalQuestions. Code, preprocessed data and pretrained model are available at https://github.com/google-research/language/tree/master/language/question_answering/bert_joint.
1 Introduction
The note presents a BERT baseline for NQ, motivated by the authors’ view that NQ may pose a harder question-answering challenge than SQuAD 2.0 and CoQA. Its key modeling choices jointly predict short and long answers and adapt training to long documents and many null instances.
1 Introduction
- The authors hypothesize that NQ may be a harder challenge than SQuAD 2.0 and CoQA because its questions precede document viewing and its documents are longer.They note BERT's strong results on SQuAD 2.0 and CoQA, with 2.5 and 6 F1 points of headroom, respectively.Source: §1 · p. 1
- 30% relative for long answers and 50% relative for short answers is the reported reduction in the gap between the original dataset-paper model F1 and the human upper bound.The authors also report 22.5 F1 points of remaining headroom for long answers and 23 F1 points for short answers.Source: §1 · p. 1
- The approach splits documents into overlapping token windows and down-samples null instances to balance training.The windows follow the original BERT approach for SQuAD; null instances are reduced aggressively during training.Source: §1 · p. 1
- At inference, the model uses the [CLS] token to predict null instances and ranks answer spans by their score difference from [CLS].This is the fourth listed modeling insight.Source: §1 · p. 1
- BERTjoint predicts short and long answers in one model rather than a two-model pipeline.The model name emphasizes this joint modeling choice.Source: §1 · p. 1
2 Data Preprocessing
NQ pairs Google queries with Wikipedia pages and annotations for long answers and short answer spans, with empty annotations representing unanswered questions or absent short answers. Preprocessing creates overlapping BERT-sized instances, assigns answer targets, balances null examples, and marks document structure.
2 Data Preprocessing
- NQ contains 307,373 training, 7,830 development, and 7,842 test examples pairing a Google query with a Wikipedia page and annotated long and short answers.Annotations may be empty; 1% of documents have a yes/no short answer rather than short spans.Source: §2 · p. 2
- Each document is split into overlapping 512-token instances with a 128-token stride, averaging 30 instances per NQ example.Each instance begins with [CLS], the tokenized question, and [SEP], followed by document tokens and a final [SEP].Source: §2 · p. 2
- Training targets point to all annotated short spans, the contained long span when short spans are absent, or [CLS] when neither span is contained.Instances with [CLS] targets are called null instances; answer types distinguish short, yes, no, long, and no-answer cases.Source: §2 · p. 2
- Because about 98% of generated instances are null, training down-samples them 50-fold to yield roughly balanced classes and about 500,000 instances.The passage attributes the high null rate to long documents and 51% of documents having no answer to the query.Source: §2 · p. 2
- Paragraph, table, and list markers identify document structure because early paragraphs and tables are more likely to contain the annotated answer.The special markers are atomic tokens.Source: §2 · p. 2
3 Model
The model represents each training instance with context, answer-span indices, and an answer-type label, then uses BERT-derived scores to predict short and long answers. At inference, it ranks spans across contexts and maps the selected short answer to a long answer.
3 Model
- A training instance is a four-tuple containing 512 wordpiece IDs, inclusive answer-span indices, and one of five answer-type labels.The labels are short, long, yes, no, and no-answer.Source: §3 · p. 2
- BERT-derived start, end, and answer-type scores yield probabilities used in the model's training loss.The passage describes each probability as a softmax over scores.Source: §3 · pp. 2–3
- At inference, the model ranks document spans across contexts by their score and returns the highest-scoring span as the short answer.The span score is the log-odds between the answer span and the [CLS] span.Source: §3 · p. 3
- The long answer is the top-level DOM-tree node containing the predicted short answer, and both predictions receive the document's maximum span score.Source: §3 · p. 3
- The baseline always predicts a single short-answer span and relies on the official evaluator to threshold predictions to long-answer-only or no-answer outputs.The authors expect possible improvements from predicting yes/no answers or multiple disjoint short-answer spans.Source: §3 · p. 3
4 Experiments
The BERT model closes part of the gap between original NQ baselines and the human upper bound, while substantial F1 headroom remains for both answer tasks.
4 Experiments
- The model was initialized from BERT finetuned on SQuAD 1.1 and further finetuned on NQ training instances; one epoch at 3 · 10^-5 was the best tested setting.Training used Adam with batch size 8.Source: §4 · p. 3
- Evaluation on the NQ development and test sets took about five hours on a single Tesla P100 GPU.Source: §4 · p. 3
- 30% and 50% relative gap closure for long and short answers, respectively, still leaves more than 20 F1 points of headroom on both tasks.The comparison is between the original NQ baseline systems and the super-annotator upper bound.Source: §4 · pp. 3–4
5 Conclusion
The paper presents a BERT-based baseline for Natural Questions as a potential starting point for further question-answering research.
5 Conclusion
- The paper presents a new BERT-based baseline for Natural Questions.Source: §5 · p. 4
- The baseline is intended as a starting point for researchers developing better Natural Questions models.Source: §5 · p. 4
- The authors also identify question-answering datasets with similar characteristics as a potential area of application.Source: §5 · p. 4