Source-linked AI summary
Learning Recurrent Span Representations for Extractive Question Answering
Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, Jonathan Berant
TL;DR
Extractive question answering must identify arbitrary answer spans in evidence documents rather than choose from predefined candidates. RASOR explicitly represents and scores all spans using recurrently shared computation and global normalization. It improves published neural baselines and reduces error against the SQuAD baseline by more than 50%, while retaining failure cases involving nearby semantically plausible spans.
Problem
Reading comprehension needs systems that answer document-based questions, but earlier approaches often relied on manually or externally defined candidate answers, whereas SQuAD permits arbitrary text spans.
Method
RASOR enumerates all candidate answer spans, builds fixed-length span representations with recurrently shared computation, and globally normalizes their scores during learning.
Results
5% exact match improvement over Wang & Jiang (2016), 3.6% predicted answer F1 improvement, and more than 50% error reduction relative to Rajpurkar et al.’s baseline against the human upper bound.
Takeaways & Limitations
Explicitly scoring complete spans enables exact decoding and improves performance over approaches that predict words or span boundaries separately.
Takeaways & Limitations
RASOR can select an answer of the correct type near a phrase overlapping the question, assigning substantial probability to an incorrect span.
Abstract
from arXiv · showhide
The reading comprehension task, that asks questions about a given evidence document, is a central problem in natural language understanding. Recent formulations of this task have typically focused on answer selection from a set of candidates pre-defined manually or through the use of an external NLP pipeline. However, Rajpurkar et al. (2016) recently released the SQuAD dataset in which the answers can be arbitrary strings from the supplied text. In this paper, we focus on this answer extraction task, presenting a novel model architecture that efficiently builds fixed length representations of all spans in the evidence document with a recurrent network. We show that scoring explicit span representations significantly improves performance over other approaches that factor the prediction into separate predictions about words or start and end markers. Our approach improves upon the best published results of Wang & Jiang (2016) by 5% and decreases the error of Rajpurkar et al.'s baseline by > 50%.
1 INTRODUCTION
Reading comprehension asks systems to answer questions about documents, but earlier datasets and models often restricted answers to externally defined candidate sets. SQuAD instead permits arbitrary answer strings from the evidence document, creating a need for models that identify answer spans directly.
- Reading comprehension is a central NLP task because it requires systems to answer questions about document contents.It is both practically relevant and considered a flagship task requiring deep language understanding.
- Earlier datasets commonly limited answers to small candidate sets defined by annotators or external NLP pipelines.Corresponding models relied on candidate locations, mention-level attention, classifiers, or network memories.
- SQuAD allows answers to be arbitrary strings drawn from the evidence document, rather than restricting them to predefined candidates.Its baseline used a syntactic parser to prune O(N^2) candidates, but made 20.7% of questions unanswerable.
- Boundary-based neural models improved on the SQuAD baseline but remained susceptible to search errors from greedy training and decoding.These models predicted individual words or answer-span boundaries rather than directly scoring complete spans.
- RASOR enumerates answer spans, builds fixed-length span representations with recurrent computation, and globally normalizes span scores during training.The approach improves over Wang & Jiang (2016) by 5% exact match and 3.6% predicted answer F1, while closing more than 50% of the gap to the human upper bound on both metrics.
2 EXTRACTIVE QUESTION ANSWERING
Extractive question answering predicts an answer span from a passage given a question. The section situates this task among lexical and syntactic SQuAD baselines, Cloze-style prediction, and sentence-level extraction, while emphasizing that SQuAD permits longer non-entity phrases.
- Extractive question answering predicts one answer span, represented by start and end indices, from a passage given a question.A learned system maps question–passage pairs to an answer span using training triples.
- The original SQuAD baseline used sparse lexical, part-of-speech, and dependency-path features to score candidate answers.Its analysis found lexical and syntactic features contributed most strongly to performance.
- The Cloze task predicts a concealed span from a declarative sentence using a supporting passage, often in datasets centered on entities.Simple models can approach the human upper bound on some Cloze datasets.
- Unlike Cloze datasets, SQuAD answers often include non-entities and longer phrases, making extractive question answering more challenging.This distinction follows the characterization of annotated SQuAD answers relative to Cloze answers.
- Sentence-level extractive question answering selects a sentence from a document given a question, with datasets including TREC evaluations and WikiQA.Recent work has applied neural architectures to this sentence-extraction setting.
3 MODEL
RASOR explicitly represents every candidate answer span while keeping computation tractable through recurrently reused computations. It incorporates question information into passage representations, scores spans jointly, and learns by maximizing the correct span's log-likelihood.
- 3 MODEL: RASOR explicitly computes fixed-length embedding representations for all candidate answer spans, enabling joint representations that can be globally normalized during learning.The architecture exploits the tractable enumeration of answer spans in extractive question answering.
- 3 MODEL: The model reduces naive cubic span-aggregation cost to quadratic size by reusing recurrent computations for shared passage-word substructures.Shared computations allow different spans with common passage words to be encoded efficiently.
- 3.1 SCORING ANSWER SPANS: RASOR defines a probability distribution over all candidate spans and selects the span with maximum likelihood, avoiding separate independent word or boundary predictions.The span representation is scored by a feed-forward network and normalized with a softmax.
- 3.2 RASOR: RECURRENT SPAN REPRESENTATION: A passage-level BiLSTM encodes left and right context so span endpoints can be concatenated to represent each span's inside and outside information.The experiments use a multi-layer BiLSTM, whose upper layers can depend on the entire passage.
- 3.3 QUESTION-FOCUSED PASSAGE WORD EMBEDDING: Question-focused passage embeddings concatenate the original passage-word embedding with passage-aligned and passage-independent question representations.The passage-independent representation is shared across all passage words, while the aligned representation uses attention to connect question and passage words.
- 3 MODEL: The model learns end-to-end by maximizing the log-likelihood of the correct answer candidates and backpropagating the resulting errors.The word representations are based on fixed pretrained embeddings for question and passage words.
4 EXPERIMENTAL SETUP
The experiments use recurrent neural components with pretrained word representations and tune architectural, regularization, and optimization choices for SQUAD.
- Representation and regularization: Question and document words use fixed 300-dimensional GloVe embeddings trained on 840bn words, with 200k-word coverage and random embeddings for OOV words.OOV words are projected onto one of 1m randomly initialized 300-dimensional embeddings.
- Representation and regularization: The model couples input and forget gates in its LSTMs and applies one dropout mask across all LSTM time-steps.
- Configuration selection: Grid searches vary LSTM state size, feed-forward width and depth, LSTM dropout, stacked layers, and learning-rate decay.The best configuration uses 50-dimensional LSTM states, two-layer BiLSTMs, and dropout of 0.1 throughout.
- Training: All models are implemented in TensorFlow and trained on SQUAD with ADAM, mini-batches of 4, and 10 asynchronous threads on one machine.
5 RESULTS
RASOR is evaluated on SQUAD against prior systems and through model variations, showing gains from explicit span modeling, task-aligned objectives, endpoint interactions, and question representations.
- Evaluation: The evaluation trains on 80k SQUAD triples and reports results on 10k development and test examples using exact match and unigram-overlap F1.The official script compares predictions with the closest labeled answer among three development-set references.
- Comparisons to other work: More than 50% error reduction over Rajpurkar et al.’s logistic-regression baseline is achieved by RASOR for both exact match and F1 relative to human performance.
- Comparisons to other work: 14% error reduction over Wang and Jiang’s best-performing Match-LSTM model results from RASOR’s explicit modeling of the quadratic set of possible answers.The comparison contrasts RASOR’s exact decoding with the other system’s greedy training and evaluation.
- Model variations: RASOR’s ablations examine the two question representations and whether the learning objective accurately reflects span prediction.
- Question representations: The passage-aligned question representation is crucial, while removing the passage-independent representation causes a drastic performance drop.
- Question representations: The passage-independent representation is less important than the passage-aligned representation, but the BiLSTM still contributes over 3% exact match and F1.
- Learning objectives: Learning objectives perform better when their labels closely align with the task of selecting one contiguous answer span.Membership and BIO objectives require modeling broader label structures, whereas RASOR’s labels directly encode answer-span semantics.
- Endpoint interactions: RASOR outperforms independent endpoint prediction by 1.1 exact match, while its F1 drops by 0.2.Endpoint interactions improve consistency and exact matching but not prediction of the correct answer region.
6 ANALYSIS
RASOR’s performance declines as predicted spans grow, while explicit modeling of span-endpoint interactions becomes increasingly important. Its attention analysis shows valid, semantically appropriate predictions but also a recurring difficulty distinguishing nearby answers with overlapping question-related phrases.
- Prediction-length analysis: RASOR and the endpoint predictor baseline both degrade as prediction lengths increase.Figure 2 compares F1 and Exact Match accuracy across different prediction lengths.
- Prediction-length analysis: Explicitly modeling interactions between end markers becomes increasingly important as the predicted span grows.
- Attention analysis: RASOR’s top predictions are valid syntactic constituents with the correct semantic category in both Figure 3 examples.The examples include predictions such as “Egyptians” and “unjust laws.”
- Attention analysis: RASOR can assign nearly as much probability to an incorrect nearby answer as to the top-scoring correct prediction.In the first example, “British” receives almost as much probability mass as “Egyptian.”
7 CONCLUSION
The paper presents RASOR for extractive question answering by explicitly representing and scoring answer spans. Its recurrent architecture reuses shared computation, while passage-independent and passage-aligned question representations benefit performance; alternate input architectures remain future work.
- RASOR explicitly represents and scores candidate answer spans for extractive question answering on SQUAD.
- A recurrent network enables shared computation across the shared substructure of different span candidates.
- Including both passage-independent and passage-aligned question representations benefits the task.
- The encoding method is orthogonal to the core contribution of efficiently computing span representations.
- Future work will explore alternate architectures that provide input to recurrent span representations.