Source-linked AI summary
Making Neural QA as Simple as Possible but not Simpler
Dirk Weissenborn, Georg Wiese, Laura Seiffe
TL;DR
The paper asks whether complex neural extractive QA architectures are justified without competitive simple baselines. It develops FastQA from a context/type-matching heuristic and finds that question-word awareness plus recurrent composition yields competitive performance, while added interaction complexity does not systematically improve it.
Problem
Existing extractive QA research lacked competitive neural baselines for assessing whether the complexity of current architectures was justified.
Method
The paper develops FastQA using a context/type-matching heuristic, question-word features, and recurrent composition, then evaluates an interaction-layer extension.
Results
FastQA is very competitive with prior state-of-the-art systems on two datasets, while FastQAExt does not yield systematic improvements from added interaction complexity.
Takeaways & Limitations
Question-word awareness and composition beyond bag-of-words are sufficient ingredients for a currently competitive neural extractive QA baseline.
Takeaways & Limitations
The bag-of-words baseline restricts answer spans to a fixed length and cannot capture language compositionality.
Abstract
from arXiv · showhide
Recent development of large-scale question answering (QA) datasets triggered a substantial amount of research into end-to-end neural architectures for QA. Increasingly complex systems have been conceived without comparison to simpler neural baseline systems that would justify their complexity. In this work, we propose a simple heuristic that guides the development of neural baseline systems for the extractive QA task. We find that there are two ingredients necessary for building a high-performing neural QA system: first, the awareness of question words while processing the context and second, a composition function that goes beyond simple bag-of-words modeling, such as recurrent neural networks. Our results show that FastQA, a system that meets these two requirements, can achieve very competitive performance compared with existing models. We argue that this surprising finding puts results of previous systems and the complexity of recent QA datasets into perspective.
1 Introduction
The paper addresses the lack of competitive neural baselines for extractive QA and uses a simple context/type-matching heuristic to guide FastQA. The heuristic targets answer type and proximity to important question words, while FastQA combines question-word awareness with recurrent composition.
- Research gap: Extractive QA systems lacked a competitive neural baseline for judging whether the complexity of existing architectures was justified.Existing ablations typically removed one component from a complex system rather than comparing against a reasonable neural baseline.
- Approach: FastQA uses the heuristic to construct simple bag-of-words and recurrent neural baselines without a complex interaction layer.Question-context interaction is modeled through computable word-level features.
- Contribution: The paper argues that FastQA’s strong performance challenges the necessity of additional interaction-layer complexity.FastQAExt is evaluated to test whether adding such a layer produces systematic improvements.
2 A Bag-of-Words Neural QA System
The bag-of-words baseline operationalizes context/type matching with lexical answer-type representations, word-in-question features, and span-level scoring. It combines type and contextual evidence to rank candidate answer spans.
- Heuristic: The heuristic requires candidate answers to match the expected lexical answer type and to occur near question-relevant context.The heuristic is used as a guideline for constructing neural systems rather than as an explicit traditional QA extraction rule.
- Type Matching: The baseline represents the lexical answer type using boundary-word embeddings and the average embedding of its words.For “what” or “which” questions, it can use the first noun phrase after the question words.
- Span representation: Each candidate span is represented using its first and last word, average span embedding, and embeddings from nearby context words.The surrounding five words on each side are included to capture contextual clues about answer type.
- Context Matching: Binary and weighted word-in-question features estimate how strongly context words match the question.The weighted feature uses embedding similarity to accommodate morphological variants, synonyms, and related words.
- Scoring: The model combines type and context scores for each span and trains on the resulting span scores with softmax cross-entropy.Context matching aggregates both features over 5-, 10-, and 20-token windows on both sides of each candidate.
3 FastQA
FastQA replaces the bag-of-words encoder with a bidirectional recurrent network while retaining question-aware features and separate span-boundary prediction. Its architecture encodes context and question, computes a single question representation, and selects spans with beam search.
- Motivation: The bag-of-words baseline cannot model language compositionality, reduces question semantics, and restricts answer spans to a fixed maximum length.These shortcomings motivate the recurrent FastQA baseline.
- Architecture: FastQA uses embedding, encoding, and answer layers, with a bidirectional recurrent network replacing the bag-of-words composition function.The answer layer separates prediction of the answer start and end.
- Question-aware encoding: Question-word awareness is added by feeding binary and weighted word-in-question features alongside embedded context words into the encoder.The resulting representation has n+2 features per context position.
- Question representation: The encoded question is compressed into one weighted representation that is computed once and is independent of individual context words.FastQA therefore does not perform additional word-by-word question-context interaction.
- Answer layer: FastQA predicts start and end distributions with feed-forward networks, trains on span cross-entropy, and uses beam search to select the highest-probability span.Beam search evaluates ends for the top-k predicted starts.
4 Comparison to Prior Architectures
Prior neural extractive QA systems commonly use embedding, encoding, interaction, and answer layers. Research especially emphasizes increasingly sophisticated interaction mechanisms for word-by-word question-context modeling.
- Common architecture: Most existing neural QA systems are organized into embedding, encoding, interaction, and answer layers.The paper uses this decomposition to compare FastQA with prior architectures.
- Embedding and encoding: The embedder maps token sequences to n-dimensional states, while the encoder applies a composition function such as a bidirectional RNN.FastQA uses a similar embedder and a bidirectional recurrent encoder.
- Interaction layer: Most prior work focuses on interaction layers that enrich encoded context with information from the question and context.Examples include attention, coattention, bidirectional attention flow, multi-perspective matching, and fine-grained gating.
- Answer layer: Answer layers range from single fully connected layers to convolutional or recurrent deep networks for predicting answer boundaries.The paper uses a simple two-layer feed-forward network with beam search.
5 FastQA Extended
FastQA is extended with an interaction layer that fuses context and question representations through attention. The extension serves as a representative of more complex architectures rather than the work’s main focus.
- FastQAExt adds intra-fusion and inter-fusion to exchange information within the context and between the question and context.Each context state receives a co-representation retrieved via attention and fused into its own representation.
- Representation fusion combines each state with its corresponding n-dimensional co-representation through weighted addition.
- The extension is described as a representative of more complex architectures, not the focus of the work.Technical details are deferred to Appendix B.
6 Experimental Setup
The experiments evaluate extractive QA models on SQuAD and NewsQA using exact match and token-based F1. Training includes span-length, preprocessing, optimization, embedding, and context-length settings.
- Datasets: Experiments use SQuAD, with over 100k questions about paragraphs from 536 Wikipedia articles, and NewsQA, with 100k answerable questions.NewsQA questions are drawn from 120k total questions based on CNN news stories.
- Evaluation: Performance is measured with exact match accuracy and mean per-answer-token F1, which accounts for partial matches.
- BoW Model: The BoW model trains on spans up to length 10, imposing upper bounds of about 95% accuracy on SQuAD and 87% on NewsQA.
- BoW Model: Inputs are lowercased and tokenized with spaCy, while the BoW word-in-question feature uses non-stopword alphanumeric lemmas.
- FastQA: FastQA uses whitespace and non-alphanumeric tokenization, 300-dimensional representations and fixed GloVe embeddings, with variational input dropout at rate 0.5.
- Context Length: NewsQA contexts longer than 400 tokens are truncated during training while retaining at least one answer and preferably all answers.
7 Results
FastQA performs competitively with more complex extractive QA systems while revealing that question-aware features and recurrent composition account for much of the gain. Additional interaction complexity offers only marginal, non-systematic improvements despite higher resource costs.
- 7.1 Model Component Analysis: ≈15% F1 comes from adding either question-aware feature, while other extensions typically improve results by 1–2% F1.Beam search slightly improves results because the most probable start is not necessarily the start of the best answer span.
- 7.1 Model Component Analysis: Question-aware encoding lets the model selectively retain question-related information and abstract mentioned entities to their types.An encoder without question knowledge must account for every possible question while processing the entire context.
- 7.2 Comparing to State-of-the-Art: The neural BoW baseline performs well on SQuAD and NewsQA but remains ≈20% F1 behind state-of-the-art systems, supporting recurrent composition beyond averaging.It outperforms feature-rich logistic regression on SQuAD development data and nearly reaches the BiLSTM baseline.
- 7.2 Comparing to State-of-the-Art: FastQA is competitive with prior state-of-the-art results on both datasets and improves on reported NewsQA results; FastQAExt slightly improves SQuAD results further.These results are notable given FastQA’s simple architecture and the complexity attributed to recent QA datasets.
- 7.3 Do we need additional interaction?: FastQA runs about twice as fast as FastQAExt and DCN and uses 2–4× less memory than FastQAExt and DCN, respectively.Comparisons found no systematic advantage for FastQAExt despite 589 FastQAExt wins versus 415 FastQA wins on SQuAD development examples.
- 7.4 Qualitative Analysis: FastQA errors commonly involve fine-grained answer types, coreference, abbreviation binding, nested syntax, semantic distinctions, and annotation preferences.About 35 of 55 mistakes (64%) were directly attributed to applying the heuristic, while 44 of 50 analyzed positive cases (88%) were covered by it.
8 Related Work
Large-scale cloze datasets enabled end-to-end neural reading-comprehension research, but earlier datasets were criticized as too easy or noisy, motivating newer extractive QA benchmarks.
- DailyMail/CNN and Children’s Book Corpus helped establish end-to-end neural architectures for reading comprehension.
- Analysis found DailyMail/CNN too easy and noisy, prompting construction of SQuAD, NewsQA, and MsMARCO.
9 Conclusion
The paper presents a simple heuristic-guided baseline strategy for extractive QA and identifies two ingredients associated with competitive performance: question-word awareness and non-bag-of-words composition.
- The context/type matching heuristic guides two neural baseline systems for extractive question answering.
- FastQA combines awareness of question words during context processing with a composition function beyond bag-of-words modeling.
- FastQA is described as an efficient neural baseline that is competitive on extractive QA.
- The authors argue that FastQA’s findings put more complex architectures and recent QA-dataset complexity into perspective.
A Weighted Word-in-Question to Term Frequency
This section connects the weighted word-in-question feature to term frequencies by redefining the similarity score and deriving the resulting feature expression.
- The analysis relates the weighted word-in-question feature to term frequency in the question and context.
- The similarity score is redefined before deriving the corresponding weighted word-in-question feature.
- Under the redefined similarity score, wiqw for context word xj becomes a combination of xj’s context and question term frequencies.
- The derived relationship holds for any finite value chosen in the redefined similarity score, not only zero.
B.1 Intra-Fusion
The section describes representation fusion for propagating co-reference information and question-conditioned context information, while noting limits from recurrent-state bottlenecks and restricted evaluation settings.
- B.1 Intra-Fusion: RNNs’ fixed-size internal states make it difficult to synthesize evidence across passages connected by co-referent entities or events.
- B.1 Intra-Fusion: Associative representation fusion retrieves co-states for context states and combines each state with its co-state through gated addition.
- B.1 Intra-Fusion: Recurrent representation fusion propagates information from associative fusion between neighboring tokens.
- B.1 Intra-Fusion: Representation fusion combines existing features rather than computing new ones.
- B.1 Intra-Fusion: Question-context fusion uses attention weights so that mainly question-related context states receive non-empty co-states before answer prediction.
- B.1 Intra-Fusion: The approach is presented as a lighter-weight interaction mechanism than concatenating attention-retrieved representations and feeding them into another RNN.
- C Generative Question Answering: On MsMARCO, training used only queries with directly extractable answers, representing 67.2% of all queries, while evaluation covered the full development and test sets.
- C Generative Question Answering: The extractive system performed strongly on MsMARCO, but comparisons were limited because other systems’ training and preprocessing details were unclear.