Source-linked AI summary
Stochastic Answer Networks for Machine Reading Comprehension
Xiaodong Liu, Yelong Shen, Kevin Duh, Jianfeng Gao
TL;DR
Machine reading comprehension requires models to answer questions from passages, while difficult questions may require multi-step reasoning. SAN uses stochastic dropout on answer-module predictions during training and averages predictions across steps at decoding, achieving robust and competitive performance across SQuAD, Adversarial SQuAD, and MS MARCO.
Problem
Machine reading comprehension asks machines to read a passage and answer questions about it, with difficult problems requiring multi-step synthesis and reasoning.
Method
SAN fixes reasoning steps during training, stochastically drops final-layer answer predictions, and averages predictions across steps during decoding.
Results
SAN achieves robust improvements on SQuAD and competitive state-of-the-art results on SQuAD, Adversarial SQuAD, and MS MARCO.
Takeaways & Limitations
Stochastic prediction dropout improves robustness and overall accuracy in the multi-step answer module.
Takeaways & Limitations
ReasoNet comparisons vary with the lower encoding layers, so scores across tables are not directly interchangeable.
Abstract
from arXiv · showhide
We propose a simple yet robust stochastic answer network (SAN) that simulates multi-step reasoning in machine reading comprehension. Compared to previous work such as ReasoNet which used reinforcement learning to determine the number of steps, the unique feature is the use of a kind of stochastic prediction dropout on the answer module (final layer) of the neural network during the training. We show that this simple trick improves robustness and achieves results competitive to the state-of-the-art on the Stanford Question Answering Dataset (SQuAD), the Adversarial SQuAD, and the Microsoft MAchine Reading COmprehension Dataset (MS MARCO).
1 Introduction
Machine reading comprehension asks models to answer questions from passages, with difficult cases requiring multi-step synthesis and reasoning. SAN addresses this by applying stochastic dropout to stepwise answer predictions and averaging the retained predictions.
- Motivation: Machine reading comprehension requires answering questions from passages and serves as a benchmark for natural language understanding.It also has applications such as conversational agents and customer service support.
- Motivation: Some difficult questions require multi-step synthesis across sentences, including resolving references before extracting an answer span.The example requires identifying that “They” refers to the galleries before selecting the relevant object phrase.
- Related work: Earlier multi-step reading-comprehension models either fixed the number of reasoning steps or used reinforcement learning to determine it dynamically.These approaches generated predictions after multiple passes through the text while integrating intermediate information.
- Contribution: SAN fixes the number of reasoning steps during training but stochastically drops answer-module predictions and averages predictions across steps during decoding.The answer module is the final prediction layer, and stochastic dropout is applied to its multinomial distributions.
- Contribution: Each reasoning step combines memory with the previous hidden state to produce a prediction, while the final result averages the predictions that remain after dropout.Figure 1 illustrates three steps in which one of three predictions is dropped.
2 Proposed model: SAN
SAN is a four-layer network that performs fixed-step multi-step reasoning and applies stochastic dropout to answer predictions during training. It averages predictions across reasoning steps during decoding to improve robustness.
- Lexicon Encoding Layer: SAN maps question and passage tokens to lexical embeddings using word, POS, NER, exact-match, and question-alignment features.A position-wise FFN normalizes the lexical encodings before contextual processing.
- Answer Module: The answer module uses a GRU over T memory steps and predicts answer-span boundaries at every reasoning step.The initial state summarizes the question, and each subsequent state attends to the working memory before span prediction.
- Contextual Encoding Layer: Shared BiLSTM layers produce contextual representations for the question and passage, while maxout reduces the resulting parameter size.The contextual representations combine pretrained CoVe vectors with lexicon embeddings.
- Memory Generation Layer: Attention builds a question-aware passage representation, self-attention rearranges passage information, and another BiLSTM generates the working memory.The self-attention mechanism drops the similarity matrix diagonal so tokens align with other passage tokens rather than themselves.
- Answer Module: During training, stochastic dropout is applied to final predictive distributions, and decoding averages all T step predictions instead of using only the final step.This design targets step bias and is reported to improve robustness; experiments use a dropout rate of 0.4.
3 Experiment Setup
The experiments evaluate SAN on SQuAD using official development-set results and two span-based metrics. The setup specifies the dataset, metrics, model dimensions, optimization schedule, and dropout settings.
- Dataset: SQuAD contains about 23K passages and 100K questions drawn from approximately 500 Wikipedia articles, with crowdsourced answer spans.Unless otherwise noted, results use the official development set.
- Evaluation Metrics: Exact Match measures exact agreement with a ground-truth span, while macro-averaged F1 measures average overlap between prediction and ground truth.Both metrics evaluate predicted answer spans against accepted ground-truth answers.
- Implementation: The implementation uses 2-layer BiLSTMs with 128 hidden units, batch size 32, Adamax optimization, and an initial learning rate of 0.002 halved every 10 epochs.The implementation also applies dropout to LSTM hidden units and the answer module.
4 Results
The experiments compare answer-module architectures under fixed lower layers and show that SAN performs best on the main SQuAD metrics while remaining competitive with top models.
- Answer-module comparisons: The comparison fixes all lower layers and varies only the answer module across standard 1-step, 5-step memory, ReasoNet3, and SAN architectures.The ReasoNet variant uses SAN’s lower layers for a fair answer-module comparison.
- Answer-module comparisons: 76.235 EM and 84.056 F1: SAN outperforms the other answer-module architectures, including standard 1-step and dynamic-step ReasoNet.Standard 1-step reaches 75.139 EM, while dynamic-step ReasoNet reaches 75.355 EM.
- Answer-module comparisons: SAN also outperforms the 5-step memory network with prediction averaging, indicating that stochastic prediction dropout contributes beyond averaging alone.The averaged memory-network variant omits stochastic dropout.
- Oracle analysis: SAN outperforms the other models on K-best oracle scores, with human performance reached at K = 2 for EM and K = 3 for F1.K-best spans are ordered by P_begin × P_end and evaluated using an oracle over K = 1 to 4.
- Comparison with prior models: SAN is competitive with top SQuAD models in both single and ensemble settings, ranking second despite its simplicity.The best-performing comparison model uses a large-scale language model as an extra contextual embedding.
5 Analysis
SAN remains strong across random initializations, reasoning-step settings, adversarial SQuAD, test-time step counts, question types, and MS MARCO. Its gains persist with fewer test-time steps, while additional reasoning steps improve performance up to about five before saturating.
- Number of reasoning steps: Increasing reasoning steps from T = 1 to T = 5 gradually improves development performance, after which gains saturate and T = 10 does not statistically differ from T = 5.The authors recommend approximate tuning rather than finding an exact optimal step count.
- Adversarial SQuAD: On adversarial SQuAD, SAN achieves state-of-the-art performance, improving F1 over the Standard 1-step answer module by +1.2 on AddSent and +0.7 on AddOneSent.The evaluation uses both AddSent and AddOneSent, which add adversarial or distracting sentences to passages.
- Train-test step mismatch: Testing a SAN trained with T = 5 using T = 1 still achieves 75.58 EM, exceeding the Standard 1-step model’s 75.14 EM at approximately equivalent prediction time.Performance is best when training and test step counts match, but T = 1 and T = 2 remain strong.
- Training dynamics: SAN improves at approximately the same rate as other systems through 10–15 epochs, then continues improving while the other models begin to saturate.Average training time per epoch is 24 minutes for SAN, compared with 22 minutes for a 5-step memory net and 30 minutes for ReasoNet on a GTX Titan X.
- Question-type breakdown: SAN appears to outperform other models uniformly across question types, except Why questions, for which the dataset is too small for strong conclusions.Question types are divided by their Wh-word, such as “who” and “where.”
- MS MARCO: On MS MARCO, SAN outperforms V-Net and becomes the new state of the art.MS MARCO contains real user queries and passages extracted from real web documents; SAN is extended to multiple passages by ranking candidate spans.
6 Related Work
MRC models increasingly use iterative multi-step reasoning, but dynamic-step approaches rely on unstable reinforcement learning. SAN instead fixes the number of steps and uses stochastic answer-module dropout, enabling back-propagation training while improving robustness.
- MRC models commonly process the same text through multiple passes to integrate intermediate information before predicting an answer.
- Dynamic multi-step reasoning adapts the number of steps to question complexity, unlike earlier models with predetermined fixed steps.
- Multi-step reasoning outperforms single-step models, while dynamic multi-step reasoning further outperforms fixed multi-step reasoning on SQuAD and MS MARCO.
- Reinforcement-learning training, including policy gradients, is difficult to implement because of instability.
- SAN fixes the reasoning-step count, applies stochastic dropout to final-layer answer predictions, and averages predictions across steps during decoding.
7 Conclusion
The paper introduces SAN as a simple, robust MRC model using stochastic answer-module dropout and reports competitive results across several benchmarks. It also identifies theoretical analysis and broader task evaluation as future directions.
- SAN uses stochastic dropout during training and prediction averaging at test time to improve robustness and overall accuracy in MRC.
- SAN outperforms fixed-step memory networks and dynamic-step ReasoNet on SQuAD.
- The model achieves state-of-the-art-competitive results on SQuAD, Adversarial SQuAD, and MS MARCO.
- Future work includes analyzing SAN’s theoretical links with memory networks and ReasoNet and testing its generalization on text classification and natural language inference.