Source-linked AI summary
Machine Comprehension Using Match-LSTM and Answer Pointer
Shuohang Wang, Jing Jiang
TL;DR
SQuAD poses machine comprehension as answering human-created questions with variable-length spans from the passage, exposing limits of candidate-based and single-token approaches. The paper combines match-LSTM with Pointer Net in two end-to-end models, and reports that both outperform Rajpurkar et al.’s feature-engineered baseline, with the boundary model reaching 67.6% exact match and 77% F1 on the test set.
Problem
SQuAD requires answering human-created questions with variable-length token sequences from the passage, unlike benchmarks with fixed candidate answers or single-token outputs.
Method
The paper combines match-LSTM for matching questions and passages with Pointer Net to select answer tokens from the passage, using sequence and boundary models.
Results
67.6% exact match and 77% F1 were achieved by the boundary model on the test dataset, outperforming the sequence model and Rajpurkar et al.’s feature-engineered model.
Takeaways & Limitations
The results support end-to-end neural models combining match-LSTM and Pointer Network as effective approaches for the SQuAD task.
Takeaways & Limitations
The authors identify low performance on some question types, such as “why” questions, and plan to test the models on other machine comprehension datasets.
Abstract
from arXiv · showhide
Machine comprehension of text is an important problem in natural language processing. A recently released dataset, the Stanford Question Answering Dataset (SQuAD), offers a large number of real questions and their answers created by humans through crowdsourcing. SQuAD provides a challenging testbed for evaluating machine comprehension algorithms, partly because compared with previous datasets, in SQuAD the answers do not come from a small set of candidate answers and they have variable lengths. We propose an end-to-end neural architecture for the task. The architecture is based on match-LSTM, a model we proposed previously for textual entailment, and Pointer Net, a sequence-to-sequence model proposed by Vinyals et al.(2015) to constrain the output tokens to be from the input sequences. We propose two ways of using Pointer Net for our task. Our experiments show that both of our two models substantially outperform the best results obtained by Rajpurkar et al.(2016) using logistic regression and manually crafted features.
1 INTRODUCTION
SQuAD makes machine comprehension more challenging by using human-created questions whose answers are variable-length text spans rather than selections from candidate answers. The paper proposes end-to-end neural models designed for this setting and reports substantial improvements over a feature-engineered baseline.
- Dataset motivation: SQuAD uses human-created questions and answers, with answers formed from any sequence of tokens in the passage rather than a fixed candidate set.This makes the dataset more realistic and challenging than automatically generated or multiple-choice-style benchmarks.
- Research gap: Existing end-to-end models relied on candidate answers or assumed single-token answers, making them unsuitable for SQuAD.
- Approach: The paper proposes an end-to-end neural architecture combining match-LSTM with Pointer Net for machine comprehension on SQuAD.The architecture uses two ways of applying Pointer Net to generate answers from the passage.
- Results: 67.9% exact match and 77.0% F1 were achieved on the unseen test dataset, outperforming the feature-engineered solution.The reported performance was also close to the stated SQuAD state of the art.
2 METHOD
The method combines contextual LSTMs, match-LSTM attention, and an Answer Pointer to select answers from passage tokens in SQuAD. It offers sequence and boundary variants, differing in whether they predict token positions or answer endpoints.
- Sequence Model: The sequence model represents an answer as passage-token positions and uses a special position P + 1 to stop generation.Its pointer mechanism can select tokens sequentially without requiring them to be consecutive in the passage.
- Boundary Model: The boundary model predicts only start and end positions, treating all passage tokens between them as the answer.This design explicitly ensures that the predicted answer is a contiguous passage subsequence.
- Architecture: The architecture has preprocessing LSTM, match-LSTM, and Answer Pointer layers, with the two models differing only in the third layer.The preprocessing layer encodes context, match-LSTM matches passage and question representations, and Answer Pointer selects passage tokens.
- Match-LSTM: Match-LSTM treats the question as a premise and the passage as a hypothesis, attending to the question while processing passage tokens sequentially.At each passage position, attention produces a weighted question representation that is combined with the current passage token before the match-LSTM update.
- Match-LSTM: A reverse match-LSTM is added so each passage token receives a representation encoding matching context from both directions.The reverse layer mirrors the forward matching process while traversing the passage in the opposite direction.
3 EXPERIMENTS
Experiments on SQuAD v1.1 evaluate the models with exact match and word-level F1, showing strong performance and revealing weaknesses by answer length, question type, and attention alignment.
- 3.1 DATA: SQuAD v1.1 contains 23,215 passages and 107,785 questions, split into training, development, and hidden test sets.The training, development, and test splits contain 87,599, 10,570, and an unspecified number of question-answer pairs, respectively.
- 3.2 EVALUATION: Performance is measured by exact-match percentage and word-level F1, using the best-matching ground-truth answer for each question.Development and test questions have around three ground-truth answers each.
- 3.3 RESULTS: The boundary model outperforms the sequence model and the logistic-regression baseline, achieving 61.1% exact match and 71.2% F1.The authors attribute the improvement over logistic regression to end-to-end neural modeling without much feature engineering.
- 3.4 FURTHER ANALYSES: For answers longer than 9 tokens, boundary-model F1 falls to around 55% and exact match to around 30%, versus close to 72% and 67% for single-token answers.These results support the authors’ hypothesis that longer answers are harder to predict.
- 3.4 FURTHER ANALYSES: The models perform best on “when” questions and worst on “why” questions, while noun-phrase answer groups such as “what,” “which,” and “where” perform relatively better.The authors suggest temporal expressions may be easier to recognize, whereas “why” answers are diverse and not restricted to particular phrase types.
- 3.4 FURTHER ANALYSES: Attention weights align some passage words with question words and support correct answer locations, but alignment is less clear for the “why” question.Examples include “German” aligned with “language” and “teacher” preceding the predicted answer “Martin Sekulic.”
4 RELATED WORK
Related work spans Cloze-style, candidate-answer, and other machine-comprehension datasets, alongside recurrent, attention-based, and memory-network models. The paper distinguishes its approach by combining match-LSTM with Pointer Network generation of multi-token passage answers.
- 4.1 DATASETS: Cloze-style datasets remove a token from an original sentence and require predicting the missing word.Examples include CNN/Daily Mail, Children’s Book Test, People Daily, and Children’s Daily datasets.
- 4.1 DATASETS: MCTest and MovieQA use human-created questions with candidate answers, whereas SQuAD uses human-created questions without providing candidate answers.SQuAD answers can instead be sequences drawn from the passage.
- 4.1 DATASETS: Other machine-comprehension datasets, including WikiReading and bAbI, differ substantially in nature from the datasets discussed above.
- 4.2 END-TO-END NEURAL NETWORK MODELS FOR MACHINE COMPREHENSION: Existing end-to-end models commonly use recurrent networks to process passages and questions, with attention mechanisms to match them.The reviewed approaches include models based on RNNs and attention over recurrent representations.
- 4.2 END-TO-END NEURAL NETWORK MODELS FOR MACHINE COMPREHENSION: This paper uses match-LSTM to match questions and passages and Pointer Network to generate multi-token answers from the passage.The Pointer Network is applied differently from prior work to select answer tokens from the given passage.
- 4.2 END-TO-END NEURAL NETWORK MODELS FOR MACHINE COMPREHENSION: Memory Networks have also been applied to machine comprehension, but their scalability on large datasets remains an issue considered by the authors.The paper therefore does not consider Memory Networks for SQuAD.
5 CONCLUSIONS
The paper develops two match-LSTM and Pointer Network models for SQuAD, with the boundary model outperforming the sequence and feature-engineered models. The authors identify low-performing question types, especially “why” questions, as future targets.
- The boundary model combines match-LSTM and Pointer Network and achieves 67.6% exact match and 77% F1 on SQuAD’s test dataset.These results are reported as better than the sequence model and Rajpurkar et al. (2016)’s feature-engineered model.
- The models address the machine comprehension problem defined by the Stanford Question Answering Dataset.
- Future work will investigate question types with low performance, such as “why” questions.
- The authors also plan to test applying the models to other machine comprehension datasets.
A APPENDIX
Figure 3 reports performance breakdowns for the sequence, boundary, and ensemble models across answer lengths and question types.
- Figure 3 shows performance by answer length and question type for the sequence, boundary, and ensemble models.