Source-linked AI summary
Multi-Perspective Context Matching for Machine Comprehension
Zhiguo Wang, Haitao Mi, Wael Hamza, Radu Florian
TL;DR
Earlier machine-comprehension datasets were either too small for end-to-end deep learning or insufficiently challenging for evaluation. The paper proposes MPCM, which matches passage contexts with questions from multiple perspectives and predicts answer boundaries on SQuAD, achieving competitive leaderboard performance.
Problem
Earlier machine-comprehension datasets were too small for end-to-end deep learning or insufficiently challenging for evaluating current techniques.
Method
MPCM weights passage representations by question relevance, matches passage contexts with the question from multiple perspectives, and predicts answer beginning and ending points.
Results
MPCM achieves EM 65.5 and F1 75.1 as a single model on the SQuAD test set, with competitive results in single and ensemble settings.
Takeaways & Limitations
The model provides an end-to-end approach for identifying answer spans in SQuAD by combining contextual matching with boundary prediction.
Takeaways & Limitations
The formulation independently predicts answer beginning and ending points because explicitly enumerating candidate spans has order O(N^2).
Abstract
from arXiv · showhide
Previous machine comprehension (MC) datasets are either too small to train end-to-end deep learning models, or not difficult enough to evaluate the ability of current MC techniques. The newly released SQuAD dataset alleviates these limitations, and gives us a chance to develop more realistic MC models. Based on this dataset, we propose a Multi-Perspective Context Matching (MPCM) model, which is an end-to-end system that directly predicts the answer beginning and ending points in a passage. Our model first adjusts each word-embedding vector in the passage by multiplying a relevancy weight computed against the question. Then, we encode the question and weighted passage by using bi-directional LSTMs. For each point in the passage, our model matches the context of this point against the encoded question from multiple perspectives and produces a matching vector. Given those matched vectors, we employ another bi-directional LSTM to aggregate all the information and predict the beginning and ending points. Experimental result on the test set of SQuAD shows that our model achieves a competitive result on the leaderboard.
1 Introduction
Machine comprehension requires models to understand passages and answer related questions, but earlier datasets were too small or insufficiently challenging. This work uses SQuAD to propose an end-to-end MPCM model that predicts answer boundaries and achieves a competitive leaderboard result.
- Machine comprehension asks systems to understand a passage and answer questions about it.
- RCTest contains 500 fictional stories and 2,000 multiple-choice questions, limiting end-to-end deep neural network training.Its state-of-the-art performance remained dominated by hand-crafted features or additional knowledge.
- SQuAD addresses earlier dataset weaknesses with substantially greater scale, human-written questions, arbitrary answer spans, and varied reasoning requirements.
- MPCM matches passage-point contexts with the question from multiple perspectives and predicts answer beginnings and endings using globally normalized distributions.The model avoids explicitly enumerating and ranking all possible spans.
- Ablation studies find that all MPCM components are crucial, while test-set experiments show competitive leaderboard performance.
2 Task Definition
The SQuAD task represents each example as a question, passage, and answer span, with the goal of estimating the conditional probability of that span. Because candidate spans grow as O(N^2), the formulation independently predicts beginning and ending positions.
- A machine-comprehension instance contains a question, a passage containing the answer, and the correct answer span.The dataset is represented as tuples (Q, P, A).
- The question has length M, the passage has length N, and the answer span is defined by beginning and ending points satisfying 1 ≤ ab ≤ ae ≤ N.
- The task estimates Pr(A|Q, P) from training data and predicts answers for test instances.
- The candidate answer set A(P) grows as O(N^2), motivating an independent prediction of the beginning and ending points.
- The beginning-point probability Pr(ab|Q, P) and ending-point probability Pr(ae|Q, P) identify the corresponding answer boundaries.
3 Multi-Perspective Context Matching Model
The MPCM model is an end-to-end architecture that filters passage words by question relevance, encodes question and passage context, matches passage positions from multiple perspectives, and predicts answer boundaries.
- Model overview: MPCM estimates separate probability distributions for the answer's beginning and ending points, sharing all layers below the prediction layer.The two distributions differ only at the final prediction layer.
- Model overview: The filter layer weights passage words by their maximum cosine similarity to question words before subsequent processing.More relevant passage words receive greater emphasis in later steps.
- Context representation: BiLSTMs encode contextual representations for question and passage sequences before matching.The context representation layer uses bidirectional LSTMs to incorporate surrounding information.
- Multi-Perspective Context Matching Layer: The core matching layer compares each passage contextual embedding with the question using dimensional weighted and directional matching strategies.Directional strategies include Full-Matching, Maxpooling-Matching, and Meanpooling-Matching.
- Multi-Perspective Context Matching Layer: Each matching perspective uses cosine similarity between weighted vectors, with trainable weights assigning different importance to vector dimensions.The resulting matching vector contains one value for each perspective.
- Prediction: A BiLSTM aggregates matching vectors across passage positions, after which separate feed-forward networks normalize beginning and ending scores with softmax.The aggregation layer lets each passage position interact with surrounding positions.
4 Experiments
Experiments evaluate MPCM on SQuAD using EM and F1, compare matching-function and layer variants, and analyze performance across answer lengths and question types. The model achieves competitive test-set results, while analyses identify longer answers and complex reasoning as difficult cases.
- Experiment Settings: 87,599 training instances, 10,570 validation instances, and a hidden test set are evaluated with Exact Match and F1 score.The model uses 300-dimensional GloVe embeddings, 100-dimensional LSTM hidden states, 50 matching perspectives, dropout 0.2, and learning rate 0.0001.
- Results on the Test Set: 65.5 EM and 75.1 F1 are achieved by the single MPCM model on the SQuAD test set.A five-model ensemble improves EM by about 3% and F1 by 2%, with competitive results in both single and ensemble settings.
- Influence of the Multi-Perspective Matching Function: Even one multi-perspective matching perspective outperforms vanilla cosine similarity, and increasing the number of perspectives improves performance significantly.The comparison varies the number of perspectives among 1, 10, 30, and 50 while keeping other options unchanged.
- Layer Ablation: Removing any MPCM component significantly decreases performance; the Aggregation Layer is most crucial, while Maxpooling-Matching has the largest effect among matching strategies.The ablation removes one layer or individual matching strategy at a time on the development set.
- Result Analysis: Performance drops as answer length increases, with EM declining faster than F1, indicating that precise boundaries are harder than approximate answer regions.The analysis interprets longer answers as harder to find and boundary identification as more difficult than locating the approximate region.
- Result Analysis: “When,” “what year,” “in what,” and “in which” questions perform much better than others, whereas “how did” questions perform poorly.The paper attributes the stronger results to detectable temporal expressions or explicit boundary words, and the weaker results to longer, more varied answers.
- Result Analysis: Among 50 randomly selected incorrect questions, 16% had acceptable predictions, 22% overlapped the correct answer, and 14% required reasoning across multiple sentences.Most remaining questions required external knowledge or complex reasoning.
5 Related Work
Prior machine-comprehension methods largely follow either chunk extraction and ranking or direct boundary identification. These approaches differ in whether they enumerate candidate answer spans or predict answer boundaries directly.
- Method Categories: The related-work discussion organizes post-SQuAD deep-learning models into chunking-and-ranking and boundary-identification classes.
- Chunking and Ranking: Chunking-and-ranking methods first extract candidate answer chunks and then rank them using learned models or hand-crafted features.Parse-tree constituents and enumerated n-grams are examples of candidate-generation strategies.
- Chunking and Ranking: Some chunking methods enumerate all possible passage chunks, encode each with a fixed-length BiLSTM representation, and score the candidates.
- Boundary Identification: Boundary-identification methods learn question-aware representations at each passage position and directly predict the answer's beginning and ending points.The cited match-LSTM approach matches passage and question representations before selecting answer positions.
6 Conclusion
The paper presents MPCM as a boundary-prediction model that matches passage time steps with the question from multiple perspectives. Its ablations find all matching components important, while SQuAD test results are competitive on the leaderboard.
- MPCM matches each passage time step with the question from multiple perspectives and predicts answer boundaries using globally normalized probability distributions.
- Ablation studies show that all matching aspects inside MPCM are crucial.
- MPCM achieves a competitive result on the SQuAD test-set leaderboard.