Source-linked AI summary
A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations
Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, Xueqi Cheng
TL;DR
Existing semantic-matching models struggle to capture contextualized local information, despite the importance of sentence matching for applications such as information retrieval and question answering. The paper introduces MV-LSTM, which matches multiple Bi-LSTM-generated positional sentence representations by aggregating their interactions with k-Max pooling and an MLP. Experiments on question answering and sentence completion show that MV-LSTM outperforms several existing baselines.
Problem
Existing single-representation and multiple-granularity models cannot well capture contextualized local information needed for semantic matching.
Method
MV-LSTM generates positional sentence representations with a Bi-LSTM, models their interactions, and aggregates them through k-Max pooling and a multilayer perceptron.
Results
MV-LSTM outperforms several existing baselines on both question answering and sentence completion, including a 11.4% relative improvement on P@1 over the strongest baseline on sentence completion.
Takeaways & Limitations
Multiple positional sentence representations let MV-LSTM capture local information while using whole-sentence context to determine the importance of different local information.
Abstract
from arXiv · showhide
Matching natural language sentences is central for many applications such as information retrieval and question answering. Existing deep models rely on a single sentence representation or multiple granularity representations for matching. However, such methods cannot well capture the contextualized local information in the matching process. To tackle this problem, we present a new deep architecture to match two sentences with multiple positional sentence representations. Specifically, each positional sentence representation is a sentence representation at this position, generated by a bidirectional long short term memory (Bi-LSTM). The matching score is finally produced by aggregating interactions between these different positional sentence representations, through $k$-Max pooling and a multi-layer perceptron. Our model has several advantages: (1) By using Bi-LSTM, rich context of the whole sentence is leveraged to capture the contextualized local information in each positional sentence representation; (2) By matching with multiple positional sentence representations, it is flexible to aggregate different important contextualized local information in a sentence to support the matching; (3) Experiments on different tasks such as question answering and sentence completion demonstrate the superiority of our model.
Introduction
Semantic matching supports applications such as information retrieval and question answering, but existing sentence-level and granularity-based models struggle with contextualized local information. MV-LSTM addresses this gap by matching multiple Bi-LSTM-generated positional sentence representations and outperforms several baselines on question answering and sentence completion.
- Semantic matching determines the degree of correspondence between sentences and supports applications including information retrieval, question answering, and paraphrase identification.
- Single-vector sentence representations can lose important local information needed to resolve which answer best matches a question.For example, “top three” and “World Cup” support different answer judgments depending on the attended phrase.
- Multiple-granularity models consider word, phrase, and sentence representations but may miss contextualized meanings because word and phrase representations rely on limited local contexts.The examples distinguish “top three” referring to football teams from “top three” referring to attendees from different countries.
- MV-LSTM matches multiple positional sentence representations, each generated by a Bi-LSTM that uses whole-sentence context at its position.This architecture is designed to capture important contextualized local information during matching.
- MV-LSTM models interactions with cosine, bilinear, and tensor operations, then uses k-Max pooling and an MLP to produce the final matching score.The model is trained end to end with parameters learned automatically from training data.
- Experiments on question answering and sentence completion show that MV-LSTM outperforms several existing baselines on both tasks.The evaluated baselines include ARC-I, ARC-II, CNTN, DeepMatch, RAE, MultiGranCNN, and LSTM-RNN.
Our Approach
MV-LSTM matches sentences through multiple positional representations generated by a Bi-LSTM, interaction functions, k-Max pooling, and an MLP. This design preserves contextualized local information and can aggregate matching evidence from multiple positions and granularities.
- Step 1: Positional Sentence Representation: MV-LSTM generates a positional sentence representation at each position using a Bi-LSTM that reflects the whole sentence from both directions.Each representation combines forward and backward hidden vectors, allowing contextual information around a position to contribute to matching.
- Step 3: Interaction Aggregation: k-Max pooling selects the strongest interactions from each interaction matrix or tensor, enabling matching from either the best position or multiple positions.With k = 1, the model uses the best matching position; larger k values aggregate multiple matching positions.
- Step 2: Interactions Between Two Sentences: Interactions between positional representations from the two sentences are computed with cosine, bilinear, or tensor-layer similarity functions.Cosine and bilinear produce interaction matrices, whereas the tensor layer produces an interaction tensor.
- MultiLayer Perception: An MLP transforms the pooled interaction feature vector into a higher-level representation and then produces the final matching score.The pooled vector q is mapped to r through a fully connected layer before a linear transformation outputs s.
- Discussions: MV-LSTM generalizes LSTM-RNN and adaptively incorporates multiple granularities through Bi-LSTM representations rather than fixed-window CNN representations.Using long- and short-term dependencies gives the model potential to capture important n-gram matching patterns.
Experiments
Experiments evaluate MV-LSTM on question answering and sentence completion using ranking metrics and multiple baselines. Results show that multiple positional representations, with k-Max pooling and Bi-LSTM context, improve matching performance and capture useful matching positions.
- Experimental setup: Experiments evaluate question answering and sentence completion as ranking tasks using P@1 and MRR.Each ranking list contains one positive sentence, which is ranked above negative sentences according to matching scores.
- Question answering: The QA experiments compare three MV-LSTM variants with random guess, BM25, single-representation models, and multiple-granularity models.The compared baselines include ARC-I, ARC-II, CNTN, LSTM-RNN, RAE, DeepMatch, and MultiGranCNN.
- Question answering: Larger k improves MV-LSTM performance on QA, while gains become limited beyond k=5, leading the authors to use k=5.Here, k=1 selects one matching position, whereas larger k aggregates multiple matching positions.
- Question answering: MV-LSTM consistently beats LSTM-RNN and Bi-LSTM-RNN, indicating that the best matching position is not always the first or last.The comparison tests models that use the last position or the first and last positions against multiple positional representations.
Conclusions
MV-LSTM matches sentences using multiple positional sentence representations, capturing local information while leveraging whole-sentence context. Experiments and case studies show benefits from selecting and aggregating interactions at different positions.
- MV-LSTM uses multiple positional sentence representations to match two sentences.The architecture is proposed as a novel deep approach for sentence matching.
- The model can capture local information while using rich context to determine keyword importance.
- With one pooled interaction, MV-LSTM outperforms single-sentence-representation methods including LSTM-RNN.This indicates that the best matching position is not always the last one.
- Aggregating multiple interactions produces better results than restricting the model to one interaction.The paper states that matching degree is usually determined by combinations of matchings at different positions.