Source-linked AI summary
Convolutional Neural Network Architectures for Matching Natural Language Sentences
Baotian Hu, Zhengdong Lu, Hang Li, Qingcai Chen
TL;DR
Sentence matching requires models that capture both the internal structure of language objects and their interactions. The paper proposes generic deep convolutional architectures that combine hierarchical sentence composition with multilevel matching patterns. Across varied matching tasks, the models outperform competitors, while ARC-I can lose details because sentences are represented independently before matching.
Problem
Sentence matching needs to model both the internal structures of language objects and their interactions across abstraction levels.
Method
The paper proposes deep convolutional architectures that combine hierarchical sentence modeling with matching patterns at multiple abstraction levels.
Results
The proposed models outperform competitors across a variety of matching tasks.
Takeaways & Limitations
ARC-II can model sentence interaction before the sentences’ high-level representations mature, while retaining separate abstraction development.
Takeaways & Limitations
ARC-I risks losing matching-relevant details because each sentence is represented without knowledge of the other during forward computation.
Abstract
from arXiv · showhide
Semantic matching is of central importance to many natural language tasks \cite{bordes2014semantic,RetrievalQA}. A successful matching algorithm needs to adequately model the internal structures of language objects and the interaction between them. As a step toward this goal, we propose convolutional neural network models for matching two sentences, by adapting the convolutional strategy in vision and speech. The proposed models not only nicely represent the hierarchical structures of sentences with their layer-by-layer composition and pooling, but also capture the rich matching patterns at different levels. Our models are rather generic, requiring no prior knowledge on language, and can hence be applied to matching tasks of different nature and in different languages. The empirical study on a variety of matching tasks demonstrates the efficacy of the proposed model on a variety of matching tasks and its superiority to competitor models.
1 Introduction
The paper develops generic convolutional architectures for matching sentences by jointly modeling sentence structure and interactions across abstraction levels. It evaluates these architectures across varied tasks and reports advantages over competing methods.
- Sentence matching must model both the internal sequential and hierarchical structures of language and the rich interaction patterns between sentences.
- The proposed deep convolutional models adapt strategies successful in image and speech processing to natural-language sentence matching.
- The architectures combine layer-by-layer sentence composition and pooling with matching patterns that progress from simple to comprehensive.
- The models require no prior language knowledge and are intended for matching tasks with different characteristics and languages.
- Extensive experiments on tasks with different scales and characteristics demonstrate superior performance over competing methods.
2 Convolutional Sentence Model
The convolutional sentence model builds fixed-length representations through repeated local convolution and pooling over sequential word embeddings. Gating handles variable-length padding, while shared local filters and feature maps support hierarchical composition.
- The model inputs sequential word embeddings and applies convolution and pooling layers until producing a fixed-length sentence representation.Convolution units use local receptive fields and shared weights, while a large feature map models rich sentence structures.
- Convolution: Each convolution unit applies a learned filter to a local segment, producing feature-map outputs at different locations.The segment contains the vectors in the convolution window, and each feature map has its own parameters and activation function.
- Max-Pooling: Max-pooling over every two-unit window halves the representation size and filters out undesirable word compositions.
- Length Variability: Zero padding and a gating function eliminate boundary artifacts caused by variable sentence lengths.The gate outputs zero when its input is all zeros, preserving a hierarchy of padding nodes that does not affect forward or backward computation.
- Relation to Recursive Models: Unlike recursive models, the architecture represents multiple composition choices through a large feature map rather than a single composition path.
- Relation to “Shallow” Convolutional Models: Compared with shallow convolutional models, the proposed model uses layered composition rather than only local template matching followed by global pooling.
3 Convolutional Matching Models
The paper proposes ARC-I and ARC-II for matching two sentences. ARC-II combines early cross-sentence interaction with hierarchical, order-preserving convolution and pooling, while subsuming ARC-I as a special case.
- 3.1 Architecture-I (ARC-I): ARC-I separately builds a convolutional representation for each sentence, then compares the two representations with a multilayer perceptron.Its Siamese design delays interaction until the final MLP.
- 3.1 Architecture-I (ARC-I): ARC-I risks losing matching-relevant details because each sentence is represented without knowledge of the other sentence.The paper gives a city name as an example of a detail that may be lost.
- 3.2 Architecture-II (ARC-II): ARC-II builds directly on the interaction space, combining sliding windows from both sentences through one-dimensional convolutions before higher-level processing.This lets the sentences interact before their high-level representations mature.
- 3.2 Architecture-II (ARC-II): ARC-II applies two-dimensional max-pooling and convolution over sentence-pair representations, selecting among compositions and local matchings while retaining location information.The two-dimensional pooling operates on non-overlapping 2 × 2 windows and can continue across additional layers.
- 3.3 Some Analysis on ARC-II: ARC-II preserves sentence order conditionally, and experiments report that it can identify the correct sentence against randomly shuffled alternatives, unlike ARC-I.The reported comparison uses triples containing an original sentence, its match, and a randomly shuffled version.
- 3.3 Some Analysis on ARC-II: ARC-II subsumes ARC-I when parameters keep the two sentence representations separate through the convolutional layers and final MLP.Under these restrictions, two-dimensional pooling reduces to one-dimensional pooling and ARC-I functionality is recovered.
- 3.3 Some Analysis on ARC-II: ARC-II blends successive within-sentence composition with matching-pattern extraction and fusion, and the paper reports superior performance across different matching tasks.The architecture is described as providing both capability and inductive bias for developing internal abstractions on each sentence.
4 Training
Training uses a discriminative large-margin objective over triples in which one candidate matches an input better than another. The models are optimized with standard back-propagation and stochastic gradient descent.
- 4 Training: The training objective is a ranking-based large-margin loss over triples (x, y+, y−), where y+ is preferred over y− for input x.The loss penalizes cases where the negative candidate's predicted matching score approaches or exceeds the positive candidate's score.
- 4 Training: The predicted matching score s(x, y) and parameters Θ cover the convolution layers and the MLP.The paper states that optimization uses standard back-propagation and stochastic gradient descent.
- 4 Training: The experiments use 50-dimensional Word2Vec embeddings trained on Wikipedia for English and Weibo data for Chinese.The paper also reports varying maximum sentence length and testing different numbers of feature maps.
5 Experiments
The experiments evaluate the proposed convolutional matching models across heterogeneous and homogeneous language-matching tasks, languages, and writing styles. ARC-II generally performs best, while convolutional models benefit from sequential modeling and show weaker results when data or task structure limits them.
- Experimental scope: The study compares proposed models with competitors across sentence completion, tweet-response matching, and paraphrase identification, spanning heterogeneous and homogeneous objects, two languages, and varied writing styles.The experiments are intended to assess broad applicability across different matching conditions.
- Sentence Completion: The two proposed models get nearly half of sentence-completion cases right, with large margins over other sentence models and models without explicit sequence modeling.ARC-II can achieve 74+% accuracy with random negatives, and ARC-II significantly outperforms ARC-I.
- Tweet-Response Matching: ARC-II again beats other models with large margins on tweet-response matching, while ARC-I and SENNA+MLP follow.The task uses four random negatives and emphasizes detailed local matching patterns such as work-overtime⇔rest.
- Paraphrase Identification: On paraphrase identification, the generic matching models achieve accuracy and F1 close to the best 2008 hand-crafted-feature performer but remain below the 76.8%/83.6% state of the art.The authors note that the model generally requires ≥100K instances to work favorably and is not tailored specifically to synonymy.
- Results: ARC-II outperforms other models significantly when training instances are relatively abundant, as in sentence completion and tweet-response matching.Its advantage is less salient when sentences have deep grammatical structures and matching relies less on local patterns.
- Discussion: Convolutional models perform favorably over bag-of-words models, indicating the importance of sequential structures for understanding and matching sentences.Simple sums of Word2Vec embeddings also yield reasonably good results on all three tasks.
6 Related Work
Related work largely represents language objects in a common latent space or uses shallow convolutional sentence models. The proposed models extend convolutional sentence representation toward deeper modeling of sentence structure and text-pair interactions.
- Matching models: Most structured-object matching methods estimate similarity within the same domain, while language methods commonly seek common latent-space representations and score pairs with inner products.Deep architectures over the interaction space for text pairs remain relatively uncommon and are often based on bag-of-words representations.
- Sentence representation: Sentence representation work includes recursive models, word-embedding sums, and a recent dynamic convolutional model tailored to classification with a carefully designed pooling strategy.The paper places its models within this broader sentence-representation literature while focusing on matching.
7 Conclusion
The paper proposes deep convolutional architectures that combine hierarchical sentence modeling with matching patterns, and reports that they outperform competitors across varied matching tasks.
- Deep convolutional architectures combine hierarchical modeling of individual sentences with patterns in their matching.
- The architectures model sentence structure and matching patterns together within a unified approach.
- Empirical studies show the models outperform competitors on a variety of matching tasks.