Source-linked AI summary
Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information
Seonhoon Kim, Inho Kang, Nojun Kwak
TL;DR
Semantic sentence matching requires logical and semantic relationship modeling, but deep recurrent and attention-based architectures face information-flow and feature-growth challenges. DRCN uses dense recurrent and co-attentive connections, with an autoencoder bottleneck, and achieves state-of-the-art performance on most evaluated tasks.
Problem
Sentence matching requires modeling logical and semantic relationships, while deeper recurrent networks and summation-based attention can limit information flow and feature retention.
Method
DRCN densely connects recurrent hidden and co-attentive features across layers and uses an autoencoder bottleneck to control feature-vector size.
Results
DRCN achieves state-of-the-art performance on most sentence-matching tasks across five competitive benchmark datasets.
Takeaways & Limitations
The model retains recurrent and attentive features across layers while using bottlenecks to enable deeper networks with controllable feature sizes.
Abstract
from arXiv · showhide
Sentence matching is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. For these tasks, understanding logical and semantic relationship between two sentences is required but it is yet challenging. Although attention mechanism is useful to capture the semantic relationship and to properly align the elements of two sentences, previous methods of attention mechanism simply use a summation operation which does not retain original features enough. Inspired by DenseNet, a densely connected convolutional network, we propose a densely-connected co-attentive recurrent neural network, each layer of which uses concatenated information of attentive features as well as hidden features of all the preceding recurrent layers. It enables preserving the original and the co-attentive feature information from the bottommost word embedding layer to the uppermost recurrent layer. To alleviate the problem of an ever-increasing size of feature vectors due to dense concatenation operations, we also propose to use an autoencoder after dense concatenation. We evaluate our proposed architecture on highly competitive benchmark datasets related to sentence matching. Experimental results show that our architecture, which retains recurrent and attentive features, achieves state-of-the-art performances for most of the tasks.
Introduction
Semantic sentence matching supports inference, paraphrase identification, and question answering, but modeling logical and semantic relationships remains difficult. DRCN addresses these challenges by retaining recurrent and co-attentive features through dense connections and controlling feature growth with an autoencoder.
- Semantic sentence matching is used for natural language inference, paraphrase identification, and question answering, where logical and semantic relationships must be identified.
- Joint matching methods capture interactive information between sentences, unlike isolated sentence encoders.
- Deep recurrent models improve long-sequence learning but face exploding or vanishing gradients, while attention can become unstable in deeper networks.
- DRCN concatenates recurrent hidden and co-attentive features from preceding layers to preserve lower-level information in deeper representations.
- An optional autoencoder bottleneck limits the feature-vector growth caused by repeated concatenation.
- Across five benchmark datasets and three sentence-matching tasks, DRCN significantly outperforms current state-of-the-art results on most tasks.
Related Work
Sentence-matching research progressed from task-specific symbolic methods toward data-driven neural architectures. Joint models add cross-sentence interaction, while residual and dense connections support deeper networks; this paper uses dense recurrent and co-attentive information without external resources.
- Earlier sentence-matching methods relied on syntactic features, transformations, or relation extraction and were restricted to specific tasks.
- Large annotated datasets and deep learning advanced learning-centered approaches to semantic sentence representation.
- Joint methods address isolated encoders’ lack of interaction by using cross-features to represent word- and phrase-level alignments.
- Residual connections commonly stabilize deeper networks, while DenseNet-style concatenation preserves lower-layer features without modification.
- Other sentence-matching systems use external resources such as syntactic parse trees, WordNet, or paraphrase-database semantics.
- DRCN belongs to the joint-model family and enhances representation power with densely connected recurrent and co-attentive information without external resources.
Methods
DRCN combines densely connected recurrent features with co-attentive information, preserving representations across layers while using an autoencoder bottleneck to control feature growth.
- DRCN comprises a word representation layer, an attentively connected RNN, and an interaction and prediction layer.
- Word representation layer: Word representations concatenate trainable and fixed word embeddings, character-level convolutional features, and an exact-match flag.
- Densely connected recurrent network: Dense connections pass recurrent hidden features directly from each layer to subsequent layers, preserving earlier features for prediction.
- Co-attention: At each recurrent layer, attention supplies soft-aligned context between sentences, which is concatenated with recurrent features to retain co-attentive information.
- Bottleneck component: An autoencoder bottleneck compresses the growing concatenated feature vectors while retaining original information and acting as a regularizer in experiments.
- Interaction and prediction layer: Step-wise max pooling produces sentence vectors, which are combined through element-wise operations before fully connected layers and softmax classification.
Experiments
The model is evaluated across three sentence-matching tasks using five established benchmark datasets.
- The experiments cover natural language inference, paraphrase identification, and answer sentence selection in question answering.
- SNLI and MultiNLI evaluate natural language inference, Quora Question Pair evaluates paraphrase identification, and TrecQA and SelQA evaluate answer sentence selection.
Implementation Details
The implementation uses pretrained and character-level representations with a five-layer recurrent stack, regularization, and a compressed bottleneck.
- Word embeddings are initialized with 300d pretrained GloVe vectors, while out-of-vocabulary embeddings are initialized randomly.
- Character embeddings are randomly initialized at 16d, and a convolutional network extracts 32d character representations.
- The densely connected recurrent component uses 5 layers with 100 hidden units each, while fully connected layers use 1000 hidden units.
- Dropout is applied after embeddings with keep rate 0.5, before fully connected layers with keep rate 0.8, and in the bottleneck with rate 0.2.
- The autoencoder bottleneck uses 200 hidden units as encoded features.
Experimental Results
DRCN delivers strong sentence-matching results across natural language inference, paraphrase identification, and answer sentence selection, including state-of-the-art performance on several benchmarks.
- Natural Language Inference: 90.1% accuracy is achieved by the DRCN ensemble on the SNLI test set, establishing new state-of-the-art performance.The ensemble uses 53 million parameters, fewer than the 85 million parameters of LM-Transformer.
- Natural Language Inference: 88.9% accuracy is obtained by single DRCN on SNLI without external contextualized knowledge, matching the competitiveness of models using such knowledge.
- Natural Language Inference: Table 2 reports classification accuracy and parameter counts for models evaluated on the SNLI test set.The parameter count is denoted by |θ|.
- Natural Language Inference: Table 3 reports MultiNLI classification accuracy, with an asterisk marking ensemble methods.
- Paraphrase Identification: 91.30% accuracy is obtained by the DRCN ensemble on Quora question pairs, surpassing the previous state-of-the-art DIIN model.The single DRCN model achieves 90.15% accuracy.
- Answer Sentence Selection: DRCN achieves new state-of-the-art performance on both TrecQA and SelQA for answer sentence selection.The model uses collective attentions over multiple layers for selecting candidate answers to questions.
Analysis
The analysis examines ablations, depth, attention, and linguistic categories to assess DRCN’s dense recurrent and co-attentive design. Results generally favor dense connections and show interpretable alignment patterns across layers.
- Ablation study: The autoencoder bottleneck reduces DRCN’s parameter count while slightly improving performance through a reported regularization effect.The ablation study specifically evaluates the bottleneck component in the model.
- Ablation study: Dense concatenation over deeper layers retains collective knowledge more effectively than residual replacement in the ablation comparisons.The comparison replaces dense connections with residual connections over recurrent and co-attentive features, with additional variants removing word-embedding dense connections.
- Ablation study: Dense connections outperform residual connections in general, while connected models remain more robust as recurrent depth increases.Models 5–9 are more robust to increased depth than models 10–11, whose performance tends to degrade.
- Interpretability: Attentive weights and max-pooled positions expose sentence-alignment clues that contribute to classification through deep supervision.Attention maps identify words such as tight, competing, and bicycle as important in an entailment example.
- Interpretability: In contradiction examples, deeper layers maintain the relevance between white building and gray building, supporting the contradiction label.The model’s attention increasingly concentrates on the semantically conflicting building descriptions.
- Linguistic Error Analysis: DRCN generally performs well across MultiNLI linguistic categories, especially Quantity/Time, while achieving the highest mean and lowest standard deviation on matched and mismatched sets.The analysis compares DRCN with ESIM, DIIN, and CAFE.
Conclusion
The paper concludes that DRCN preserves recurrent and co-attentive features across layers while controlling feature size with bottlenecks. It reports state-of-the-art performance on most datasets across three sentence-matching tasks.
- Conclusion: DRCN connects recurrent and co-attentive features from bottom to top without deforming lower-layer information.The resulting multi-layer features are described as collective semantic knowledge.
- Conclusion: Autoencoder bottlenecks reduce the network size and provide controllable feature sizes as the architecture expands to deeper layers.The paper presents this as a generalized DenseRNN version.
- Conclusion: DRCN achieves state-of-the-art performance on most datasets spanning natural language inference, paraphrase identification, and answer sentence selection.The conclusion also presents attentive weights and max-pooled positions as interpretability tools.
Supplementary Material
The supplementary dataset descriptions cover sentence-pair resources for inference, paraphrase identification, and question answering. They specify dataset scale, task labels, genres, and collection or preprocessing details where available.
- Datasets: SNLI contains 570k human-written sentence pairs for natural language inference with entailment, neutral, and contradiction labels.Its pairs are based on image captioning, with data splits provided by the original dataset work.
- Datasets: MultiNLI contains 433k sentence pairs modeled closely on SNLI and spanning ten distinct genres.The passage identifies MultiNLI as Multi-Genre NLI.
- Datasets: Quora consists of over 400k question pairs from actual Quora questions, each labeled as paraphrase or non-paraphrase.The dataset includes training, development, and test splits.
- Datasets: TrecQA was collected from TREC Question Answering tracks 8–13 and has clean and raw versions created through different preprocessing methods.The passage attributes the dataset to the TREC Question Answering tracks.
Visualization on the comparable models
The visualization compares DRCN with two residual variants across layers using attentive-weight maps. The examples illustrate how dense multi-layer features support earlier or progressively emerging alignments.
- Entailment example: Figure 4 compares attentive maps for DRCN, Res1, and Res2 on an entailment example across layers.Res1 removes additional dense word-embedding connections, while Res2 replaces dense recurrent and co-attentive connections with residual ones.
- Entailment example: DRCN can retain a lower-layer rationale instead of searching for the same alignment at the upper layer.The residual variants must find correct alignments at the top layer and show a reported misalignment in Res2.
- Contradiction example: Figure 5 compares attentive maps for DRCN, Res1, and Res2 on a contradiction example involving white and gray buildings.The caption identifies the premise and hypothesis used for the contradiction example.
- Contradiction example: By its fifth layer, DRCN assigns substantially higher attentive weights to gray building and white building, whereas residual variants spread attention across several positions.The example shows DRCN progressively finding the alignment when lower layers do not identify the clues.