Source-linked AI summary
ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs
Wenpeng Yin, Hinrich Schütze, Bing Xiang, Bowen Zhou
TL;DR
Sentence-pair tasks need models that account for mutual sentence influence rather than representing each sentence in isolation. The paper introduces ABCNN, which integrates attention into CNNs to create interdependent representations across multiple granularities. ABCNNs outperform non-attention CNNs across AS, PI, and TE, with linguistic features adding further improvements, while generation-heavy tasks remain an open boundary.
Problem
Sentence-pair modeling is important for AS, PI, and TE, but prior work often represents sentences separately or depends on task-specific linguistic features.
Method
ABCNN integrates mutual sentence influence into CNNs through attention schemes that model related parts at multiple granularities.
Results
ABCNNs perform better than CNNs without attention across AS, PI, and TE; linguistic features further improve all three tasks.
Takeaways & Limitations
Interdependent attention-based CNN representations provide a general sentence-pair architecture that reaches or surpasses state-of-the-art without linguistic features in AS and TE.
Takeaways & Limitations
CNNs have not been used for tasks with a strong generation component, leaving such applications as future work.
Abstract
from arXiv · showhide
How to model a pair of sentences is a critical issue in many NLP tasks such as answer selection (AS), paraphrase identification (PI) and textual entailment (TE). Most prior work (i) deals with one individual task by fine-tuning a specific system; (ii) models each sentence's representation separately, rarely considering the impact of the other sentence; or (iii) relies fully on manually designed, task-specific linguistic features. This work presents a general Attention Based Convolutional Neural Network (ABCNN) for modeling a pair of sentences. We make three contributions. (i) ABCNN can be applied to a wide variety of tasks that require modeling of sentence pairs. (ii) We propose three attention schemes that integrate mutual influence between sentences into CNN; thus, the representation of each sentence takes into consideration its counterpart. These interdependent sentence pair representations are more powerful than isolated sentence representations. (iii) ABCNN achieves state-of-the-art performance on AS, PI and TE tasks.
1 Introduction
Modeling sentence pairs is central to answer selection, paraphrase identification, and textual entailment, but prior approaches often represent sentences independently. ABCNN addresses this by using attention in CNNs to let each sentence influence the other across multiple granularities.
- Sentence-pair modeling is critical for answer selection, paraphrase identification, and textual entailment.
- Most prior work derives each sentence’s representation separately, rarely incorporating the other sentence’s influence.
- Different sentence counterparts can determine which parts of each sentence require attention for AS, PI, and TE.
- ABCNN is a general attention-based CNN that models sentence pairs through interdependent sentence representations.
- Its CNN-based attention models relatedness automatically and supports multiple granularities through stacked convolution layers.
- ABCNN incorporates attention into CNNs and reports state-of-the-art AS and TE performance, competitive PI performance, and further gains with linguistic features.
2 Related Work
Related work spans shallow matching, neural sentence-pair models, and attention-based systems, while ABCNN extends attention to CNNs for general NLP sentence-pair modeling.
- Shallow approaches use lexical, syntactic, semantic, tree-based, or alignment features, but some alignment methods require substantial computation.
- Neural systems model answer selection, paraphrase identification, and textual entailment with CNNs, recursive networks, or LSTMs.
- Earlier paraphrase systems generally keep one sentence’s representation independent of the other, unlike ABCNN’s attention-based model.
- General matching architectures target phrase-level or multigranular matching across tasks such as paraphrase identification, sentence completion, and answer selection.
- Attention-based CNNs had appeared in computer vision, including visual question answering, image classification, caption generation, segmentation, and localization.
- Attention-based NLP research mostly centers on RNNs and encoder-decoder systems, whereas ABCNN explores attention mechanisms in CNNs.
3 BCNN: Basic Bi-CNN
BCNN is a non-attention Siamese baseline that processes the two sentences with weight-sharing CNNs and combines their representations for the pair task.
- BCNN uses two weight-sharing CNNs, one for each sentence, followed by an output layer for the sentence-pair task.
- The implementation pads both sentences to a common length s = max(s0, s1).
- BCNN comprises input, convolution, average-pooling, and output layers.
- Words enter as embedding feature maps, and convolution produces phrase representations from windows of concatenated word embeddings.
- Average pooling extracts robust features, with all-column pooling producing one representation vector for each sentence.
- Stacked convolution-pooling blocks extract increasingly abstract features, while pooling outputs from multiple layers supplies different abstraction levels to the output layer.
- The final output layer is task-dependent; binary classification uses logistic regression.
4 ABCNN: Attention-Based BCNN
ABCNN introduces three attention-based architectures that model sentence pairs interdependently, using one sentence to influence the representation of the other at convolution and pooling stages.
- ABCNN defines three architectures—ABCNN-1, ABCNN-2, and ABCNN-3—that introduce attention mechanisms into the BCNN for sentence-pair modeling.
- ABCNN-1: ABCNN-1 constructs an attention feature matrix by matching units across sentences, then combines attention and representation feature maps before convolution.Attention values represent how units in one sentence relate to units in the other, at word and higher phrase levels.
- ABCNN-2: ABCNN-2 computes attention weights from convolution outputs and uses them to reweight those outputs during pooling.Unlike ABCNN-1, its attention directly influences pooling rather than convolution input.
- Comparison: ABCNN-1 uses more parameters than ABCNN-2 and is therefore more vulnerable to overfitting, while the two architectures attend at different processing granularities.ABCNN-1 affects convolution at input granularity; ABCNN-2 affects pooling over larger units such as phrases.
- ABCNN-3: ABCNN-3 stacks ABCNN-1 and ABCNN-2 so attention operates on both convolution and pooling, across input and more abstract output granularities.
5 Experiments
The experiments evaluate ABCNN variants and baselines on answer selection, paraphrase identification, and textual entailment. Attention improves CNN performance across tasks, with ABCNN-3 generally strongest, while deeper convolution provides limited additional benefit.
- The experiments evaluate the proposed architectures on answer selection, paraphrase identification, and textual entailment.
- The networks use word2vec initialization, stacked convolution-pooling blocks for increasingly abstract features, and a final logistic-regression layer.Representations from each block produce similarity scores that feed the classifier.
- 5.1 Answer Selection: The non-attention BCNN outperforms shared baselines, while adding attention improves performance by several points.ABCNN-2 is slightly better than ABCNN-1, and combining both mechanisms in ABCNN-3 yields further improvement.
- 5.2 Paraphrase Identification: ABCNN-3 reaches state-of-the-art accuracy and F1 for paraphrase identification, while two convolution layers provide only small improvements over one.
- 5.3 Textual Entailment: ABCNNs outperform A-LSTM and the top three SemEval systems on textual entailment, with ABCNN-3 improving 1.6 points over the previous state of the art.Attention consistently improves over BCNN, and ABCNN-3 is better than ABCNN-1 and ABCNN-2.
- Attention Analysis: The attention maps assign higher values to semantically corresponding words and phrases, including matches across multiple granularities.Higher-level attention also identifies longer phrase correspondences and possible coreference across sentences.
6 Summary
The paper introduces three attention mechanisms for integrating sentence-pair interactions into CNNs and evaluates their performance across answer selection, paraphrase identification, and textual entailment. Attention-based CNNs outperform non-attentive CNNs, while linguistic features provide additional gains across all three tasks.
- ABCNN integrates attention into CNNs through three mechanisms for general sentence-pair modeling.
- Attention-based CNNs perform better than CNNs without attention mechanisms on AS, PI, and TE.
- ABCNN-2 generally outperforms ABCNN-1, while ABCNN-3 surpasses both variants.
- Two convolution layers provide no major improvement over one, possibly because the training data are limited.
- Linguistic features improve AS by 0.0321 (MAP) and 0.0338 (MRR), PI by 3.8 (acc) and 2.1 (F1), and TE by 1.6 (acc).
- Attention-based CNNs remain able to reach or surpass state-of-the-art performance without linguistic features in AS and TE.
- Attention-based CNNs have not been used for tasks with a strong generation component, such as machine translation.