Source-linked AI summary
Attentive Pooling Networks
Cicero dos Santos, Ming Tan, Bing Xiang, Bowen Zhou
TL;DR
Pair-wise ranking and classification models often construct representations of paired inputs independently, limiting how their interactions inform representation learning. The paper proposes Attentive Pooling, which uses two-way segment-level attention with CNNs and RNNs; experiments on three answer-selection benchmarks report strong performance, including new state-of-the-art results on InsuranceQA and TREC-QA.
Problem
Existing pair-wise models commonly construct input representations independently before ranking or classification, motivating a mechanism that jointly models paired inputs.
Method
Attentive Pooling lets each input influence the other's representation by learning segment-level similarities and using them to compute bidirectional attention for pooling.
Results
The models are effective on three answer-selection benchmarks; the paper reports new state-of-the-art results for InsuranceQA and TREC-QA and the best non-handcrafted-feature results for WikiQA.
Takeaways & Limitations
AP is a general two-way attention framework applicable to CNNs and RNNs, and it helps CNNs cope with large input texts.
Abstract
from arXiv · showhide
In this work, we propose Attentive Pooling (AP), a two-way attention mechanism for discriminative model training. In the context of pair-wise ranking or classification with neural networks, AP enables the pooling layer to be aware of the current input pair, in a way that information from the two input items can directly influence the computation of each other's representations. Along with such representations of the paired inputs, AP jointly learns a similarity measure over projected segments (e.g. trigrams) of the pair, and subsequently, derives the corresponding attention vector for each input to guide the pooling. Our two-way attention mechanism is a general framework independent of the underlying representation learning, and it has been applied to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) in our studies. The empirical results, from three very different benchmark tasks of question answering/answer selection, demonstrate that our proposed models outperform a variety of strong baselines and achieve state-of-the-art performance in all the benchmarks.
1. Introduction
Attentive Pooling (AP) is a two-way attention mechanism for pair-wise ranking and classification that lets paired inputs jointly influence their representations and pooling. Applied to CNNs and RNNs, it supports matching variable-length inputs and improves answer-selection models across three benchmarks.
- Motivation and approach: Attentive Pooling jointly learns pairwise representations and similarity measurements, improving discriminative ranking or classification models.The pooling layer becomes aware of the current input pair rather than representing each item independently.
- Motivation and approach: AP computes segment-level similarities between paired inputs, converts them into attention vectors in both directions, and uses those vectors for pooling.The segments may be projected trigrams or CNN-convolved sequences, while biLSTM variants use hidden states.
- Benefits: AP projects paired inputs into a common representation space, enabling more plausible comparisons when the inputs are not always semantically comparable.The paper specifically identifies question-answer pairs as an example.
- Benefits: AP is effective for matching input pairs with significant length variations and makes CNN models more robust to large input texts.This addresses reported scaling problems of CNN representations as input text grows.
- Benefits: The two-way attention mechanism is independent of the representation learner and applies to both CNNs and RNNs.The experiments use AP-CNN and AP-biLSTM models for answer selection.
- Evaluation: AP-CNN achieves state-of-the-art results across InsuranceQA, TREC-QA, and WikiQA, while AP-CNN representations require fewer convolutional filters.The resulting representations are more compact and may help speed training.
2. Neural Networks for Answer Selection
QA-CNN and QA-biLSTM independently encode questions and candidate answers into fixed-length vectors, then score pairs by cosine similarity. They are trained with pairwise ranking using positive and negative answers.
- Architectures: QA-CNN and QA-biLSTM compute independent fixed-length representations for the question and candidate answer before cosine scoring.Both architectures use shared processing parameters for the paired inputs, while differing in how they process embedding sequences.
- Convolution: QA-CNN applies convolution to word-embedding sequences, with each feature column derived from a context window around a question or answer token.The convolution uses learned parameters, while the filter count and context-window size are hyperparameters.
- Bidirectional LSTM: QA-biLSTM processes each sequence in forward and reverse directions, concatenating the two directional outputs to represent both previous and future context.The resulting matrices contain one bidirectional hidden state per sequence position.
- Pooling and scoring: Column-wise max-pooling followed by a non-linearity produces the fixed-length question and answer vectors used for pair scoring.The same pooling formulation is applied to the matrices produced by either architecture.
- Scoring and training: Training minimizes a pairwise ranking loss on positive and negative question-answer pairs, selecting the highest-scoring of 50 sampled negative answers for each update.Optimization uses stochastic gradient descent with backpropagation.
3. Attentive Pooling Networks for Answer Selection
Attentive Pooling makes representations of a question-answer pair mutually dependent by learning segment-level similarities and using them to guide pooling. The framework applies to both convolutional and biLSTM outputs while retaining cosine scoring and pairwise-loss training.
- Attentive Pooling: Attentive Pooling lets each input directly influence the computation of the other input’s representation through two-way attention.The mechanism produces pair-aware representations rather than independently pooled vectors.
- Similarity modeling: AP learns a similarity measure over projected segments, using convolved sequences for AP-CNN and biLSTM hidden states for AP-biLSTM.The similarity measure has a bilinear form followed by a non-linearity.
- Similarity modeling: The model computes G from the question and answer matrices, with entries representing soft alignment scores between their contextual segments or hidden states.For CNNs, alignments compare k-sized context windows; for biLSTMs, they compare token-level hidden vectors.
- Attention construction: Row-wise and column-wise max-pooling over G yields importance vectors for answer and question contexts, respectively.Each vector reflects the importance of one input’s context with regard to the other input.
- Attention-guided pooling: Softmax converts the importance vectors into attention vectors, which are combined with convolutional or biLSTM outputs to form the final representations.The resulting representations are cosine-scored and trained with the same pairwise loss as the baseline networks.
4. Related Work
Earlier answer-selection models often represented questions and answers separately, while attention-based approaches typically focused on one input. Attentive Pooling instead models interdependence between both inputs and learns segment similarities for two-way attention.
- Earlier answer-selection systems commonly generated separate question and answer representations before applying a similarity metric.
- Attention mechanisms had succeeded across NLP tasks but generally focused on selected parts of one input.
- Tan et al. used question-conditioned attention during answer encoding, whereas AP models interdependence between questions and answers.
- AP differs from related two-way methods by learning segment similarity rather than relying on simple Euclidean distance.
5. Experimental Setup
The experiments evaluate AP-CNN and AP-biLSTM on three answer-selection datasets with varied domains, scales, and text lengths. The setup uses established splits, pretrained embeddings, tuned hyperparameters, and Theano implementations.
- Datasets: AP-CNN and AP-biLSTM are compared with QA-CNN and QA-biLSTM on InsuranceQA, TREC-QA, and WikiQA.The datasets differ in domain, scale, complexity, and question-answer length ratios.
- Datasets: InsuranceQA provides training, validation, and two test sets, with 500 candidate answers per development or test question.
- Datasets: TREC-QA uses 1,162 training, 65 development, and 68 test questions after removing questions containing only positive or negative answers.
- Datasets: WikiQA evaluation uses questions with at least one correct answer, comprising 20,360 training, 1,130 development, and 2,352 test pairs.
- Implementation: The experiments use 100-dimensional InsuranceQA embeddings and 300-dimensional TREC-QA and WikiQA embeddings, with context windows of 3 and 4, respectively.
- Implementation: The four neural architectures are implemented in Theano, with selected hyperparameters typically producing best results after 15–25 training epochs.
6. Experimental Results
Across InsuranceQA, TREC-QA, and WikiQA, attentive-pooling models outperform their corresponding baselines and reported state-of-the-art systems. AP-CNN also uses fewer filters, trains faster, and remains more stable on longer answers.
- InsuranceQA: Both AP-CNN and AP-biLSTM outperform QA-CNN and QA-biLSTM on InsuranceQA, while also surpassing the listed state-of-the-art systems.
- InsuranceQA: AP-CNN uses 400 filters versus QA-CNN’s 4,000 on InsuranceQA, a tenfold reduction, while 800 AP-CNN filters give similar results.
- InsuranceQA: AP-CNN training is twice as fast as QA-CNN; one epoch takes about 16 minutes while processing 1.5 million text segments.
- InsuranceQA: For InsuranceQA answers, QA-CNN accuracy continues declining with length, whereas AP-CNN stabilizes after approximately 90 tokens.
- TREC-QA: AP-CNN outperforms QA-CNN on both MAP and MRR for TREC-QA, and AP-biLSTM outperforms QA-biLSTM but trails AP-CNN.
- TREC-QA: AP-CNN surpasses the listed state-of-the-art TREC-QA systems on both MAP and MRR.
- WikiQA: On WikiQA, AP-CNN outperforms QA-CNN and AP-biLSTM outperforms QA-biLSTM, while AP-CNN also exceeds two recent systems on both metrics.
- Analysis: Heat maps show larger attention weights on answer segments interacting with the question in correctly answered InsuranceQA examples exceeding 100 words.
7. Conclusions
The paper presents attentive pooling as a two-way attention mechanism for discriminative pair-wise models and reports strong benchmark outcomes across answer selection tasks. AP achieves new state-of-the-art results on InsuranceQA and TREC-QA, and the best reported WikiQA results among methods without handcrafted features.
- AP is presented as a two-way attention mechanism for discriminative model training.
- 0.6921(MAP) and 0.7108(MRR) are reported for a comparison system using handcrafted features alongside word embeddings.
- The experiments demonstrate that AP helps CNNs cope with large input texts.
- AP achieves new state-of-the-art results on InsuranceQA and TREC-QA datasets.
- For WikiQA, AP reports the best results so far among methods that do not use handcrafted features.