Source-linked AI summary
A Convolutional Neural Network for Modelling Sentences
Nal Kalchbrenner, Edward Grefenstette, Phil Blunsom
TL;DR
Accurately representing sentence semantics is central to language understanding, but sentence models must capture relations across varying-length inputs. The paper introduces a Dynamic Convolutional Neural Network with dynamic k-max pooling and reports strong performance across sentiment and question-classification tasks, including over 25% error reduction for Twitter sentiment prediction.
Problem
Sentence modelling requires representations of semantic content for classification and generation across many language-understanding tasks.
Method
The Dynamic Convolutional Neural Network combines convolutional layers with dynamic k-max pooling to process varying-length sentences and induce an internal feature graph.
Results
The network outperforms other approaches on binary and multi-class sentiment, matches state-of-the-art accuracy on six-way question classification, and achieves over 25% error reduction on Twitter sentiment prediction.
Takeaways & Limitations
The induced feature graph captures word relations of varying size without requiring external parser-derived features or other resources.
Takeaways & Limitations
The paper notes that the Max-TDNN’s feature-detector range is limited to the span of its weights, and deeper or wider configurations increase sentence-margin neglect.
Abstract
from arXiv · showhide
The ability to accurately represent sentences is central to language understanding. We describe a convolutional architecture dubbed the Dynamic Convolutional Neural Network (DCNN) that we adopt for the semantic modelling of sentences. The network uses Dynamic k-Max Pooling, a global pooling operation over linear sequences. The network handles input sentences of varying length and induces a feature graph over the sentence that is capable of explicitly capturing short and long-range relations. The network does not rely on a parse tree and is easily applicable to any language. We test the DCNN in four experiments: small scale binary and multi-class sentiment prediction, six-way question classification and Twitter sentiment prediction by distant supervision. The network achieves excellent performance in the first three tasks and a greater than 25% error reduction in the last task with respect to the strongest baseline.
1 Introduction
The paper introduces a Dynamic Convolutional Neural Network for semantic sentence modelling, combining convolution with dynamic k-max pooling to process variable-length inputs. Its layered operations induce an internal feature graph that captures relations among distant phrases, and experiments show strong results across sentiment and question-classification tasks.
- Motivation: Sentence modelling targets semantic representation for classification or generation across tasks requiring natural-language comprehension.Applications include sentiment analysis, paraphrase detection, entailment recognition, summarisation, discourse analysis, machine translation, grounded language learning, and image retrieval.
- Architecture: The DCNN handles variable-length input sequences by interleaving one-dimensional convolutional layers with dynamic k-max pooling layers.Dynamic k-max pooling generalises max pooling, while convolutional filters extract features from n-grams independently of their positions.
- Feature graph: Multiple convolutional and dynamic pooling layers induce a structured feature graph whose higher-layer filters can relate distant, noncontinuous phrases.The graph is hierarchical and internally learned, resembling a syntactic parse tree without being tied to purely syntactic relations.
- Experiments: The network is evaluated in four settings spanning binary and multi-class movie-review sentiment, six-way TREC question classification, and Twitter sentiment prediction by distant supervision.It outperforms other approaches on both movie-review sentiment tasks and matches state-of-the-art methods on six-way question classification.
2 Background
The background reviews neural sentence models, contrasting Neural Bag-of-Words and parse-tree-based recursive models with convolutional and TDNN architectures. It also introduces one-dimensional convolution and the Max-TDNN, including their mechanisms and key properties.
- Related neural sentence models: Neural Bag-of-Words models combine word, sub-word, or n-gram embeddings component-wise before classification through fully connected layers.
- Related neural sentence models: Recursive Neural Networks use an external parse tree to combine left and right child contexts recursively, producing a sentence representation at the root.
- Related neural sentence models: Convolutional sentence models build on one-dimensional convolution and the TDNN architecture, concepts identified as central to the DCNN.
- One-dimensional convolution: A one-dimensional convolution applies a learned filter to each m-gram of a sentence sequence, producing a new sequence of feature values.
- One-dimensional convolution: Wide convolution supports arbitrary sentence and filter lengths, includes margin words through zero-padding, and lets every filter weight reach the entire sentence.
- Max-TDNN: Max-TDNN applies narrow convolution to sentence embeddings, then takes each row’s maximum to obtain a fixed-sized vector for classification.The model is sensitive to word order and does not require external dependency or constituency parse trees.
3 Convolutional Neural Networks with Dynamic k-Max Pooling
The Dynamic Convolutional Neural Network alternates wide convolutional layers with dynamic k-max pooling, allowing sentence representations to adapt to input length and network depth. Its repeated convolution, pooling, and nonlinear operations produce increasingly higher-order feature maps with variable-range dependencies.
- Architecture: The DCNN alternates wide convolutional layers with dynamic k-max pooling, whose intermediate feature-map widths vary with sentence length.Dynamic pooling determines k as a function of sentence length and network depth.
- Pooling: k-max pooling selects the k highest activations while preserving their original order and ignoring their specific positions.This allows active features separated by multiple positions to be pooled together.
- Dynamic k-Max Pooling: Dynamic pooling assigns different pooling sizes across layers, so lower-order features can occur more often than higher-order features in a sentence.For example, with three convolutional layers, ktop = 3, and sentence length s = 18, the first-layer pooling parameter is k1 = 12.
- Feature Extraction: Convolution and nonlinear transformation extract first-order features, while stacking the operations yields feature maps of increasing order and network depth.Multiple feature maps of the same order can be computed in parallel using distinct filter sets.
- Feature Extraction: Together, convolution and pooling provide position invariance and make the range of higher-order features variable.The architecture therefore supports feature extraction across changing positional spans.
4 Properties of the Sentence Model
The DCNN is sensitive to word order while preserving relative ordering and inducing position invariance through convolution and generalized pooling. Its internal feature graph captures variable-range dependencies, including long-distance relations, without relying on an external parse tree.
- Word-order sensitivity: Wide convolution learns to recognize specific n-grams up to the first-layer filter width, which experiments often set relatively large, such as 10.Generalized pooling preserves n-gram order and relative positions while making representations invariant to absolute positions.
- Comparison with other sentence models: Unlike NBoW, RNN, Max-TDNN and recursive models, the DCNN combines word-order sensitivity with internally induced variable-range structure rather than a fixed chain, fixed range or external parse tree.The cited comparisons characterize NBoW as order-insensitive, RNNs as biased toward recent words, Max-TDNN as fixed-range, and RecNNs as parse-tree structured.
- Induced feature graph: The DCNN’s convolution and pooling layers induce an internal feature graph whose edges connect computationally involved nodes across successive layers.Pooling drops unselected nodes, leaving the graph determined by forward propagation through the network.
- Variable-range dependencies: Global dynamic k-max pooling can connect features for words many positions apart, producing higher-order features with short, focused or global, long ranges.The resulting subgraphs may be localized to sentence parts or spread widely across the sentence.
5 Experiments
The experiments evaluate the DCNN on movie-review sentiment, six-way question classification, and distantly supervised Twitter sentiment. Across these tasks, the DCNN outperforms competing neural and non-neural models while learning detectors for sentiment-modifying and syntactic patterns.
- Movie-review sentiment: The first two experiments predict binary or five-way sentiment on Stanford Sentiment Treebank movie reviews.The binary splits contain 6920/872/1821 training, development, and test sentences; fine-grained splits contain 8544/1101/2210.
- Movie-review sentiment: The DCNN significantly outperforms the other neural and non-neural movie-review models.NBoW performs similarly to n-gram classifiers, while Max-TDNN likely suffers from excessive max pooling that discards sentiment features.
- Question classification: Six-way question classification uses 5452 labelled training questions and 500 test questions from the TREC dataset.The classes include location, person, and numeric-information questions.
- Twitter sentiment: The final experiment trains on 1.6 million emoticon-labelled tweets and tests on about 400 hand-annotated tweets.Tweets are automatically labelled positive or negative from their emoticons and minimally preprocessed.
- Learned feature detectors: Feature detectors learn positive, negative, negating, potentiating, and other syntactic-semantic patterns within n-grams.Detected constructs include ‘not’, ‘too’, ‘all’, ‘or’, ‘with...that’, and ‘as...as’.
6 Conclusion
The paper presents a dynamic convolutional neural network using dynamic k-max pooling to model word relations of varying sizes. It achieves high performance on question and sentiment classification without external parser-derived or other resource-based features.
- 6 Conclusion: The dynamic convolutional neural network uses dynamic k-max pooling as a non-linear subsampling function.The architecture induces a feature graph over the input.
- 6 Conclusion: The induced feature graph captures word relations of varying sizes.This enables the network to represent relations across different scales.
- 6 Conclusion: The network achieves high performance on question and sentiment classification without external parser-derived or other resource-based features.Its performance does not depend on features supplied by parsers or other external resources.