Source-linked AI summary
Molding CNNs for text: non-linear, non-consecutive convolutions
Tao Lei, Regina Barzilay, Tommi Jaakkola
TL;DR
Standard text CNNs linearly map concatenated consecutive n-grams, missing direct multi-way interactions and often requiring several layers to combine spaced phrases. This paper replaces that mapping with low-rank tensor features over non-consecutive n-grams, reporting top performance on sentiment and document categorization.
Problem
Method
Results
Takeaways & Limitations
The results support tensor-based non-consecutive n-gram mapping as a strong-performing approach on standard sentiment and document categorization tasks.
Source: §8 · p. 9
Abstract
from arXiv · showhide
The success of deep learning often derives from well-chosen operational building blocks. In this work, we revise the temporal convolution operation in CNNs to better adapt it to text processing. Instead of concatenating word representations, we appeal to tensor algebra and use low-rank n-gram tensors to directly exploit interactions between words already at the convolution stage. Moreover, we extend the n-gram convolution to non-consecutive words to recognize patterns with intervening words. Through a combination of low-rank tensors, and pattern weighting, we can efficiently evaluate the resulting convolution operation via dynamic programming. We test the resulting architecture on standard sentiment classification and news categorization tasks. Our model achieves state-of-the-art performance both in terms of accuracy and training speed. For instance, we obtain 51.2% accuracy on the fine-grained sentiment classification task.
1 Introduction
Standard text CNNs linearly map concatenated n-grams, which initially misses multi-way interactions and can require several layers to assemble non-consecutive phrases. The paper proposes low-rank tensor feature mapping over non-consecutive n-grams and evaluates it on sentiment and news categorization.
1 Introduction
- Typical text CNNs linearly map concatenated consecutive word or character representations, initially missing n-gram interactions and requiring several layers to assemble useful non-consecutive phrases.Examples include phrases such as “not that good” and “not nearly as good.”Source: §1 · p. 1
- The proposed feature mapping replaces linear operations on stacked vectors with tensor products to capture non-linear interactions between adjacent word features.Low-rank tensor parameters offset the accompanying parameter growth; weighted non-consecutive n-grams can be evaluated in linear time via dynamic programming.Source: §1 · p. 1
- 51.2% fine-grained and 88.6% binary test accuracy on Stanford Sentiment Treebank; Chinese news categorization reaches 80.0% accuracy versus a 79.2% closest baseline.The sentiment results outperform the best published numbers from a deep recursive model and a convolutional model.Source: §1 · p. 2
- The model is evaluated on sentence sentiment classification and news categorization, two settings where its reported results are strong.The paper reports state-of-the-art performance among neural networks on Stanford Sentiment Treebank in accuracy and training cost.Source: §1 · p. 2
2 Related Work
Prior NLP systems use varied neural architectures, while text CNNs have adapted image-oriented convolution with pooling and filter variations. This paper distinguishes its approach by modeling non-consecutive n-grams and nonlinear interactions within them.
2 Related Work
- NLP models span recurrent, recursive, and convolutional neural networks, with cited examples across language modeling, sentiment analysis, parsing, and machine translation.The passage cites Bengio et al. (2003) and Mikolov et al. (2010) for language modeling; Socher et al. (2013), Iyyer et al. (2015), and Le and Zuidema (2015) for sentiment analysis; and further work for parsing and translation.Source: §2 · p. 2
- Collobert et al. (2011) used max-over-time pooling, and that variant was applied to semantic parsing and information retrieval; Kalchbrenner et al. (2014) proposed dynamic k-max pooling, while Kim (2014) combined filter widths with static or fine-tuned word vectors.Source: §2 · p. 2
- Unlike traditional CNN models, this method uses non-consecutive n-grams and tensor-based nonlinear interactions rather than direct linear projection, expanding representation capacity.The paper reports that these advancements result in improved performance.Source: §2 · p. 2
3 Background
The background describes a text input as a sequence of word vectors and the traditional convolution as a linear mapping over concatenated n-gram vectors. The architecture can replace that mapping with a feature operation over non-consecutive n-grams.
3 Background
- The input sequence comprises word vectors, and a consecutive n-gram vector ending at a position is formed by concatenating the corresponding word vectors.Out-of-index words are set to zero.Source: §3 · p. 2
- Traditional convolution applies a filter matrix to each concatenated n-gram vector, producing a sequence of feature representations.The resulting features may undergo nonlinearities and then be aggregated or reduced by sum-over or max-pooling operations.Source: §3 · p. 2
- The architecture can replace its basic n-gram vectors and convolution with other feature mappings, including a tensor-algebra mapping over non-consecutive n-grams.Source: §3 · p. 2
4 Model
The model replaces standard linear n-gram convolution with low-rank tensor feature mappings that capture word interactions, including patterns across non-consecutive words. Dynamic programming makes the weighted feature aggregation linear in sequence length, and stacked feature layers combine representations from multiple levels.
Tensor-based feature mapping
- Tensor-product n-grams replace concatenation with multi-way coordinate interactions, and filters map those interaction terms to features.Source: §4 · pp. 2–3
- Low-rank factorization controls the parameter explosion of full tensor filters, with projected word vectors combined element-wise to produce higher-order terms.Source: §4 · p. 3
Non-consecutive n-gram features
- Weighted non-consecutive 3-grams capture variable-spacing patterns such as “not ... good,” while λ downweights triples spanning more intervening words.Source: §4 · pp. 3–4
Aggregating features via dynamic programming
- Dynamic programming reduces n-gram aggregation to O(Ln) matrix operations, linear in sequence length; aggregating 1-, 2-, and 3-grams works better than 3-grams alone.Source: §4 · p. 4
The overall architecture
- Each tensor feature sequence receives ReLU and feeds stacked mappings; the architecture averages each layer’s outputs and concatenates them for prediction.Source: §4 · p. 4
5 Learning the Model
The model is trained on labeled sequences by minimizing cross-entropy and updating its parameters with AdaGrad. L2 regularization and dropout are used to reduce overfitting.
Training objective
- Training minimizes cross-entropy on each labeled sequence, with m denoting the number of possible output labels.Source: §5 · p. 4
Parameter updates
- Model parameters are updated by stochastic gradient descent using AdaGrad.Source: §5 · pp. 4–5
Model architecture
- The architecture stacks feature-map layers, averages outputs within each layer, concatenates the layers, and applies a softmax for prediction.Source: Figure 1 · p. 5
Regularization
- L2 regularization is applied to all parameters with the same weight, and dropout randomly removes output feature units at each level.Source: §5 · p. 5
6 Experimental Setup
Experiments evaluate sentence sentiment classification on the Stanford Sentiment Treebank and news categorization on Sogou Chinese news, alongside tuned baselines and pretrained word vectors.
6 Experimental Setup
- The evaluation uses Stanford Sentiment Treebank sentiment classification and Sogou Chinese news categorization.The sentiment benchmark includes fine-grained and binary variants; Sogou has 10 news categories.Source: §6 · p. 5
- Baselines include SVM, neural bag-of-words, and a CNN with traditional linear filters; sentiment comparisons also cover recursive and convolutional models.Recursive models use parse-tree inputs and can use phrase-level annotations; sequence models are also evaluated with and without phrase annotations.Source: §6 · pp. 5–6
- Sentiment uses fixed 300-dimensional GloVe vectors, while Chinese news uses fixed 200-dimensional word2vec vectors trained on 1.6 million articles.Both vector sets are normalized to unit norm and are not fine-tuned.Source: §6 · p. 6
- Hyperparameters for the proposed model, linear-filter CNN, and SVM are searched across task-specific settings and multiple random initializations.Each neural configuration is run three times; searched settings include n-gram order, feature dimension, dropout, and length decay.Source: §6 · pp. 6–7
- The implementation uses Python and Theano, and reported baseline training times come from prior work or authors, while implementation timings use one processor core.The implementation timings were measured on a 2.6GHz Intel i7 processor.Source: Table 1 · p. 6, §6 · p. 7
7 Results
The full model leads on Stanford Sentiment Treebank accuracy, trains faster than top-performing models, and reaches 80.0% accuracy on Chinese news categorization. Hyperparameter analyses report gains from deeper architectures, non-consecutive n-grams, and ReLU; local predictions capture negation and mixed sentiment.
7.1 Overall Performance
- 51.2% fine-grained and 88.6% binary test accuracy; the full model has the highest Stanford Sentiment Treebank accuracy on both development and test sets, with around 0.5% standard deviation across initializations and dropout rates.Table 2 reports average accuracy and standard deviation for sentiment classification.Source: §7.1 · p. 7, Table 2 · p. 7
- 28 seconds per epoch with root labels and 445 seconds with all labels; CNN-MC takes over 2400 seconds per epoch.The full model's times are averages; the CNN-MC figure is per training epoch.Source: §7.1 · p. 7
- 80.0% accuracy on Chinese news categorization, outperforming SVM, NBoW, and traditional CNN baselines by a 0.8% absolute margin.The model's best configuration uses 2 layers and n = 2; the authors hypothesize this task requires fewer composition or context interactions than sentiment analysis.Source: §7.1 · p. 7, Table 3 · p. 7
7.2 Hyperparameter Analysis
- Adding higher-level features improves fine-grained sentiment accuracy across hyperparameter settings and initializations.Figure 2 plots development accuracy against test accuracy for independent runs.Source: §7.2 · p. 7, Figure 2 · p. 8
- Non-consecutive n-gram features outperform consecutive-only variants, and applying ReLU above the tensor-based feature map improves sentiment-classification performance.The consecutive-only setting uses λ = 0, while non-consecutive features use λ > 0; Figure 4 marks runs without activation as blue circles.Source: Figure 3 · p. 8, Figure 4 · p. 9
7.3 Example Predictions
- Local sentiment predictions identify negation and double negation, as well as positive and negative segments within a sentence.The plotted examples include synthetic inputs and two real test-set inputs.Source: Figure 5 · p. 8, §7.3 · p. 9
8 Conclusion
The proposed CNN feature mapping models n-gram interactions with tensor products, evaluates non-consecutive n-grams, and maintains parameters as a low-rank tensor for efficient feature extraction via dynamic programming.
8 Conclusion
- The operator models n-gram interactions with tensor products and evaluates non-consecutive n-gram vectors.These design choices replace the standard convolution feature mapping.Source: §8 · p. 9
- Low-rank tensor parameters enable efficient feature extraction via dynamic programming.Source: §8 · p. 9
- The model achieves top performance on standard sentiment classification and document categorization tasks.Source: §8 · p. 9