Source-linked AI summary
Multichannel Variable-Size Convolution for Sentence Classification
Wenpeng Yin, Hinrich Schütze
TL;DR
Sentence classification must capture complex structures and compose information from words and variable-size phrases, while small datasets make overfitting a concern. MVCNN combines diverse pretrained word embeddings with variable-size convolution filters and uses pretraining; it achieves the best results against baselines across four sentence-classification tasks.
Problem
Sentence classification requires modeling complex structures and variable-size phrases, but many tasks provide small training sets that make overfitting a danger for large models.
Method
MVCNN combines multiple pretrained embedding versions as multichannel inputs with variable-size convolution filters for multigranular phrase features.
Results
MVCNN’s best configuration gets the best results, beating all baselines across four sentence classification tasks.
Takeaways & Limitations
Multichannel initialization and variable-size filters enhance performance on sentiment and subjectivity classification tasks, with pretraining having a bigger impact than mutual learning.
Takeaways & Limitations
Embedding complementarity is likely more useful for some tasks than others; POS tagging may benefit less because a single embedding version may suffice.
Abstract
from arXiv · showhide
We propose MVCNN, a convolution neural network (CNN) architecture for sentence classification. It (i) combines diverse versions of pretrained word embeddings and (ii) extracts features of multigranular phrases with variable-size convolution filters. We also show that pretraining MVCNN is critical for good performance. MVCNN achieves state-of-the-art performance on four tasks: on small-scale binary, small-scale multi-class and largescale Twitter sentiment prediction and on subjectivity classification.
1 Introduction
MVCNN addresses sentence classification by combining diverse pretrained word embeddings with variable-size convolution filters, while pretraining helps control overfitting on small datasets.
- Sentence classification requires representing sequential and hierarchical structure while composing features from words and variable-size phrases.
- Fixed-size CNN filters struggle to identify a good phrase granularity across sentences, motivating variable-size filters for multigranular phrase features.
- Different embedding versions capture varying semantic and syntactic characteristics, motivating their combination as distinct word descriptions.
- MVCNN uses multichannel initialization, where each channel corresponds to a different embedding version.
- Pretraining MVCNN on unlabeled data addresses overfitting risks from its many parameters when classification training sets are small.
- The architecture combines variable-size convolution filters and multiple embedding versions, with mutual learning and pretraining used to further enhance performance.
2 Related Work
Related work covers neural sentence composition, convolutional architectures, and studies of embedding combinations. MVCNN extends these directions by combining diverse embedding versions and developing variable-size filters.
- Earlier approaches composed word embeddings through element-wise operations, recursive autoencoders, or phrase-level composition.
- Convolutional sentence models used sliding-window filters with pooling, while deeper models combined stacked convolution with dynamic k-max pooling.
- Kim’s multichannel model used two copies of one pretrained Word2Vec embedding, whereas MVCNN incorporates diverse embedding versions.
- Sentence-level parameter vectors can produce generic representations that encode no task-specific features.
- Prior embedding studies found benefits from combining representations, and MVCNN draws on diversity across corpora and learning principles.
3 Model Description
MVCNN combines multiple embedding channels with variable-size convolution filters to represent phrases at different granularities. Its architecture uses stacked convolution, dynamic k-max pooling, and a fully connected classifier, with wide convolution and pretraining-oriented design choices supporting sentence classification.
- Multichannel Input: MVCNN initializes each sentence with multiple embedding versions, reducing unknown words because representations missing from one channel can be supplied by others.In the binary sentiment vocabulary, only 1824 words lacked embeddings from every channel, despite thousands being unknown in individual versions.
- Convolution Layer: The convolution layer concatenates embeddings from l consecutive words and projects them through tanh to produce local phrase features.The resulting feature vector contains local features across the sentence, with l controlling filter width.
- Variable-Size Convolution: MVCNN extends multi-kernel convolution by using variable-size filters in the same layer, allowing feature maps to capture regions with different granularities.This design addresses the limitation that traditional same-size kernels cannot detect features from differently sized regions.
- Convolution Layer: Convolution combines multiple input feature maps through learned kernels, modeling dependencies across representation dimensions and adjacent words without a folding step.The model uses rich feature maps and flexible convolution to extract variable-range phrase features.
- Dynamic k-Max Pooling: Dynamic k-max pooling retains the top activations at each layer, enabling stacked convolution layers to extract increasingly abstract sentence features.The top layer uses an empirically fixed ktop value of 4.
- Classification: The final pooled features pass through a fully connected sentence-representation layer and then logistic regression for classification.MVCNN combines this classifier with variable-range feature extraction and multichannel initialization.
4 Model Enhancements
MVCNN uses mutual learning to complete missing embedding representations and pretraining to initialize model parameters and word embeddings before supervised classification. The pretraining objective predicts component words from sentence representations, and is especially helpful for unknown words.
- Mutual-Learning of Embedding Versions: Mutual learning predicts unknown-word representations in each embedding version through projections from other versions, giving all versions the same vocabulary.Projections are trained on overlapping vocabularies, with squared error between observed and projected representations.
- Mutual-Learning of Embedding Versions: For an unknown word, MVCNN averages projected representations from the embedding versions where that word is known to estimate its representation in the missing space.The method treats each embedding version as a distinct source description and averages projected vectors in the target space.
- Pretraining: Pretraining uses unlabeled task data to predict a sentence's middle word from its sentence representation and surrounding-word representations.The averaged vector is trained with noise-contrastive estimation to identify the true middle word.
- Pretraining: During pretraining, multichannel word inputs, convolution parameters, and fully connected parameters are updated before fine-tuning the same parameters for supervised classification.Word inputs use multichannel initialization, while context and target words use random initialization.
- Training Effects: Pretraining is designed to produce good initial values for model parameters and word embeddings, with particular benefit for embeddings of unknown words.On the binary sentiment dataset, 5022 words unknown in at least one embedding version had coverage in another version and could benefit from mutual learning.
5 Experiments
MVCNN is evaluated on four sentence-classification tasks using five embedding versions, variable filter widths, and variants that isolate embedding, filter, training-trick, and depth choices. Its overall configuration achieves the best results against the reported baselines, while pretraining, all embedding versions, and multiple filter sizes contribute to performance.
- Tasks and Setup: MVCNN is tested on four classification tasks: binary and five-class movie-review sentiment, Twitter sentiment, and subjectivity classification.The Twitter dataset contains 1.6 million automatically labeled training tweets and about 400 hand-annotated test tweets.
- Pretrained Word Vectors: The experiments use five pretrained embedding versions, with embeddings trained using different corpora and algorithms.The authors use the diversity of these embedding sets as a source of complementary information.
- Pretrained Word Vectors: About 30% of words in each task have partially initialized embeddings, and mutual learning initializes the missing embeddings through projections.For Sentiment140, 236,484 of 387,877 words, or 61%, have no hit in any embedding version, making pretraining especially relevant.
- Results and Analysis: MVCNN overall achieves the best results against all reported baselines across the four sentence-classification tasks.The overall configuration uses five embedding versions, filter sizes 3, 5, 7, and 9, mutual learning, pretraining, and task-dependent convolution depth.
- Results and Analysis: Two convolution layers perform best on Standard Sentiment Treebank and Subjectivity Classification, whereas Sentiment140 performs best with three layers.The authors speculate that the larger Sentiment140 dataset may benefit from deeper networks.
- Results and Analysis: Pretraining has a bigger performance impact than mutual learning, every embedding version is crucial, and filter sizes 5 and 7 are most important.The system benefits from filters of each tested size, especially width 7.
6 Conclusion
MVCNN combines diverse pretrained word embeddings with variable-size convolution filters to extract multigranular phrase features for sentence classification. The paper reports that these components enhance performance on sentiment and subjectivity classification.
- MVCNN is a CNN architecture for sentence classification that combines multichannel initialization with variable-size filters.Different pretrained embedding versions initialize the channels, while variable-size filters extract features from multigranular phrases.
7 Future Work
The authors identify several unresolved factors behind MVCNN’s multichannel success, including embedding algorithms, training corpora, and task-dependent complementarity. They leave these effects for future investigation.
- 7 Future Work: The multichannel approach may succeed through several distinct effects rather than a single mechanism.The authors specifically point to differences among embedding learning algorithms and corpora.
- 7 Future Work: Embedding learning algorithms differ in word-order sensitivity, objective function, and treatment of ambiguity.These differences complicate attribution of multichannel gains to embedding diversity alone.
- 7 Future Work: Corpus size and genre may substantially affect embedding performance, but this effect was not analyzed in the paper.
- 7 Future Work: Embedding complementarity is likely more useful for some tasks than others.The authors contrast sentiment, which draws on heterogeneous information, with POS tagging, where one embedding version may be sufficient.
- 7 Future Work: The authors plan to pursue these questions in future work.