Source-linked AI summary
Convolutional Neural Networks for Sentence Classification
Yoon Kim
TL;DR
The paper examines whether simple convolutional neural networks built on pretrained word vectors can perform effectively across sentence-level classification benchmarks. It finds that static vectors already yield competitive results, while task-specific fine-tuning provides further improvements.
Problem
The paper examines whether simple CNNs built on pretrained word vectors can perform effectively across sentence-level classification benchmarks.
Method
The authors train a one-layer CNN over unsupervised pretrained word vectors, comparing static vectors with task-specific fine-tuning and a combined architecture.
Results
Static vectors produce competitive benchmark results, and fine-tuning them for each task yields further improvements over the static model.
Takeaways & Limitations
The results support unsupervised pretraining of word vectors as an important ingredient in deep learning for NLP.
Takeaways & Limitations
The multichannel architecture results are mixed, and further work on regularizing fine-tuning is warranted.
Abstract
from arXiv · showhide
We report on a series of experiments with convolutional neural networks (CNN) trained on top of pre-trained word vectors for sentence-level classification tasks. We show that a simple CNN with little hyperparameter tuning and static vectors achieves excellent results on multiple benchmarks. Learning task-specific vectors through fine-tuning offers further gains in performance. We additionally propose a simple modification to the architecture to allow for the use of both task-specific and static vectors. The CNN models discussed herein improve upon the state of the art on 4 out of 7 tasks, which include sentiment analysis and question classification.
1 Introduction
This section motivates applying convolutional neural networks to NLP and introduces a simple CNN built on publicly available pre-trained word vectors. With little hyperparameter tuning and initially static vectors, the model achieves excellent results across multiple benchmarks.
- Motivation: Deep learning in NLP has commonly combined learned word-vector representations with composition over those vectors for classification.The introduction places this work within broader progress in deep learning for computer vision, speech recognition, and NLP.
- Motivation: CNNs, originally developed for computer vision, apply convolving filters to local features and have shown strong performance across several NLP tasks.Reported applications include semantic parsing, search query retrieval, sentence modeling, and other traditional NLP tasks.
- Approach: The work trains a simple one-layer CNN on word vectors obtained from an unsupervised neural language model.The vectors were trained by Mikolov et al. on 100 billion words of Google News and are publicly available.
- Results: 100 billion words: the pre-trained vectors come from Google News, while the initial model keeps them static and learns only its other parameters.The model achieves excellent results on multiple benchmarks despite little hyperparameter tuning.
2 Model
The model applies convolutional filters to sentence word vectors, max-over-time pooling to extract salient features, and a softmax layer to predict labels. A multichannel variant combines static and fine-tuned word vectors, with dropout and constrained weight norms used for regularization.
- Architecture: The model represents a padded sentence as a sequence of k-dimensional word vectors and applies filters over windows of h words to generate features.Each filter has weights in R^(hk) and produces a feature from each possible sentence window.
- Feature extraction: Max-over-time pooling selects the highest-valued activation from each feature map, capturing the most important feature while handling variable sentence lengths.For a feature map c, the pooled feature is ˆc = max{c}.
- Classification: Multiple filters with varying window sizes produce features that form the penultimate layer and feed a fully connected softmax layer yielding label probabilities.The model combines the pooled outputs of multiple filters before classification.
- Multichannel variant: The multichannel variant applies each filter to a static word-vector channel and a fine-tuned channel, then adds the channel results before otherwise following the single-channel architecture.The static channel remains unchanged during training, whereas the other channel is updated through backpropagation.
- Regularization: Regularization uses dropout on the penultimate layer together with an l2-norm constraint on weight vectors.Dropout randomly sets a proportion p of hidden units to zero during forward and backpropagation, while weight vectors exceeding norm s are rescaled after gradient descent.
3 Datasets and Experimental Setup
The experiments span seven sentiment, subjectivity, opinion-polarity, and question-classification benchmarks, using a shared CNN configuration selected on SST-2 with minimal dataset-specific tuning. Models compare random, static, fine-tuned, and multichannel word-vector treatments, initialized where applicable with 300-dimensional word2vec vectors trained on 100 billion Google News words.
- Experimental setup: The shared configuration uses ReLU units, filter windows of 3, 4, and 5 with 100 feature maps each, dropout p=0.5, l2 constraint s=3, and mini-batches of 50.These values were chosen through a grid search on the SST-2 development set.
- Experimental setup: The study avoids dataset-specific tuning beyond early stopping, using a randomly selected 10% development set when no standard dev set exists.Training uses stochastic gradient descent over shuffled mini-batches with the Adadelta update rule.
- Word vectors: Pre-trained representations use publicly available word2vec vectors trained on 100 billion Google News words with dimensionality 300.The vectors were trained using the continuous bag-of-words architecture.
- Model variants: The model variants are CNN-rand, CNN-static, CNN-non-static, and CNN-multichannel, respectively comparing random, fixed pre-trained, task-fine-tuned, and jointly static/fine-tuned channels.In the multichannel model, both channels are initialized with word2vec, but gradients update only one channel.
4 Results and Discussion
Pre-trained word vectors substantially improve CNN sentence classification, with static vectors already competitive and task-specific fine-tuning providing further gains. Additional analyses show mixed multichannel results, task-specific representation learning, and consistent benefits from dropout and word2vec.
- Model comparisons: CNN-static achieves competitive results against deeper models using complex pooling or precomputed parse trees, while CNN-non-static improves further through task-specific fine-tuning.The authors interpret pre-trained vectors as broadly reusable feature extractors across datasets.
- Model comparisons: Multichannel results are mixed, and the authors identify regularizing fine-tuning as an area for further work.The architecture was intended to limit deviation from original vectors and prevent overfitting, especially on smaller datasets.
- Representation analysis: Fine-tuning makes non-static vectors more task-specific, shifting sentiment-related neighbors such as good toward nice rather than great.On SST-2, good is initially most similar to bad in word2vec but changes after fine-tuning.
- Representation analysis: Fine-tuning also gives unseen tokens meaningful representations, associating exclamation marks with effusive expressions and commas with conjunctive usage.These tokens were randomly initialized because they were absent from the pre-trained vectors.
- Further observations: 45.0% is obtained by the authors’ CNN on SST-1 with randomly initialized words, versus 37.4% for Kalchbrenner et al.’s Max-TDNN.The authors attribute the discrepancy to their CNN’s greater capacity from multiple filter widths and feature maps.
- Further observations: 2%–4% relative performance is consistently added by dropout, while word2vec gives far superior performance to Collobert et al.’s publicly available vectors.Dropout allowed the use of a larger-than-necessary network, and the source of word2vec’s advantage remains unclear.
5 Conclusion
The work evaluates convolutional neural networks built on word2vec and finds that a simple one-layer CNN performs remarkably well with little hyperparameter tuning. The results reinforce the importance of unsupervised pre-training of word vectors for deep learning in NLP.
- The experiments use convolutional neural networks built on top of word2vec.
- A simple CNN with one convolutional layer performs remarkably well despite little hyperparameter tuning.
- The results add to evidence that unsupervised pre-training of word vectors is important for deep learning in NLP.