Source-linked AI summary
A Sensitivity Analysis of (and Practitioners' Guide to) Convolutional Neural Networks for Sentence Classification
Ye Zhang, Byron Wallace
TL;DR
CNN sentence-classification performance depends on architectural and hyperparameter choices, but the relative importance of those choices was not well established. The paper evaluates one-layer CNN configurations across nine datasets, finding task-dependent representation effects and substantial stochastic variation, then derives practical guidance for tuning. Its recommendations are intended primarily for sentence datasets resembling those studied.
Problem
CNNs require many architecture and hyperparameter decisions, yet their sensitivity to these configurations for sentence classification was unclear and exhaustive tuning was expensive.
Method
The study performs extensive experiments on one-layer CNN configurations across nine sentence-classification datasets, using replicated cross-validation and reporting accuracy/AUC means and ranges.
Results
Word-vector representations affect performance, with GloVe and word2vec varying by task and generally outperforming one-hot vectors; stochastic inference alone produces accuracy ranges up to 1.5 points and irony-dataset AUC ranges up to 3.4 points.
Takeaways & Limitations
Practitioners should experiment with different pre-trained word vectors and report replicated performance ranges rather than relying only on mean scores.
Takeaways & Limitations
The practical suggestions apply only to datasets with properties similar to those studied, and some datasets may produce contrary findings.
Abstract
from arXiv · showhide
Convolutional Neural Networks (CNNs) have recently achieved remarkably strong performance on the practically important task of sentence classification (kim 2014, kalchbrenner 2014, johnson 2014). However, these models require practitioners to specify an exact model architecture and set accompanying hyperparameters, including the filter region size, regularization parameters, and so on. It is currently unknown how sensitive model performance is to changes in these configurations for the task of sentence classification. We thus conduct a sensitivity analysis of one-layer CNNs to explore the effect of architecture components on model performance; our aim is to distinguish between important and comparatively inconsequential design decisions for sentence classification. We focus on one-layer CNNs (to the exclusion of more complex models) due to their comparative simplicity and strong empirical performance, which makes it a modern standard baseline method akin to Support Vector Machine (SVMs) and logistic regression. We derive practical advice from our extensive empirical results for those interested in getting the most out of CNNs for sentence classification in real world settings.
1 Introduction
CNNs offer strong sentence-classification performance with relatively simple architectures, but practitioners face expensive training and a large hyperparameter space. This work empirically identifies which settings merit tuning across nine datasets and reports performance ranges as well as means.
- Simple one-layer CNNs achieve strong sentence-classification results and may replace established baselines such as SVMs or logistic regression.
- About 1 hour is required for 10-fold cross-validation on SST-1 using a similar CNN configuration.
- CNN configuration requires specifying representations, filter sizes, feature-map counts, activations, pooling, and regularization.
- The architecture and hyperparameter search space is too large for practitioners to tune exhaustively, even with emerging optimization methods.
- The study tests CNN configurations across nine datasets, identifying settings worth tuning and reporting accuracy/AUC means and ranges from replicated cross-validation.
2 Background and Preliminaries
The one-layer CNN converts tokenized sentences into matrices of word vectors, applies filters across adjacent words, pools feature maps into fixed-length vectors, and classifies them. Its trainable components include filters, biases, softmax weights, and optionally word vectors.
- Tokenized sentences become matrices whose rows are word-vector representations, allowing convolutional filters to operate over the input.
- Filter width equals the word-vector dimensionality, while filter height determines the number of adjacent rows processed jointly.
- A filter repeatedly produces an output sequence from sentence-matrix submatrices, with region size h determining its parameter count h · d.
- Multiple filters can learn complementary features at one region size or across different region sizes.
- Pooling converts variable-length feature maps into fixed-length vectors that are concatenated and passed to a softmax classifier.
- Training minimizes categorical cross-entropy using SGD and back-propagation, with non-static models also tuning word vectors.
3 Datasets
The study evaluates sentence-classification CNNs on nine datasets, including sentiment benchmarks such as MR, SST-1, and SST-2. SST-1 uses sentences בלבד rather than the phrases-and-sentences setup used in earlier work.
- The experiments use nine sentence-classification datasets, seven of which were also used by Kim (2014).
- MR is a sentence-polarity dataset, while SST-1 is the Stanford Sentiment Treebank.
- For SST-1, the study trains and tests only on sentences to keep input representations consistent across tasks.
4 Baseline Models
The analysis finds that several CNN design choices materially affect sentence-classification performance, while others offer limited or inconsistent gains. Region size and input representations merit tuning, whereas pooling, dropout, and l2 regularization are comparatively less promising.
- Input word vectors: Non-static word vectors uniformly outperform static vectors, while GloVe versus word2vec performance depends on the dataset.Simply concatenating GloVe and word2vec representations does not necessarily help.
- Filter region size: Each dataset has its own optimal filter region size, suggesting a coarse search over sizes 1 to 10, with larger sizes possible for longer sentences.For CR, whose maximum sentence length is 105, the optimal region size may exceed the range useful for shorter sentiment datasets.
- Filter region size: Combining region sizes near the best single size can improve performance, whereas adding distant or suboptimal sizes can hurt it.On MR, near-optimal sets such as (5,6,7), (7,8,9), and (6,7,8,9) performed best, while a single good size could outperform several poor ones.
- Number of feature maps: Increasing feature maps beyond 600 yields marginal or negative returns and increases training time, supporting a search range of 100 to 600.The performance loss is attributed as likely overfitting, though more than 600 maps may help in some cases.
- Pooling: 1-max pooling consistently outperforms alternative pooling strategies, while average pooling performs much worse and runs more slowly.The authors suggest that predictive contexts may be location-independent and that individual n-grams can be more predictive than the full sentence jointly.
- Regularization: Dropout provides little benefit overall, and l2 norm constraints generally do not improve performance and can reduce it on some datasets.Non-zero dropout helps slightly at some rates and datasets, while l2 regularization adversely affects CR in at least one setting.
5 Conclusions
The analysis identifies which one-layer CNN design choices matter for sentence classification and translates those findings into practical deployment guidance. It also emphasizes reporting stochastic variation and limiting conclusions to datasets with similar sentence properties.
- Main empirical findings: Stochastic inference can change mean accuracy by up to 1.5 points and irony-dataset AUC by up to 3.4 points.The authors recommend repeated cross-fold validation and reporting ranges or variances to avoid potentially spurious relative-performance conclusions.
- Main empirical findings: Input word-vector choice affects performance, with word2vec and GloVe generally outperforming one-hot vectors for sentence classification.The better representation differs across tasks, and the one-hot comparison may change with sufficiently large training data.
- Main empirical findings: The filter region size can substantially affect performance and should be tuned.A reasonable initial search range is 1–10, with larger sizes worth exploring for very long sentences.
- Main empirical findings: Increasing the number of feature maps can improve performance but also increases training time.Practitioners are advised to explore roughly 100–600 feature maps while considering the resulting runtime trade-off.
- Main empirical findings: 1-max pooling uniformly outperforms the alternative pooling strategies evaluated, whereas regularization has relatively little effect.The guidance therefore prioritizes 1-max pooling and suggests stronger regularization when additional feature maps begin reducing performance.
- Practical guidance: The simple one-layer CNN is positioned as a standard baseline to consider before more sophisticated models, with the findings also informing random-search or Bayesian-optimization ranges.These suggestions are intended as practical starting points rather than universal rules.
Appendix
The appendix collects reference tables covering dataset lengths, baseline logistic regression, CNN configurations, and sensitivity analyses across architecture and regularization choices.
- Dataset and baselines: The appendix includes a table of average and maximum sentence lengths for the seven datasets.This provides dataset-level length statistics for the evaluated sentence-classification tasks.
- Dataset and baselines: A separate appendix table reports logistic-regression performance.The table is identified as a baseline performance reference.
- CNN sensitivity analyses: The appendix reports CNN performance by single filter region size, feature-map count, activation function, pooling strategy, dropout rate, and L2-norm constraint.Several tables distinguish non-static and static word2vec-CNN configurations.
- CNN sensitivity analyses: Additional tables cover activation functions and alternative pooling strategies, including global k-max and local max pooling.These tables are identified as performance analyses using non-static or static word2vec-CNNs where specified.
- Regularization analyses: The appendix also tabulates the effects of dropout rates at different model locations and feature-map settings.The listed tables separately examine dropout-rate effects for static and non-static word2vec-CNNs.
- Regularization analyses: Two further tables report the effect of L2-norm constraints for non-static and static word2vec-CNNs.These tables extend the appendix’s regularization analysis beyond dropout.