Source-linked AI summary
Very Deep Convolutional Networks for Text Classification
Alexis Conneau, Holger Schwenk, Loïc Barrault, Yann Lecun
TL;DR
NLP lacked evidence that very deep convolutional architectures could deliver the gains seen in computer vision, especially for sentence representation and classification. The paper introduces a character-level VDCNN built from small convolutions and pooling, and reports that increasing depth to 29 layers improves performance over prior approaches across text-classification datasets.
Problem
NLP had not yet shown the large gains from deep architectures observed in computer vision, while LSTMs were viewed as generic sequence learners lacking task-specific structure.
Method
VDCNN operates directly on characters and uses a deep stack of small convolutions and max-pooling operations to learn hierarchical sentence representations jointly with classification.
Results
VDCNN performance improves with depth up to 29 convolutional layers and outperforms previous convolutional approaches across eight text-classification datasets.
Takeaways & Limitations
The paper provides evidence that increasing convolutional depth benefits text processing and reports this benefit across the evaluated NLP classification datasets.
Takeaways & Limitations
The study focuses on sentence classification, leaving whether deeper convolutional encoders benefit other sequence tasks such as neural machine translation for future investigation.
Abstract
from arXiv · showhide
The dominant approach for many NLP tasks are recurrent neural networks, in particular LSTMs, and convolutional neural networks. However, these architectures are rather shallow in comparison to the deep convolutional networks which have pushed the state-of-the-art in computer vision. We present a new architecture (VDCNN) for text processing which operates directly at the character level and uses only small convolutions and pooling operations. We are able to show that the performance of this model increases with depth: using up to 29 convolutional layers, we report improvements over the state-of-the-art on several public text classification tasks. To the best of our knowledge, this is the first time that very deep convolutional nets have been applied to text processing.
1 Introduction
NLP seeks to transform and analyze text across granularities, but mainstream sequence models have not matched the large gains seen from deep architectures in vision. VDCNN addresses this gap by learning hierarchical sentence representations with up to 29 convolutional layers.
- NLP tasks range from character- and subword-level processing to word- and sentence-level analysis, including tagging, parsing, translation, and summarization.
- Deep neural networks have performed well in NLP but have not yet achieved the large state-of-the-art margins observed in computer vision and speech recognition.
- LSTMs model long-range dependencies but are described as generic sequence learners lacking task-specific structure.
- Characters combine into increasingly larger units, motivating convolutional architectures that learn hierarchical text representations.
- VDCNN applies many convolutional layers to jointly learn hierarchical representations of whole sentences and the classification task, reaching up to 29 layers.
- The proposed deep convolutional network reports significantly better results than previous ConvNets on several sentence classification tasks.
2 Related work
Prior sentence-classification systems used handcrafted features, embeddings, recursive or recurrent networks, and relatively shallow ConvNets. VDCNN extends character-level convolutional modeling substantially deeper than earlier sentence-classification work.
- Earlier sentence-classification methods used handcrafted features such as bag-of-words, n-grams, and TF-IDF before classification.
- Embedding-based methods can lose token-order information when they combine word or character representations by element-wise averaging.
- Recursive networks use an external parser to determine how word embeddings are combined, while recurrent networks combine representations sequentially.
- Previous ConvNets for NLP were generally word-based and ranged from one to six convolutional layers, with varying filter widths and pooling schemes.
- Hybrid convolutional-recurrent models used up to five convolutional layers to learn features that were then supplied to an LSTM.
- Before VDCNN, no reported sentence-classification work had used VGG-like or ResNet-like architectures deeper than six convolutional layers.
3 VDCNN Architecture
VDCNN represents fixed-length character sequences and processes them through stacked small convolutions, pooling, and convolutional blocks. Down-sampling builds progressively wider feature representations before k-max pooling and classification.
- The model begins with a look-up table that produces character embeddings for a fixed sequence of 1024 characters.
- The network uses size-3 convolutions and convolutional blocks, keeping feature-map counts constant within a temporal resolution and doubling them when resolution halves.
- Three pooling operations halve temporal resolution and create levels with 128, 256, and 512 feature maps.
- Stacking four size-3 convolutions yields a nine-token span, allowing the network to learn broader relations without using larger kernels.
- For classification, k-max pooling selects the k most important features independently of their sentence positions before a three-layer fully connected classifier.
- Convolutional Block: Each convolutional block contains two size-3 convolutional layers, each followed by temporal BatchNorm and ReLU; small filters make greater depth feasible.
- Convolutional Block: The experiments vary network depth across 9, 17, 29, and 49 convolutional layers by adding or removing convolutional blocks.
4 Experimental evaluation
The experiments evaluate VDCNN across eight large-scale text-classification datasets, varying network depth and pooling while using character-level inputs without data augmentation. Increasing depth generally improves performance, with the 29-layer model outperforming prior ConvNets across the datasets, whereas excessive depth without shortcuts degrades accuracy.
- 4.1 Tasks and data: Eight large-scale datasets cover sentiment analysis, topic classification, and news categorization, with 120k to 3.6M training examples.The datasets have balanced training and test classes, and some tasks, especially sentiment analysis, are ambiguous.
- 4.2 Model configurations: The experiments compare depths 9, 17, and 29 alongside stride-2 convolution, k-max pooling, and temporal max-pooling.The evaluation focuses on increasing depth and alternative pooling configurations using small temporal convolution filters.
- 4.3 Experimental results: For depth 9, VDCNN already beats Zhang’s convolutional baselines on Yelp Full, Yahoo Answers, and Amazon Full and Polarity.The largest classification-error decrease occurs on Amazon Full, which has more than 3 million training samples.
- 4.3 Experimental results: 1% absolute error reduction occurs on Amazon Full when increasing depth from 9 to 17 and 29; the best 29-layer model reaches 37.0 test error versus 40.43% previously.This corresponds to a 3.43% absolute accuracy gain and 6.5K additional correctly classified test samples on Amazon Full.
- 4.3 Experimental results: Temporal max-pooling performs best at small depth, while max-pooling is best overall and k-max pooling is significantly inferior to the other two approaches.The authors suggest that applying whole-sentence k-max pooling at intermediate layers hurts performance, except on the smallest datasets.
- 4.3 Experimental results: VDCNN obtains state-of-the-art results on all datasets except AG’s News and Sogou News, and significantly surpasses prior convolutional models.The very deep models match n-gram-based results on Yelp Review Polarity and DBPedia, without data augmentation.
- 4.4 Very deep networks: Without shortcut connections, increasing depth from 29 to 49 layers raises Yelp Full test error from 35.28 to 37.41.Shortcut connections improve 49-layer training and test errors and reduce underfitting, but still do not reach state-of-the-art results.
5 Conclusion
The paper presents a character-level architecture built from deep local operations for sentence classification. Across eight datasets, increasing depth to 29 convolutional layers steadily improves performance, while broader sequence-processing applications remain future work.
- The architecture operates on characters and uses convolutions and max-pooling of size 3 to learn hierarchical sentence representations.
- Eight freely available large-scale datasets were used to evaluate the architecture.
- Increasing depth to 29 convolutional layers steadily improves performance and outperforms previously published convolutional networks on all datasets.
- Applying very deep convolutional models to sequence-processing tasks such as neural machine translation is left for future research.