Source-linked AI summary
Character-level Convolutional Networks for Text Classification
Xiang Zhang, Junbo Zhao, Yann LeCun
TL;DR
Character-level ConvNets address limited evidence on using raw characters for text classification, especially with large-scale data. The study finds they become stronger than traditional n-gram TFIDF methods at million-sample scale, while performance depends on dataset characteristics.
Problem
Open text-classification datasets are often too small to assess character-level ConvNets, which typically require large-scale data.
Method
The paper applies one-dimensional ConvNets directly to character-level text and evaluates them on several newly built large-scale classification datasets.
Results
At several-million-sample scale, character-level ConvNets outperform traditional n-gram TFIDF methods, which remain strong on datasets up to several hundred thousand samples.
Takeaways & Limitations
Character-level ConvNets are effective for text classification, but the best method depends on dataset size, text curation, and alphabet choice.
Takeaways & Limitations
No single model works best across all datasets, so method selection depends on application-specific factors.
Abstract
from arXiv · showhide
This article offers an empirical exploration on the use of character-level convolutional networks (ConvNets) for text classification. We constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results. Comparisons are offered against traditional models such as bag of words, n-grams and their TFIDF variants, and deep learning models such as word-based ConvNets and recurrent neural networks.
1 Introduction
The article treats text as a raw character-level signal for temporal ConvNets, using classification to demonstrate their ability to understand text. It argues that, with large-scale datasets, character-only ConvNets can avoid word knowledge while naturally learning abnormal character combinations.
- Background: Earlier text-classification methods were predominantly word-based, with statistics over ordered word combinations such as n-grams usually performing best.Related ConvNet work used distributed or discrete word embeddings without explicit syntactic or semantic structure and remained competitive with traditional models.
- Approach: The study applies one-dimensional ConvNets directly to character sequences, using classification to exemplify their ability to understand text.The authors build several large-scale datasets because ConvNets usually require substantial training data, and compare against traditional and other deep-learning models.
- Related work: Prior character-level approaches combined character n-grams with linear classifiers or extracted character features within word- or word n-gram-based ConvNet representations.These methods reported improvements for part-of-speech tagging and information retrieval, but did not apply ConvNets only to characters.
- Novelty: The article is presented as the first application of ConvNets exclusively to characters, showing that deep ConvNets trained on large-scale datasets do not require word knowledge.This extends prior findings that ConvNets need not encode syntactic or semantic language structure.
- Implications: Character-only processing can simplify engineering for multilingual systems because characters are always available even when word segmentation is not.The passage also notes that abnormal combinations, including misspellings and emoticons, may be learned naturally.
2 Character-level Convolutional Networks
The model uses modular character-level ConvNets built from temporal convolutions, temporal max-pooling, rectifiers, and fully connected layers. Character sequences are one-hot encoded, while temporal max-pooling enables substantially deeper networks.
- Architecture: Character-level ConvNets are designed as modular text-classification models optimized by back-propagation.The architecture is modular, with gradients obtained through back-propagation.
- Architecture: Temporal convolution computes one-dimensional convolutions across input features using parameterized kernel functions and produces output features by summing convolutions.The convolution uses stride d and an offset c = k − d + 1.
- Architecture: Temporal max-pooling enabled training ConvNets deeper than 6 layers, whereas other approaches failed at that depth.It is the one-dimensional counterpart of max-pooling used in computer vision.
- Optimization: The models use rectifier nonlinearities and stochastic gradient descent with minibatches of 128, momentum 0.9, and an initial step size of 0.01 halved every 3 epochs for 10 times.Each epoch samples a fixed number of random training examples uniformly across classes.
- Input representation: Inputs are sequences of one-hot encoded characters from a 70-character alphabet, padded or truncated to fixed length l0, with unknown characters mapped to zero vectors.Characters exceeding l0 are ignored, and characters outside the alphabet, including blanks, become all-zero vectors.
3 Comparison Models
The comparison study evaluates traditional feature-based classifiers and representative deep learning models against character-level ConvNets. It also examines whether distinguishing uppercase from lowercase letters affects performance.
- Comparison design: The experiments compare competitive traditional and deep learning models, with results reported faithfully without model selection.Traditional models use hand-crafted feature extractors with multinomial logistic regression classifiers.
- Traditional models: Traditional baselines include bag-of-words, TFIDF, bag-of-ngrams, n-gram TFIDF, and bag-of-means word-embedding features.Bag-of-words selects 50,000 frequent words, bag-of-ngrams selects 500,000 n-grams up to 5-grams, and bag-of-means uses 300-dimensional embeddings with 5,000 means.
- Deep learning models: The deep learning comparisons use a word-based ConvNet and a word-based LSTM recurrent neural network.The word-based ConvNet comparison includes pretrained word2vec embeddings and end-to-end learned lookup tables, both with embedding size 300.
- Deep learning models: The LSTM forms a feature vector by averaging all cell outputs, then applies multinomial logistic regression; its output dimension is 512.The model uses pretrained 300-dimensional word2vec embeddings and a common vanilla LSTM variant.
- Case sensitivity: Distinguishing uppercase from lowercase letters usually worsens results, although not universally.The authors suggest that case-invariant semantics make case distinction a useful source of regularization.
4 Large-scale Datasets and Results
The authors built several large-scale text-classification datasets because existing open datasets were generally small or had disproportionately small training sets. They report testing errors across applicable models on these datasets, including news, ontology, question-answer, and review classification tasks.
- Motivation: Large-scale datasets were constructed because ConvNets using low-level character features generally benefit from large training sets, while many existing text datasets are small.The authors specifically motivate building new datasets rather than relying on datasets with substantially smaller training than testing sets.
- Datasets: The news benchmarks include AG’s corpus with 496,835 articles, Sogou with 2,909,551 articles, and datasets formed by selecting major labeled categories.AG’s uses four classes with 30,000 training and 1,900 testing samples per class; Sogou combines SogouCA and SogouCS and selects five categories.
- Datasets: DBPedia provides a 14-class ontology task, with 40,000 training and 5,000 testing samples randomly selected per class from DBpedia 2014.The inputs are the title and abstract of each Wikipedia article.
- Datasets: Yelp and Amazon reviews support both full-score and polarity prediction, with Amazon spanning 34,686,770 reviews and Yelp containing 1,569,264 samples.The Yelp polarity task treats stars 1 and 2 as negative and stars 3 and 4 as positive; Amazon similarly constructs full-score and polarity datasets.
- Datasets: Yahoo! Answers contributes a 10-category topic-classification task from 4,483,032 questions and answers, using 140,000 training and 5,000 testing samples per class.The fields include question title, question content, and best answer.
- Results: Testing errors are reported for all applicable models, with no thesaurus-augmentation results for Sogou News because a Chinese thesaurus was unavailable.The best and worst results are marked in blue and red, respectively.
5 Discussion
The discussion identifies dataset scale, text curation, alphabet design, and task characteristics as factors influencing which classification method performs best. It also finds that bag-of-means underperforms consistently and that no single model works best across all datasets.
- Dataset size: At datasets up to several hundreds of thousands, n-gram TFIDF remains strong, while character-level ConvNets outperform only at several-million-example scale.Larger datasets generally perform better across the plots in Figure 3.
- User-generated data: Character-level ConvNets work better on less curated user-generated texts than on more carefully written user-generated data.The comparison concerns word-based deep models on million-scale datasets, including Amazon reviews and Yahoo! Answers.
- Alphabet choice: For million-scale datasets, treating uppercase and lowercase letters as indistinguishable usually works better than distinguishing them.The authors suggest a possible regularization effect but state that this explanation requires validation.
- Task semantics: The distinction between sentiment analysis and topic classification does not seem to determine which method performs better.The datasets include sentiment tasks for Yelp and Amazon reviews and topic-classification tasks for the remaining datasets.
- Bag-of-means: Bag-of-means performs worse in every case, suggesting that this simple use of distributed word representations offers no advantage for text classification.The authors do not generalize this finding to other language-processing tasks or other uses of word2vec.
- Model selection: No single machine-learning model works best for all datasets, so application-specific factors can determine the strongest method.The discussion presents this as a “no free lunch” result.
6 Conclusion and Outlook
The study finds character-level convolutional networks effective for text classification, while performance depends on dataset and text characteristics. Future work will extend them to language-processing tasks requiring structured outputs.
- The article empirically evaluates character-level ConvNets for text classification against traditional and deep learning models across several large-scale datasets.The comparisons include a large number of models and datasets.
- Character-level ConvNets are effective, but comparative performance varies with dataset size, text curation, and alphabet choice.The conclusion identifies these factors as determinants of how well the model performs in comparisons.
- Future work will apply character-level ConvNets to a broader range of language-processing tasks, especially those requiring structured outputs.