Source-linked AI summary
Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers
Yijun Xiao, Kyunghyun Cho
TL;DR
Word-level document classification has limitations with shared word structure and rare or unseen words, motivating character-level modeling. The paper introduces a convolution-recurrent architecture that combines convolutional layers with a recurrent layer for character inputs and evaluates it on eight tasks. It reports comparable results to convolution-only models with much fewer convolutional layers, especially when the number of classes is large, training data is small, or two or three convolutional layers are used.
Problem
Word-level approaches handle shared roots, prefixes, and suffixes inefficiently and map rare or unseen words to unknown tokens, limiting handling of typos and domain mismatch.
Method
The paper processes character sequences with convolutional layers followed by a single recurrent layer, using recurrent units such as GRU or LSTM to capture long-term dependencies.
Results
Comparable results were achieved on eight large-scale document classification tasks with much fewer convolutional layers than the convolution-only architecture.
Takeaways & Limitations
The model generally performs better when the number of classes is large, training size is small, and the convolutional depth is two or three layers.
Takeaways & Limitations
The paper identifies recurrent-layer substitution for pooling and applications beyond document classification as directions for future exploration rather than established results.
Abstract
from arXiv · showhide
Document classification tasks were primarily tackled at word level. Recent research that works with character-level inputs shows several benefits over word-level approaches such as natural incorporation of morphemes and better handling of rare words. We propose a neural network architecture that utilizes both convolution and recurrent layers to efficiently encode character inputs. We validate the proposed model on eight large scale document classification tasks and compare with character-level convolution-only models. It achieves comparable performances with much less parameters.
1 Introduction
Document classification traditionally uses word-level representations, but character-level inputs address shortcomings involving shared word structure and rare or unseen words. The paper proposes a convolution-recurrent architecture that captures long-term dependencies with fewer convolutional layers and achieves comparable performance to a convolution-only model.
- Motivation: Word-level document classification commonly represents documents with word vectors before jointly extracting features and predicting categories.Conventional pipelines may instead use TF-IDF features followed by a classifier.
- Motivation: Word-level models treat related words separately, despite shared roots, prefixes, or suffixes, and external segmentation is language-dependent.The paper describes this as statistically inefficient.
- Motivation: Word-level models map unseen or rare words to an unknown token, making typos and domain transfer difficult.The paper specifically mentions informal social-network postings and mismatch between training and target domains.
- Character-level approaches: Character-level inputs can replace word-level inputs without changing the general neural-network setup and have been used for document classification.Prior work includes character-level convolutional networks with repeated convolution and max-pooling layers.
- Problem: Small convolutional receptive fields require many layers to capture long-term dependencies in character sequences.The cited convolution-only model used six convolutional layers followed by two fully connected layers.
- Proposed approach: The proposed hybrid processes character sequences with convolutional layers followed by one recurrent layer, enabling long-term dependency capture with fewer convolutional layers.The recurrent layer may use GRU or LSTM units.
- Evaluation: On eight large-scale document-classification tasks, the proposed model achieves comparable performance to the convolution-only baseline with significantly fewer parameters.The comparison is against the character-level convolutional model from Zhang et al. (2015).
2 Basic Building Blocks: Neural Network Layers
The paper builds document classifiers from embedding, convolutional, recurrent, and classification layers. These components transform symbol sequences into contextual representations and then fixed-dimensional category probabilities, while gated and bidirectional recurrent variants address long-range dependency and information-balance issues.
- Embedding Layer: Documents are represented as sequences of T one-hot vectors, one for each symbol in the vocabulary.Each vector has one active element corresponding to its symbol.
- Embedding Layer: The embedding layer maps each one-hot symbol vector into a d-dimensional continuous vector sequence.The mapping uses a learned weight matrix W.
- Convolutional Layer: A convolutional layer applies d′ filters over receptive fields of size r at every time step, followed by a nonlinear activation.The resulting sequence is denoted F = (f1, f2, . . . , fT).
- Convolutional Layer: The convolutional output is max-pooled over windows of size r′, producing a shorter sequence.Max pooling operates elementwise on the vectors.
- Recurrent Layer: A recurrent layer recursively combines each input vector with the previous hidden state to produce a sequence of hidden states.The initial hidden state is often initialized as an all-zero vector.
- Recurrent Layer: The naive recurrent function uses weight matrices for the current input and previous hidden state but is known to suffer from vanishing gradients.This limitation motivates more complicated recurrent units.
- Recurrent Layer: LSTM units control information flow through input, output, and forget gates plus a candidate memory cell to capture long-term dependencies more easily.The recurrent layer computes the memory cell and then the output activation.
- Recurrent Layer: A bidirectional recurrent layer uses two recurrent layers in opposite directions to reduce the imbalance in information available to hidden states across time.Earlier forward states otherwise observe fewer lower-layer vectors than later states.
3 Character-Level Convolutional-Recurrent Network
The proposed ConvRec combines convolutional and recurrent layers for character-level document classification, using recurrence to capture long-term dependencies with fewer convolutional layers. It processes embedded character sequences through convolutional layers, a bidirectional recurrent layer, and classification.
- 3 Character-Level Convolutional-Recurrent Network: Convolutional layers efficiently extract higher-level, locally translation-invariant features from character sequences.
- 3 Character-Level Convolutional-Recurrent Network: Long character-level documents require many convolutional layers to capture long-term dependencies because convolution and pooling are local.
- 3 Character-Level Convolutional-Recurrent Network: A single recurrent layer can capture long-term dependencies, but recurrent computation is more expensive and less parallelizable than convolution.
- 3 Character-Level Convolutional-Recurrent Network: ConvRec combines convolutional and recurrent layers so recurrence assists convolutional layers in capturing document-wide dependencies more efficiently.
- 3.2 Model Description: The model embeds one-hot character inputs, applies multiple convolutional layers, feeds the resulting feature sequence into a bidirectional recurrent layer, and classifies a fixed-dimensional vector.
- 3.3 Related Work: Unlike related convolution-only models, ConvRec uses recurrence to support long-term dependency capture across the whole document; this purpose differs from hierarchical Conv-GRNN processing.
4 Experiment Settings
The experiments evaluate character-level architectures on eight large-scale document classification tasks using varied convolutional configurations and regularized training. Model selection uses validation data, with test error reported for the lowest-validation-error model.
- 4.1 Task Description: Eight large-scale tasks span sentiment analysis, ontology classification, question type classification, and news categorization, with datasets containing 200,000 to 4,000,000 documents.
- 4.2 Model Settings: Character inputs use a 96-character vocabulary and character embeddings of size d = 8.
- 4.2 Model Settings: The experiments test architectures with two to five convolutional layers, 128 filters per layer, and receptive fields of five or three.
- 4.2 Model Settings: Dropout is applied after the last convolutional layer and after the recurrent layer, with dropout probability set to 0.5.
- 4.3 Training and Validation: Validation sets match test-set sizes and are class-balanced; early stopping selects the model with the lowest validation error for test evaluation.
- 4.3 Training and Validation: Training minimizes regularized cross-entropy loss with weight decay λ = 5 × 10^-4 and uses AdaDelta with batch size 128.
5 Results and Analysis
Across eight datasets, the hybrid model matched the convolution-only model with significantly fewer parameters. Its relative advantage generally increased with more classes and smaller training sets, while accuracy peaked at two or three convolutional layers.
- Comparable performance across all eight datasets came with significantly fewer parameters than the best character-level convolutional baseline.
- Number of classes: The hybrid model generally performed better as the number of classes increased, including on DBPedia, while convolution-only models scored better on two-class polarity tasks.
- Number of training examples: The proposed model generally performed better when the training set was small, which the authors associate with its compactness and reduced overfitting.
- Number of convolutional layers: Accuracy peaked at two or three convolutional layers and decreased with additional layers, indicating an optimal amount of local features for the recurrent layer.
- Number of filters: Increasing the model to 1,024 filters yielded limited validation-error gains despite a 70-fold increase in parameters.Validation error changed from 8.75% to 8.39% for AG’s news and from 29.48% to 28.62% for Yahoo! Answers.
6 Conclusion
The paper concludes that a character-level hybrid of convolutional and recurrent layers can achieve comparable results with fewer convolutional layers across eight large-scale classification tasks. It also identifies settings where the model generally performs better and points to broader applications for the architecture.
- The proposed architecture combines convolutional layers with a single recurrent layer to encode character-level documents.
- Across eight large-scale document classification tasks, the model achieved comparable results using much fewer convolutional layers than the convolution-only architecture.
- The model generally performed better with more classes, smaller training sets, and two or three convolutional layers.
- The architecture is presented as applicable beyond document classification and natural-language inputs, with future applications suggested for machine translation and music information retrieval.