Source-linked AI summary
HDLTex: Hierarchical Deep Learning for Text Classification
Kamran Kowsari, Donald E. Brown, Mojtaba Heidarysafa, Kiana Jafari Meimandi, Matthew S. Gerber, Laura E. Barnes
TL;DR
Growing document collections and category hierarchies challenge conventional multi-class document classifiers. HDLTex performs hierarchical classification by combining deep learning architectures specialized to different hierarchy levels. On Web of Science documents, RNN-at-higher-level combinations with DNN or CNN below achieved consistently higher accuracy than conventional naïve Bayes or SVM approaches.
Problem
Growing document collections and expanding numbers of categories make traditional multi-class document classification increasingly challenging.
Method
HDLTex combines deep learning architectures specialized for different levels of a document hierarchy, using parent-level outputs to define lower-level models.
Results
RNN at the higher level combined with DNN or CNN at the lower level produced accuracies consistently higher than conventional naïve Bayes or SVM approaches.
Takeaways & Limitations
Deep learning methods can improve hierarchical document classification on the Web of Science document dataset evaluated in this paper.
Abstract
from arXiv · showhide
The continually increasing number of documents produced each year necessitates ever improving information processing methods for searching, retrieving, and organizing text. Central to these information processing methods is document classification, which has become an important application for supervised learning. Recently the performance of these traditional classifiers has degraded as the number of documents has increased. This is because along with this growth in the number of documents has come an increase in the number of categories. This paper approaches this problem differently from current document classification methods that view the problem as multi-class classification. Instead we perform hierarchical classification using an approach we call Hierarchical Deep Learning for Text classification (HDLTex). HDLTex employs stacks of deep learning architectures to provide specialized understanding at each level of the document hierarchy.
I. INTRODUCTION
Growing document volumes and increasingly specialized fields make conventional classification harder, especially when documents must be organized across fields and sub-fields. HDLTex addresses this challenge through hierarchical deep learning that specializes models by hierarchy level.
- 2.5 million articles were published in 2014 across 28,100 active scholarly journals, while publication growth was accelerating.
- Growing corpora and expanding fields and sub-fields make automatic classification increasingly challenging.
- Hierarchical classification organizes documents by overall field and accompanying sub-field, rather than by specialized area alone.
- Few existing approaches rapidly organize documents into correct sub-fields, while combined top-level and sub-field labels create a combinatorial classification challenge.
- HDLTex combines deep learning architectures to provide overall and specialized learning at different hierarchy levels.
III. BASELINE TECHNIQUES
The paper reviews conventional and deep-learning baselines for document classification, including SVM variants and multi-class extensions for problems with several labels.
- Six baseline methods include term-weighted SVM, multi-word SVM, naïve Bayes, and three deep learning methods used in the implementation.
- A. Support Vector Machines (SVMs): Nonlinear SVMs use kernels to address more complex classification problems.
- Multi-Class SVM: The original SVM solves binary classification, so document classification requires extensions when several classes are present.
- Multi-Class SVM: Multi-class SVM constructs a decision function across all k classes, using pairwise comparisons or one-versus-all classification.
Stacking Support Vector Machines (SVM):
The baseline methods include stacking SVMs and naïve Bayes, contrasting an ensemble of SVM classifiers with a fast, interpretable probabilistic classifier.
- Stacking Support Vector Machines (SVM): Stacking SVMs combine individual SVM classifiers in an ensemble and generally produce more accurate results than single-SVM models.
- Naïve Bayes classification: Naïve Bayes is a fast and interpretable supervised learning technique often used for information retrieval.
- Naïve Bayes classification: The study’s naïve Bayes classifier uses word-level classification with word-specific parameters.
IV. FEATURE EXTRACTION
HDLTex extracts text features differently for its deep learning architectures: CNNs and RNNs use 100-dimensional GloVe vectors, while DNNs use count-based and tf-idf N-gram features.
- CNN and RNN models use 100-dimensional GloVe vector-space representations of document text.
- The vector-space representation maps words in each document to word-embedding vectors.
- DNN feature extraction uses count-based and term frequency–inverse document frequency features over N-grams.
- Feature count (1) represents individual words with their document frequencies.
- Feature count (2) includes both 1-grams and 2-grams, including individual words and adjacent word pairs.
- The resulting DNN feature space indexes counts by N-gram size and supports N-gram features within deep learning models.
V. DEEP LEARNING NEURAL NETWORKS
Deep learning neural networks use layered nonlinear processing to generalize from text features, and HDLTex extends these architectures for hierarchical classification.
- Deep learning networks combine nonlinear processing elements in layers to form computational models that generalize to new data.
- The baseline DNN uses fully connected layers, with text features as input and classification labels represented in the output layer.
- The architecture also includes recurrent neural networks whose child-level models receive parent-level document outputs as inputs.
- HDLTex trains a first-level DNN on all documents and child-level DNNs on documents from the predicted parent domain.
- The DNNs use back-propagation with sigmoid and ReLU activations, while the output layer uses softmax.
B. Recurrent Neural Networks (RNN)
RNNs process sequential text by allowing recurrent feedback, while LSTM and GRU variants modify this architecture to address its limitations. HDLTex uses an extended RNN architecture for hierarchical classification.
- RNN: RNNs allow a layer’s output to reenter that layer as input, supporting text processing over sequential states.The formulation defines x_t as the state at time t and u_t as the input at step t.
- RNN: The implementation reformulates the RNN with recurrent and input weight matrices, a bias, and an element-wise activation function.The modified architecture is extended for hierarchical classification.
- LSTM: LSTM preserves long-term dependencies more effectively than basic RNNs and is particularly effective at mitigating vanishing gradients.RNNs can encounter vanishing and exploding gradients during backpropagation.
- LSTM: LSTM regulates information entering each node state through multiple gates in a chain-like cell structure.The gates include input, cell memory, forget, and output functions.
- CNN contrast: CNN max-pooling is described as a way to determine discriminative phrases and address the bias toward later words in RNNs.This passage presents CNNs as a contrasting approach discussed elsewhere in the paper.
- GRU: GRU is a simplified LSTM variant with two gates, no internal memory, and no second tanh non-linearity.The paper identifies GRU as a gating mechanism for RNNs introduced in 2014.
C. Convolutional Neural Networks (CNN)
CNNs are used for text classification by connecting convolutional layers to local subsets of inputs and preceding-layer representations.
- CNN: CNNs, originally developed for image processing, have also been effectively used for text classification.The paper describes CNN as its final deep learning approach for hierarchical document classification.
- CNN: A CNN convolutional layer connects to a small input subset, typically of size 3 × 3, while subsequent layers connect locally to the preceding layer.This layered local connectivity defines the basic convolutional structure described here.
D. Hierarchical Deep Learning
HDLTex addresses document classification with architectures specialized for levels of a document hierarchy rather than treating the task as ordinary multi-class classification. The implementation combines RNN and CNN configurations and uses a hierarchical cost function.
- Contribution: The paper’s primary contribution is hierarchical document classification for settings where performance drops as the number of classes increases.The authors contrast this with traditional multi-class classification.
- Architecture: HDLTex creates deep learning architectures that specialize approaches for their respective levels of the document hierarchy.The passage identifies RNN, GRU, and LSTM as components of the implementation.
- Implementation: The RNN implementation uses GRU and LSTM with two hidden layers, while the CNN uses filter sizes of {3, 4, 5, 6, 7}.These are implementation configurations reported for the hierarchical model.
- Implementation: The CNN contains 8 hidden layers with layer sizes of {128, 128, 128} and max-pooling sizes of {5, 5, 35}.The reported CNN configuration also uses max-pooling of 5 after the filters.
- Training: All models use batch size 128, learning parameters 0.001, β1=0.9, β2=0.999, ϵ = 1e08, and decay 0.0.Dropout is 0.5 for DNN and 0.25 for CNN and RNN.
- Training: The deep learning models use a cost function defined across hierarchical levels, classes at each level, and classes in each child level.ϱ denotes the number of levels, k the number of classes at each level, and Ψ the number of classes in the child’s level.
F. Optimization
The paper uses RMSProp and Adam as stochastic gradient optimizers for its deep learning models. Adam addresses sparse-gradient issues that affect RMSProp while handling non-stationary objectives.
- RMSProp and Adam are the two stochastic gradient optimization methods used for the deep learning models.
- SGD updates parameters using a momentum term on a rescaled gradient.The learning parameter is θ, α is the learning rate, and J(θ, x_i, y_i) is the objective or cost function.
- Adam averages the first two gradient moments, m and v, using bias-corrected estimates.The first and second moments are denoted m_t and v_t, respectively.
- Adam handles non-stationary objectives and can overcome the sparse-gradient issue identified as a drawback in RMSProp.
VI. RESULTS
The experiments use Web of Science documents organized into hierarchical label sets and evaluate HDLTex alongside baseline methods. The study uses CPU/GPU hardware and Python-based CUDA, Keras, and TensorFlow implementations.
- A. Data: The document collection contains 46,985 published Web of Science papers and uses two levels of target labels.The top level contains seven fields, while the child levels contain 17, 16, 19, 9, 11, 53, and 9 specific topics, respectively.
- A. Data: Three datasets are used: the full WOS-46985 collection and the WOS-11967 and WOS-5736 subsets.WOS-11967 has five subclasses under each of seven level-1 classes; WOS-5736 has four subclasses under each of two higher-level classes and a different final structure.
- B. Hardware and Implementation: The approaches were implemented in Python with CUDA, Keras, and TensorFlow using CPU and GPU resources.The hardware included a 32-core Xeon E5-2640 system with 64GB memory and Nvidia Quadro K620 and Tesla K20c GPUs.
- B. Hardware and Implementation: The experiments compare HDLTex and baseline accuracies across three WOS datasets.Table III is identified as the source of these accuracy comparisons.
C. Empirical Results
Deep learning improves general document classification over traditional approaches, while stacked HDLTex architectures provide superior performance for hierarchical classification. The authors identify broader datasets, deeper hierarchies, and unlabeled documents as future application directions.
- RNN outperforms the other tested methods on all three WOS datasets, with CNN generally second-best.Traditional and deep learning baselines are compared across the three datasets.
- HDLTex stacked architectures provide superior performance for classifying documents within a hierarchy.For WOS-11967, the best architecture combines RNN at the first level with DNN at the next level.
- CNN at both levels wins on WOS-5736, although two other HDLTex architectures obtain similar scores.The alternatives are DNN with CNN and RNN with CNN.
- Across the Web of Science evaluation, RNN at the higher level combined with DNN or CNN at the lower level consistently exceeds conventional naїve Bayes and SVM approaches.
- Future work includes testing additional hierarchical datasets, extending the hierarchy beyond two levels, and applying HDLTex to unlabeled documents.