Hierarchical Taxonomy-Aware and Attentional Graph Capsule RCNNs for Large-Scale Multi-Label Text Classification

Hao Peng; Jianxin Li; Qiran Gong; Senzhang Wang; Lifang He; Bo Li; Lihong Wang; Philip S. Yu

Source-linked AI summary

Hierarchical Taxonomy-Aware and Attentional Graph Capsule RCNNs for Large-Scale Multi-Label Text Classification

Hao Peng, Jianxin Li, Qiran Gong, Senzhang Wang, Lifang He, Bo Li, Lihong Wang, Philip S. Yu

arXiv:1906.04898v1Jun 9, 2019cs.IRcs.LGstat.ML

TL;DR

Large-scale multi-label text classification requires models that jointly capture long-distance and sequential semantics and exploit hierarchical label relations. The paper proposes HE-AGCRCNN with graph-based document modeling, attentional capsule recurrent CNNs, and taxonomy-aware loss, reporting improved performance across benchmark evaluations. Its stated future work targets richer attention mechanisms, newer architectures, and broader datasets and applications.

Problem
Existing deep models often separately address non-consecutive, long-distance, or sequential semantics and do not effectively leverage hierarchical dependencies among labels.
Method
HE-AGCRCNN models documents as word-order-preserved graph-of-words, learns features with attentional capsule recurrent CNNs, and uses hierarchical taxonomy-aware weighted margin loss.
Results
The proposed approach outperforms traditional shallow and recent deep learning methods on three benchmark datasets, with best results reported on RCV1 and EUR-Lex.
Takeaways & Limitations
The framework provides an end-to-end approach for combining multiple textual semantics with hierarchical label information in large-scale multi-label classification.
Takeaways & Limitations
Future work proposes extending the framework to more sophisticated architectures and broader text-classification datasets and applications.

Abstract

from arXiv · show

CNNs, RNNs, GCNs, and CapsNets have shown significant insights in representation learning and are widely used in various text mining tasks such as large-scale multi-label text classification. However, most existing deep models for multi-label text classification consider either the non-consecutive and long-distance semantics or the sequential semantics, but how to consider them both coherently is less studied. In addition, most existing methods treat output labels as independent methods, but ignore the hierarchical relations among them, leading to useful semantic information loss. In this paper, we propose a novel hierarchical taxonomy-aware and attentional graph capsule recurrent CNNs framework for large-scale multi-label text classification. Specifically, we first propose to model each document as a word order preserved graph-of-words and normalize it as a corresponding words-matrix representation which preserves both the non-consecutive, long-distance and local sequential semantics. Then the words-matrix is input to the proposed attentional graph capsule recurrent CNNs for more effectively learning the semantic features. To leverage the hierarchical relations among the class labels, we propose a hierarchical taxonomy embedding method to learn their representations, and define a novel weighted margin loss by incorporating the label representation similarity. Extensive evaluations on three datasets show that our model significantly improves the performance of large-scale multi-label text classification by comparing with state-of-the-art approaches.

1 INTRODUCTION

The paper addresses the difficulty of jointly modeling long-distance, non-consecutive, and local sequential text semantics while incorporating hierarchical dependencies among labels. It proposes HE-AGCRCNN, combining graph-based document modeling, attentional capsule recurrent CNNs, and taxonomy-aware weighted margin loss, and reports strong results on benchmark datasets.

Existing RNNs, CNNs, and attention models capture only partial textual semantics, leaving long-distance relationships or sequential structure insufficiently modeled.
Hierarchical dependencies among labels remain insufficiently leveraged in existing deep learning models for multi-label classification.
HE-AGCRCNN combines word-order-preserved graph-of-words, attentional capsule recurrent CNNs, and hierarchical taxonomy-aware weighted margin loss.
Extensive evaluations on three benchmark datasets report improved efficiency and effectiveness over traditional shallow and recent deep learning methods.
The document model preserves non-consecutive, long-distance, and local sequential semantics for richer textual feature extraction.
The proposed attention recurrent network learns local sequential semantics, while the taxonomy-aware loss measures class distances in the label hierarchy.

2 WORD ORDER PRESERVED GRAPH-OF-WORDS FOR DOCUMENT MODELING

The document representation models words as a positional, weighted graph, selects central words, extracts bounded neighborhood subgraphs, and normalizes them into an arranged words-matrix. This representation preserves local sequential, non-consecutive, and long-distance information for subsequent neural processing.

Each document is represented as a weighted directed graph whose vertices are words, edges encode sliding-window co-occurrence, and node attributes record positional indexes.
Top N central words are selected using closeness centrality, with larger centrality indicating greater graph importance.
For every central word, BFS and DFS extract a neighborhood subgraph capped at K nodes, retaining local sequential, non-consecutive, and long-distance information.
Each extracted subgraph is normalized to a length-T word sequence, padded when necessary, and concatenated into an arranged words-matrix.
Word2vec embeddings convert the words-matrix into a 3-D tensor, with zero vectors representing padding, for later convolutional, recurrent, and capsule networks.

3 ATTENTIONAL CAPSULE RECURRENT CNN

The attentional capsule recurrent CNN learns document features from a 3-D tensor while combining non-consecutive, long-distance, and local sequential semantics. Its recurrent convolution layers feed a capsule network that preserves subgraph-specific representations and supports class prediction.

Architecture: The model takes an N × T × D document tensor as input and outputs a 3-D feature map for capsule processing.N is the number of central words, T the normalized sequence length, and D the word-embedding dimension.
Attentional recurrent CNN: Two attentional recurrent convolution layers combine convolutional feature extraction with recurrent modeling of horizontal word sequences.The first layer uses kernels of size 1 × 3 × D, while the second uses kernels of size 1 × 3 × k1.
Attentional recurrent CNN: The attentional recurrent CNN integrates long-distance, non-consecutive, and local sequential semantics from each corresponding subgraph.Words within each block share an attention parameter, allowing different blocks to receive different impacts during local sequential semantic learning.
Capsule network: The capsule network independently learns feature representations for each subgraph from the recurrent convolution output.Dynamic routing combines prediction vectors from lower-level capsules, while the architecture preserves subgraph independence.
Capsule network: The final capsule vector length represents the presence probability of each class and is used to calculate the classification loss.The final DigitCaps layer contains capsules associated with the classes.

4 HIERARCHICAL TAXONOMY-AWARE WEIGHTED MARGIN LOSS

The method embeds labels using the taxonomy graph and uses their semantic distances to weight the margin loss. This incorporates parent-child and graph relations among labels into multi-label training.

Taxonomy representation: The label taxonomy is represented as a directed graph whose vertices are classes and whose edges encode hierarchical parent-child relations.The taxonomy may contain cycles, so it is modeled as a hierarchical graph-of-labels rather than a strict hierarchy.
Label embedding: Two meta-paths, “Child1 - Father0 - Child2” and “Father1 - Child0 - Father2”, guide random walks through lower and upper taxonomy layers.The two meta-paths are sampled with equal probability during random walking.
Label embedding: Skip-gram with negative sampling encodes the generated label sequences into taxonomy-aware label representations.The method uses negative sampling to approximate the large-network normalization term and optimizes the objective with stochastic gradients.
Weighted margin loss: The semantic distance between labels is computed from the cosine distance between their embedding vectors.This distance supplies the label-relation information used by the weighted loss.
Weighted margin loss: The weighted margin loss assigns each negative label a weight based on its minimum semantic distance from the positive-label set.An adjustment factor normalizes the weighted objective across labels.

5 EXPERIMENTS

The experiments evaluate the framework on three datasets using standard metrics, traditional and deep-learning baselines, hierarchical models, sequence-generation models, and architectural variants.

Datasets: Experiments use RCV1 and EUR-Lex for large-scale multi-label classification, plus Reuters-21578 to test transfer from single-label to multi-label classification.Reuters-21578 training and development use single-label documents, while testing uses multi-label documents.
Metrics: Micro-F1 weights all instances equally through overall precision and recall, whereas Macro-F1 gives equal weight to each class label.
Baselines: The comparison includes flat TF-IDF baselines, word-sequence and graph-of-words models, hierarchical classifiers, and sequence-generation methods.Flat baselines ignore relations among words and labels, while other families model textual or label structure differently.
Baselines: Capsule baselines use N-gram convolutions followed by dynamic or static routing to learn relationships among text features.Capsule-B uses parallel networks with different filter sizes and performs better than Capsule-A.
Model Variants: The proposed variants progressively combine graph CNNs, recurrent CNNs, attention, capsule networks, and hierarchical weighted margin loss.The variants include TGCNN, TGRCNN, GCCNN, GCRCNN, AGCRCNN, and hierarchical taxonomy-aware versions.

5.3 Experimental Settings

The experiments use fixed computational, document-modeling, optimization, and capsule-network settings, while Fig. 5 compares taxonomy-embedding reconstruction across thresholds.

Implementation Settings: Experiments run on 64 CPU cores, 512GB RAM, and 8 NVIDIA Tesla P100 GPUs using Ubuntu, Python, and PyTorch.
Document Modeling: Document modeling selects 100 central words for RCV1 and Reuters-21578, 200 for EUR-Lex, subgraph size 25, sequence length 20, and word-embedding dimension 50.
Evaluation Settings: Fig. 5 compares label-network reconstruction produced by two hierarchical taxonomy embedding methods across multiple thresholds.
Objective Functions: Capsule models use either the original margin loss or the hierarchical taxonomy embedding-based weighted margin loss.Non-capsule models use cross entropy or the hierarchical weighted margin loss.

5.4 Evaluation on Label Embedding

The label-embedding evaluation reconstructs label relations from cosine distances and compares meta-path-guided and traditional random walks using Macro-F1 and Micro-F1.

Label Embedding: Meta-path-guided and traditional random walks generate label sequences, which skip-gram converts into label vectors for reconstructing hierarchical label relations.
Relation Reconstruction: Edges are added when the cosine distance between two label vectors exceeds a threshold, and reconstruction is evaluated with Macro-F1 and Micro-F1.
Results: For RCV1, meta-path taxonomy embedding reaches Macro-F1 0.337 and Micro-F1 0.310, versus 0.275 and 0.272 for traditional random-walk embedding.The best meta-path thresholds are 0.660 and 0.940.
Results: Overall, the meta-path-guided random walk performs better at capturing hierarchical taxonomy semantics than the traditional random walk.

5.5 Performance Evaluation on RCV1

On RCV1, the proposed components improve classification performance over established baselines, with graph-based representations, sequential modeling, capsules, attention, and hierarchical loss each contributing to the results.

Baseline Comparisons: Deep recurrent baselines are comparable to SVM and logistic regression, while RCNN performs worse on both evaluation settings.The passage attributes recurrent-model limitations on RCV1 to long-document representation compression.
Baseline Comparisons: DCNN improves over XML-CNN by 9% in Macro-F1 and 4% in Micro-F1, while graph CNN models improve over Capsule-B by 4% and 3%.The improvements are reported for Macro-F1 and Micro-F1, respectively.
Proposed Models: HE-AGCRCNN outperforms HR-DGCNN-3 by 8% in Macro-F1, while TGCNN(No-R) achieves comparable Macro-F1 and Micro-F1 with DGCNN-3.
Proposed Models: TGRCNN and TAGRCNN improve Macro-F1 over HR-DGCNN-3 by 5%-6%, supporting the importance of local sequential semantics in the reported text features.
Ablation Findings: Capsule networks provide an average 1% gain in both Macro-F1 and Micro-F1, while hierarchical weighted margin loss adds 2% Macro-F1 and 1% Micro-F1.

5.6 Performance Evaluation on EUR-Lex

On EUR-Lex, the proposed model improves classification over HR-DGCNN-3, while component comparisons indicate benefits from sequential modeling, attention, capsules, and hierarchical label modeling.

Results: About 6% Macro-F1 and 4% Micro-F1 gains over HR-DGCNN-3 are achieved by HE-AGCRCNN on EUR-Lex.The passage reports these improvements directly for the proposed model.
Ablation findings: LSTM units, attentional LSTM units, capsule networks, and hierarchical taxonomy-aware weighted margin loss each improve classification performance.The reported configuration comparisons attribute improvements to these components collectively.
Ablation findings: TGCNN reaches Macro-F1 0.257 and Micro-F1 0.655 without hierarchical dependencies, LSTM units, attentional LSTM units, or capsules.These values are reported for the simplified model relative to HR-DGCNN-3.
Ablation findings: Ordering words in sub-graphs improves TGCNN from Macro-F1 0.244 and Micro-F1 0.648, demonstrating the value of local sequential semantics.The comparison is between TGCNN(No-R) and TGCNN.
Interpretation: The framework combines non-consecutive, long-distance, and sequential semantics for topical multi-label text classification.This conclusion connects the arranged words-matrix representation and the proposed deep models to the reported semantic coverage.

5.7 Performance Evaluation on Reuters-21578

On Reuters-21578 transfer from single-label to multi-label classification, HE-AGCRCNN outperforms the reported baselines, with proposed variants improving Micro-F1 over Capsule-B.

Transfer performance: About 5%-7% Micro-F1 improvements over the existing best baseline Capsule-B are achieved by the proposed graph-capsule recurrent CNN variants.The passage lists GCCNN, GCRCNN, AGCRCNN, HE-GCCNN, HE-GCRCNN, and HE-AGCRCNN among these variants.
Baseline comparison: LSTM, BiLSTM, and CNN-non-static perform worst among the compared models in the transfer experiment.This comparison is explicitly reported against capsule-based and proposed models.
Evaluation setting: RCV1 testing uses multi-core CPUs because its test set contains about 34 times more samples than its training set.This is the stated testing setup for the training-efficiency evaluation.
Training efficiency: Most evaluated models reach a promising classification result in under 3 hours, while LSTM or capsule units increase training time.The efficiency discussion also reports faster convergence for TGCNN and HE-TGCNN.

6 RELATED WORK

Related work covers traditional feature-engineered classification and neural text models, then situates graph methods while identifying limitations in coherent semantic and label-dependency modeling.

Organization: The paper’s related-work discussion is organized around text classification, textual deep learning models, and graph convolution networks.The section explicitly identifies these three aspects.
Traditional methods: Traditional text classification uses feature engineering and selection, including LDA topic representations and bag-of-words features.The related-work discussion contrasts LDA with BOW as vocabulary size changes.
Deep learning models: RNNs, CNNs, capsule models, attention mechanisms, and self-attention architectures have all been applied to text classification.The cited approaches target long documents, important words or sentences, long-range dependencies, and local dependencies.
Research gap: Prior textual models often model N-grams or sequences while ignoring high-level non-consecutive and long-distance semantics and label dependencies.This limitation is stated in the comparison with the paper’s approach.
Graph methods: Graph convolution research develops graph filters, adjacency-based updates, localized spectral operations, pooling, and graph embedding approaches.The passage places graph classification methods such as PSCN and GCAPS-CNN within this development.

7 CONCLUSION AND FUTURE WORK

The paper concludes that its hierarchical taxonomy-aware graph capsule recurrent CNN framework combines multiple textual semantics and label relations, with strong results across evaluated datasets and transfer settings.

Contributions: The proposed framework represents documents with an arranged words-matrix preserving non-consecutive, long-distance, and local sequential semantics.This document model is the first major component described in the conclusion.
Contributions: HE-AGCRCNN uses masked attentional LSTM units to model differing impacts across word-sequence blocks and enhance sequential feature learning.The conclusion describes this as part of the feature-learning design.
Contributions: A hierarchical taxonomy-aware weighted margin loss incorporates hierarchical label relations to improve multi-label classification performance.The loss is presented as the mechanism for incorporating label hierarchy.
Conclusions: The proposed models obtain the best results on all RCV1 and EUR-Lex datasets in the comparative evaluation.The conclusion also reports effectiveness in transferring from single-label to multi-label classification.
Future work: Future work includes subgraph-level attention capsules, self-attention RNN/CNN models, variable-size convolution kernels, BERT pre-training, and broader datasets and applications.These are the explicitly listed future directions.