Source-linked AI summary
Investigating Capsule Networks with Dynamic Routing for Text Classification
Wei Zhao, Jianbo Ye, Min Yang, Zeyang Lei, Suofei Zhang, Zhou Zhao
TL;DR
Text classification needs models that capture information beyond individual word meanings, while multi-label classification remains challenging because its expanded label space requires more training and costly data collection. This paper applies capsule networks with dynamic routing and three stabilization strategies to text classification, achieving the best results on 4 of 6 benchmarks and significant improvement when transferring from single-label to multi-label classification.
Problem
Text classification requires modeling higher-level text information beyond individual word meanings, and multi-label classification remains challenging because its label space expands from n to 2^n and large datasets are costly to collect and annotate.
Method
The paper applies capsule networks with dynamic routing to text classification and proposes three strategies to reduce disturbance from noisy capsules.
Results
Capsule networks achieve the best results on 4 of 6 text-classification benchmarks and show significant improvement over strong baselines when transferring from single-label to multi-label classification.
Takeaways & Limitations
Capsule networks are effective for text classification and have stronger transferring capability from single-label to multi-label classification than conventional deep neural networks.
Abstract
from arXiv · showhide
In this study, we explore capsule networks with dynamic routing for text classification. We propose three strategies to stabilize the dynamic routing process to alleviate the disturbance of some noise capsules which may contain "background" information or have not been successfully trained. A series of experiments are conducted with capsule networks on six text classification benchmarks. Capsule networks achieve state of the art on 4 out of 6 datasets, which shows the effectiveness of capsule networks for text classification. We additionally show that capsule networks exhibit significant improvement when transfer single-label to multi-label text classification over strong baseline methods. To the best of our knowledge, this is the first work that capsule networks have been empirically investigated for text modeling.
1 Introduction
Text classification requires modeling more than individual words, including compositions, hierarchies, structures, and spatial patterns. Capsule networks are introduced as a text-modeling approach that uses iterative routing and stabilization strategies to address limitations of existing models and noisy capsules.
- Text categorization demands modeling compositions, hierarchies, and structures beyond individual word meanings.
- CNNs and LSTMs model texts as sequences and spatial patterns, while other approaches treat words as order-insensitive collections.
- Spatially sensitive CNNs incur inefficiency from replicating feature detectors across grids, with detector size or labeled-data requirements potentially growing exponentially with dimensionality.
- Capsule networks use iterative routing to assign credit between lower- and higher-layer nodes, potentially encoding part–whole spatial relationships.
- The study proposes three strategies to stabilize dynamic routing by reducing disturbance from noisy capsules containing background or category-irrelevant information.
2 Our Model
The model combines n-gram convolution, capsule layers, transformation matrices, and dynamic routing to represent text while stabilizing routing against noisy capsules.
- Overall architecture: The architecture comprises n-gram convolutional, primary capsule, convolutional capsule, and fully connected capsule layers.Two capsule frameworks integrate these four components differently.
- N-gram convolutional layer: The n-gram convolutional layer extracts features at different sentence positions using convolutional filters and nonlinear activation.Filters slide with stride 1 over word windows to produce feature maps.
- Primary capsule layer: Primary capsules replace scalar CNN feature detectors with vector outputs that preserve local word order and semantic representations.Each capsule is produced by multiplying a sliding-window n-gram vector by a shared filter matrix.
- Child-parent relationships: Transformation matrices convert child capsules into prediction vectors for parent capsules, using either shared or non-shared weights.The shared design uses W t1 across child capsules, whereas the alternative uses W t2 matrices indexed by child and parent positions.
- Dynamic routing: Dynamic routing iteratively adjusts child-parent connection strengths and aggregates prediction vectors so outputs are sent to appropriate parent capsules.The model uses routing by agreement in local convolutional capsule regions and the fully connected capsule layer.
- Routing stabilization: Three strategies reduce noisy-capsule disturbance: an orphan category, Leaky-Softmax, and amendment using child-capsule existence probabilities.The orphan category captures background text such as stop words, while Leaky-Softmax routes noise to an extra dimension without added parameters or computation.
3 Experimental Setup
The experiments evaluate capsule networks on six text-classification benchmarks using standard word embeddings, mini-batch training, Adam optimization, and comparisons with strong neural baselines.
- Experimental Datasets: Experiments cover six benchmarks spanning sentiment classification, question categorization, and news categorization.The datasets are MR, SST-2, Subj, TREC, CR, and AG’s news.
- Training Configuration: 300-dimensional word2vec vectors initialize the embedding vectors for all experiments.The embeddings are pretrained word representations used to initialize the model.
- Training Configuration: Adam optimization uses a 1e-3 learning rate, with batch size 50 for AG’s news and 25 for the other datasets.The reported routing configuration uses three iterations for all datasets.
- Baselines: The evaluation compares capsule networks with LSTM, Bi-LSTM, Tree-LSTM, LR-LSTM, CNN variants, VD-CNN, and CL-CNN baselines.These include both recurrent and convolutional neural architectures.
4 Experimental Results
Capsule networks achieve the best results on four of six benchmarks under classification accuracy, outperform simpler neural baselines, and benefit from routing strategies that reduce noise-capsule disturbance.
- Quantitative Evaluation: Capsule networks achieve the best results on 4 out of 6 benchmarks using classification accuracy.The results are summarized in Table 2.
- Quantitative Evaluation: Capsule networks substantially and consistently outperform LSTM, Bi-LSTM, and CNN-rand on all experimental datasets.They also achieve competitive results against LR-LSTM, Tree-LSTM, VD-CNN, and CL-CNN.
- Quantitative Evaluation: Capsule-B consistently performs better than Capsule-A because it learns more meaningful and comprehensive text representations.Capsule-B uses parallel n-gram networks with filter windows of 3, 4, and 5.
- Ablation Study: All three proposed dynamic-routing strategies contribute to Capsule-B’s effectiveness by alleviating disturbance from noisy capsules.The cited noise capsules may contain background information such as stop words or words unrelated to specific categories.
5 Single-Label to Multi-Label Text Classification
The study evaluates capsule networks for transferring from single-label to multi-label text classification, using single-label training data and multi-label test settings. Results show improvements over strong baselines, while visualizations and routing curves examine how capsules support multi-category prediction.
- Transfer Setting: The study trains on single-label Reuters documents and evaluates transfer to Reuters-Multi-label and Reuters-Full test sets.Reuters-21578 contains documents with either multiple or single labels; training and development use only single-label documents.
- Evaluation: Micro-averaged Precision, Recall, F1, and Exact Match Ratio evaluate multi-label classification.Exact Match Ratio counts only fully correct samples, treating partially correct predictions as incorrect.
- Results: Capsule networks substantially improve all four metrics over strong baselines on Reuters-Multi-label and Reuters-Full.The larger improvement occurs on Reuters-Multi-label, while Reuters-Full results indicate robust performance on single-label documents.
- Connection Strength Visualization: Connection strengths between primary capsules and fully connected capsules act like parallel attention over N-gram phrases.The visualization is designed to show which primary capsules are important for text categories and how multiple categories can be recognized from single-label training.
- Connection Strength Visualization: A fully correct Reuters-Multi-label example predicts Interest Rates and Money/Foreign Exchange with high confidence, and routing directs votes into both categories.The example uses p > 0.8; stronger connection strengths appear as larger tag-cloud fonts.
- Routing Convergence: Capsule-B with 3 or 5 routing iterations optimizes training loss faster and reaches lower final loss than one iteration.The comparison is shown through learning curves for the Reuters-Multi-label dataset.
6 Related Work
Related work progresses from feature-engineered text classifiers to neural models using distributed representations, recurrent networks, and convolutional networks. Capsule networks are presented as a newer approach intended to address representational limitations of CNNs and RNNs.
- Traditional Methods: Early text classifiers used bag-of-words, n-grams, and TF-IDF with SVM, logistic regression, or naive Bayes.These methods often relied on laborious feature engineering or extensive linguistic resources.
- Capsule Networks: Capsule networks replace CNN scalar-output feature detectors with vector-output capsules and max-pooling with routing-by-agreement.The related work describes capsules as preserving instantiated parameters and learning part-whole relationships through transformation matrices.
7 Conclusion
The paper investigates dynamic-routing capsule networks for text classification and proposes strategies to reduce noise-capsule disturbance. Experiments across six benchmarks report strong classification performance and improved single-label-to-multi-label transfer.
- Conclusion: The paper investigates capsule networks with dynamic routing for text classification and proposes three strategies to alleviate disturbance from noisy capsules.The noise capsules may contain background information or may not have been successfully trained.
- Conclusion: Experiments on six text classification benchmarks show the effectiveness of capsule networks for text classification.The conclusion also reports significant improvement when transferring from single-label to multi-label classification over strong baseline methods.
Supplementary Material
The supplementary material analyzes capsule connections through unigram rankings and includes examples of fully and partially correct multi-label predictions. It also reports additional component variations for Capsule-B on the MR dataset.
- Supplementary Material: Top-20 unigrams from Orphan, Trade, Money Exchange, and Interest Rates categories are sorted by connection strength.The supplementary analysis uses primary capsules representing unigrams and compares their category-specific connection strengths.
- Supplementary Material: A fully corrected case shows correct category labels despite category-specific phrases appearing only once.The example is described as having weak confidence, with 0.4 < p < 0.6.
- Supplementary Material: A partially corrected case predicts Interest Rates correctly but confuses Money/Foreign Exchange with Mergers/Acquisitions.The category-specific phrases are subtle and not mentioned directly in the example.
- Supplementary Material: Appendix ablations vary routing iterations, leaky softmax, shared child-parent weights, and the orphan category on the MR dataset.The standard routing condition omits leaky softmax and the orphan category in the last capsule layer.