Source-linked AI summary
Text Classification using Capsules
Jaeyoung Kim, Sion Jang, Sungchul Choi, Eunjeong Park
TL;DR
The paper investigates whether capsule networks can be used for text classification, where their validity had not been established despite effectiveness in image classification. It applies capsule architectures with dynamic and static routing, and finds comparable benchmark performance while static routing provides higher accuracy with less computation.
Problem
The paper addresses the limited evidence for capsule networks in text classification despite their demonstrated effectiveness in image classification.
Method
The authors apply capsule networks to text classification, add an ELU-gate, and compare dynamic routing with a proposed static-routing variant across seven benchmark datasets.
Results
Capsule networks provide comparable results on seven benchmark datasets, while static routing achieves higher classification accuracy with less computation than dynamic routing.
Takeaways & Limitations
Capsule networks are useful for text classification, and the proposed static-routing variant preserves this utility while reducing computational cost.
Takeaways & Limitations
The model uses six capsules, selected from held-out-data experiments, and this count is much lower than the 1,152 capsules used for image classification.
Abstract
from arXiv · showhide
This paper presents an empirical exploration of the use of capsule networks for text classification. While it has been shown that capsule networks are effective for image classification, their validity in the domain of text has not been explored. In this paper, we show that capsule networks indeed have the potential for text classification and that they have several advantages over convolutional neural networks. We further suggest a simple routing method that effectively reduces the computational complexity of dynamic routing. We utilized seven benchmark datasets to demonstrate that capsule networks, along with the proposed routing method provide comparable results.
1 Introduction
This paper applies capsule networks to text classification, motivated by their success in image classification and the dominance of CNNs and RNNs in text. It reports comparable results and proposes routing and gating adaptations for this setting.
- Deep learning text-classification methods have largely centered on convolutional and recurrent neural networks.
- Capsule networks, effective for image classification, are applied here to text classification to test their advantages in this domain.
- The paper applies dynamic-routing capsule networks to text classification and achieves comparable results to previous methods.
- The authors propose an alternative routing method that achieves higher accuracy than dynamic routing.
- The model also uses an ELU-gate to propagate relevant information.
2 Related Work
Related work establishes CNNs, RNNs, and self-attention as major approaches to text classification, while capsule networks address representation and pooling limitations through vectors and routing.
- CNNs have been applied to text at the sentence and character levels, with dynamic convolutional networks adding length-dependent pooling.
- RNNs exploit text's sequential structure, and LSTM-based approaches have achieved strong results despite weaker performance from naive RNNs.
- Self-attention networks provide another successful text-classification approach without convolutions or recurrence.
- CNN pooling reduces computation and captures local-feature invariance but can lose spatial relationships and cause orientation- or proportion-based misclassification.
- Capsules encode entities and their properties as vectors, while dynamic routing learns relationships between lower- and upper-level capsules.
3 Model
The proposed text model combines an ELU-gated convolutional feature map with convolutional and text capsule layers. It evaluates dynamic and static routing, with static routing motivated by the weaker need for positional hierarchies in language.
- 3.1 Architecture: Documents are represented as vectors because capsules express attributes of partial entities and semantic meaning in a wider space than scalar representations.The input document is D ∈ R^l×e, where l is document length and e is embedding size.
- 3.1 Architecture: The ELU-gate selects activated features without discarding spatial information as pooling does.
- 3.1.1 Capsule network with dynamic routing: Dynamic routing updates coupling coefficients iteratively to direct lower-level capsules toward upper-level capsules according to their similarity.
- 3.1 Architecture: The architecture passes each document through a gate layer, convolutional capsule layer, and text capsule layer.
- 3.1.2 Capsule network with static routing: Static routing is proposed because documents permit substantial variation in expression and do not require the same positional hierarchy as images.
- 3.1.2 Capsule network with static routing: Static routing multiplies capsule inputs by a weight matrix, applies squashing, and represents documents as vectors.
4 Experimental Settings
The experiments evaluate the capsule model on seven benchmark datasets spanning sentiment, topic, news, and question classification, using published dataset statistics and tuned CNN comparisons. Training uses pretrained GloVe vectors, Adam optimization, dropout, and a six-capsule configuration selected on held-out data.
- Datasets: The datasets vary in vocabulary size, pretrained-vocabulary coverage, and average tokenized sequence length, as summarized in Table 1.Table 1 defines |V| as vocabulary size, |Vpre| as words present in pretrained vectors, and l_avg as average sequence length.
- Hyperparameters and training: Training uses 840 billion-word pretrained GloVe vectors, Adam, a 0.99 per-epoch learning-rate decay, dropout p = 0.5, and embedding size e = 300.The model was trained on a GPU using TensorFlow, with additional hyperparameters listed in Table 2.
- Hyperparameters and training: The capsule count is set to 6 using held-out-data experiments, far below the 1,152 capsules used by Sabour et al. for image classification.The authors conjecture that text benchmark feature maps have lower complexity, allowing an appropriate representation with fewer capsules.
- Hyperparameters and training: The CNN classification model from Kim (2014) serves as the experimental baseline, with dataset-specific parameter tuning reported in Table 3.The baseline comparisons use tuned CNN parameters for each dataset.
5 Results and analysis
The experiments compare capsule-network routing and vector representations with CNN-based approaches across benchmark text-classification settings. Static routing preserves capsule properties, improves some comparisons over dynamic routing, and supports robustness and interpretability analyses.
- Routing comparisons: Static routing achieved higher accuracy than dynamic routing, which the authors attribute to greater complexity in the second convolutional feature-map layer.The comparison is reported in Table 4.
- Capsule representations: Increasing capsule-vector dimension increased accuracy while keeping the number of trainable parameters fixed.The authors interpret this as greater capacity to represent entity information and attributes.
- Capsule representations: Perturbation tests showed that static-routing capsules represented properties such as rotation, thickness, and scale, retaining essential capsule characteristics.The authors contrast this vector-based representation with CNN computation by independent neurons.
- Capsule networks over CNNs: Unlike CNN max-pooling, the static-routing model learned syntactic word representations without discarding positional context.Max-pooling updates only the highest-activation context, whereas this model does not use max-pooling.
- Robustness to text changes: With pretrained word vectors, static routing achieved much higher accuracy than dynamic routing after word-order changes in TREC-QA samples.The evaluation used 50 samples each from the ENTITY and HUMAN classes.
- Routing and gating: Removing the coupling factor reduced computational complexity and improved generalization relative to dynamic routing, while ELU-gating supported global word updates during fine-tuning.The ELU-gate comparison was evaluated in an ablation setting.
6 Conclusion
The paper applies capsule networks to text classification, proposes static routing, and finds capsule models useful across seven benchmark datasets.
- Capsule networks are useful for text classification based on seven popular benchmark datasets.
- Static routing is proposed as an alternative to dynamic routing for capsule-based text classification.
- Static routing achieves higher classification accuracies with less computation than dynamic routing.