Source-linked AI summary
Towards Scalable and Reliable Capsule Networks for Challenging NLP Applications
Wei Zhao, Haiyun Peng, Steffen Eger, Erik Cambria, Min Yang
TL;DR
Capsule networks for NLP face challenges in scaling to large output spaces and achieving reliable instance-level routing. This paper introduces an agreement-based routing evaluation, an adaptive optimizer, capsule compression, and partial routing; it reports improvements over strong competitors on two NLP tasks and best results in low-resource settings.
Problem
Capsule networks for NLP need to scale to large output spaces while making routing more reliable at the instance level.
Source: §2 · p. 2Method
The framework evaluates routing with an agreement score, adapts iteration counts per example, and uses capsule compression and partial routing for scalability.
Source: §1 · p. 2Results
The framework outperforms strong baselines on multi-label text classification and question answering and achieves superior generalization in low-resource settings.
Source: §1 · p. 2Takeaways & Limitations
The paper concludes that capsule networks can be extended with scalability, reliability, and generalizability advantages for two NLP tasks.
Source: §5 · p. 8
Abstract
from arXiv · showhide
Obstacles hindering the development of capsule networks for challenging NLP applications include poor scalability to large output spaces and less reliable routing processes. In this paper, we introduce: 1) an agreement score to evaluate the performance of routing processes at instance level; 2) an adaptive optimizer to enhance the reliability of routing; 3) capsule compression and partial routing to improve the scalability of capsule networks. We validate our approach on two NLP tasks, namely: multi-label text classification and question answering. Experimental results show that our approach considerably improves over strong competitors on both tasks. In addition, we gain the best results in low-resource settings with few training instances.
1 Introduction
The introduction motivates capsule networks as a possible route to generalizing across varied sentence forms, while noting that routing reliability and computational cost hinder their use in NLP. It presents instance-level routing evaluation, an adaptive optimizer, and scalability techniques, and reports results over strong baselines on two NLP tasks and in low-resource settings.
1 Introduction
- Capsule networks have generalized across viewpoints in image classification, motivating the possibility of abstracting over different surface realizations in NLP.The paper suggests extrapolation from examples with few viewpoints as a basis for this possibility.Source: §1 · p. 1
- Existing routing algorithms use the same iteration count for every example, even though system-level training loss can improve while an individual example remains unconverged.The paper identifies this mismatch as a reliability concern.Source: §1 · p. 2
- Capsule networks can be difficult to train because large capsule counts and output spaces make routing computationally demanding.The introduction frames scalability and routing reliability as obstacles to mature NLP applications.Source: §1 · p. 2
- The framework uses a negative-agreement objective to evaluate routing per instance and an adaptive optimizer to adjust iterations for each example.The optimizer is intended to improve instance-level convergence and routing reliability.Source: §1 · p. 2
- Capsule compression and partial routing are proposed to improve scalability when datasets have large output spaces.These techniques target the computational demands of capsule networks.Source: §1 · p. 2
- The framework outperforms strong baselines on multi-label text classification and question answering, with superior generalization in low-resource settings.The introduction reports these results without giving task-specific metrics here.Source: §1 · p. 2
2 NLP-Capsule Framework
NLP-Capsule is a unified framework designed to scale capsule networks to large output spaces and improve routing reliability at the instance level. It combines capsule compression and partial routing with KDE-based routing and an adaptive optimizer.
2.1 Convolutional Layer
- The convolutional layer extracts document features by sliding a filter over document embeddings and concatenating feature maps from filters with window sizes 2, 4, and 8.The resulting maps are passed to the primary capsule layer.Source: §2.1 · p. 2
2.2 Primary Capsule Layer
- Group convolution transforms each feature-map scalar into a d-dimensional primary capsule using d filters, with a squashing function constraining capsule length to [0, 1].Capsule length indicates the probability that the capsule is useful for the task.Source: §2.2 · pp. 2–3
- Capsule Compression condenses the large number of document-dependent primary capsules into fewer weighted-sum capsules, merging similar capsules and removing outliers.The weights are learned by supervision.Source: §2.2 · p. 3
2.3 Aggregation Layer
- Routing builds candidate vectors for each high-level capsule from condensed capsules, then forms each output capsule with iteratively updated coupling weights.The coupling coefficients are updated by a clustering-like method.Source: §2.3 · p. 3
- The negative agreement score evaluates routing at instance level using KDE without parametric distribution assumptions, and Mean Shift minimizes this objective.An adaptive optimizer adjusts iteration counts for individual examples using their negative agreement scores; leaky-softmax decreases noisy capsules' strength.Source: §2.3 · pp. 3–4
2.4 Representation Layer
- Partial routing restricts each example to real and randomly selected negative outputs, both far smaller than the full output space, reducing routing cost for large label sets.The top-level layer has one final capsule per output, making computation expensive as the output space grows.Source: §2.4 · pp. 4–5
3 Experiments
Experiments evaluate scalability and low-resource generalization on multi-label classification and question answering. The approach performs strongly on both tasks, with reported advantages under reduced training data and competitive computational efficiency.
3 Experiments
- The experiments target large-label-scale classification and data-imbalanced QA; EUR-Lex has 3,956 labels and 15.59 examples per label, while RCV1 has 103 labels.The authors leave datasets with extremely large label sets, such as Wikipedia-500K, for future study.Source: §3 · p. 5, §3.1 · p. 5
3.1 Multi-label Text Classification
- With capsule compression and partial routing, the model uses 14.06M parameters and takes 0.25 seconds per batch, versus XML-CNN’s 22.52M parameters and 0.08 seconds.Without acceleration, the capsule approach takes 1.7 seconds per batch.Source: §3.1 · p. 5
3.1 Multi-label Text Classification; 3.2 Question Answering
- NLP-Cap shows a noticeable margin over strong baselines on EUR-Lex and competitive results on RCV1, and achieves the best MAP performance on TREC QA.The QA results also exceed CNN, LSTM, and NTN-LSTM by a noticeable margin.Source: §3.1 · p. 5, Table 2 · p. 6, Table 6 · p. 7
3.1 Multi-label Text Classification
- About 5% improvement over XML-CNN on 4 of 6 metrics is reported for NLP-Cap trained with 70% of examples versus XML-CNN trained with 100%.The study varies EUR-Lex training fractions from 50% to 100%.Source: §3.1 · p. 6, Table 3 · p. 6
- On the EUR-Lex dataset, the authors report that their routing process performs best on all metrics among the routing processes considered, as reported in Table 4.They speculate that the improvement comes from enhanced reliability of routing processes at instance level.Source: Table 4 · p. 6, §3.1 · p. 6
3.2 Question Answering
- On TREC QA, the approach uses 17.84M parameters and takes 0.06 seconds per accelerated batch, versus HD-LSTM’s 34.51M parameters and 0.03 seconds.Without acceleration, it takes 0.12 seconds per batch.Source: §3.2 · p. 7
4 Related Work
Related work surveys approaches to large-scale multi-label classification and question answering, then situates this paper among capsule-network research. The paper distinguishes its focus on instance-level routing reliability and scalability to large output spaces.
4.1 Multi-label Text Classification
- Tree-based classifiers partition feature or instance spaces, while label-embedding methods compress labels into a lower-dimensional representation before decompression.FastXML optimizes an NDCG-based objective at tree nodes; SLEEC is an example of label embedding.Source: §4.1 · p. 7
- Other multi-label approaches include efficient FastText, CNN variants such as Bow-CNN and XML-CNN, and the PD-Sparse linear classifier.FastText uses averaged word embeddings and ignores word order; PD-Sparse addresses data sparsity.Source: §4.1 · pp. 7–8
4.2 Question and Answering
- IR-based QA retrieves relevant passages before answer extraction, whereas knowledge-based QA matches logical forms against structured knowledge bases.Neural attention and question-answer interaction models are also prominent in the surveyed approaches.Source: §4.2 · p. 8
4.3 Capsule Networks
- Earlier capsule work introduced vector-output capsules and routing-by-agreement (Sabour et al., 2017), followed by EM routing and NLP applications.This paper focuses on instance-level routing evaluation and reliability, plus capsule compression and partial routing for large output spaces.Source: §4.3 · p. 8
5 Conclusion
The conclusion frames the work around generalization from few data points and reports an extended capsule framework for NLP. Experiments demonstrate effectiveness on multi-label text classification and question answering.
5 Conclusion
- The work addresses generalization from few data points as one aspect of making computers perform more like humans.The conclusion distinguishes this aim from matching human performance levels and robustness to adversarial examples.Source: §5 · p. 8
- The extended capsule framework is presented as offering scalability, reliability, and generalizability, with demonstrated effectiveness on multi-label text classification and question answering.The authors hope the modifications make capsule networks more suitable for large-scale problems and plan to study language modeling and text generation.Source: §5 · p. 8