Source-linked AI summary
Efficient-CapsNet: Capsule Network with Self-Attention Routing
Vittorio Mazzia, Francesco Salvetti, Marcello Chiaberge
TL;DR
CNNs use substantial feature-detector capacity and augmentation to represent object transformations, while capsule-network efficiency has received limited attention. The paper proposes a compact Efficient-CapsNet with non-iterative self-attention routing and reports state-of-the-art results across three datasets with about 160K parameters.
Problem
CNN transformation handling can require redundant feature detectors, while the efficiency and transformation-representation benefits of capsule networks have received limited attention.
Method
Efficient-CapsNet combines an extreme low-parameter capsule architecture with a non-iterative, highly parallelizable self-attention routing algorithm.
Results
Efficient-CapsNet achieves state-of-the-art results on three distinct datasets with barely 160K parameters and substantially fewer operations than previous implementations.
Takeaways & Limitations
The experiments support efficient capsule-based visual representations that retain capsule-network properties while reducing parameter and computational requirements.
Abstract
from arXiv · showhide
Deep convolutional neural networks, assisted by architectural design strategies, make extensive use of data augmentation techniques and layers with a high number of feature maps to embed object transformations. That is highly inefficient and for large datasets implies a massive redundancy of features detectors. Even though capsules networks are still in their infancy, they constitute a promising solution to extend current convolutional networks and endow artificial visual perception with a process to encode more efficiently all feature affine transformations. Indeed, a properly working capsule network should theoretically achieve higher results with a considerably lower number of parameters count due to intrinsic capability to generalize to novel viewpoints. Nevertheless, little attention has been given to this relevant aspect. In this paper, we investigate the efficiency of capsule networks and, pushing their capacity to the limits with an extreme architecture with barely 160K parameters, we prove that the proposed architecture is still able to achieve state-of-the-art results on three different datasets with only 2% of the original CapsNet parameters. Moreover, we replace dynamic routing with a novel non-iterative, highly parallelizable routing algorithm that can easily cope with a reduced number of capsules. Extensive experimentation with other capsule implementations has proved the effectiveness of our methodology and the capability of capsule networks to efficiently embed visual representations more prone to generalization.
1 Introduction
CNNs rely on augmentation and expanded feature-detector capacity to handle transformations, but this creates redundancy and can lose object localization. Efficient-CapsNet addresses capsule-network efficiency with a compact architecture and self-attention routing.
- CNNs replicate learned feature detectors across spatial locations, supporting translation handling but progressively losing object localization information.
- Data augmentation, increased feature-map counts, residual connections, and normalization only partially address CNN limitations and detector redundancy.These mechanisms help enlarge filter capacity but do not fully support recognizing unseen object transformations.
- Capsules encode different properties of the same entity in vector outputs, explicitly representing the entity’s pose and relationships between object parts and wholes.This repurposes learned weights toward relationships rather than only feature detectors.
- Efficient-CapsNet investigates capsule efficiency using an architecture with barely 160K parameters and reports state-of-the-art results on three datasets.
- The proposed routing algorithm is non-iterative, highly parallelizable, and uses self-attention to route a reduced number of capsules.It exploits similarities between low-level capsules to cluster and route them toward promising high-level capsules.
- The paper provides open-source training and testing code and evaluates the method on MNIST, smallNORB, and MultiMNIST.
2 Related Works
Related work introduced capsules to represent entity properties and pose, developed routing methods, and applied attention to capsule routing. Efficient-CapsNet differs by using self-attention as a substitute for dynamic routing.
- Capsules represent multiple properties of an entity, while CapsNet incorporated capsules into a CNN and introduced routing-by-agreement through dynamic routing.
- Prior capsule research explored modified activations, optimization-based dynamic routing, group capsules, and Expectation-Maximization routing.
- Attention mechanisms dynamically emphasize relevant features, and prior work applied attention to capsule routing with feed-forward, non-iterative operations.
- The architecture uses depthwise separable convolution for primary capsules and an initial convolutional stack that maps inputs into a higher-dimensional space.
- Efficient-CapsNet uses self-attention as a substitute routing algorithm between capsules rather than merely adding attention to dynamic routing.
3 Methods
Efficient-CapsNet builds capsule representations from convolutional features, then routes lower-level predictions to higher-level capsules using self-attention and learned priors. Its capsule outputs encode entity presence through vector length and retain instantiation properties for reconstruction.
- Efficient-CapsNet architecture: The architecture extracts local image features, creates primary capsules with depthwise separable convolution, and routes them through a capsule hierarchy.The depthwise separable operation reduces the parameters required for capsule creation.
- Primary capsule creation: HConv maps the input image to a higher-dimensional space, while a linear depthwise separable convolution converts those features into the primary capsule layer.The convolution uses filters and kernel dimensions aligned with HConv output dimensions to obtain the primary capsules.
- Capsule activation: Capsule activation preserves vector orientation while squashing vector length between zero and one, allowing length to encode entity-existence probability.The activation shrinks short vectors toward zero and long vectors toward a length slightly below one; its variant is more sensitive to small changes near zero.
- Prediction generation: Each lower-level capsule applies learned weight matrices to predict properties of every capsule in the layer above, embedding affine transformations between adjacent layers.The higher-level capsule input is a weighted sum of these prediction vectors.
- Self-attention routing: Self-attention routing computes coupling coefficients from agreement scores, combines them with learned log priors, and assigns lower-level capsules to the higher-level entities they represent.Coupling coefficients from each lower-level capsule to the layer above sum to one, and the procedure remains unchanged for multiple capsule layers.
- Output and training objectives: The final capsule layer represents each class with a vector whose length indicates entity presence, while margin loss and reconstruction regularization train class scores and meaningful properties.The reconstruction decoder adds a scaled mean L2 loss between the input image and decoder output to the margin loss.
4 Results
Experiments evaluate Efficient-CapsNet’s efficiency, accuracy, reconstruction behavior, and affine-transformation encoding across MNIST, smallNORB, and MultiMNIST. The model uses a substantially smaller architecture while retaining competitive or state-of-the-art results and interpretable capsule representations.
- Experimental settings: The experiments assess generalization and efficiency against traditional CNNs and capsule-based methods on MNIST, smallNORB, and MultiMNIST.The study also examines how capsule representations encode transformations and reconstruct input information.
- Experimental settings: Efficient-CapsNet uses a much lower parameter count and computational cost than comparable methodologies.The comparison is intended to expose the efficiency and generalization capability of capsules relative to traditional CNNs.
- MNIST results: 161K parameters yield a MNIST mean accuracy of 0.9974, with minimum 0.9971 and maximum 0.9978.The model is comparable with capsule methodologies despite its small size; reconstruction regularization improves a vectorial-output network’s performance.
- MNIST results: Averaging Efficient-CapsNet predictions above 0.9973 produces a 0.16 final test error, compared with 0.26 for the mean single network.The ensemble reduces uncertainty in ambiguous digits by selecting the most probable answer across models.
- smallNORB results: Efficient-CapsNet reaches 0.974 mean accuracy on smallNORB, with values ranging from 0.97 to 0.983, using half the parameters of similar methodologies.An ensemble using two networks and 40-patch prediction obtains a test accuracy of 1.23 as reported in the passage.
- Affine transformations embedding: Perturbation tests show that Efficient-CapsNet encodes combinations of digit properties, while affine-transformation analysis finds higher linearity than original CapsNet.The analyses examine shape, position, orientation, translations, rotations, and cumulative variance in output capsule space.
5 Conclusion
Efficient-CapsNet demonstrates strong capsule generalization with very few parameters, achieving state-of-the-art results while using efficient self-attention routing for reduced capsule sets.
- 5 Conclusion: Efficient-CapsNet achieves state-of-the-art results on three distinct datasets with a very limited number of parameters.The paper reports considerably fewer needed operations than previous implementations.
- 5 Conclusion: The model highlights stronger knowledge representation and generalization capabilities than traditional CNNs after training.
- 5 Conclusion: The proposed routing algorithm uses self-attention to efficiently route a reduced number of capsules between subsequent layers without iteration.
- 5 Conclusion: Future work will use a synthetic dataset to study viewpoint generalization and internal feature representations in greater depth.
Author contributions statement
The author contributions statement assigns responsibilities across conceptualization, methodology, software, validation, analysis, investigation, writing, visualization, supervision, administration, and funding.
- Author contributions statement: V.M. and F.S. share most research, implementation, analysis, writing, visualization, and supervision responsibilities.
- Author contributions statement: M.C. contributes resources, project administration, and funding acquisition alongside V.M. and F.S.