Source-linked AI summary
DeepCaps: Going Deeper with Capsule Networks
Jathushan Rajasegaran, Vinoj Jayasundara, Sandaru Jayasekara, Hirunima Jayasekara, Suranga Seneviratne, Ranga Rodrigo
TL;DR
Capsule networks had not fully matched CNN performance on complex benchmark datasets, motivating a deeper capsule architecture. DeepCaps introduces 3D-convolution-based dynamic routing and a class-independent decoder, surpassing capsule-network state of the art on several datasets while reducing parameters and improving physical-attribute control.
Problem
Capsule networks showed limited performance on complex image datasets, while existing dynamic routing was restricted to computationally inefficient fully connected architectures.
Method
DeepCaps combines a deep capsule architecture with 3D-convolution-based localized dynamic routing, skip connections, and a class-independent decoder.
Results
DeepCaps surpasses capsule-network state-of-the-art results on CIFAR10, SVHN, and Fashion-MNIST, including a 3% CIFAR10 accuracy improvement over with 68% fewer parameters.
Takeaways & Limitations
The class-independent decoder provides consistent cross-class control over physical attributes represented by instantiation parameters and supports data-generation applications.
Takeaways & Limitations
Existing decoders lack guaranteed cross-class correspondence between instantiation parameters and physical properties, making controlled generation of attributes such as thickness or skewness difficult.
Abstract
from arXiv · showhide
Capsule Network is a promising concept in deep learning, yet its true potential is not fully realized thus far, providing sub-par performance on several key benchmark datasets with complex data. Drawing intuition from the success achieved by Convolutional Neural Networks (CNNs) by going deeper, we introduce DeepCaps1, a deep capsule network architecture which uses a novel 3D convolution based dynamic routing algorithm. With DeepCaps, we surpass the state-of-the-art results in the capsule network domain on CIFAR10, SVHN and Fashion MNIST, while achieving a 68% reduction in the number of parameters. Further, we propose a class-independent decoder network, which strengthens the use of reconstruction loss as a regularization term. This leads to an interesting property of the decoder, which allows us to identify and control the physical attributes of the images represented by the instantiation parameters.
1. Introduction
DeepCaps addresses the limitations of shallow and fully connected capsule networks by going deeper with 3D-convolution-based routing, skip connections, and a class-independent decoder. The architecture targets complex image datasets while reducing routing costs and improving control over represented physical attributes.
- Motivation: CapsNet performs well on MNIST but is not competitive with CNNs on complex datasets such as CIFAR10.The limitation is attributed to the greater complexity of CIFAR10 objects compared with MNIST.
- Motivation: Stacking fully connected capsule layers creates computationally expensive routing, weak middle-layer learning, and no localized routing mechanism.Small coupling coefficients can dampen gradient flow, while fully connected capsules cannot exploit localized correlations.
- Decoder: The class-independent decoder strengthens reconstruction-based regularization and makes physical attributes consistently controllable across classes.A given instantiation parameter is guaranteed to represent the same physical property for every class.
- DeepCaps Architecture: DeepCaps combines deeper capsule layers with 3D-convolution-based dynamic routing and skip connections to support convolutional capsule processing.The routing design provides parameter sharing and localized routing, while skip connections aid training of deeper networks.
- Results: DeepCaps significantly outperforms existing state-of-the-art capsule architectures while using fewer parameters.For CIFAR10, it reports a 3% accuracy improvement over and a 68% reduction in parameters.
2. Related Work
Related work establishes dynamic routing as a way to preserve capsule-specific relationships, while highlighting the computational and architectural barriers to deeper capsule networks. DeepCaps differs by using 3D routing that votes for capsules separately along the capsule dimension.
- Deep architectures: ResNets, Highway Networks, and stochastic depth address gradient attenuation in deep networks through identity paths or layer dropping.These approaches motivate architectural mechanisms for maintaining signal propagation across depth.
- Capsule routing: Dynamic routing was introduced to route capsules by agreement and provide equivariance, while EM routing extends the idea to pose matrices.The passage contrasts capsule routing with CNN pooling, which provides invariance.
- Related routing methods: SegCaps applies 2D convolution to capsule voting, mixing information across the capsule depth dimension.It achieves state-of-the-art segmentation results on LUNA16, but its voting operation differs from DeepCaps.
- Related routing methods: DeepCaps uses 3D-convolution-based routing with depth strides equal to the capsule dimension, so capsules along depth are voted separately.This design preserves separate capsule-specific transformations during voting.
- Instantiation parameters: Earlier capsule work showed that instantiation parameters can encode physical variations, but did not establish which parameter controls which change.This leaves controllable reconstruction and data generation unresolved.
3. DeepCaps
DeepCaps replaces fully connected capsule routing with localized 3D convolution-based routing and uses deeper capsule layers to handle complex visual data. Its architecture combines iterative routing, ConvCaps and FlatCaps layers, and fully connected capsules.
- Motivation: Fully connected dynamic routing is computationally expensive and difficult to extend deeply, motivating a convolutional routing framework.Stacking fully connected capsule layers can also produce poor learning in middle layers.
- 3D Convolution Based Dynamic Routing: The routing mechanism reshapes capsule tensors, applies 3D convolutions to produce intermediate votes, and reorganizes them into predictions for iterative routing.The reshaped votes have shape (w_l+1, w_l+1, n_l+1, c_l+1, c_l), with c_l+1 predicted capsule tensors for each input capsule tensor.
- 3D Convolution Based Dynamic Routing: 3D convolutional kernels transform groups of adjacent capsules into localized votes for higher-level capsules.A 3 × 3 × 8 kernel, for example, transforms nine adjacent capsules into one vote.
- 3D Convolution Based Dynamic Routing: Coupling coefficients are initialized through a 3D softmax, predictions are weighted and squashed, and later iterations update coefficients using agreement.The routing algorithm is empirically run for i = 3 iterations.
- DeepCaps Architecture: DeepCaps uses ConvCaps layers to preserve capsule outputs while going deeper, with one routing iteration in the earliest large feature maps to reduce computational cost.ConvCaps reshapes convolutional feature maps into capsule tensors and applies a squash function.
- DeepCaps Architecture: FlatCaps removes spatial relationships between adjacent ConvCaps capsules before fully connected capsules preserve part-whole relationships.A tensor shaped (w_l, w_l, c_l, n_l) is reshaped into a matrix with a_l = w_l × w_l × c_l rows.
4. Class Independent Decoder Network
The proposed class-independent decoder feeds only the predicted class's activity vector into a deconvolutional network, strengthening reconstruction regularization and aligning instantiation parameters across classes. This shared representation also exposes dataset variations through activity-vector variance.
- Decoder architecture: Deconvolutional layers reconstruct input data from DeepCaps instantiation parameters while capturing more spatial relationships than the prior fully connected decoder.The decoder uses binary cross entropy as its reconstruction loss.
- Existing decoder: The existing decoder masks activity vectors by the highest-probability class, indirectly encoding class information through dimension-specific positions.The vectorized masked matrix contains nonzero values only in the dimensions assigned to the selected class.
- Proposed decoder: The proposed decoder sends only the predicted class's activity vector into the decoder, rather than masking all other class vectors.During training, the selected vector corresponds to the true label; during inference, it corresponds to the class with the largest capsule norm.
- Class independence: Because each instantiation parameter is learned from the same joint distribution, its encoded entity is intended to remain consistent across image labels.This design constrains the activity vectors to a shared R^b space and supports class-independent decoding.
- Interpreting parameters: Rotation-related instantiation parameters have higher variance than parameters associated with localized character changes in MNIST.The variance reflects rotation and elongation as dominant variations, while localized changes are less dominant.
5. Experiments and Results
DeepCaps is evaluated across several benchmark datasets and compared with capsule-network baselines and state-of-the-art results. Experiments also examine its class-independent decoder, showing shared and controllable relationships between instantiation parameters and reconstructed image attributes.
- Benchmark setup: DeepCaps is evaluated on CIFAR10, SVHN, Fashion-MNIST, and MNIST, with CIFAR10 and SVHN images resized to 64×64×3.Other datasets retain their original image sizes.
- Classification results: DeepCaps surpasses existing capsule-network models on CIFAR10, SVHN, and Fashion-MNIST, while achieving on-par state-of-the-art results on MNIST.Reported improvements over comparison models are 3.25% on CIFAR10, 1.86% on SVHN, and 1.62% over HitNet on Fashion-MNIST.
- Classification results: 7.22 million parameters let DeepCaps achieve 91.01% on CIFAR10, compared with CapsNet’s 22.48 million parameters and 89.40% seven-ensemble accuracy.Inference takes 1.38 ms for DeepCaps on a 64×64×3 image versus 2.86 ms for CapsNet on a 32×32×3 image.
- Class-independent decoder: The class-independent decoder places corresponding attributes at shared instantiation-parameter locations across digit classes.Examples include boldness, rotation, and skewness; the paper reports that the 28th parameter controls vertical elongation and the 1st controls thickness.
- Class-independent decoder: Varying the 28th activity-vector dimension produces consistent vertical elongation across classes with the proposed decoder, unlike class-specific variations from the prior decoder.The perturbation range is [-0.075,0.075].
- Class-independent decoder: Higher-variance instantiation parameters show separable global changes, whereas lower-variance parameters produce mixed or localized changes.The parameter space is not orthogonal, so some parameters share an image attribute.
6. Conclusion
The paper presents DeepCaps as a deeper capsule-network architecture built with skip connections and 3D convolutions, together with a class-independent decoder. It reports stronger capsule-domain performance on complex datasets and controllable attribute representations in reconstruction.
- Conclusion: DeepCaps combines skip connections and 3D convolutions to support deeper capsule networks with reduced computational complexity.Skip connections improve gradient flow, while 3D convolutions provide localized dynamic routing.
- Conclusion: DeepCaps surpasses state-of-the-art capsule-network performance on CIFAR10, SVHN, and Fashion-MNIST, and achieves state-of-the-art performance on MNIST within the capsule-network domain.The conclusion frames these results as evidence of improved performance on comparatively complex datasets.
- Conclusion: The class-independent decoder regularizes DeepCaps and places a given image change at the same instantiation-parameter location across classes.The paper identifies potential applications in data generation with specific requirements.