Source-linked AI summary
Capsule Network Performance on Complex Data
Edgar Xi, Selina Bing, Yang Jin
TL;DR
CNNs do not model spatial hierarchies or rotational invariance well, motivating capsule networks for more spatially aware recognition. This study tests capsule-network configurations on CIFAR-10 and reports 71.550% validation accuracy with a 4-model ensemble and two convolution layers, below the original paper’s CIFAR-10 performance.
Problem
CNNs have limited handling of spatial hierarchies and rotational invariance, motivating evaluation of capsule networks on more complex data than MNIST.
Method
The study adapts Hinton’s MNIST capsule model to CIFAR-10 and compares architectural, ensemble, reconstruction-loss, activation, and category modifications.
Results
71.550% validation accuracy was achieved after 50 epochs with a 4-model ensemble and 2 convolution layers, while several other modifications underperformed the baseline.
Takeaways & Limitations
Adding convolution and ensemble averaging were the most promising tested modifications, whereas the study’s best result remained below the original paper’s CIFAR-10 performance.
Takeaways & Limitations
Computational constraints prevented testing the combination of a 7-model ensemble and a 64-capsule network, and comparisons were made at 50 epochs rather than absolute convergence.
Abstract
from arXiv · showhide
In recent years, convolutional neural networks (CNN) have played an important role in the field of deep learning. Variants of CNN's have proven to be very successful in classification tasks across different domains. However, there are two big drawbacks to CNN's: their failure to take into account of important spatial hierarchies between features, and their lack of rotational invariance. As long as certain key features of an object are present in the test data, CNN's classify the test data as the object, disregarding features' relative spatial orientation to each other. This causes false positives. The lack of rotational invariance in CNN's would cause the network to incorrectly assign the object another label, causing false negatives. To address this concern, Hinton et al. propose a novel type of neural network using the concept of capsules in a recent paper. With the use of dynamic routing and reconstruction regularization, the capsule network model would be both rotation invariant and spatially aware. The capsule network has shown its potential by achieving a state-of-the-art result of 0.25% test error on MNIST without data augmentation such as rotation and scaling, better than the previous baseline of 0.39%. To further test out the application of capsule networks on data with higher dimensionality, we attempt to find the best set of configurations that yield the optimal test error on CIFAR10 dataset.
1 Introduction
The study tests capsule networks on the higher-dimensional CIFAR-10 dataset after their strong MNIST result without rotation or scaling augmentation.
- 0.25% MNIST test error was achieved without rotation or scaling augmentation, improving on the previous 0.39% baseline.
- The study searches for capsule-network configurations that minimize test error on CIFAR-10.
2 Overview of Capsule Networks
Capsule networks use vector-valued capsules and dynamic routing to represent feature relationships, while reconstruction regularization encourages more general image representations.
- Capsule networks distinguish themselves from CNNs through layer-based squashing and dynamic routing.Capsule outputs are squashed as entire vectors rather than individual neurons.
- Dynamic routing adjusts connection strength toward potential parent capsules based on routing by agreement.This routing is presented as more effective for adding invariance than max-pooling.
- Reconstruction autoencoding regularizes capsule networks by reconstructing the input from the correct digit’s activity vector during training.The reconstruction objective encourages a more general image representation.
- Capsules were introduced to address CNN representational limitations using transformation matrices that model part-whole relationships and novel viewpoints.
- The classification ability of capsule networks on datasets more complex than MNIST remained unestablished in the cited overview.
3 Methodology
The study begins with a color-channel adaptation of Hinton’s MNIST model and evaluates architectural, ensemble, reconstruction-loss, activation, and classification-category modifications for CIFAR-10.
- The baseline is Hinton’s MNIST model adapted to three color channels, with experiments varying capsule layers and other parameters.
- 3 Methodology: Additional capsule layers are tested to represent increasingly intricate feature relationships in higher-dimensional CIFAR-10 data.
- 3 Methodology: Increasing primary-capsule types is intended to learn richer features as data dimensionality increases.
- 3 Methodology: Ensemble averaging trains multiple networks and averages their test predictions to capture distinct features.The cited capsule-network work used an ensemble of seven models with 10.6% test error.
- 3 Methodology: The experiments vary reconstruction-loss scaling to study its influence on overfitting and convergence rate.
- 3 Methodology: The study adds convolutional layers before capsules to create a more complex image encoding for CIFAR-10.
- 3 Methodology: A customized activation function replaces squash, with the hypothesis that sensitivity to small input changes may improve class separation.
- 3 Methodology: An eleventh “none of the above” category is tested because capsule networks may explain everything in the image.
4 Datasets
The evaluation uses CIFAR-10, a labeled 32x32 color-image dataset with ten mutually exclusive classes, to test capsule networks on more complex data than MNIST.
- CIFAR-10 contains 32x32 colored images from ten classes, with 6,000 images per class.The classes are airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.
- The dataset is divided into 50,000 training images and 10,000 validation images.
- CIFAR-10 is selected because it is established, supports many classification reports, and is sufficiently complex for higher-dimensionality testing.
- The cited capsule-network CIFAR-10 result used three routing iterations, 24 × 24 image patches, and 64 primary-capsule types.The passage also reports a 4.50% state-of-the-art validation error using fractional max-pooling within a convolutional architecture.
5 Results
The reported results compare representative capsule-network configurations trained for 50 epochs, including ensemble and architectural modifications. Reconstructions show clear structure for MNIST but blurry, indistinct features for CIFAR10.
- Representative models were selected after experimenting with combinations of the methods, including an ensemble with a “none of the above category” optimization.
- 50 epochs was used for training because of resource limitations and validation-accuracy plateauing around that point.
- Most modifications underperformed the baseline, although adding a convolution layer and ensemble averaging were successful.
- MNIST reconstructions exhibit clear structure and distinct features, whereas CIFAR10 reconstructions are blurry and lack class-specific features.
6 Discussion and Analysis
Experiments improved CIFAR10 validation accuracy over the baseline through convolution and ensembling, but the best tested configuration remained below the original result. The discussion attributes this gap partly to reconstruction regularization and points to pose matrices as a future direction for viewpoint invariance.
- Model Comparisons: 1.85% validation accuracy improvement came from a 4-model ensemble, while adding a convolution layer improved validation accuracy by 0.41% over baseline.
- Summary: Computational constraints required comparing models at epoch 50 rather than training them until absolute convergence.
- Summary: 71.550% validation accuracy was achieved after 50 epochs with a 4-model ensemble and 2 convolution layers, improving 2.57% over the baseline MNIST model.
- Summary: The tested results fell short of the original CIFAR-10 performance using a 70-model ensemble and 64 capsules.
- Reconstruction Loss: The performance gap between MNIST and CIFAR10 may reflect a 2D reconstruction regularizer applied to CIFAR10 objects with multiple viewpoints.
- Future Work: Matrix capsules using EM routing introduce a 4 x 4 pose matrix to represent object-viewer relations and support viewpoint invariance as a future direction.