Source-linked AI summary
Dynamic Routing Between Capsules
Sara Sabour, Nicholas Frosst, Geoffrey E Hinton
TL;DR
Convolutional networks struggle to generalize to novel viewpoints and recognize highly overlapping objects. This paper introduces capsules with dynamic routing-by-agreement, achieving state-of-the-art MNIST performance and a 5.0% error rate on highly overlapping digit pairs.
Problem
The paper asks how visual systems can generalize to novel viewpoints and recognize complex or overlapping entities without replicating detectors or greatly expanding labeled data.
Method
The system represents entities as activity vectors and uses transformation-based predictions with iterative routing-by-agreement to connect lower-level capsules to appropriate parents.
Results
5.0% classification error on highly overlapping digit pairs, matching a sequential attention model on a much easier task and exceeding the baseline convolutional model.
Takeaways & Limitations
The results provide an early indication that capsules are a direction worth exploring for recognizing and segmenting overlapping digits.
Takeaways & Limitations
Capsules perform worse when image clutter is too varied for a reasonably sized network to model, as in CIFAR-10.
Abstract
from arXiv · showhide
A capsule is a group of neurons whose activity vector represents the instantiation parameters of a specific type of entity such as an object or an object part. We use the length of the activity vector to represent the probability that the entity exists and its orientation to represent the instantiation parameters. Active capsules at one level make predictions, via transformation matrices, for the instantiation parameters of higher-level capsules. When multiple predictions agree, a higher level capsule becomes active. We show that a discrimininatively trained, multi-layer capsule system achieves state-of-the-art performance on MNIST and is considerably better than a convolutional net at recognizing highly overlapping digits. To achieve these results we use an iterative routing-by-agreement mechanism: A lower-level capsule prefers to send its output to higher level capsules whose activity vectors have a big scalar product with the prediction coming from the lower-level capsule.
1 Introduction
The introduction models visual parsing with capsules in a fixed multilayer network, where capsule activity vectors encode entities and their properties. Dynamic routing sends capsule outputs toward parent capsules whose transformed predictions agree.
- 1 Introduction: A single fixation is assumed to provide more than one identified object and its properties, motivating structured visual parsing.The paper frames human vision as selectively processing a tiny fraction of the optic array at highest resolution.
- 1 Introduction: A parse tree is carved from a fixed multilayer neural network whose layers contain capsules corresponding to active tree nodes.The construction uses iterative routing rather than dynamically allocating memory during parsing.
- 1 Introduction: Capsule neuron activities represent entity properties including pose, deformation, velocity, albedo, hue, and texture, along with existence.Existence is identified as a special property of the instantiated entity.
- 1 Introduction: Dynamic routing initially distributes each capsule output across possible parents using coupling coefficients that sum to 1.Each capsule forms a prediction vector for each parent by multiplying its output by a weight matrix.
- 1 Introduction: A capsule’s vector output enables routing to an appropriate parent when its transformed prediction has a large scalar product with that parent’s output.This mechanism replaces scalar-output feature detectors and max-pooling while retaining spatial replication of learned knowledge.
2 How the vector inputs and outputs of a capsule are computed
The section defines capsule outputs by transforming lower-level vectors into predictions, combining them with dynamically routed coupling coefficients, and applying a squashing function. Routing iteratively increases coupling to higher-level capsules whose outputs agree with the predictions.
- Vector inputs and outputs: The total input s_j is converted into output v_j by a squashing function that shrinks short vectors toward zero and long vectors to lengths slightly below 1.This makes vector length represent the probability that the capsule’s entity is present.
- Vector inputs and outputs: A capsule’s output vector is computed from lower-level prediction vectors formed by multiplying each lower capsule output u_i by a transformation matrix W_ij.For higher-than-first layers, the total input s_j is a weighted sum over prediction vectors.
- Dynamic routing: The coupling coefficients c_ij determine how strongly capsule i contributes to each higher-level capsule and are determined by dynamic routing.The coefficients linking each lower capsule to higher-level capsules sum to 1 under a routing softmax.
- Dynamic routing: Routing begins with learned, input-independent log priors based on capsule location and type, then iteratively refines couplings using prediction–output agreement.The agreement is computed as the scalar product a_ij = v_j.û_j|i and added to b_ij before recomputing coupling coefficients.
- Dynamic routing: The routing procedure alternates softmax coupling, weighted-input summation, squashing, and logit updates for a specified number of iterations.Each logit update adds the scalar product between a prediction vector and the current higher-level output.
3 Margin loss for digit existence
The model represents digit existence with capsule-vector length and uses a separate margin loss for each digit class, summed across all digit capsules.
- Digit existence representation: Capsule-vector length represents the probability that a digit entity exists, with the class-k capsule long if and only if digit k is present.A separate loss L_k is used for each digit capsule to support multiple digits.
- Margin loss: The margin loss uses T_k = 1 exactly when a digit of class k is present, with positive and negative margins m+ = 0.9 and m− = 0.1.The loss penalizes short vectors for present digits and long vectors for absent digits.
- Margin loss: λ = 0.5 down-weights loss for absent digit classes, preventing early learning from shrinking all digit-capsule activity-vector lengths.The total loss is the sum of the losses for all digit capsules.
4 CapsNet architecture
The CapsNet is a shallow three-layer architecture that transforms convolutional features into primary capsules and then class-specific digit capsules. Routing occurs only between capsule layers, while a reconstruction decoder regularizes digit-capsule representations.
- Architecture: The three-layer network comprises two convolutional layers followed by one fully connected capsule layer.Conv1 uses 256 9 × 9 kernels with stride 1 and ReLU activation.
- PrimaryCapsules: PrimaryCapsules contains 32 channels of 8D convolutional capsules, producing 32 × 6 × 6 capsule outputs.Each capsule uses a 9 × 9 kernel with stride 2, and capsules share weights across the 6 × 6 grid.
- DigitCaps: DigitCaps contains one 16D capsule per digit class, with each capsule receiving input from all capsules in PrimaryCapsules.The activity-vector length indicates the presence of each digit class and contributes to classification loss.
- Routing: Routing is applied only between consecutive capsule layers because Conv1 outputs are one-dimensional and lack an orientation space for agreement.Routing logits are initialized to zero, so each primary-capsule output initially reaches all ten digit capsules with equal probability.
- Reconstruction: An auxiliary reconstruction loss trains the correct digit capsule’s activity vector to reconstruct the input image through a three-layer fully connected decoder.Training masks all other digit capsules and minimizes the sum of squared differences between reconstructed and input pixels.
5 Capsules on MNIST
On shifted MNIST, a single three-layer CapsNet achieves 0.25% test error, while routing and reconstruction regularization are important to performance. Digit capsules learn class-specific representations of handwriting variation and show moderate robustness to small affine transformations.
- Performance: 0.25% test error is achieved by a single three-layer CapsNet on shifted MNIST without model averaging.Training uses 60K images and testing uses 10K images, with shifts of up to 2 pixels and no other augmentation.
- Performance: Routing and the reconstruction regularizer are important components of the tested CapsNet setups.The supplied passage states that Table 1 shows their importance and that adding the reconstruction regularizer boosts performance, but its sentence is truncated.
- Capsule representations: Each DigitCaps capsule learns a robust representation for its digit class, spanning variations such as stroke thickness, skew, width, and digit-specific structure.Decoder reconstructions expose dimensions associated with localized features, including the ascender length of a 6 and the size of its loop.
- Robustness: CapsNet is moderately robust to small affine transformations because handwritten digits naturally vary in skew, rotation, and style.Robustness was evaluated after training on padded, translated MNIST digits placed on 40 × 40 black backgrounds and testing on affNIST4.
6 Segmenting highly overlapping digits
Dynamic routing enables capsules to attend selectively to active lower-level capsules, supporting segmentation and recognition of highly overlapping digit pairs. On MultiMNIST, CapsNet outperforms the convolutional baseline and achieves a 5.0% classification error while reconstructing the two original digits.
- Routing mechanism: Dynamic routing acts as parallel attention, allowing higher-level capsules to attend selectively to active lower-level capsules and ignore others.This mechanism is intended to help recognize multiple overlapping objects.
- Dataset: MultiMNIST overlays two differently labeled digits, shifts each by up to 4 pixels, and produces 36×36 images with approximately 80% average bounding-box overlap.The dataset contains 60M training examples and 10M test examples.
- Results: 5.0% classification error: the 3-layer CapsNet matches the sequential attention model’s error on a much easier task with less than 4% overlap.CapsNet also achieves higher test classification accuracy than the convolutional baseline.
- Segmentation and reconstruction: CapsNet segments overlapping images into the two original digits despite pixels shared by both digits, encoding each digit’s position and style in DigitCaps.The decoder reconstructs each digit from its capsule encoding.
- Segmentation procedure: The two most active DigitCaps capsules are decoded separately, and pixels with non-zero intensity are assigned to each digit to obtain segmentation results.This produces one segmentation image for each decoded digit.
7 Other datasets
The capsule model achieved competitive results on smallNORB and SVHN, but CIFAR-10 performance was poorer because varied backgrounds were difficult to model. CIFAR-10 reached 10.6% error using an ensemble and routing adjustments.
- CIFAR-10: 10.6% error was achieved on CIFAR-10 using an ensemble of seven models trained with three routing iterations on 24 × 24 patches.Each model used three color channels, 64 primary-capsule types, and a “none-of-the-above” routing category.
- CIFAR-10: CIFAR-10 performance was poorer because its varied backgrounds were too difficult for a reasonably sized capsule network to model.Capsules tend to account for everything in an image rather than rely on an orphan category.
- smallNORB: 2.7% test error was achieved on smallNORB, on par with the state of the art, using the same architecture as for MNIST.The 96 × 96 stereo grayscale images were resized to 48 × 48; training used random 32 × 32 crops and testing used the central crop.
- SVHN: 4.3% test-set error was achieved on SVHN using a smaller network trained on only 73,257 images.The network used 64 first-layer channels, 16 6D primary capsules, and an 8D final capsule layer.
8 Discussion and previous work
The discussion presents capsules as a representation designed to avoid convolutional networks’ viewpoint-generalization inefficiencies by encoding instantiation parameters and routing predictions between parts and wholes. It argues that capsules offer advantages for multiple affine transformations and segmentation, while remaining an early-stage approach whose initial overlapping-digit results are promising but not yet broadly dominant.
- Viewpoint generalization: Capsules address convolutional networks’ difficulty generalizing to novel viewpoints by converting recognized fragments into instantiation-parameter vectors and transforming them to predict larger fragments.Learned transformation matrices encode intrinsic part–whole spatial relationships, yielding viewpoint-invariant knowledge that generalizes to novel viewpoints.
- Representational assumption: Capsules assume at most one instance of each represented entity type at each image location, allowing a distributed activity vector to encode its instantiation parameters.The assumption is motivated by visual crowding and is described as eliminating the binding problem.
- Multiple transformations: Unlike normalization methods such as spatial transformer networks, capsules let neural activities vary with viewpoint and can handle multiple affine transformations of different objects or parts simultaneously.The comparison specifically identifies spatial transformer networks as a normalization-based alternative.
- Segmentation: Capsules support segmentation through instantiation-parameter vectors and routing-by-agreement, with dynamic routing also supported by biologically plausible models of invariant visual pattern recognition.The passage identifies segmentation as a difficult vision problem and links dynamic routing to visual-cortex models.
- Research status: Capsule research remains at an early stage, but a simple system already achieves unparalleled performance at segmenting overlapping digits, providing an initial indication of the approach’s promise.The authors compare its development stage with recurrent neural networks for speech recognition at the beginning of this century and say further insights are probably needed to outperform mature technology.
A How many routing iterations to use?
Routing logits change negligibly by the fifth iteration, while additional iterations increase capacity and tendency to overfit. Based on these observations, the authors recommend three routing iterations for all experiments.
- A How many routing iterations to use?: By routing iteration 5, the logits change by only 1e −5 on average, indicating negligible change from the start of training.The average change in the second routing pass settles at 0.007 after 500 training epochs.
- A How many routing iterations to use?: More routing iterations increase network capacity and tend to overfit the training dataset.This comparison was made between one and three routing iterations on CIFAR10.
- A How many routing iterations to use?: The authors suggest three routing iterations for all experiments.On CIFAR10, three iterations optimize the training loss faster and converge to a lower final loss than one iteration.