Sparse Unsupervised Capsules Generalize Better

1. Introduction

• Dynamic routing forms a consensus mechanism across capsule layers, using feedback so capsules agree on entities’ pose or instantiation parameters.It is conceptually related to message passing, but is not numerically equivalent to belief propagation.Source: §1 · p. 1

1.1. Equivariances

• Capsule outputs are vectors whose continuously changing components represent transformations of inputs, making them equivariant rather than merely invariant.Learned specialization allows capsules to parameterize properties such as stroke width and digit skew.Source: §1.1 · p. 1
• Sabour et al. (2017) reported 79% classification accuracy when training on MNIST and testing on affNIST, motivating equivariance as a potentially generalizable representation.The result was attributed to capsules describing affine-transformed digits through their equivariant representations.Source: §1.1 · p. 1

1.2. Unsupervised Learning

• Prior capsule systems combine unsupervised routing with supervised classification and masked reconstruction losses that select a single winning capsule to represent the input.The reconstruction decoder is trained through the capsule layers, while the classification loss is applied at the deepest capsule layer.Source: §1.2 · p. 2
• This paper creates a fully unsupervised capsule network because labelled training data can be costly and difficult to produce at scale.The motivation also reflects the expectation that unsupervised training may produce more regularized models that generalize better, although not universally.Source: §1.2 · p. 2
• The broader motivation includes understanding and emulating natural neural networks without relying on biological equivalents of gradient-based credit assignment.Supervised discriminatory optimization can improve task-specific performance but has also been associated with susceptibility to adversarial attacks.Source: §1.2 · p. 2

2.1. Dense Unsupervised Capsules

• Dense unsupervised training produces good reconstructions but turns the capsule layer into an autoencoder in which all capsules contribute broadly.Removing SUPCAPS’ mask and margin loss causes capsules to lose selective responses, equivariances, and functional routing consensus.Source: §2.1 · p. 2, Figure 2 · p. 3

2.2. Sparse Unsupervised Capsules

• Sparse training restores selective routing, allowing dynamic routing to activate relevant capsules rather than assigning similar coefficients to all capsules.The resulting specialization enables capsules to model subsets of the input space instead of sharing every reconstruction.Source: Figure 3 · p. 3, §2.2 · p. 3
• Sparse training selects latent capsules by ranking their routing support and applying a mask to capsule outputs.Routing coefficients are aggregated across convolutional locations, and the mask is applied elementwise to the latent-capsule output.Source: §2.2 · pp. 3–4, §2.3.1 · p. 4
• Using L = 16 latent capsules and γ = 12 leaves approximately one capsule fully active and two partially active for each input.Mask values below 0.01 are forced to zero for numerical stability.Source: §2.2 · p. 4

2.3. Online Lifetime Sparsity Constraint

• Online lifetime sparsity boosting keeps capsule rank-zero frequencies around the target range of 0.04 to 0.1.Capsule utilization is tracked across batches, and boost values are updated every N = 50 batches when frequencies fall outside the specified bounds.Source: §2.3 · p. 4, §2.3.2 · p. 5

2.3.1. PER-IMAGE

• The sparse-mask and boosting operations affect only reconstruction output during capsule training, not the later classification procedure.The masked latent output v′ replaces v for classification and reconstruction, while boost statistics are updated after batches.Source: §2.3 · p. 4, §2.3.1 · p. 4, §2.3.2 · p. 5

3.1. Generalization to affNIST

• On the affNIST task, SPARSECAPS achieved 90% accuracy, exceeding the 79% achieved by SUPCAPS, although notes qualify the SPARSECAPS result.A supervised SVM with a Radial Basis Function kernel was added to the output of the pre-trained SPARSECAPS representation.Source: §3.1 · p. 5, §3 · p. 5
• SPARSECAPS was evaluated with SVMs trained either on affNIST outputs or on MNIST outputs, testing distinct forms of representation generalization.Training on affNIST outputs tests whether the representation captures transformed digits; training only on MNIST outputs yields poor affNIST classification.Source: §3.1 · p. 5

3.2. affNIST Benchmark

• SPARSECAPS trained only on MNIST was similar or better than most reported convolutional-network results on affNIST.Comparisons vary in preprocessing and test conditions; Chang et al. used scale and translation variation during training rather than full affine transformation.Source: §3.2 · p. 6, Table 2 · p. 6

3.3. Equivariances in Sparse Latent Capsules

• Varying one parameter of the most active capsule produced continuous reconstructed changes, including skew, stroke width, loop opening, and digit-form mutations.These discovered equivariances differ from the predefined meanings of SUPCAPS digit capsules.Source: §3.3 · p. 6

3.4. Equivariances in Dense Latent Capsules

• Without sparsening, capsule parameters made scattered minor edits, illustrating a degenerate representation without learned equivariances.The dense unsupervised condition was trained on MNIST and failed to generalize to affNIST even when the classifier saw affNIST outputs.Source: §3.1 · pp. 5–6, §3.4 · p. 6

3.5. Other Metrics

• 99% MNIST classification accuracy was not significantly affected by replacing supervised with sparse unsupervised capsules, while sparse training restored selective activation in dynamic routing.Batch mean square error was 7.75 on MNIST and 18.49 on affNIST.Source: §3.5 · p. 6

• Without supervised masking, latent capsule layers lose desirable capsule characteristics and may cause deeper capsule networks to behave like convolutional autoencoders.The authors observed this failure using SUP-CAPS dynamic routing and expect activation to become more uniform with increasing depth without supervised sparsening.Source: §4 · p. 6
• Sparse unsupervised training restored capsule qualities and produced arguably superior generalization performance on affNIST, suggesting it could enable deeper capsule networks.The paper’s conclusions connect sparsity with both recovered capsule behavior and broader generalization.Source: §4 · p. 6
• The study plans to integrate SPARSECAPS with EM-routing and evaluate it on SmallNORB, which tests 3D object recognition under pose and illumination changes.These experiments are identified as future work rather than completed evaluations.Source: §4 · p. 7
• Robustness to adversarial examples was not tested, and the expected robustness of SPARSECAPS therefore requires empirical confirmation.The expectation is based on reported apparent robustness of EM-routing capsules, not on a direct SPARSECAPS experiment.Source: §4 · p. 7
• The lifetime capsule-frequency hyperparameters were the only parameters empirically optimized, with future work proposing capsule-count-dependent ranges or online adaptation.The current frequency bounds were derived from a uniform-frequency midpoint and empirical optimization.Source: §4 · p. 7

• Full source code for reproducing the results is available in the authors’ GitHub repository.The repository is identified as sparse-unsupervised-capsules.Source: §A.1 · p. 9
• The repository is a fork of the SUPCAPS code from Sabour et al. (2017), whose original implementation is also linked.The source-code section distinguishes the authors’ fork from the upstream repository.Source: §A.1 · p. 9