Quaternion Equivariant Capsule Networks for 3D Point Clouds

1 Introduction

• Irregular point clouds and arbitrary, non-commutative SO(3) rotations make 3D embeddings require both transformation equivariance and point-permutation invariance.The introduction frames these as central challenges in extending neural networks to 3D point sets.Source: §1 · p. 1
• The network uses sparse local reference frames and quaternion dynamic routing to build latent representations that disentangle local orientation from object-existence activations.It preserves rotation equivariance, translation and permutation invariance, and supports classification and orientation estimation without rotation supervision or rotation augmentation.Source: §1 · p. 2
• The paper proposes an SO(3)-equivariant capsule module, connects its routing to generalized Weiszfeld iterations, and evaluates joint classification and orientation-estimation capabilities on ModelNet10 and ModelNet40.The authors argue for convergence of rotation estimation on votes.Source: §1 · p. 2

2 Related Work

• PointNet and its extensions process unordered point clouds, but common permutation-invariant set operators are equivariant only to permutations and translations, not other groups.The cited work also extends local receptive fields.Source: §2 · p. 3
• Equivariant-network approaches include finite-group convolutions, steerable filters, and harmonic-basis filters, with cited drawbacks including dense group coverage and reduced filter interpretability or expressiveness.The passage contrasts spatial filters with frequency-space bases that do not span the full space of spatial filters.Source: §2 · p. 3
• For 3D, voxel grids incur storage costs, while spherical-harmonic approaches require projection to the unit sphere and create additional problems for unstructured point clouds.Other work defines regular structures on the sphere for equivariant convolutions.Source: §2 · p. 3
• Existing rotation-equivariant representations act on input data or bottlenecks, whereas this work uses an explicit latent rotation parametrization alongside invariant features.Tensor Field Networks achieve localized equivariant filters over continuous groups, but the passage notes memory demands and lack of explicit orientations.Source: §2 · pp. 3–4
• Capsule-network research has connected routing by agreement to EM, KL-regularized clustering, and equivariance, and has applied capsules to graphs, point clouds, and medical images.The passage traces the recent line of work to dynamic routing by agreement.Source: §2 · p. 4

3.1 Equivariance

• Under the paper's stricter equivariance form, applying a group transformation to the input transforms the output by the same group element; stacking equivariant layers yields a globally equivariant network.The general definition permits different group actions on domain and codomain, provided they satisfy the stated homomorphism condition.Source: §3.1 · p. 4

3.2 The Quaternion Group H1

• Quaternions represent rotations as four-vectors, avoiding the singularities and unbounded redundancies of three-vector formulations, while their unit form parametrizes orientations on S3.The paper notes that quaternion means can be differentiated analytically or automatically, and that manifold means on the Lie algebra require iterative tangent-space updates.Source: §3.2 · pp. 4–5
• The quaternion matrix representation is an injective homomorphism into orthonormal matrices and linearizes Hamilton-product group composition.The representation has determinant one and preserves orthonormality.Source: §3.2 · p. 5

3.3 3D Point Clouds

• A point cloud is a discrete subset sampled from a 3D surface, and each local reference frame is an ordered orthonormal tangent basis whose first vector is the surface normal.The method assumes the second frame direction can be computed uniquely and repeatably for the point sets considered, relying on routing robustness otherwise.Source: §3.3 · p. 5

4.1 Equivariant Quaternion Mean

• The weighted quaternion mean is the maximum-eigenvalue eigenvector of M; its operator is left-equivariant and permutation-invariant, and the geodesic distance is preserved by rotations.Source: §4.1 · pp. 6–7

4.2 Equivariant Weiszfeld Dynamic Routing

• Routing iteratively clusters capsule votes, assigning each output capsule a weighted group mean with weights inversely proportional to vote-to-center distance.Input activations initialize the weights; the refined activations are final inlier scores.Source: §4.2 · p. 7, Figure 2 · p. 8
• For σ(x, y) = δ(x, y)^q−2, the routing procedure is a variant of affine-subspace Weiszfeld for robust Lq geometric-median computation.Source: §4.2 · p. 7
• Under mild assumptions, the sequence of inner-loop routing iterates almost surely converges to a critical point.The paper summarizes the procedure as robust IRLS rotation averaging on votes, routing information according to refined inlier scores.Source: §4.2 · p. 8

5.1 QEC Module

• The QEC module maps canonicalized local point sets through a shared continuous kernel to quaternion transformations and capsule votes, then routes votes into output orientations and activations.The module takes point coordinates, local reference-frame rotations, and activations as input.Source: §5.1 · pp. 9–10

5.2 Network Architecture

• The architecture stacks QEC modules so each layer reduces the number of points and increases the receptive field.The first layer uses FLARE-computed local reference frames as primary capsules.Source: §5.2 · p. 10
• In the two-level design, pooling centers connect local patches to intermediate capsules, while the final module pools capsules into one point with an estimated frame and class-specific outputs.The figure describes classification and, in the siamese case, orientation estimation as supervision cues for the transform kernels.Source: Figure 3 · p. 10, §5.2 · p. 11

Implementation

• LRFs use surface normals from local plane fits and FLARE second axes; training uses Adam at 0.001 and three routing iterations per layer.For ModelNet40, the network has 0.047M parameters and computational cost O(MKL).Source: §6 · pp. 11–12

Evaluation setup

• Classification uses ModelNet40, while pose estimation uses well-classified ModelNet10 shapes tested with five rotations and five resamplings.The pose-estimation protocol evaluates category-level generalization to shapes not seen during training.Source: §6 · pp. 11–13

Classification results

• In ModelNet40 NR/AR classification, the equivariant model outperforms all other methods, including equivariant spherical CNNs, by at least 5%.The comparison uses training on aligned shapes and testing under arbitrary rotations.Source: §6 · p. 12

Rotation estimation

• Both canonical and siamese pose-estimation modes outperform the baselines in relative angular error, with symmetric-object ambiguity motivating separate No Sym results.The corresponding pose capsule supplies the rotation estimate for the maximally activated class.Source: §6 · p. 13, Figure 4 · p. 13

Robustness against point and LRF resampling

• The point-density ablation reports robustness to changes in sampled LRFs and point density, while evaluating classification and pose estimation on well-classified non-symmetric objects.The ablation varies sampled patches from 10K LRFs and also computes LRFs from 2K or 1K points.Source: §6 · p. 14, Table 3 · p. 14

Conclusion

• The framework uses sparse LRF rotations and quaternion capsules to obtain permutation-invariant, SO(3)-equivariant representations, with orientation estimated without pose supervision.The paper connects Weiszfeld dynamic routing to convergence results for rotation estimation on capsule votes.Source: §7 · p. 14

Limitations

• Shape symmetries severely affect performance; LRF computation is sensitive to point-density changes, and LRFs may be ambiguous or non-unique.The activation-vector length depends on the number of classes, with a sufficiently descriptive latent vector requiring a significant number of classes.Source: §7 · p. 14

Future work

• Future work includes Lie-algebra equivariances, tangent-direction invariance, 6DoF object detection, and equivariances to point resampling.Source: §7 · p. 14

Left equivariance

• Transforming the inputs by g transforms the operator output by the same g, establishing left equivariance.The proof uses orthogonality of G in the quaternion action.Source: A Proof of Proposition 1 · p. 20

Permutation invariance

• Permuting the quaternion inputs and their weights leaves the operator unchanged because the outer-product summation is symmetric.Source: A Proof of Proposition 1 · p. 20

Distance preservation

• The group transformations preserve geodesic distance because the corresponding matrix G is orthonormal.Source: A Proof of Proposition 1 · p. 20

• A unit quaternion is treated as the normal of a four-dimensional hyperplane through the origin, with projection matrix I − qq⊤.Source: B Proof of Lemma 1 · p. 21
• The quaternion mean used in the main paper is equivalent to the Euclidean Weiszfeld mean on affine quaternion subspaces.Source: B Proof of Lemma 1 · p. 22
• The affine-subspace Lq-Weiszfeld algorithm iteratively minimizes a q-norm variant of the cost.Source: B Proof of Lemma 1 · p. 22
• For q = 2, the procedure reduces to an unweighted mean with a closed-form solution, yielding the unweighted Quaternion mean.Source: B Proof of Lemma 1 · pp. 22–23

• Under the stated assumptions, the Weiszfeld sequence converges to a critical point unless an iterate equals one of the input votes.Source: C Proof of Theorem 1 · p. 23
• For q = 1, the critical point lies on one of the specified subspaces and is a geometric median.Source: C Proof of Theorem 1 · p. 23
• The bounded convexity assumption prevents a guarantee of convergence from every starting point, although the authors observed no convergence issues in experiments.Source: C Proof of Theorem 1 · pp. 23–24

D Further Discussions

• The Weiszfeld rate is unestablished and conjectured to be at best linear; the implementation runs three iterations, sufficient in practice for a point good enough to explain the data.Source: D Further Discussions · p. 24
• A QEC module has complexity O(L+KCMLP +CDR), with CDR = M(K + 2(k + 1)L); stacking modules adds complexity proportional to downsampled points.Source: D Further Discussions · pp. 24–25
• The weighted quaternion average uses differentiable SVD, which can raise runtime when implemented naively; batch-wise CUDA kernels enable efficient averaging.Source: D Further Discussions · p. 25
• Inference takes 0.036s and 1107M of GPU memory per instance; training takes 0.226s and 1939M per batch of size 8.Source: D Further Discussions · p. 25
• Using LRFs to restrict the rotation group to certain elements permits significantly fewer parameters; the network’s count depends on class count, with 0.047M parameters for ModelNet10.Source: D Further Discussions · p. 25
• Mapping one quaternion hemisphere onto the other sacrifices manifold closeness and may slightly distort linearization near the equator, but caused no serious shortcomings in the considered operations and LRFs.Source: D Further Discussions · p. 25
• Canonicalized poses cluster within non-symmetric classes, while symmetric objects form multiple clusters; this indicates consistent relative-pose information within classes.Source: D Further Discussions · p. 26

E Our Siamese Architecture

• For supervised relative-pose estimation, a Siamese network processes two point sets whose latent capsule representations jointly contribute to pose regression.Source: E Our Siamese Architecture · p. 26

F Additional Details on Evaluations

• For ModelNet40, the official split uses 9,843 training and 2,468 different test shapes; for ModelNet10, it uses 3,991 training and 908 test shapes; 10K points are randomly sampled from each shape’s mesh surface [55,57].Source: F Additional Details on Evaluations · p. 26
• Additional pairwise shape alignments are produced directly by the Siamese network’s forward pass, without ICP.The examples cover more ModelNet10 categories.Source: Figure 8 · p. 27
• Training uses a single well-aligned orientation without random rotation augmentation, while testing averages performance across randomly generated arbitrary SO(3) rotations.The same protocol is used for the authors’ algorithms and the baselines.Source: F Additional Details on Evaluations · p. 27
• The first five, less rotationally symmetric ModelNet10 categories have higher classification accuracy than their more rotationally symmetric counterparts.The authors report that classification and rotation estimation reinforce one another in their algorithm.Source: F Additional Details on Evaluations · p. 27
• The Siamese architecture performs best at object rotation estimation in the reported comparison.The comparison includes recalculated-LRF variants and rotation-only versions of IT-Net and PointNetLK; the authors note that their networks were trained without rotation augmentation, while those baselines used it.Source: Figure 9 · p. 28, F Additional Details on Evaluations · p. 28