Source-linked AI summary
3D Point Capsule Networks
Yongheng Zhao, Tolga Birdal, Haowen Deng, Federico Tombari
TL;DR
The paper addresses the loss of spatial arrangements and the limitations of single-vector latent representations in 3D point-cloud processing. It introduces 3D-PointCapsNet, a dynamically routed capsule auto-encoder with multiple latent basis functions, and reports improvements across reconstruction, local features, transfer learning, and segmentation while enabling part-based applications.
Problem
Existing 3D architectures often discard spatial arrangements, while common auto-encoders use a single latent vector despite point clouds comprising multiple basis functions.
Method
3D-PointCapsNet uses PointNet-like input layers, dynamic routing, and latent capsules representing multiple shape features and composable basis functions.
Results
The evaluation reports improvements in reconstruction, local feature extraction, transfer learning, and part segmentation, with accuracy up to 85% using 1% of training data.
Takeaways & Limitations
Latent capsules support part-specific interpolation and replacement alongside broader 3D shape-processing and understanding tasks.
Takeaways & Limitations
The decoder formulation assumes conditions such as an invertible C−1 or sufficient MLP capacity for universal reconstruction of arbitrary 3D surfaces.
Abstract
from arXiv · showhide
In this paper, we propose 3D point-capsule networks, an auto-encoder designed to process sparse 3D point clouds while preserving spatial arrangements of the input data. 3D capsule networks arise as a direct consequence of our novel unified 3D auto-encoder formulation. Their dynamic routing scheme and the peculiar 2D latent space deployed by our approach bring in improvements for several common point cloud-related tasks, such as object classification, object reconstruction and part segmentation as substantiated by our extensive evaluations. Moreover, it enables new applications such as part interpolation and replacement.
1. Introduction
3D point-cloud architectures often discard spatial arrangements that express parts-to-whole relationships. The paper proposes 3D-PointCapsNet, a capsule-based auto-encoder whose routed latent capsules improve several tasks and support part-specific applications.
- 3D data provide rich geometric structure, supporting applications such as reconstruction and detection of texture-less objects.
- Existing 3D architectures address order invariance, dimensional complexity, and density variation but often discard spatial arrangements important to parts-to-whole relationships.
- 3D-PointCapsNet uses unsupervised dynamic routing to organize multiple max-pooled feature maps into latent capsules that represent shape features and their likelihood.
- Latent capsules provide explicit control over composable basis functions and can specialize to meaningful shape parts with limited task-specific supervision.
- The evaluation reports gains in local feature extraction, point-cloud reconstruction, transfer learning, and generalization, reaching accuracy up to 85% with 1% of training data.
- The approach enables part replacement and part-by-part animation through latent-space interpolation, applications described as non-trivial for standard 1D-latent architectures.
2. Related Work
Related work spans sparse, permutation-invariant point-cloud architectures, grid-based decoders, and capsule networks in 2D and volumetric 3D. The paper positions 3D-PointCapsNet as the first capsule network designed specifically for sparse, unstructured 3D point clouds.
- Point Clouds in Deep Networks: Point-cloud architectures exploit sparsity and permutation invariance, while some methods reduce local projections to two-dimensional convolutions.
- Point Clouds in Deep Networks: Unsupervised point-cloud methods include PU-Net, FoldingNet, PPF-FoldNet, AtlasNet, and PointGrow, which improve decoding, local feature extraction, representation, or generation.
- 2D Capsule Networks: 2D capsule-network extensions address segmentation, action understanding, and generative modeling, while later work develops alternative routing mechanisms.
- 3D Capsule Networks: Prior 3D capsule research focused on rigid-motion equivariance, volumetric medical data, or video, rather than sparse point clouds.
- 3D Point Capsule Networks: Figure 2 depicts an encoder-decoder pipeline that forms primary point capsules, routes them into latent capsules, and decodes multiple 2D-grid point patches.
- 3D Capsule Networks: The paper claims the first capsule network specifically devised for 3D point clouds, exploiting their sparse and unstructured nature to represent 3D surfaces.
3. Method
The method unifies 3D point-cloud auto-encoders as choices of domains, distances, parameterizations, and parameters, then introduces 3D-PointCapsNet with multiple latent capsules and local surface patches.
- Unified formulation: 3D auto-encoders generate surfaces by learning parameterization functions Ψ(U | θ) that reconstruct an input point cloud X as ˆX ≈ X.The formulation treats θ as a lower-dimensional parameterization of the functions.
- Unified formulation: Common decoders differ in their input domain, reconstruction distance, parameterization functions, and parameterization parameters.The unified view explicitly identifies four design choices: U, d(X, ˆX), Ψ, and θ.
- Existing decoders: PointNet uses one latent feature without a grid, FoldingNet warps a fixed 2D grid, and AtlasNet uses multiple MLPs to reconstruct distinct local surface patches.AtlasNet's multiple charts increase learning capacity by modeling different local surface patches.
- 3D-PointCapsNet: 3D-PointCapsNet replaces the single latent vector with latent features {fi}, each paired with a distinct 2D grid Pi for a specific local patch.This design addresses the limitation that one latent representation governs multiple basis functions in a point cloud.
- 3D-PointCapsNet: The encoder extracts point-wise features, diversifies attention through independent convolutional layers, and uses dynamic routing to form 64 capsules of dimension 64.The input contains 2048 points, and the independent kernels produce primary point capsules before routing to higher-level capsules.
- 3D-PointCapsNet: The decoder replicates capsules with unique random grids, reconstructs 64-point local patches through MLPs, and joins 32 patches into a 2048-point shape.The reconstructed shape is represented as ˆX = ∪iΨi(Pi|{fi}).
- Optional supervision: Optional supervision classifies latent capsules into object parts, treating segmentation as per-capsule classification rather than per-point classification.Ground-truth capsule labels are obtained by reconstructing each local part, retrieving nearest-neighbor point labels, and taking their mode.
4. Experiments
Experiments evaluate 3D-PointCapsNet across local feature extraction, reconstruction, transfer learning, segmentation, and capsule-based shape editing. The network reports stronger benchmark performance and supports part-specific operations through latent capsules.
- The evaluation covers local feature extraction, classification, reconstruction, part segmentation, interpolation, and part replacement across several 3D benchmarks.Datasets include ShapeNet-Core, ShapeNet-Part, ModelNet40, and 3DMatch.
- 3D Local Feature Extraction: 10% average improvement over state of the art was reported for local feature extraction using 2K points instead of PPF-FoldNet’s 5K.Performance remained above competing methods except on the Kitchen sequence, where PPFNet performed remarkably well.
- Rotated Feature Matching: ∼12% improvement over the state of the art was reported on the rotated 3DMatch benchmark in the 2K-point setting.The authors attribute the result to operation on 4D point-pair features while retaining theoretical invariances.
- 3D Reconstruction: The capsule auto-encoder achieved lower Chamfer reconstruction distances than the compared AtlasNet-based methods on ShapeNet Core v2.The comparison uses the same splits and evaluation metric as AtlasNet, with Chamfer distances averaged across categories for N > 2K points.
- Part Segmentation: With 1% labeled training data, the method performed 7% better than SO-Net, and with 5% data it still surpassed SO-Net by 2% for part segmentation.The method classifies latent capsules rather than 3D points and uses a smaller network.
- Capsule Behavior and Applications: Dynamic routing concentrates capsules on semantic or local regions, while replacing routing with PointNet-like layers harms reconstruction and spreads capsules across shapes.Semi-supervision further steers capsules toward meaningful parts and enables part exchange through latent-capsule switching.
5. Conclusion
The conclusion presents 3D Point-Capsule Network as a flexible framework that represents shapes with dynamically routed latent capsules rather than a single latent vector. It reports success across several 3D tasks and highlights part interpolation and replacement as enabled applications.
- The network summarizes point clouds as a union of disjoint latent basis functions learned through dynamic routing.This contrasts with auto-encoders that parameterize the latent space with a one-dimensional embedding.
- The method succeeds on 3D reconstruction, local feature extraction, and part segmentation in extensive evaluation.
- A latent capsule set enables part interpolation and replacement applications.The conclusion identifies pose estimation and 3D object detection as future deployment targets.
A. Semi-supervised Classification
Semi-supervised classification experiments compare the network with FoldingNet under standard and scarce-data settings. The reported predictions are on par with or better than FoldingNet, including approximately 85% accuracy with limited training data.
- The network generates semi-supervised classification predictions on par with or better than FoldingNet.
B. Part Segmentation
The part-segmentation extension predicts a part label for each latent capsule and reconstructs objects using the labeled capsules. Its qualitative results are reported as comparable to ground truth.
- The part association network takes latent capsules and a one-hot object-category vector as input, then outputs a part prediction for each capsule.It uses cross-entropy loss and Adam optimization.
- The pre-trained decoder reconstructs objects from labeled capsules, producing results qualitatively comparable to ground truth.Visualizations are shown for objects from ShapeNet-Part.
C. Part Interpolation
The network interpolates parts by matching their capsules across source and target shapes, then linearly interpolating those capsule representations to generate unseen intermediate shapes.
- Part prediction uses capsule-wise association instead of point-wise labeling, reducing data annotation efforts.
- Part interpolation matches capsules representing the selected part in the source and target shapes.
- Linear interpolation between corresponding source and target capsules generates visually pleasing intermediate shapes.The network produces intermediate shapes it has never seen during training.
D. Part Replacement
Part replacement exchanges selected parts through latent-space capsule arithmetic, producing coherent point clouds rather than the disconnected shapes caused by direct 3D cut-and-paste.
- Latent-space capsule arithmetic exchanges selected parts between object pairs while respecting the resulting point cloud’s overall coherence.The decoder’s high-fidelity shape generation supports the replacement.
- Compared with cut-and-paste in 3D space, capsule-based replacement avoids undesired disconnected shapes.
- The replacement evaluation presents numerous object pairs with a selected part exchanged between the shapes.
E. Ablation Study
The ablation compares reconstruction using dynamic routing against a PointNet-like convolutional alternative, while the comparison module operates on primary capsules to produce latent-dimensional vectors.
- The reconstruction ablation replaces dynamic routing with a PointNet-like set of convolutional layers.
- The study uses reconstruction results to compare the dynamic-routing design with the PointNet-like alternative.
- The comparison module operates on primary point capsules and generates vectors matching the latent capsule output dimensionality of dynamic routing.
F. A Discussion on the Local Spatial Attention
Local attention arises from primary capsule organization and potentially the decoder, enabling shape instantiation control in a latent space for interpolation and replacement.
- F. A Discussion on the Local Spatial Attention: Multiple MLPs act on individual capsules, with part information encoded inside capsules rather than in the MLPs themselves.
- F. A Discussion on the Local Spatial Attention: Local attention stems from primary point-capsule organization through dynamic routing and potentially from the decoder.
- F. A Discussion on the Local Spatial Attention: Latent-space control represents shape instantiation through part interpolation and replacement.
- F. A Discussion on the Local Spatial Attention: The paper also presents part segmentation under limited training data as an application of capsule-based processing.
- C. Part Interpolation: Part interpolation produces smooth intermediate topologies through linear interpolation of corresponding capsules.
- D. Part Replacement: Latent-space part exchange yields more natural replacements without the detachment problems of simple cut-and-paste.