Source-linked AI summary
Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds from Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction
Zhizhong Han, Xiyang Wang, Yu-Shen Liu, Matthias Zwicker
TL;DR
Unsupervised point-cloud feature learning is limited by weak local-geometry supervision in reconstruction-focused methods. MAP-VAE combines multi-angle half-to-half prediction with variational self-reconstruction, and reports outperforming results across four shape-analysis tasks versus state-of-the-art methods.
Problem
Existing unsupervised point-cloud methods relying on global self-reconstruction suffer from ineffective local-geometry learning, limiting feature discriminability.
Method
MAP-VAE jointly learns local and global geometry through multi-angle front-to-back-half prediction, RNN-based sequence modeling, and variational self-reconstruction.
Results
MAP-VAE reports outperforming results across four shape-analysis tasks, learning more discriminative global or local features than state-of-the-art methods.
Takeaways & Limitations
Joint local and global self-supervision enables MAP-VAE to learn fine-grained point-cloud geometry and structure within the reported evaluation scope.
Abstract
from arXiv · showhide
Unsupervised feature learning for point clouds has been vital for large-scale point cloud understanding. Recent deep learning based methods depend on learning global geometry from self-reconstruction. However, these methods are still suffering from ineffective learning of local geometry, which significantly limits the discriminability of learned features. To resolve this issue, we propose MAP-VAE to enable the learning of global and local geometry by jointly leveraging global and local self-supervision. To enable effective local self-supervision, we introduce multi-angle analysis for point clouds. In a multi-angle scenario, we first split a point cloud into a front half and a back half from each angle, and then, train MAP-VAE to learn to predict a back half sequence from the corresponding front half sequence. MAP-VAE performs this half-to-half prediction using RNN to simultaneously learn each local geometry and the spatial relationship among them. In addition, MAP-VAE also learns global geometry via self-reconstruction, where we employ a variational constraint to facilitate novel shape generation. The outperforming results in four shape analysis tasks show that MAP-VAE can learn more discriminative global or local features than the state-of-the-art methods.
1. Introduction
MAP-VAE addresses limitations in unsupervised point-cloud feature learning by jointly using local and global self-supervision. Its multi-angle analysis creates semantic regional supervision for learning fine-grained geometry and structure.
- Motivation: Unsupervised point-cloud learning avoids intensive manual labeling, but existing self-supervised methods have limited effectiveness in learning local geometry.Prior approaches rely on self-reconstruction, distribution approximation, or local patches to learn point-cloud structure.
- MAP-VAE: MAP-VAE jointly employs local and global self-supervision to learn fine-grained point-cloud geometry and structure.The model combines multi-angle half-to-half prediction with global self-reconstruction under a variational constraint.
- Multi-angle analysis: Multi-angle analysis splits each point cloud into front and back halves across incrementally varying angles, enabling prediction of back-half sequences from complementary front-half sequences.The prediction targets local geometric and structural information across varying views.
- Contributions: The method introduces a consistent view-angle-based split that exposes semantic regions and fine-grained discriminative information in point-cloud regions.This splitting strategy is presented as a way to explore local geometry and structure.
2. Related work
Related work distinguishes supervised feature learning from unsupervised approaches that derive training signals directly from point clouds. Existing unsupervised methods use reconstruction, distribution approximation, and local-region modeling to capture structure.
- Unsupervised feature learning: Supervised point-cloud feature learning requires labels, whereas unsupervised methods mine self-supervision from point clouds without additional supervised information.The reviewed approaches target feature learning while avoiding manually obtained shape or segmentation labels.
- Generative methods: Unsupervised generative methods learn point-cloud structure through self-reconstruction, distribution approximation, or adversarial generation.These strategies are implemented with autoencoders or generative adversarial networks.
- Local structure modeling: Methods seeking finer global structure first learn local information from point-cloud patches and then use it to reconstruct the full point cloud.The reviewed approaches obtain local structure through patch-based processing such as clustering or conditional prediction.
3. Overview
MAP-VAE uses three coordinated branches to learn from multi-angle point-cloud halves while reconstructing the complete shape. Aggregation supports both prediction and reconstruction, while separate decoders produce the back-half sequence and original point cloud.
- Architecture: MAP-VAE jointly performs half-to-half prediction and self-reconstruction through aggregation, reconstruction, and prediction branches.Branches A and P perform half-to-half prediction, while branches A and R perform self-reconstruction.
- Training sample: Each training sample contains a front-half sequence, its complementary back-half sequence, and the original point cloud.Corresponding sequence elements are obtained by splitting the original cloud from a specific angle.
- Aggregation branch: The aggregation branch encodes local point-cloud geometry and spatial relationships by sequentially aggregating front-half features across angles.It combines global and local encoder features into an angle-specific representation.
- Decoding branches: The reconstruction branch decodes an angle-specific feature into the original point cloud, while the prediction branch decodes it into the paired back-half sequence.The two branches provide global self-reconstruction and local half-to-half prediction targets.
4. Multi-angle splitting
MAP-VAE uses multi-angle splitting to create local self-supervision by pairing front halves with complementary back halves, while preserving semantic regions and their spatial relationships.
- Multi-angle splitting: Multi-angle analysis provides front-half inputs whose corresponding back halves must be inferred, encouraging detailed shape representations.The approach uses multiple viewpoints to expose correspondences between different shape regions.
- Multi-angle splitting: Incrementally varying angles lets MAP-VAE observe different semantic parts while retaining their spatial relationships.This multi-angle construction supports learning relationships among regions across the shape.
- Multi-angle splitting: Front and back halves are defined from each viewpoint using the N nearest and N farthest points, respectively.The viewpoints are arranged around the point cloud on a circle.
- Geodesic splitting: Euclidean sorting can produce discontinuous semantic front halves, so MAP-VAE instead sorts points by geodesic distance from the viewpoint-nearest point.The geodesic map is used to form a continuous front half and its complementary back half.
- Half-to-half sequence pairs: Each angle yields a front-half sequence paired elementwise with its complementary back-half sequence, preserving correspondence across spatially varying views.The method selects W uniformly covering angles and forms training samples from the resulting sequence pairs.
5. MAP-VAE
MAP-VAE combines aggregation, reconstruction, and prediction branches to learn global and local point-cloud geometry. Its variational reconstruction supports novel shape generation, while prediction models correspondence and spatial relationships among halves.
- Aggregation branch A: Aggregation branch A encodes global geometry and local front-half geometry, then combines them with an aggregation RNN into an angle-specific feature.The encoders follow a PointNet++ architecture, and the aggregation RNN uses a 512-dimensional hidden state.
- Reconstruction branch R: Reconstruction branch R decodes each angle-specific feature into the original point cloud and optimizes Earth Mover’s Distance.The decoder reconstructs the full point cloud, with EMD selected as more faithful to visual quality than Chamfer Distance.
- Reconstruction branch R: A variational constraint transforms angle-specific features into latent vectors near a unit Gaussian, enabling novel shapes to be generated by sampling and decoding.The latent vector is formed from estimated mean and variance with sampled Gaussian noise.
- Prediction branch P: Prediction branch P uses an RNN to predict each back half from the corresponding front-half sequence, learning half-to-half correspondence and spatial relationships.It predicts the back-half elements in sequence order and also predicts their low-level features.
- Objective and representation: MAP-VAE jointly minimizes the costs of its branches and aggregates angle-specific features with max pooling to represent the point cloud globally.The objective combines the branch costs using balance parameters, and the final global feature is pooled over all training samples.
6. Experimental results and analysis
MAP-VAE is evaluated through parameter studies and four shape-analysis tasks, combining global reconstruction with local multi-angle self-supervision. Results indicate strong feature learning, generation, and completion performance across the evaluated settings.
- Parameter analysis: The best parameter settings are β = 1000, α = 0.01, W = 6, and Z = 128, with performance degrading when these values become too large or small.Uniformly distributing six angles performs best because fewer angles provide less local information while more angles increase redundancy.
- Shape classification: MAP-VAE outperforms all point-cloud methods on ModelNet40 and achieves the best performance among compared methods on ModelNet10 classification.A view-based method is slightly better on ModelNet40, but the paper reports MAP-VAE as stronger than all point-cloud-based methods.
- Ablation study: Half-to-half prediction improves point-cloud understanding, while geodesic distance is reported as superior to Euclidean distance for selecting semantic front halves.The comparison uses encoder variants with and without the variational constraint.
- Shape segmentation: MAP-VAE significantly outperforms LGAN and LGAN1 in both mIoU and classification accuracy for shape segmentation.The paper attributes the result to captured local geometry that supports more discriminative per-point features.
- Shape generation: MAP-VAE generates sharp, high-fidelity novel shapes across ModelNet10 and ShapeNet part classes, with learned local geometry details.The experiments sample class-specific latent vectors and report high-quality generated point clouds.
- Point cloud completion: The lowest EMD distance shows that MAP-VAE outperforms all competitors for point-cloud completion while preserving more local geometry details.The evaluation uses airplane and chair partial point clouds with complete clouds as ground truth.
7. Conclusions
The paper concludes that MAP-VAE jointly leverages local and global self-supervision for unsupervised 3D point-cloud feature learning. Its multi-angle analysis learns local geometry and structure, while results across applications indicate more discriminative global or local features than state-of-the-art methods.
- Conclusions: MAP-VAE jointly leverages local and global self-supervision to learn fine-grained point-cloud geometry and structure.The method uses semantic local self-supervision from multi-angle analysis together with global self-supervision.
- Conclusions: Results across various applications show that MAP-VAE learns more discriminative global or local point-cloud features than state-of-the-art methods.