Source-linked AI summary
Multi-Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds from Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction
Zhizhong Han, Xiyang Wang, Yu-Shen Liu, Matthias Zwicker
TL;DR
Unsupervised point-cloud feature learning is limited by weak local-geometry supervision in reconstruction-focused methods. MAP-VAE combines multi-angle half-to-half prediction with variational self-reconstruction, and reports outperforming results across four shape-analysis tasks versus state-of-the-art methods.
Problem
Existing unsupervised point-cloud methods relying on global self-reconstruction suffer from ineffective local-geometry learning, limiting feature discriminability.
Source: §1 · p. 1Method
Results
MAP-VAE reports outperforming results across four shape-analysis tasks, learning more discriminative global or local features than state-of-the-art methods.
Source: §6 · pp. 6–7, §7 · p. 8Takeaways & Limitations
Abstract
from arXiv · showhide
Unsupervised feature learning for point clouds has been vital for large-scale point cloud understanding. Recent deep learning based methods depend on learning global geometry from self-reconstruction. However, these methods are still suffering from ineffective learning of local geometry, which significantly limits the discriminability of learned features. To resolve this issue, we propose MAP-VAE to enable the learning of global and local geometry by jointly leveraging global and local self-supervision. To enable effective local self-supervision, we introduce multi-angle analysis for point clouds. In a multi-angle scenario, we first split a point cloud into a front half and a back half from each angle, and then, train MAP-VAE to learn to predict a back half sequence from the corresponding front half sequence. MAP-VAE performs this half-to-half prediction using RNN to simultaneously learn each local geometry and the spatial relationship among them. In addition, MAP-VAE also learns global geometry via self-reconstruction, where we employ a variational constraint to facilitate novel shape generation. The outperforming results in four shape analysis tasks show that MAP-VAE can learn more discriminative global or local features than the state-of-the-art methods.
1. Introduction
MAP-VAE addresses limitations in unsupervised point-cloud feature learning by jointly using local and global self-supervision. Its multi-angle analysis creates semantic regional supervision for learning fine-grained geometry and structure.
Motivation
- Unsupervised point-cloud learning avoids intensive manual labeling, but existing self-supervised methods have limited effectiveness in learning local geometry.Prior approaches rely on self-reconstruction, distribution approximation, or local patches to learn point-cloud structure.Source: §1 · p. 1
MAP-VAE
- MAP-VAE jointly employs local and global self-supervision to learn fine-grained point-cloud geometry and structure.The model combines multi-angle half-to-half prediction with global self-reconstruction under a variational constraint.Source: §1 · pp. 1–2
Multi-angle analysis
- Multi-angle analysis splits each point cloud into front and back halves across incrementally varying angles, enabling prediction of back-half sequences from complementary front-half sequences.The prediction targets local geometric and structural information across varying views.Source: §1 · pp. 1–2
Contributions
- The method introduces a consistent view-angle-based split that exposes semantic regions and fine-grained discriminative information in point-cloud regions.This splitting strategy is presented as a way to explore local geometry and structure.Source: §1 · p. 2
2. Related work
Related work distinguishes supervised feature learning from unsupervised approaches that derive training signals directly from point clouds. Existing unsupervised methods use reconstruction, distribution approximation, and local-region modeling to capture structure.
Unsupervised feature learning
- Supervised point-cloud feature learning requires labels, whereas unsupervised methods mine self-supervision from point clouds without additional supervised information.The reviewed approaches target feature learning while avoiding manually obtained shape or segmentation labels.Source: §2 · p. 2
Generative methods
- Unsupervised generative methods learn point-cloud structure through self-reconstruction, distribution approximation, or adversarial generation.These strategies are implemented with autoencoders or generative adversarial networks.Source: §2 · p. 2
Local structure modeling
- Methods seeking finer global structure first learn local information from point-cloud patches and then use it to reconstruct the full point cloud.The reviewed approaches obtain local structure through patch-based processing such as clustering or conditional prediction.Source: §2 · p. 2
3. Overview
MAP-VAE uses three coordinated branches to learn from multi-angle point-cloud halves while reconstructing the complete shape. Aggregation supports both prediction and reconstruction, while separate decoders produce the back-half sequence and original point cloud.
Architecture
- MAP-VAE jointly performs half-to-half prediction and self-reconstruction through aggregation, reconstruction, and prediction branches.Branches A and P perform half-to-half prediction, while branches A and R perform self-reconstruction.Source: §3 · p. 2
Training sample
- Each training sample contains a front-half sequence, its complementary back-half sequence, and the original point cloud.Corresponding sequence elements are obtained by splitting the original cloud from a specific angle.Source: §3 · p. 2
Aggregation branch
- The aggregation branch encodes local point-cloud geometry and spatial relationships by sequentially aggregating front-half features across angles.It combines global and local encoder features into an angle-specific representation.Source: §3 · pp. 2–3
Decoding branches
- The reconstruction branch decodes an angle-specific feature into the original point cloud, while the prediction branch decodes it into the paired back-half sequence.The two branches provide global self-reconstruction and local half-to-half prediction targets.Source: §3 · p. 3
4. Multi-angle splitting
MAP-VAE uses multi-angle splitting to create local self-supervision by pairing front halves with complementary back halves, while preserving semantic regions and their spatial relationships.
Multi-angle splitting
- Multi-angle analysis provides front-half inputs whose corresponding back halves must be inferred, encouraging detailed shape representations.The approach uses multiple viewpoints to expose correspondences between different shape regions.Source: §4 · p. 3
- Incrementally varying angles lets MAP-VAE observe different semantic parts while retaining their spatial relationships.This multi-angle construction supports learning relationships among regions across the shape.Source: §4 · p. 3
- Front and back halves are defined from each viewpoint using the N nearest and N farthest points, respectively.The viewpoints are arranged around the point cloud on a circle.Source: §4 · p. 3
Geodesic splitting
- Euclidean sorting can produce discontinuous semantic front halves, so MAP-VAE instead sorts points by geodesic distance from the viewpoint-nearest point.The geodesic map is used to form a continuous front half and its complementary back half.Source: §4 · p. 3
Half-to-half sequence pairs
- Each angle yields a front-half sequence paired elementwise with its complementary back-half sequence, preserving correspondence across spatially varying views.The method selects W uniformly covering angles and forms training samples from the resulting sequence pairs.Source: §4 · pp. 3–4
5. MAP-VAE
MAP-VAE combines aggregation, reconstruction, and prediction branches to learn global and local point-cloud geometry. Its variational reconstruction supports novel shape generation, while prediction models correspondence and spatial relationships among halves.
Aggregation branch A
- Aggregation branch A encodes global geometry and local front-half geometry, then combines them with an aggregation RNN into an angle-specific feature.The encoders follow a PointNet++ architecture, and the aggregation RNN uses a 512-dimensional hidden state.Source: §5 · p. 4
Reconstruction branch R
- Reconstruction branch R decodes each angle-specific feature into the original point cloud and optimizes Earth Mover’s Distance.The decoder reconstructs the full point cloud, with EMD selected as more faithful to visual quality than Chamfer Distance.Source: §5 · p. 4
- A variational constraint transforms angle-specific features into latent vectors near a unit Gaussian, enabling novel shapes to be generated by sampling and decoding.The latent vector is formed from estimated mean and variance with sampled Gaussian noise.Source: §5 · p. 4
Prediction branch P
- Prediction branch P uses an RNN to predict each back half from the corresponding front-half sequence, learning half-to-half correspondence and spatial relationships.It predicts the back-half elements in sequence order and also predicts their low-level features.Source: §5 · p. 4
Objective and representation
- MAP-VAE jointly minimizes the costs of its branches and aggregates angle-specific features with max pooling to represent the point cloud globally.The objective combines the branch costs using balance parameters, and the final global feature is pooled over all training samples.Source: §5 · p. 4
6. Experimental results and analysis
MAP-VAE is evaluated through parameter studies and four shape-analysis tasks, combining global reconstruction with local multi-angle self-supervision. Results indicate strong feature learning, generation, and completion performance across the evaluated settings.
Parameter analysis
- The best parameter settings are β = 1000, α = 0.01, W = 6, and Z = 128, with performance degrading when these values become too large or small.Uniformly distributing six angles performs best because fewer angles provide less local information while more angles increase redundancy.Source: §6 · p. 5, Table 1 · p. 5, Table 2 · p. 5
Shape classification
- MAP-VAE outperforms all point-cloud methods on ModelNet40 and achieves the best performance among compared methods on ModelNet10 classification.A view-based method is slightly better on ModelNet40, but the paper reports MAP-VAE as stronger than all point-cloud-based methods.Source: §6 · p. 6, Table 6 · p. 6
Ablation study
- Half-to-half prediction improves point-cloud understanding, while geodesic distance is reported as superior to Euclidean distance for selecting semantic front halves.The comparison uses encoder variants with and without the variational constraint.Source: §6 · p. 6
Shape segmentation
- MAP-VAE significantly outperforms LGAN and LGAN1 in both mIoU and classification accuracy for shape segmentation.The paper attributes the result to captured local geometry that supports more discriminative per-point features.Source: §6 · p. 7
Shape generation
- MAP-VAE generates sharp, high-fidelity novel shapes across ModelNet10 and ShapeNet part classes, with learned local geometry details.The experiments sample class-specific latent vectors and report high-quality generated point clouds.Source: §6 · p. 7
Point cloud completion
- The lowest EMD distance shows that MAP-VAE outperforms all competitors for point-cloud completion while preserving more local geometry details.The evaluation uses airplane and chair partial point clouds with complete clouds as ground truth.Source: §6 · p. 8
7. Conclusions
The paper concludes that MAP-VAE jointly leverages local and global self-supervision for unsupervised 3D point-cloud feature learning. Its multi-angle analysis learns local geometry and structure, while results across applications indicate more discriminative global or local features than state-of-the-art methods.
Conclusions
- MAP-VAE jointly leverages local and global self-supervision to learn fine-grained point-cloud geometry and structure.The method uses semantic local self-supervision from multi-angle analysis together with global self-supervision.Source: §7 · p. 8
- Results across various applications show that MAP-VAE learns more discriminative global or local point-cloud features than state-of-the-art methods.Source: §7 · p. 8
Related papers
This paper cites
- Auto-Encoding Variational Bayes2013
- PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space2017
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling2016
- SO-Net: Self-Organizing Network for Point Cloud Analysis2018
- PCN: Point Completion Network2018
- Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-based Sequence to Sequence Network2018