Source-linked AI summary
PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition
Haoxuan You, Yifan Feng, Rongrong Ji, Yue Gao
TL;DR
Existing point-cloud methods leave relationships among local structure features underexplored, while point-cloud and multi-view representations have received little joint treatment. PVNet integrates both modalities through embedding attention fusion and achieves superior 3D shape recognition performance on ModelNet40 classification and retrieval tasks.
Problem
Point-cloud methods do not adequately model relationships among local structure features, and joint use of point-cloud and multi-view data remains limited.
Method
PVNet projects multi-view global features into the point-cloud feature space and uses embedding attention fusion to generate soft masks that refine local geometric features.
Results
PVNet achieves superior performance on ModelNet40 classification and retrieval tasks, including 89.5% retrieval performance and gains of 0.7% overall accuracy and 0.2% mean class accuracy over late fusion.
Takeaways & Limitations
Jointly modeling complementary point-cloud and multi-view representations with attention improves 3D shape recognition beyond separate branches and direct late fusion.
Takeaways & Limitations
The paper places no limit on the methods used to obtain different views.
Abstract
from arXiv · showhide
3D object recognition has attracted wide research attention in the field of multimedia and computer vision. With the recent proliferation of deep learning, various deep models with different representations have achieved the state-of-the-art performance. Among them, point cloud and multi-view based 3D shape representations are promising recently, and their corresponding deep models have shown significant performance on 3D shape recognition. However, there is little effort concentrating point cloud data and multi-view data for 3D shape representation, which is, in our consideration, beneficial and compensated to each other. In this paper, we propose the Point-View Network (PVNet), the first framework integrating both the point cloud and the multi-view data towards joint 3D shape recognition. More specifically, an embedding attention fusion scheme is proposed that could employ high-level features from the multi-view data to model the intrinsic correlation and discriminability of different structure features from the point cloud data. In particular, the discriminative descriptions are quantified and leveraged as the soft attention mask to further refine the structure feature of the 3D shape. We have evaluated the proposed method on the ModelNet40 dataset for 3D shape classification and retrieval tasks. Experimental results and comparisons with state-of-the-art methods demonstrate that our framework can achieve superior performance.
1 INTRODUCTION
PVNet jointly uses point-cloud and multi-view representations for 3D shape recognition, addressing unexplored relationships among local point-cloud structures. Its embedding attention fusion uses global view features to generate soft masks that refine point-cloud features.
- Point-cloud methods preserve spatial and local structure but leave relationships among different local features unexplored, limiting 3D-shape representation.
- PVNet is introduced as a joint framework that directly considers point-cloud and multi-view data for 3D shape recognition.
- Global multi-view features are projected into the point-cloud feature subspace and used to learn relationships among local point-cloud features.
- Soft attention masks quantify local-structure significance, enhancing discriminative point-cloud features while suppressing less useful features.
- The framework is evaluated on ModelNet40 for 3D-shape classification and retrieval, with reported better performance on both tasks.
2 RELATED WORK
Prior work develops separate view-based and point-cloud-based models, while multimodal methods often use fusion strategies that do not directly preserve both modalities. PVNet is presented as a direct fusion of point-cloud and multi-view streams.
- View Based Models: View-based models represent 3D objects with images captured from different angles and process them using established 2D CNN architectures.
- Point Cloud Based Models: Point-cloud models process unordered 3D points and increasingly extract local geometric features, but existing methods do not exploit correlations among different geometric features.
- Multimodal Fusion Methods: Multimodal approaches fuse representations such as volumetric, view, image, depth, optical-flow, textual, and visual features using varied architectures.
- Multimodal Fusion Methods: Earlier fusion work may project point clouds into images for CNN processing, whereas PVNet directly fuses point-cloud and multi-view streams.
3 POINT-VIEW NETWORK
PVNet jointly processes raw point clouds and rendered multi-view images, using multi-view global features to guide attention over point-cloud structure features. Its attention embedding fusion combines the modalities through adaptive masks, residual refinement, and final feature fusion for classification and retrieval.
- Input and branches: PVNet uses 1,024 raw points and 12 rendered views to represent each 3D shape.The point-cloud and multi-view inputs are processed as two separate modalities.
- Input and branches: Point-cloud and multi-view inputs are processed in separate branches, with multi-view global features incorporated into the point-cloud branch through embedding attention fusion.The resulting fused feature supports both classification and retrieval.
- Attention embedding fusion: The fusion mask path repeats embedded global view features across points, concatenates them with point features, and uses an MLP to produce relationship descriptors.The convention path uses EdgeConv to produce local point features, while the mask path models relationships between view-global and point-local features.
- Attention embedding fusion: The normalized descriptors generate soft attention masks in [0, 1] that represent the significance of different local structures.The normalization uses abs and log before sigmoid to avoid outputs approaching 0 or 1 because of large inputs.
- Attention embedding fusion: Residual fusion multiplies original point features by the attention mask and adds them back to refine discriminative structures while suppressing less useful features.This residual design is intended to avoid corrupting original features and to improve feature representation robustness and efficiency.
- Implementation and training: The implementation uses a spatial transform network, two EdgeConv layers, two attention fusion blocks, and a fully connected embedding layer, with k = 20 neighbors.Training uses end-to-end optimization with an initial stage that freezes the multi-view branch before updating all parameters.
4 EXPERIMENTS AND DISCUSSIONS
Experiments on ModelNet40 evaluate PVNet for classification, retrieval, component effectiveness, and robustness to missing views or points. PVNet achieves strong benchmark performance, benefits from embedding attention fusion, and maintains performance when one modality is incomplete.
- 3D Shape Classification and Retrieval: PVNet achieves 93.2% classification accuracy and 89.5% retrieval mAP on ModelNet40, the best performance among compared methods.The evaluation compares models using volumetric, multi-view, and point-cloud representations.
- 3D Shape Classification and Retrieval: 89.5% mAP is achieved without Mahalanobis metric learning, while PVNet significantly outperforms MVCNN with metric learning and other methods.PVNet directly uses the 512-dimensional feature after its first fully connected layer for retrieval.
- Ablation Studies: Embedding attention fusion improves overall accuracy by 0.7% and mean class accuracy by 0.2% over late fusion.The module uses multi-view global features to quantify local point-cloud feature significance as soft attention masks applied residually.
- On The Number of Views and Points: With only 128 points, PVNet drops about 1% in overall accuracy and 0.5% in mean class accuracy, while DGCNN falls below 20% overall accuracy.With only 32 points, PVNet still achieves 88.5% overall accuracy, attributed to compensation from the complete multi-view modality.
5 CONCLUSION
PVNet jointly uses point-cloud and multi-view data, with embedding attention fusion to capture geometric-property relationships and refine point-cloud features. Experiments on ModelNet40 support its effectiveness and robustness across model components, view counts, and point-cloud counts.
- PVNet is a convolutional network that jointly employs point-cloud and multi-view data for 3D shape recognition.
- Embedding attention fusion uses global multi-view features to model correlations and significance among geometric properties extracted from point clouds.
- The method quantifies discriminative information as a soft attention mask to capture attention-aware point-cloud features.
- Experiments on ModelNet40, including component studies and tests with different view and point-cloud counts, demonstrate the framework’s effectiveness and robustness.