Source-linked AI summary
Multi-view Convolutional Neural Networks for 3D Shape Recognition
Hang Su, Subhransu Maji, Evangelos Kalogerakis, Erik Learned-Miller
TL;DR
The paper asks whether 3D shapes must be recognized from native 3D representations or can be effectively represented by rendered 2D views. It uses CNNs to classify and aggregate multiple views, finding that view-based recognition outperforms direct 3D classifiers and yields compact descriptors.
Problem
The paper examines whether 3D shapes should be recognized from native 3D representations or from rendered 2D views.
Method
It introduces a multi-view CNN that processes rendered views independently, aggregates them with view pooling, and produces a compact shape descriptor.
Results
8% higher accuracy (77% →85%) than the best direct-3D models was achieved from a single rendered view, with further gains from additional views.
Takeaways & Limitations
Collections of 2D views can support highly accurate 3D recognition, compact retrieval descriptors, and high-accuracy retrieval from hand-drawn sketches.
Takeaways & Limitations
The experiments primarily use 3D polygon mesh models, leaving recognition from real-world multi-view objects or video for future investigation.
Abstract
from arXiv · showhide
A longstanding question in computer vision concerns the representation of 3D shapes for recognition: should 3D shapes be represented with descriptors operating on their native 3D formats, such as voxel grid or polygon mesh, or can they be effectively represented with view-based descriptors? We address this question in the context of learning to recognize 3D shapes from a collection of their rendered views on 2D images. We first present a standard CNN architecture trained to recognize the shapes' rendered views independently of each other, and show that a 3D shape can be recognized even from a single view at an accuracy far higher than using state-of-the-art 3D shape descriptors. Recognition rates further increase when multiple views of the shapes are provided. In addition, we present a novel CNN architecture that combines information from multiple views of a 3D shape into a single and compact shape descriptor offering even better recognition performance. The same architecture can be applied to accurately recognize human hand-drawn sketches of shapes. We conclude that a collection of 2D views can be highly informative for 3D shape recognition and is amenable to emerging CNN architectures and their derivatives.
1. Introduction
The paper argues that 2D rendered views can outperform direct 3D representations for 3D shape recognition, even from a single test view. It introduces a multi-view CNN that compiles view information into a compact descriptor and improves performance by combining informative views.
- Motivation and result: 2D rendered-view classifiers can dramatically outperform classifiers built directly on 3D representations, even when testing uses only one view.The CNN is trained on a fixed set of rendered views but receives a single view at test time.
- Motivation and result: Voxel-based deep networks may require reduced resolution because full-resolution 3D representations are difficult to train with available samples and reasonable computation.The paper contrasts this constraint with the relative efficiency of 2D representations.
- Motivation and result: 2D representations enable pre-training on advances in image descriptors and massive labeled image databases before fine-tuning for 3D model projections.The introduction identifies ubiquitous images and abundant labeled datasets as a significant advantage while comparable 3D training data remain limited.
- Multi-view CNN: The multi-view CNN compiles multiple 2D views into a compact object descriptor that is at least as informative for classification and slightly more informative for retrieval than all view descriptors.The architecture is presented as a way to combine information across views rather than classify each view independently.
- Multi-view CNN: The multi-view CNN learns to combine views instead of averaging, using informative views for prediction while ignoring others and improving performance.The introduction relates this process to jittering, where transformed copies help learn invariance to transformations such as rotation or translation.
2. Related Work
Prior work divides 3D shape descriptors into native 3D and view-based approaches, while CNNs provide learned image descriptors for recognition tasks. The paper addresses the limited prior work combining view-based descriptors by introducing a multi-view CNN with view pooling.
- Shape descriptors: 3D shape descriptors operate on native representations such as polygon meshes, voxel grids, point clouds, or implicit surfaces, whereas view-based descriptors describe how objects look across rendered views.This distinction organizes prior work into two broad descriptor categories.
- Shape descriptors: 3D descriptors were largely hand-designed around geometric properties, with Wu et al. providing a recent voxel-based exception using 3D convolutional networks.Examples include representations based on surface normals, curvatures, distances, and angles.
- Shape descriptors: View-based descriptors are relatively low-dimensional, efficient to evaluate, robust to representation artifacts, and directly comparable with 2D images, silhouettes, and hand-drawn sketches.The cited artifacts include holes, imperfect polygon-mesh tessellations, and noisy surfaces.
- Convolutional neural networks: CNNs trained on large datasets such as ImageNet learn general-purpose image descriptors for object detection, scene recognition, texture recognition, and fine-grained classification.The paper relates its approach to adapting these deep architectures to domain-specific imagery, including shaded illustrations of 3D objects.
- Multi-view combination: Prior work rarely learned to combine view-based descriptors for 3D recognition, instead using exhaustive pairwise comparisons or concatenation of ordered views; this work learns multi-view recognition through a view-pooling layer.The architecture uses image-based CNNs while processing each shape in the context of its other views.
3. Method
The method represents 3D shapes through rendered views and learns to aggregate them with a unified multi-view CNN into a compact descriptor. It uses controlled camera setups, view-level image features, and view pooling for classification, retrieval, and perturbed-image integration.
- Multi-view CNN: MVCNN processes views through shared CNN1 branches, applies element-wise maximum view pooling, and passes the aggregate through CNN2 to produce one compact shape descriptor.This learned aggregation avoids exhaustive pairwise comparisons and improves retrieval from 62.8% →70.1% over separate CNN image descriptors.
- View generation: 3D meshes are rendered with Phong shading under perspective projection, with shapes uniformly scaled to fit the viewing volume.Pixel colors are determined by interpolating reflected intensity at polygon vertices.
- View generation: The method uses virtual-camera viewpoints, including 12 cameras spaced every 30 degrees around upright-oriented meshes and elevated 30 degrees from the ground.The first setup assumes shapes share a consistent upright axis.
- View-based descriptors: Each view receives an image descriptor, using either Fisher vectors with multi-scale SIFT or CNN activation features, and separate-view classification sums SVM decisions across 12 views.The CNN uses a fine-tuned VGG-M network whose 4096-dimensional fc7 representation serves as the image descriptor.
- View-based descriptors: For retrieval, shape distance symmetrically averages each projection’s nearest-view distance, while alternative minimum-pair and averaged-descriptor distances perform worse.Individual image distances are ℓ2 distances between feature vectors.
- Multi-view CNN: MVCNN also provides a general framework for integrating perturbed image samples, illustrated through sketch recognition.The architecture is designed to synthesize information from multiple views into a single descriptor.
4. Experiments
Experiments on ModelNet40 show that view-based CNN descriptors outperform established 3D and Fisher-vector baselines for shape classification and retrieval. Aggregating transformed views also improves sketch recognition and enables sketch-to-3D retrieval.
- 3D shape recognition: ModelNet40 provides 12,311 shapes across 40 common categories using the training and test split from.The broader ModelNet dataset contains 127,915 3D CAD models from 662 categories.
- 3D shape recognition: 78.8% single-view classification accuracy from Fisher vectors exceeds the 77.3% result of learned 3D descriptors.The Fisher-vector baseline uses a single rendered view.
- 3D shape recognition: 88.6% classification accuracy and 62.8% retrieval mAP are achieved by a fine-tuned CNN using all 12 views.This fine-tuned CNN improves substantially over the ImageNet-trained CNN and Fisher-vector baseline.
- 3D shape recognition: 90.1% classification accuracy and 70.4% retrieval mAP are achieved by MVCNN without assuming consistent upright orientation.With the first camera setup and fine-tuning, MVCNN achieves 89.9% classification accuracy and 70.1% retrieval mAP.
- Sketch recognition: 87.2% sketch-classification accuracy is achieved with VGG-VD, while six-view jittering yields 86.3% with MVCNN and 85.5% with a regular CNN.The six samples use rotations of −45°, 0°, and 45° plus horizontal reflections.
- Sketch-to-3D retrieval: 36.1% mAP is obtained for retrieving 3D objects from hand-drawn sketch queries without fine-tuning on sketches or 3D shapes.The experiment uses 193 sketches and 790 CAD models from 10 shared categories, with descriptors from 12 rendered viewpoints and six perturbed query sketches.
5. Conclusion
The paper shows that rendered 2D views can support highly effective 3D shape recognition, outperforming prior direct-3D methods. Aggregating multiple views yields compact, efficient, accurate descriptors and enables high-accuracy sketch-based retrieval, while future work targets view selection and real-world objects.
- Rendered images achieved performance better than any previously published results, including methods operating on direct 3D shape representations.
- Aggregating information from multiple 2D projections produced compact, efficient descriptors with better accuracy than naive multi-view usage.
- Relating 3D shapes to sketches enabled high-accuracy retrieval and leveraged implicit 3D-shape knowledge contained in 2D views.
- Future work should study informative view combinations, required view counts, on-the-fly view selection, and aggregation for real-world objects or video.
Appendix
The appendix provides additional evaluations and visualizations of the multi-view CNN, covering 3D shape classification, view-based saliency, and hand-drawn sketch classification.
- The appendix includes a confusion matrix for 3D shape classification.
- It presents additional view-based saliency maps for the multi-view CNN.
- It shows correctly and wrongly classified examples of hand-drawn sketches.
A. 3D shape classification
MVCNN with fine-tuning on 12 views is evaluated for ModelNet40 3D shape classification, where the largest confusions involve visually similar or ambiguous object categories.
- A. 3D shape classification: MVCNN with fine-tuning on 12 views is used for ModelNet40 3D shape classification.The corresponding confusion matrix is reported in Figure 6.
- A. 3D shape classification: 45% of flower pots are confused with plants, the largest reported confusion.Other major confusions include table → desk at 32% and flower pot → vase at 20%.
- A. 3D shape classification: The reported confusions include plant → flower at 19% and stool → chair at 15%, with some distinctions ambiguous even for humans.These errors reflect category pairs that can be difficult to distinguish visually.
B. Image-specific class saliency visualization across views
The saliency maps highlight both canonical views across views and discriminative parts within individual views.
- Image-specific class saliency visualization across views: Saliency maps emphasize canonical views, such as a bench’s front view, and discriminative parts, such as a bathtub’s faucet and sink hole.Additional examples are shown in Figure 7.
C. Sketch classification
The sketch-classification results include both correctly and incorrectly classified hand-drawn sketches. Misclassifications often involve visually similar component layouts, while some sketches are difficult for humans to recognize as well.
- Figure 8 shows examples of correctly and wrongly classified hand-drawn sketches.
- Misclassified sketches often contain components visually similar to those of the target class.Spider and crab sketches can share a similar leg layout.
- Some misclassified sketches are difficult to recognize even for humans.
D. Document changelog
The paper’s revisions update experimental consistency, add metric-learning results and a second camera setup, and slightly revise reported performance figures.
- D. Document changelog: v2 adopts an updated ModelNet40 training/test split consistent with, causing most methods’ performance to drop slightly because the training set is smaller.The v1 experiments used the full ModelNet40 dataset.
- D. Document changelog: v2 adds results using low-rank Mahalanobis metric learning.
- D. Document changelog: v3 adds a second camera setup without the upright-orientation assumption and slightly changes some accuracy and mAP numbers due to a mesh-rendering issue related to specularities.