Source-linked AI summary
FusionNet: 3D Object Classification Using Multiple Data Representations
Vishakh Hegde, Reza Zadeh
TL;DR
3D object classification needs methods suited to 3D structure rather than relying only on 2D pixel representations. FusionNet combines multi-view pixel CNNs with compact volumetric CNNs, yielding significantly better classification than either representation alone and the best reported combination on ModelNet10 and ModelNet40.
Problem
3D model features differ from RGB-image features because relevant information lies on surfaces and mesh orientations, while CAD models may lack color.
Method
FusionNet combines multi-view pixel CNNs with two compact volumetric CNN architectures trained on multiple object orientations.
Results
FusionNet is the best-performing combination obtained for both ModelNet10 and ModelNet40, and combining the representations significantly outperforms either individual network.
Takeaways & Limitations
The two representations learn partly nonoverlapping object features, so their combination produces a stronger classifier than either representation alone.
Takeaways & Limitations
The method assumes CAD models are aligned along the gravity axis and uses 60 rendered orientations for voxel construction.
Abstract
from arXiv · showhide
High-quality 3D object recognition is an important component of many vision and robotics systems. We tackle the object recognition problem using two data representations, to achieve leading results on the Princeton ModelNet challenge. The two representations: 1. Volumetric representation: the 3D object is discretized spatially as binary voxels - $1$ if the voxel is occupied and $0$ otherwise. 2. Pixel representation: the 3D object is represented as a set of projected 2D pixel images. Current leading submissions to the ModelNet Challenge use Convolutional Neural Networks (CNNs) on pixel representations. However, we diverge from this trend and additionally, use Volumetric CNNs to bridge the gap between the efficiency of the above two representations. We combine both representations and exploit them to learn new features, which yield a significantly better classifier than using either of the representations in isolation. To do this, we introduce new Volumetric CNN (V-CNN) architectures.
1. Introduction
3D model classification matters for emerging vision and robotics applications, but 3D data requires representations and features tailored to surfaces and mesh orientations. FusionNet addresses this by combining compact volumetric CNNs with multi-view pixel CNNs to improve classification.
- Motivation: 3D model classification supports applications including self-driving cars and augmented reality as publicly available 3D content increases.The growing supply of 3D data benefits CNN-based methods that rely on large datasets.
- Challenge: Features effective for RGB image classification may be insufficient for 3D models because relevant information lies on surfaces and mesh orientations.The dataset’s CAD models may lack color, requiring models to learn other shape-defining features.
- Prior Work: Prior work used volumetric voxels, multi-view projections, and cylindrical panoramic projections for 3D classification, with pixel-based methods often outperforming volumetric inputs.Volumetric CNNs partly narrowed this performance difference.
- Contribution: FusionNet introduces two parameter-efficient volumetric CNNs, including one inspired by GoogLeNet’s inception module, and combines them with an ImageNet-pretrained multi-view CNN.The combined networks complement one another and improve state-of-the-art classification accuracy.
2. Related Work
Related work spans handcrafted 3D shape descriptors and CNNs applied to 2D images and 3D model representations. CNNs provide end-to-end automated feature learning, while learned features can generalize across datasets and image categories.
- 3D Shape Descriptors: Earlier 3D shape descriptors represented objects using surface-derived histograms, bag-of-features models, light fields, heat kernel signatures, and SPH.These approaches covered voxel, point-cloud, and surface-based descriptions among other representations.
- CNNs in Vision: CNNs have been widely used for 2D image classification, detection, segmentation, and scene recognition.Their success established CNNs as a common framework for visual recognition tasks.
- CNNs for 3D Models: CNNs were later applied to 3D CAD-model classification and retrieval, combining end-to-end training with automated feature learning.Learned features can generalize to datasets with substantially different image categories.
3. Methods
The method combines voxel and pixel representations through complementary CNNs. Multi-view projections provide pixel features, while compact volumetric networks learn features from multiple object orientations.
- Method Overview: Most 3D classification pipelines choose a data representation and train a CNN on that representation.The proposed method instead combines voxel and pixel inputs through relatively weak classifiers.
- Representations: The pixel branch follows Multi View CNN by aggregating multiple projected views of each 3D model.For voxel data, the authors train their own networks on multiple orientations to learn partly complementary features.
- Architecture: One volumetric CNN uses about 3.5 million parameters, compared with 60 million for AlexNet.This provides a substantially smaller volumetric model than the cited 2D RGB-image baseline.
4. Dataset and Accuracy Measure
The study benchmarks 3D CAD classification on ModelNet, using ModelNet40 and ModelNet10 with the dataset’s prescribed splits. Performance is reported as average per-class classification accuracy.
- Datasets: ModelNet contains 662 object classes and 127,915 CAD models, with ModelNet40 and ModelNet10 serving as benchmark subsets.These datasets helped address the limited availability of large-scale 3D training sets.
- ModelNet40: ModelNet40 contains 12,311 CAD models across 40 classes, using 9,843 models for training and 2,468 for testing.The study uses the dataset authors’ prescribed train-test split.
- ModelNet10: ModelNet10 is a ModelNet40 subset with 4,899 CAD models and gravity-axis-aligned orientations, split into 3,991 training and 908 testing models.The same dataset-provided split is used for ModelNet10.
- Accuracy Measure: Both datasets are evaluated using average per-class classification accuracy, the measure used in most previous work.Figure 1 illustrates voxelized ModelNet40 examples including bathtubs, stools, toilets, and wardrobes.
5. Volumetric CNN (V-CNN)
The V-CNNs classify voxelized 3D models from multiple orientations, using shared networks and max-pooling to aggregate orientation-specific features. V-CNN II adds an inception-style multi-scale module, while training also uses deformation augmentation and ModelNet40-to-ModelNet10 fine-tuning.
- Data representation and augmentation: The volumetric pipeline renders 30 × 30 × 30 voxels from 60 orientations sampled using polar and azimuth angles around the gravity axis.The approach assumes CAD models are aligned along the gravity axis and uses random angle sampling to address rotational variation.
- V-CNN I: V-CNN I uses three 3D convolution layers and two fully connected layers, with 3 × 3 kernels and a final fully connected classifier.ReLU, pooling, and dropout provide nonlinear processing, local redundancy reduction, and over-fitting control.
- Architecture and orientation aggregation: V-CNNs combine multiple voxel orientations by learning shared features and max-pooling activations across 60 views before classification.The same network weights process all orientations, and pooled first fully connected-layer activations feed the final classifier.
- Training and augmentation: Training augments voxel data with randomly displaced mesh vertices, and ModelNet10 fine-tunes weights learned from ModelNet40 instead of using random initialization.The paper motivates augmentation by the limited number of CAD models and uses Softmax loss with end-to-end optimization.
6. Multi-View CNN
The Multi-View CNN represents each CAD model with projected grayscale images from 20 icosahedron-corner cameras and uses AlexNet transfer learning. Its performance is attributed partly to transferable ImageNet features, contrasting with the data limitations of volumetric CNNs.
- Classification comparison: MV-CNN achieves substantially higher accuracy than the volumetric CNNs while using only 20 views, according to the reported comparison.The paper contrasts this with an earlier multi-view setup that used additional camera rotations for each view.
- Multi-view representation: MV-CNN renders grayscale projections from cameras placed on the 20 corners of an icosahedron and uses all 20 views for training and testing.The grayscale images are replicated across three channels for compatibility with AlexNet.
- Architecture size: The V-CNN I architecture contains about 3.5M parameters, whereas V-CNN II contains about 55M parameters.These parameter counts describe the two volumetric architectures compared in the classification section.
- Transfer learning: AlexNet is initialized with ImageNet-pretrained weights before being fine-tuned for the projected 2D images.The training uses Softmax loss, a base learning rate of 0.001, and momentum of 0.9.
- Interpretation of results: The paper links MV-CNN's advantage partly to transferable early-layer features from large 2D datasets, which are unavailable for volumetric CNNs because large 3D datasets are scarce.Accordingly, MV-CNN freezes layers before fc6 while fine-tuning later fully connected layers.
7. Experiments
Experiments compare individual volumetric and multi-view networks, then combine their complementary predictions in FusionNet. FusionNet is reported as the best-performing combination for both ModelNet10 and ModelNet40.
- Individual networks: V-CNN I and V-CNN II perform similarly but learn slightly different features, enabling a slightly better combined classifier.
- Individual networks: Multi-View CNN based on AlexNet performs significantly better than Volumetric CNN.
- Fine-tuning: 1.1 percent gain is observed when AlexNet is fine-tuned on only its last three fully connected layers for ModelNet40.
- Network combination: About 1.6 percent gain is observed when fine-tuned MV-CNN based on AlexNet is combined with V-CNN I.
- FusionNet: FusionNet aggregates V-CNN I, augmented V-CNN I, V-CNN II, and ImageNet-pretrained MV-CNN, achieving the best performance for ModelNet10 and ModelNet40.
- FusionNet: The networks combine after their final fully connected layers by linearly weighting class scores, with weights selected through cross-validation.
8. Conclusions and Future Work
The paper concludes that voxel and pixel representations provide complementary information for 3D CAD classification when used together. Future work considers reducing volumetric test orientations and exploring distance-field representations.
- Conclusions: Combining voxel- and pixel-based networks yields significantly better performance than either individual network.
- Conclusions: The individual representations perform well on partially nonoverlapping object sets, which the authors associate with representation-dependent learned features.
- Future work: Applying Next Best View prediction to volumetric representations could require fewer test orientations while maintaining similar classification accuracy.
- Future work: Signed or unsigned distance fields are proposed as alternative 3D representations for possible classification enhancement.