PaperlayerView exact revision on arXiv

Source-linked AI summary

Volumetric and Multi-View CNNs for Object Classification on 3D Data

Charles R. Qi, Hao Su, Matthias Niessner, Angela Dai, Mengyuan Yan, Leonidas J. Guibas

arXiv:1604.03265v2cs.CVcs.AI

TL;DR

The paper addresses the performance gap between volumetric and multi-view CNNs for 3D object classification. It analyzes architecture and resolution, introduces improved networks and 3D multi-resolution filtering, and reports stronger performance for both CNN types while identifying resolution as a bottleneck.

  • Problem

    Existing volumetric CNN architectures underuse 3D representations, performing worse than multi-view CNNs despite the importance and growing availability of 3D data.

  • Method

    The paper analyzes network architecture and volume resolution, proposes two volumetric CNN architectures, and adds a 3D multi-resolution component to multi-view CNNs.

  • Results

    The proposed volumetric CNNs achieve comparable performance to multi-view CNNs at 30 × 30 × 30 resolution, while the multi-resolution component improves multi-view CNNs.

  • Takeaways & Limitations

    The analysis indicates that 3D resolution is likely a bottleneck for volumetric CNN performance and motivates efficient architectures that scale to higher resolutions.

  • Takeaways & Limitations

    The method assumes volumetric point clouds or meshes are already pre-segmented by 3D bounding boxes.

Abstract

from arXiv · show

3D shape models are becoming widely available and easier to capture, making available 3D information crucial for progress in object classification. Current state-of-the-art methods rely on CNNs to address this problem. Recently, we witness two types of CNNs being developed: CNNs based upon volumetric representations versus CNNs based upon multi-view representations. Empirical results from these two types of CNNs exhibit a large gap, indicating that existing volumetric CNN architectures and approaches are unable to fully exploit the power of 3D representations. In this paper, we aim to improve both volumetric CNNs and multi-view CNNs according to extensive analysis of existing approaches. To this end, we introduce two distinct network architectures of volumetric CNNs. In addition, we examine multi-view CNNs, where we introduce multi-resolution filtering in 3D. Overall, we are able to outperform current state-of-the-art methods for both volumetric CNNs and multi-view CNNs. We provide extensive experiments designed to evaluate underlying design choices, thus providing a better understanding of the space of methods available for object classification on 3D data.

1. Introduction

The paper studies 3D object classification, where volumetric CNNs trail multi-view CNNs despite the growing availability and importance of 3D data. It analyzes this gap and proposes improved volumetric and multi-view architectures, including evaluation on real-world reconstructions.

  • 3D object classification matters across vision applications, while improved scanning and model collection have increased access to 3D data.
  • Existing volumetric CNNs face computational complexity and data sparsity, and perform significantly worse than multi-view CNNs.
  • The method assumes pre-segmented volumetric point clouds or meshes and predicts the category label of each volumetric instance.
  • The paper analyzes network architecture and volume resolution, then proposes two volumetric CNN architectures to improve 3D shape classification.
  • The proposed volumetric networks use auxiliary part classification or long anisotropic kernels, with data augmentation and multi-orientation pooling improving both.
  • A new multi-resolution component improves multi-view CNN performance, while experiments examine design choices and adaptation from synthetic CAD data to real-world reconstructions.

2. Related Work

Related work spans handcrafted 3D shape descriptors, CNNs for RGB-D and volumetric data, and approaches that reuse 2D CNNs through rendered views. Volumetric and multi-view representations provide two common tensor-based inputs for 3D classification.

  • Earlier 3D shape descriptors use surface geometry, distances, angles, volumes, local diameters, heat kernels, or extensions of SIFT and SURF.
  • CNNs learned general-purpose image descriptors from large RGB datasets, advancing multiple vision tasks beyond handcrafted features.
  • Depth-aware CNN methods incorporate RGB-D information for feature learning, object detection, and related recognition tasks.
  • 3DShapeNets and VoxNet represent shapes with binary voxel grids, enabling CNN processing of LiDAR, RGB-D, and CAD data.
  • Multi-view approaches extract 2D data from 3D shapes, including panoramic or multiple rendered views processed jointly by CNNs.

3. Analysis of state-of-the-art 3D Volumetric CNN versus Multi-View CNN

The analysis attributes the volumetric-versus-multi-view performance gap mainly to input resolution and network architecture. Even when both networks receive similarly detailed inputs, the volumetric CNN remains weaker, motivating architectural improvements and multi-resolution filtering.

  • The two representations both support CNN training, but their classification performance differs substantially despite comparable encoded information.
  • A voxel-occupancy volumetric CNN is 7.3% worse than a multi-view CNN on ModelNet40 classification.
  • The proposed analysis identifies input resolution and network architecture as the two factors associated with the performance gap.
  • The multi-view CNN uses 227 × 227 rendered views, whereas the volumetric CNN uses a 30×30×30 occupancy grid to maintain similar computational cost.
  • Even with similar object detail from 30 × 30 × 30 sphere renderings, the volumetric CNN is 4.8% worse than the multi-view CNN.
  • A multi-view CNN reaches 89.5% accuracy at 30 × 30 × 30 resolution, indicating that low-frequency 3D information can support classification.

4. Volumetric Convolutional Neural Networks

The paper improves volumetric CNNs with new architectures, augmentation, and orientation pooling. The designs target overfitting, local feature use, long-range interactions, and sensitivity to object orientation.

  • Overview: The authors improve volumetric CNNs through new network structures, data augmentation, and feature pooling.
  • Network 1: Auxiliary Training by Subvolume Supervision: Auxiliary subvolume supervision predicts object labels from partial regions, encouraging local discriminative feature learning without additional annotations.
  • Data Augmentation and Multi-Orientation Pooling: Azimuth and elevation augmentation exposes networks to varied orientations, while multi-orientation pooling aggregates orientation-specific features for holistic object recognition.
  • Network 2: Anisotropic Probing: The anisotropic network projects 3D volumes to 2D through an X-ray-like mechanism before classification with an adapted NIN network.
  • Network 2: Anisotropic Probing: Anisotropic probing uses elongated kernels to capture global 3D structure and aggregate long-range interactions with fewer parameters than large isotropic kernels.
  • Network 2: Anisotropic Probing: The anisotropic architecture scales to higher resolutions while maintaining quadratic rather than cubic computation.

5. Multi-View Convolutional Neural Networks

The paper extends multi-view CNNs with 3D multi-resolution filtering and stronger viewpoint augmentation. This extension captures information across scales and is designed to improve robustness, including on real-world scans.

  • Multi-View CNNs: Multi-view CNNs render 3D shapes from multiple viewpoints, extract image features per view, and pool those features before classification.
  • Multi-Resolution Extension: 3D multi-resolution filtering captures information at multiple scales by rendering spheres at different volume resolutions.
  • Multi-Resolution Extension: Sphere-based discretization is view-invariant and helps regularize noise or irregularities in real-world scanned data relative to synthetic training data.
  • Data Augmentation: Training augmentation varies both azimuth and elevation, and AlexNet is used instead of VGG for efficiency.

6. Experiments

Experiments evaluate the proposed volumetric and multi-view CNNs on ModelNet40 and a new real-world reconstruction dataset. The proposed volumetric models close the performance gap with multi-view CNNs at 3D resolution 30, while multi-resolution filtering improves multi-view performance.

  • Datasets: The evaluation uses ModelNet40 and a new real-world reconstruction dataset, with classification accuracy reported on ModelNet40.ModelNet40 contains 12,311 models from 40 categories; the real-world benchmark contains 243 objects from 12 categories and is used as a test set.
  • Comparison with State-of-the-Art Methods: Both proposed volumetric CNNs significantly outperform state-of-the-art volumetric CNNs and match the multi-view CNN at 3D resolution 30 on ModelNet40.The compared volumetric models are SubvolumeSup and AniProbing, trained with data augmentation and multi-orientation pooling.
  • Comparison with State-of-the-Art Methods: The 3D multi-resolution multi-view CNN achieves state-of-the-art performance on ModelNet40.It combines fc7 features from sphere renderings at multiple 3D resolutions.
  • 3D Resolution: At tested resolutions, volumetric and multi-view CNN performance is on par, while multi-view performance increases as 3D resolution grows.The volumetric CNN is tested at 3D resolutions 10 and 30 because of computational cost.
  • Data Augmentation and Multi-Orientation Pooling: Combining azimuth and elevation augmentation with multi-orientation pooling improves classification accuracy by 4.8%, whereas translation jittering has only marginal influence.Azimuth-only augmentation improves accuracy by 1.4%; the combined result is 87.8% − 83.0% = 4.8%.
  • More Evaluations: On real-world reconstructions, the proposed networks outperform state-of-the-art methods, and 3D multi-resolution filtering is particularly effective.The authors suggest that low-resolution filtering may remove noisy micro-structures in real-world data.

7. Conclusion and Future work

The paper addresses 3D object classification with volumetric and multi-view CNNs, analyzing the performance gap through architecture and 3D resolution. Its proposed volumetric architectures match multi-view CNN performance at resolution 30, while higher-resolution volumetric processing remains an open direction.

  • The proposed volumetric CNN architectures outperform state-of-the-art volumetric CNNs and achieve comparable performance to multi-view CNNs at 30 × 30 × 30 resolution.
  • The analysis indicates that 3D resolution is likely a bottleneck for volumetric CNN performance.The authors identify efficient volumetric architectures that scale to higher resolutions as future work.

A.1. Batch Normalization

The appendix examines batch normalization for volumetric CNN training. Adding batch normalization improves the reported single-orientation classification accuracy.

  • Adding batch normalization raises single-orientation classification accuracy from 87.2% to 88.8% for the subvolume supervision model.The base network is 3D-NIN; complete results are reported in Table 5.

A.2. Spatial Transformer Networks

Spatial Transformer Networks are used to improve volumetric CNNs from single-orientation inputs, addressing the computational cost of preparing multiple views or orientations. The module predicts a transformation, generates a sampling grid, and transforms the input volume before CNN processing.

  • Multi-view or multi-orientation methods require multiple prepared inputs, making them computationally more expensive than single-input processing.
  • The spatial transformer regresses transformation parameters, generates a sampling grid, and transforms the occupancy volume before the original volumetric CNN.It is inserted directly after the data layer and before the volumetric CNN.
  • Figure 10 compares input and output voxel grids for table models, with point colors determined by depth.The visualization illustrates the spatial transformer's alignment effect across examples.

A.3. Details on Model Training

The paper trains volumetric and multi-view models using rotated voxelizations or rendered views, and evaluates them on synthetic and reconstructed real-world 3D data. Retrieval uses CNN features with nearest-neighbor search and a learned low-rank Mahalanobis metric.

  • Volumetric CNN training: 60 voxelizations per model augment training with uniformly sampled azimuth and elevation rotations.Azimuths are sampled from [0, 360] and elevations from [−45, 45], in degrees.
  • Volumetric CNN training: 20 orientations of each CAD-model occupancy grid are input to MO-VCNN during testing for class prediction.Azimuths are equally distributed, while elevation is uniformly sampled from [−45, 45].
  • Multi-view CNN training: MVCNN training renders 20 views per CAD model across ten azimuth angles and two elevation angles.The procedure renders ordinary or spherical models, using elevations of −30 and 30 degrees and four fixed point lights.
  • Multi-view CNN training: AlexNet is pretrained separately on ordinary and multi-resolution sphere renderings before initializing MVCNN for multi-view fine-tuning.The same pretrained-network initialization strategy is used for the multi-view model.
  • Data representations: The volumetric approaches generalize to signed and unsigned distance fields, with ModelNet classification accuracy differing by only around 0.5% to 1.0%.Signed distance fields come from virtual scanning with volumetric fusion, while distance fields are generated directly from model surfaces.
  • Evaluation and retrieval: Retrieval ranks models by L2 distance between CNN features and further optimizes performance with a low-rank Mahalanobis metric.The evaluation includes real-world models reconstructed from RGB-D frame sequences using Voxel Hashing and segmented in post-processing.
Loading 1604.03265v2…