Source-linked AI summary
3D ShapeNets: A Deep Representation for Volumetric Shapes
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, Jianxiong Xiao
TL;DR
Category recognition has made limited use of 3D shape, and recovering unseen structure from a single depth view remains a challenge. 3D ShapeNets learns voxel-based shape distributions and applies them to recognition, completion, and view planning. The paper reports significant improvements over existing approaches on recognition tasks and effectiveness for next-best-view prediction.
Problem
3D shape has been underused in category recognition, and a single-view depth map leaves the possible unseen 3D structure to be inferred.
Source: §1 · p. 1Method
3D ShapeNets uses a convolutional deep belief network to learn probability distributions over binary 3D voxels and apply them to recognition and shape completion from depth maps.
Source: §1 · p. 2, §3 · p. 3, §4.1 · p. 4Results
Takeaways & Limitations
The paper presents voxel-based shape learning as a way to jointly recognize and reconstruct objects from a single-view 2.5D depth map.
Source: §7 · p. 8When the 3D shape is fully observed, adding views will not reduce recognition uncertainty for algorithms based purely on 3D shapes, including ShapeNets.
Source: §4.2 · p. 5
Abstract
from arXiv · showhide
3D shape is a crucial but heavily underutilized cue in today's computer vision systems, mostly due to the lack of a good generic shape representation. With the recent availability of inexpensive 2.5D depth sensors (e.g. Microsoft Kinect), it is becoming increasingly important to have a powerful 3D shape representation in the loop. Apart from category recognition, recovering full 3D shapes from view-based 2.5D depth maps is also a critical part of visual understanding. To this end, we propose to represent a geometric 3D shape as a probability distribution of binary variables on a 3D voxel grid, using a Convolutional Deep Belief Network. Our model, 3D ShapeNets, learns the distribution of complex 3D shapes across different object categories and arbitrary poses from raw CAD data, and discovers hierarchical compositional part representations automatically. It naturally supports joint object recognition and shape completion from 2.5D depth maps, and it enables active object recognition through view planning. To train our 3D deep learning model, we construct ModelNet -- a large-scale 3D CAD model dataset. Extensive experiments show that our 3D deep representation enables significant performance improvement over the-state-of-the-arts in a variety of tasks.
1. Introduction
Although 3D shape is an important recognition cue, its use in category recognition has been limited by the lack of a generic representation. 3D ShapeNets learns voxel-based shape distributions to support recognition, completion, and next-view planning.
1. Introduction
- 3D shape had been largely absent from state-of-the-art category recognition, despite its importance as a cue, because a good generic representation was lacking.The availability of inexpensive 2.5D depth sensors renewed interest in depth-based recognition and the need for a strong 3D shape representation.Source: §1 · p. 1
- The paper targets shape completion from a single 2.5D depth map, where the unseen 3D structure may have multiple plausible forms.It also seeks to infer full volume without prior knowledge of object category or pose, and to select a subsequent view when initial recognition is uncertain.Source: §1 · pp. 1–2
- 3D ShapeNets represents shapes as probability distributions over binary voxel grids and uses a Convolutional Deep Belief Network to learn them.The approach is intended to learn complex shapes across categories and poses from raw 3D data, rather than hand-coding shapes by parts.Source: §1 · p. 2
- Experiments report generalization to real-world NYU depth data, significantly better single-view 2.5D recognition than existing approaches, and effectiveness for next-best-view prediction.The work also constructs ModelNet, a large-scale dataset of 3D CAD models.Source: §1 · p. 2
2. Related Work
Prior 3D shape methods often rely on part assembly, small shape variation, or expensive annotations, while reconstruction and depth-based approaches have other scope limits. The paper positions voxel-based deep learning and view planning as alternatives for broader 3D shape variation and active recognition.
2. Related Work
- Assembly-based models are limited to specific shape classes with small variation, and surface correspondence is a key challenge.The paper notes that part annotation is tedious and expensive, making assembly-based modeling cumbersome for varied objects.Source: §2 · p. 2
- Smooth interpolation or extrapolation methods address small scanning holes, while template-based methods depend on template quality and may lack different semantic interpretations.The passage distinguishes their limits for corrupted scanning input and reconstruction.Source: §2 · p. 3
- Earlier 2D deep models for depth maps use depth as an extra image channel, whereas ShapeNets models a shape distribution over a voxel grid.The paper describes its work as the first 3D deep learning model to its knowledge, adapting convolution to high-resolution voxels.Source: §2 · p. 3
- Active-recognition methods often plan views from 2D color, while the 3D problem can involve intra-class variation that instance-level assumptions [1, 2] exclude.ShapeNets also uses mutual information for next-best-view selection, but considers voxel-level contributions to reducing recognition uncertainty.Source: §2 · p. 3
3. 3D ShapeNets
3D ShapeNets models a shape as binary voxel variables and uses convolutional layers to learn their joint distribution. Its design shares weights without pooling, and its training combines layer-wise pretraining with generative fine-tuning.
3. 3D ShapeNets
- Shapes are binary voxel tensors, with 1 for voxels inside the mesh and 0 for empty space; convolution shares weights to limit parameters, while hidden-layer pooling is omitted to avoid added reconstruction uncertainty.The experiments use a 30 × 30 × 30 grid.Source: §3 · p. 3
- The architecture stacks three convolutional layers, a fully connected RBM, and a final associative-memory layer combining label and feature variables.The first three layers use 48 filters of size 6, 160 of size 5, and 512 of size 4, respectively; the fourth and fifth layers have 1200 and 4000 hidden units.Source: §3 · p. 4
- Training uses layer-wise pretraining followed by generative fine-tuning, with Contrastive Divergence for the first four layers and Fast Persistent Contrastive Divergence for the top layer.Fine-tuning resembles wake-sleep learning, ties the weights, and uses bottom-up data propagation in the wake phase and a persistent top-layer chain in the sleep phase.Source: §3 · p. 4
- First-layer pretraining ignores empty receptive fields and adds sparsity regularization, while the top-layer label units are duplicated tenfold to increase their significance.The authors report that ignoring empty-space learning signals resulted in more meaningful filters.Source: §3 · p. 4
4. 2.5D Recognition and Reconstruction
3D ShapeNets uses a learned joint distribution over voxel data and object categories to recognize objects and complete missing shape regions from a single depth view. For view planning, it evaluates candidate viewpoints by their potential to reduce recognition uncertainty.
4.1. View-based Sampling
- The model learns the joint distribution p(x, y) and estimates object category from observed free-space and surface voxels while treating occluded voxels as missing.The depth map is converted to a volumetric representation, and recognition estimates p(y|x_o).Source: §4.1 · p. 4
- Gibbs sampling alternates between sampling a category label and voxel values while clamping observed voxels to produce shape completions.The procedure uses 50 up-down iterations; multiple particles yield completions associated with potentially different classes, and the most frequently sampled class is selected.Source: §4.1 · pp. 4–5
4.2. Next-Best-View Prediction
- The planner selects from a finite list of candidate views the one with the greatest potential to reduce recognition uncertainty.It uses hypothesized shapes conditioned on current observations to simulate depth maps from different views and estimate that potential reduction.Source: §4.2 · p. 5
- After physically moving the camera to the candidate view and capturing another object surface, surfaces from previous views are merged into a new observation xo, allowing the view-planning scheme to run again.If the 3D shape is fully observed, adding more views will not help reduce recognition uncertainty for algorithms based purely on 3D shapes, including 3D ShapeNets.Source: §4.2 · pp. 5–6
5. ModelNet: A Large-scale 3D CAD Dataset
ModelNet was constructed to provide the broad collection of 3D shapes needed to train a representation that captures intra-class variation. It contains 151,128 CAD models across 660 categories, substantially more than the cited prior dataset.
5. ModelNet: A Large-scale 3D CAD Dataset
- ModelNet addresses the limited category variety and per-category example counts of previous CAD datasets, which constrained training for intra-class shape variation.The authors identify a large collection of 3D shapes as necessary for training a representation that captures intra-class variance.Source: §5 · p. 6
- The authors gathered CAD models from 3D Warehouse, Yobi3D, and the Princeton Shape Benchmark, then filtered category labels and removed irrelevant, unrealistic, or duplicate models.Categories were queried from the SUN database, and Mechanical Turk screening plus manual checks were used to clean the models.Source: §5 · p. 6
- 151,128 CAD models in 660 categories make ModelNet 22 times larger than the cited dataset of 6,670 models in 161 categories.Figure 5 shows examples of major categories and dataset statistics.Source: §5 · p. 6
6. Experiments
Experiments evaluate 3D ShapeNets on 3D shape classification and retrieval, single-view depth recognition and completion, and next-best-view planning. The reported results favor ShapeNets over the tested alternatives, including on NYU depth data and view selection.
6. Experiments
- The experiments train on 40 ModelNet categories, with 100 CAD models per category rotated into 12 poses.The classification and retrieval experiments use 38,400 training and 9,600 test models from the enlarged 48,000-model set.Source: §6 · p. 6, §6.1 · p. 6
6.1. 3D Shape Classification and Retrieval
- Classification compares learned ShapeNets features with LFD and SPH; retrieval evaluates test-shape similarity using precision-recall and mean average precision.The 3D classification evaluation uses average category accuracy.Source: §6.1 · pp. 6–7
- 3D ShapeNets outperforms LFD and SPH by a large margin despite using significantly lower-resolution meshes.The authors infer that ShapeNets learned rotation invariance during feature learning.Source: §6.1 · p. 7, Figure 7 · p. 7
6.2. View-based 2.5D Recognition
- For NYU recognition, the authors train on 4,899 CAD models from 10 categories and apply the model to segmented real-world depth data.They discriminatively fine-tune on NYU data, using 1,390 instances for training and 495 for testing.Source: §6.2 · p. 7
6.3. Next-Best-View Prediction
- Shape completions produce reasonable shapes across categories and are not merely nearest-neighbor copies of training examples.Figure 8 compares input depth maps, ground truth, completions, and nearest-neighbor results.Source: §6.3 · p. 7, Figure 8 · p. 7
6.2. View-based 2.5D Recognition
- Over 10%: fine-tuned 3D ShapeNets achieves the best NYU recognition accuracy, outperforming approaches using depth alone or depth and color.The comparison uses 495 test instances; the generative model performs reasonably well before fine-tuning.Source: §6.2 · p. 7, Table 2 · p. 8
6.3. Next-Best-View Prediction
- View planning evaluates eight candidate views on 200 test examples, selecting a second view to improve recognition accuracy.Baselines select randomly, maximize new visibility, or maximize camera-center distance from the first view.Source: §6.3 · p. 8
- Entropy-based view planning outperforms the other tested strategies in recognition accuracy from two views.The strategies use the same 3D ShapeNets recognizer.Source: §6.3 · p. 8, Table 3 · p. 8
7. Conclusion
The conclusion presents 3D ShapeNets as a voxel-grid probability model that jointly recognizes and reconstructs objects from single-view depth. It reports improved recognition across tasks and identifies next-best-view planning as a promising application.
7. Conclusion
- 3D ShapeNets represents shapes as voxel-grid probability distributions and jointly recognizes and reconstructs objects from single-view 2.5D depth maps.The paper introduces ModelNet, a large-scale 3D CAD model dataset, for training the model.Source: §7 · p. 8
- The authors report significant improvements over existing approaches across recognition tasks and describe the model as promising for next-best-view planning.The conclusion also notes that source code and the dataset are available on the project website.Source: §7 · p. 8