Source-linked AI summary
VConv-DAE: Deep Volumetric Shape Learning Without Object Labels
Abhishek Sharma, Oliver Grau, Mario Fritz
TL;DR
Noisy and incomplete depth scans motivate learning volumetric shape representations without object labels. The paper proposes an end-to-end fully convolutional volumetric auto-encoder that predicts voxel occupancy grids from corrupted inputs. It reports stronger denoising and shape-completion results than prior work, competitive classification, and promising interpolation.
Problem
Noisy, occluded, and incomplete depth scans make 3D geometry acquisition difficult, while prior deep representations are tied to object labels.
Method
VConv-DAE learns volumetric representations by predicting missing voxels with an end-to-end fully convolutional auto-encoder.
Results
VConv-DAE outperforms prior supervised ShapeNet work on denoising and shape completion, achieves competitive classification, and shows promising interpolation.
Takeaways & Limitations
The learned embedding supports recognition and interpolation while the label-free model provides stronger denoising and completion results.
Takeaways & Limitations
Evaluation relies on CAD models with simulated noise because ground-truth real-world missing parts and holes are unavailable.
Abstract
from arXiv · showhide
With the advent of affordable depth sensors, 3D capture becomes more and more ubiquitous and already has made its way into commercial products. Yet, capturing the geometry or complete shapes of everyday objects using scanning devices (e.g. Kinect) still comes with several challenges that result in noise or even incomplete shapes. Recent success in deep learning has shown how to learn complex shape distributions in a data-driven way from large scale 3D CAD Model collections and to utilize them for 3D processing on volumetric representations and thereby circumventing problems of topology and tessellation. Prior work has shown encouraging results on problems ranging from shape completion to recognition. We provide an analysis of such approaches and discover that training as well as the resulting representation are strongly and unnecessarily tied to the notion of object labels. Thus, we propose a full convolutional volumetric auto encoder that learns volumetric representation from noisy data by estimating the voxel occupancy grids. The proposed method outperforms prior work on challenging tasks like denoising and shape completion. We also show that the obtained deep embedding gives competitive performance when used for classification and promising results for shape interpolation.
1 Introduction
The paper addresses noisy and incomplete 3D scans by learning volumetric shape representations without object labels. It proposes an end-to-end convolutional auto-encoder and reports advantages for completion, denoising, recognition, and interpolation.
- Noisy, occluded, or incomplete scans make 3D geometry acquisition difficult and hinder VR/AR interaction, robotic planning, printing, and manufacturing.
- Existing CAD-shape methods often rely on part annotations, while ShapeNet learns shape distributions from raw voxel data using object-label-based modeling.
- VConv-DAE learns deep volumetric shape embeddings through a fully convolutional auto-encoder trained without object labels.
- The method outperforms prior supervised ShapeNet work on denoising and shape completion, while achieving competitive classification and promising interpolation results.
- At test time, the method is at least two orders of magnitude faster than ShapeNet.
2 Related Work
Related work spans part-based synthesis, deep learning for 3D data, denoising auto-encoders, and learnable upsampling. VConv-DAE differs by learning voxel representations end to end without additional supervision.
- Part and Symmetry based Shape Synthesis: Part- and symmetry-based shape methods are typically class-specific and require expensive part annotations.
- Deep learning for 3D data: ShapeNet learns a probability distribution over class labels and voxel grids with convolutional RBMs, then fine-tunes the model for shape completion.
- Deep learning for 3D data: VConv-DAE is trained completely unsupervised by predicting missing voxels, producing embeddings useful for classification and interpolation.
- Denoising Auto-Encoders: Unlike stacked convolutional auto-encoders, the model is learned end to end without layer-wise fine-tuning or pre-training.
- Learnable Upsampling Layer: Learnable deconvolutional layers provide trainable upsampling for reconstructing spatial outputs.
3 Unsupervised Learning of Volumetric Representation by Completion
The method learns voxel-to-voxel volumetric representations by reconstructing missing occupancy from corrupted shapes. Its encoder produces a compact embedding, while deconvolutional layers reconstruct the input grid.
- The model learns shape distributions by predicting missing voxels and later uses the embedding for recognition, interpolation, denoising, and completion.
- Shapes are represented as binary voxel occupancy grids derived from CAD meshes, using a 24^3 voxel cube with padding.
- VConv-DAE: Fully Convolutional Denoising Auto Encoder: VConv-DAE maps a complete voxel grid to a reconstructed voxel grid through an end-to-end two-class auto-encoder whose labels are voxel occupancies, not object classes.
- VConv-DAE: Fully Convolutional Denoising Auto Encoder: The decoder uses 3D deconvolutions, which act as learnable local upsampling units until the original input size is restored.
- VConv-DAE: Fully Convolutional Denoising Auto Encoder: An input dropout layer augments volumetric data and helps avoid overfitting.
- VConv-DAE: Fully Convolutional Denoising Auto Encoder: The encoder uses 3D convolutions and a length-6912 bottleneck that serves as the shape embedding for classification.
- VConv-DAE: Fully Convolutional Denoising Auto Encoder: The architecture uses two encoder convolutional layers to extract robust features at multiple scales and capture correlations between voxels.
- Experiments use ModelNet CAD data, including a 10-class subset, and train the network end to end from scratch with dropout noise.
4 Experiments
The experiments evaluate unsupervised volumetric features for classification and shape interpolation, comparing them with prior 3D methods. The learned representation supports competitive classification and plausible interpolations.
- Experimental design: The experiments evaluate unsupervised representation learning for 3D classification and qualitative shape interpolation, alongside comparisons with ShapeNet and other descriptors.The network is also evaluated on its intended generative tasks in the broader experimental program.
- Classification: 75% accuracy is achieved by Ours-UnSup on 40 classes using a completely unsupervised representation.The bottleneck feature vector has 6912 dimensions and is classified with a linear SVM.
- Classification: Ours-FT outperforms ShapeNet on both 10 and 40 classes in the fine-tuned comparison.The comparison uses the same fine-tuning setup described for ShapeNet.
- Classification: On 40 classes, the proposed accuracy is only 3% below fully supervised VoxNet, while MVCNN achieves higher performance using rendered image views.VoxNet uses pooling layers suited to classification but not reconstruction, whereas the proposed architecture targets reconstruction and denoising.
- Shape interpolation: Linear interpolation between encoded source and target shapes produces mostly connected intermediate shapes and plausible transitions, including for highly non-convex shapes.The encoder maps shapes to 6912-dimensional vectors, and the decoder converts interpolated vectors back into volumes.
5 Denoising and Shape Completion
The denoising and completion experiments simulate sensor noise using complete CAD models because real-world ground truth for missing geometry is unavailable. The proposed network performs significantly better than the CAE and ShapeNet baselines, while completion remains harder than random-noise denoising.
- Experimental setting: Real-world denoising and completion lack ground-truth data, so experiments use complete CAD models with simulated noise characteristics.The study uses a 10-class ModelNet subset and trains on 9600 CAD models for these experiments.
- Metrics and baselines: The evaluation compares the proposed network with a no-noise convolutional autoencoder and ShapeNet using normalized voxel reconstruction error.The error counts differing voxels between reconstruction and the original, normalized by the voxel grid size.
- Results: The proposed network performs significantly better than both the CAE and ShapeNet on the reported denoising and completion evaluations.Qualitative results also show more complete reconstructions than ShapeNet for substantial missing regions.
- Results: Completion under slicing noise is more challenging than denoising random noise because the slicing perturbation removes structured portions of objects.The test noise is not seen during training, and 30% slicing noise removes a significant chunk of an object.
6 Qualitative Comparison
The qualitative comparison presents ground truth, noisy inputs, VConv-DAE reconstructions, and ShapeNet reconstructions for shape completion under slicing and random noise. VConv-DAE fills substantial missing object regions, while slicing noise is especially challenging and can smooth edges and boundaries.
- Comparison setup: Each comparison row shows ground truth, a noise-corrupted input, the network reconstruction, and the ShapeNet reconstruction.Table 6 covers 30% slicing noise, while Table 4 covers 50% random noise.
- Shape completion: VConv-DAE fills significant missing portions of objects compared with ShapeNet in the qualitative shape-completion results.
- Noise conditions: Slicing noise is the most challenging scenario, partly because the network was not trained for slicing noise.
- Reconstruction effects: Some reconstructions smooth out edges and boundaries to some extent.
- Runtime comparison: The runtime experiment compares training and testing times for VConv-DAE and ShapeNet on an Nvidia K40 GPU.
- Runtime comparison: 3ms test time makes VConv-DAE 200x faster than ShapeNet, partly because it requires no sampling at test time.
7 Conclusion and Future Work
The paper concludes that unsupervised volumetric representation learning by completion supports denoising, shape completion, recognition, and interpolation. It identifies deformable objects and larger scenes as directions for future work.
- Conclusion: The proposed unsupervised approach learns volumetric representations by completing voxel shapes.
- Conclusion: The learned embedding delivers comparable recognition results and promising shape-interpolation results.
- Conclusion: The method obtains stronger denoising and shape-completion results without labels.
- Future work: Future work will extend the approach to deformable objects and larger scenes.