Source-linked AI summary
3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction
Christopher B. Choy, Danfei Xu, JunYoung Gwak, Kevin Chen, Silvio Savarese
TL;DR
Traditional reconstruction methods struggle with sparse, wide-baseline, textureless, or non-Lambertian views and often require annotations or calibration. 3D-R2N2 learns an image-to-voxel-shape mapping with a recurrent architecture, achieving strong single-view performance and reconstruction where SFM/SLAM fails. Its reported scope is limited for highly textured objects and highly detailed reconstructions from more than 30 views.
Problem
Correspondence-based 3D reconstruction is restricted by dense views, small baselines, texture, Lambertian appearance, and feature-matching assumptions.
Method
3D-R2N2 uses a recurrent neural network to learn an end-to-end mapping from one or more object images to a 3D occupancy grid with minimal supervision.
Results
3D-R2N2 outperforms a state-of-the-art single-view method and reconstructs objects in settings where traditional SFM/SLAM methods fail.
Takeaways & Limitations
The unified framework supports single- and multi-view reconstruction from arbitrary viewpoints without requiring image annotations or object class labels.
Takeaways & Limitations
The method reconstructs fewer details than MVS beyond 30 views and performs worse on highly textured objects, partly because ShapeNet training models are mostly low-texture.
Abstract
from arXiv · showhide
Inspired by the recent success of methods that employ shape priors to achieve robust 3D reconstructions, we propose a novel recurrent neural network architecture that we call the 3D Recurrent Reconstruction Neural Network (3D-R2N2). The network learns a mapping from images of objects to their underlying 3D shapes from a large collection of synthetic data. Our network takes in one or more images of an object instance from arbitrary viewpoints and outputs a reconstruction of the object in the form of a 3D occupancy grid. Unlike most of the previous works, our network does not require any image annotations or object class labels for training or testing. Our extensive experimental analysis shows that our reconstruction framework i) outperforms the state-of-the-art methods for single view reconstruction, and ii) enables the 3D reconstruction of objects in situations when traditional SFM/SLAM methods fail (because of lack of texture and/or wide baseline).
1 Introduction
3D-R2N2 addresses reconstruction settings that challenge correspondence-based methods by learning 3D shape directly from images, unifying single- and multi-view reconstruction with minimal supervision. It accepts arbitrary-view image sequences and incrementally refines voxelized reconstructions, outperforming a state-of-the-art single-view method and operating where traditional SFM/SLAM methods fail.
- Motivation: Traditional reconstruction methods often require dense, small-baseline views, Lambertian appearance, rich texture, or accurate feature correspondences.Large baselines, self-occlusions, textureless surfaces, and specular reflections make correspondence estimation difficult.
- Motivation: Shape-prior methods reduce reliance on cross-view correspondences and can operate with fewer images and fewer reflectance assumptions.Priors may be represented by simple 3D primitives or learned from 3D CAD repositories.
- Approach: 3D-R2N2 learns an end-to-end mapping from object images to underlying 3D shapes using deep convolutional and recurrent networks.The approach learns appropriate intermediate representations from a large collection of training data rather than matching a hand-designed shape prior.
- Approach: The network accepts one or more images from arbitrary, uncalibrated viewpoints and outputs a voxelized 3D occupancy reconstruction.The reconstruction is incrementally refined as additional views are processed.
- Contributions: 3D-R2N2 unifies single- and multi-view reconstruction while requiring no segmentation, keypoints, viewpoint labels, camera calibration, or object class labels.The stated supervision requirement is limited to bounding boxes, with no image annotations or class labels needed for training or testing.
- Results: The framework outperforms a state-of-the-art single-view reconstruction method and reconstructs objects in settings where traditional SFM/SLAM methods fail.The failure settings involve lack of texture and wide baselines.
2 Recurrent Neural Network
The paper reviews recurrent units that regulate information flow through hidden states. It contrasts LSTM’s memory cell and separate gates with GRU’s fewer computations, combined update gate, and reset gate.
- LSTM: LSTM explicitly controls information flow between inputs and outputs, helping the network overcome the vanishing-gradient problem.An LSTM contains a memory cell, hidden state, input gate, output gate, and forget gate.
- LSTM: The LSTM memory cell combines retained prior state with gated candidate information at each time step.The input and forget gates regulate the contributions to the memory cell.
- LSTM: LSTM gates and states are defined as input, forget, and output gates, plus memory cell and hidden state, with matrices transforming current and previous activations.Element-wise multiplication combines gated quantities, while biases complete the transformations.
- GRU: GRU reduces computation relative to standard LSTM by using one update gate for both input and forget functions.It also applies a reset gate before the nonlinear transformation.
- GRU: The GRU hidden state interpolates between the previous hidden state and a transformed current input controlled by update and reset gates.The reset gate modulates the previous hidden state before nonlinear transformation.
3 3D Recurrent Reconstruction Neural Network
3D-R2N2 combines a 2D image encoder, spatially structured recurrent units, and a 3D decoder to reconstruct object occupancy from single or multiple views. Its recurrence retains prior observations and selectively updates spatial regions as new views arrive.
- 3 3D Recurrent Reconstruction Neural Network: The architecture includes shallow and deep residual encoder variants, plus 3D-LSTM and GRU recurrence variants.The residual encoder uses identity mappings and 1 × 1 convolutions to match channels in residual connections.
- 3 3D Recurrent Reconstruction Neural Network: The network performs both single- and multi-view 3D reconstruction using a 2D-CNN, 3D-LSTM recurrence module, and 3D-DCNN decoder.The encoder maps each image to low-dimensional features, recurrence processes them, and the decoder produces the reconstruction.
- 3 3D Recurrent Reconstruction Neural Network: Input gates and forget gates let the recurrence selectively update or retain memory as potentially conflicting information from different viewpoints becomes available.This supports updating previously occluded regions while retaining states for other object parts.
- 3 3D Recurrent Reconstruction Neural Network: The 3D-LSTM is arranged as spatially distributed units, each responsible for reconstructing a particular part of the output voxel space.Each unit has an independent hidden state, while restricted connections link it to spatial neighbors.
- 3 3D Recurrent Reconstruction Neural Network: Removing output gates reduces parameters because the network extracts its output only at the end.The implementation uses a 3D-LSTM grid with spatial resolution N = 4.
- 3 3D Recurrent Reconstruction Neural Network: After an image sequence, the decoder upsamples the final hidden state through 3D convolutions, nonlinearities, and unpooling to the target resolution.Voxel-wise softmax converts the final two-channel activation into occupancy probabilities, trained with summed voxel-wise cross-entropy.
4 Implementation
Training uses rendered 3D CAD models and corresponding voxel occupancy maps, with randomized image augmentation and viewpoints. Input sequences have variable lengths beginning with one image.
- 4 Implementation: Training data consist of rendered 3D CAD-model images paired with ground-truth voxel occupancy maps.Inputs use transparent-background renders augmented with random PASCAL VOC 2012 crops, color tinting, and translations.
- 4 Implementation: Viewpoints are sampled randomly, and training uses variable-length input sequences ranging from one image onward.
5 Experiments
Experiments evaluate architecture variants, single-view reconstruction, multi-view reconstruction, real-world images, and comparisons with MVS. Results show improved reconstruction with more views, strong single-view performance, and robustness where MVS fails, alongside limits on highly textured objects and very many views.
- 5.2 Network Structures Comparison: Experiments compare five 3D-R2N2 variants using five ShapeNet views and evaluate reconstruction with cross-entropy loss and voxel IoU.The variants use GRU or LSTM units, different convolution kernels, and a residual architecture.
- 5.2 Network Structures Comparison: GRU units, 3 × 3 × 3 recurrent connections, and the deep residual architecture each improve reconstruction performance over their alternatives.The reported comparison finds GRU-based networks outperform LSTM-based networks, neighboring recurrent connections outperform 1 × 1 × 1 connections, and residual networks further boost performance.
- 5.3 Single Real-World Image Reconstruction: Our approach outperforms Kar et al. in every PASCAL VOC category despite that method receiving ground-truth segmentation masks and keypoint labels.The network also trains and reconstructs without knowing the object category and does not require those annotations as inputs.
- 5.4 Multi-view Reconstruction Evaluation: Reconstruction quality improves as the number of ShapeNet views increases, although the marginal gain decreases as additional views provide less information.Quality improves for every category, but performance varies across categories; bulky, less variable classes such as cabinets, cars, and speakers perform best.
- 5.4 Multi-view Reconstruction Evaluation: The network refines reconstructions as additional views arrive, correcting plausible but incorrect initial interpretations in examples such as trucks and couches.This behavior appears in both ShapeNet testing examples and real-world Online Products images trained using synthetic data.
- 5.6 Multi View Stereo(MVS) vs. 3D-R2N2: Our model works with one view and across texture levels, while MVS fails completely below 20 views with IoU=0.The comparison uses identical image sets and evaluates voxel IoU; the network was fine-tuned with samples containing at most 24 views.
- 5.6 Multi View Stereo(MVS) vs. 3D-R2N2: With more than 30 views, our method reconstructs fewer details than MVS and performs worse on highly textured objects because ShapeNet training models are mostly low-texture.The method was fine-tuned only on samples with a maximum of 24 views.
6 Conclusion
The proposed architecture unifies single- and multi-view 3D reconstruction, improving reconstructions with additional views and handling settings where MVS fails.
- The network unifies single- and multi-view 3D reconstruction within one framework.
- It outperforms Kar et al. for single-view reconstruction using real-world images.
- Reconstructions improve incrementally as the network receives more object views.
- The method produces accurate reconstructions when MVS fails, including images with insufficient texture or wide-baseline viewpoints.
- The network produces plausible reconstructions without requiring a minimum number of input images.