Source-linked AI summary
3D Object Reconstruction from a Single Depth View with Adversarial Learning
Bo Yang, Hongkai Wen, Sen Wang, Ronald Clark, Andrew Markham, Niki Trigoni
TL;DR
The paper tackles the challenge of recovering complete 3D geometry from a single arbitrary depth view despite occlusion, ambiguity, and high-resolution modeling difficulty. It proposes 3D-RecGAN, combining an autoencoder with conditional adversarial learning to generate complete occupancy grids. Experiments report accurate fine-grained reconstructions, state-of-the-art improvements, and plausible shapes for unseen categories.
Problem
Single-view 3D reconstruction must infer occluded geometry from incomplete depth observations, while existing methods often operate at low resolution or require additional views or information.
Method
3D-RecGAN combines an autoencoder and conditional GAN to map a single occupancy-grid depth view to a complete 3D occupancy grid.
Results
The method outperforms the state of the art in single-view shape completion and produces plausible complete shapes for multiple and previously unseen object categories.
Takeaways & Limitations
The results support reconstructing accurate, fine-detailed 3D shapes from one depth view without object class labels during training or testing.
Takeaways & Limitations
Overall reconstruction metrics do not adequately evaluate geometric details, so the paper supplements them with qualitative visualizations.
Abstract
from arXiv · showhide
In this paper, we propose a novel 3D-RecGAN approach, which reconstructs the complete 3D structure of a given object from a single arbitrary depth view using generative adversarial networks. Unlike the existing work which typically requires multiple views of the same object or class labels to recover the full 3D geometry, the proposed 3D-RecGAN only takes the voxel grid representation of a depth view of the object as input, and is able to generate the complete 3D occupancy grid by filling in the occluded/missing regions. The key idea is to combine the generative capabilities of autoencoders and the conditional Generative Adversarial Networks (GAN) framework, to infer accurate and fine-grained 3D structures of objects in high-dimensional voxel space. Extensive experiments on large synthetic datasets show that the proposed 3D-RecGAN significantly outperforms the state of the art in single view 3D object reconstruction, and is able to reconstruct unseen types of objects. Our code and data are available at: https://github.com/Yang7879/3D-RecGAN.
1. Introduction
The paper addresses complete 3D reconstruction from a single arbitrary depth view, where occlusion and ambiguity make conventional methods inadequate. 3D-RecGAN combines autoencoding and conditional adversarial learning to predict high-resolution shapes and reports strong reconstruction and generalization results.
- Motivation: Single-view depth reconstruction is needed because scanning multiple object surfaces is often infeasible, leaving occluded regions in incomplete models.The problem is relevant to AR/VR, semantic understanding, robot grasping, and obstacle avoidance.
- Problem: A single partial depth observation can correspond to infinitely many 3D models, while interpolation methods recover only limited holes or missing regions.The cited limitations include quantization artifacts, sensor noise, and insufficient geometry information.
- Problem: Existing deep reconstruction methods typically use less than 40^3 voxel grids, producing coarse shapes and making higher-resolution learning difficult.Increasing resolution exponentially expands the search space of possible 2.5D-to-3D mappings and complicates neural-network convergence.
- Method: 3D-RecGAN encodes a single 2.5D view into a latent vector, decodes a likely complete shape, and refines it with a conditional discriminator.The model combines an autoencoder with GAN-based plausibility assessment for full 3D reconstruction.
- Method: The model uses only an occupancy-grid input without class labels or annotations and predicts a 64^3 voxel grid.The contribution describes the model as end-to-end trainable and broadly applicable.
- Method: The latent-distribution discriminator stabilizes GAN training, whereas standard binary discriminator outputs can cause training to crash easily.The discriminator-based refinement is applied to high-dimensional real or reconstructed 3D shapes.
- Results: Extensive synthetic-data experiments report performance better than the state of the art and reconstruction of previously unseen object categories.The evaluation uses virtually scanned 3D CAD models because real single-view scans lack complete ground truth for occluded regions.
2. Related Work
Prior work spans geometric fitting, multi-view reconstruction, and single-view completion. These approaches face limitations involving handcrafted shape assumptions, multiple observations, specialized encodings, low resolution, or missing adversarial refinement.
- 3D Model/Shape Fitting: Geometric fitting methods complete small missing regions or holes using plane fitting, symmetry, or other predefined regularities.Their structure space is limited to hand-crafted shapes, and large missing or occluded regions remain difficult.
- 3D Model/Shape Fitting: Database-based fitting methods assume identical or very similar shapes, limiting generalization to novel objects or categories.This assumption constrains applicability beyond the available shape database.
- Multi-view Reconstruction: Traditional and learned multi-view methods recover 3D structure from collections of images but can fail on featureless regions such as white walls.Their pipelines use feature matching, reprojection-error minimization, or deep networks operating on multiple images.
- Single-view Reconstruction: Single-view methods include templates, random forests, deep voxel predictors, point-cloud generators, denoising networks, and skeleton estimators.The surveyed methods differ in output and efficiency, with some producing multiple solutions or failing to recover complete accurate geometry.
- Single-view Reconstruction: Some higher-resolution approaches use tree structures or Hilbert Maps, but their networks contain only 3D encoders and decoders without adversarial learning.The passage contrasts these architectures with approaches that exploit adversarial refinement.
3. 3D-RecGAN
3D-RecGAN reconstructs a complete 3D occupancy grid from one arbitrary 2.5D depth view by combining an autoencoder generator with conditional adversarial learning. Its latent-distribution discriminator and joint objectives are designed to preserve geometric detail and stabilize training.
- 3.1. Overview: The method represents input and output as aligned 64^3 occupancy grids, mapping a single 2.5D depth view to a complete 3D shape.Occupied voxels are encoded as 1 and empty voxels as 0; training pairs are generated by virtually scanning ModelNet40 objects.
- 3.2. Architecture: The autoencoder generator learns correlations between partial and complete structures, then produces a rough reconstruction conditioned on the input view.It uses a U-net-style architecture with encoder–decoder skip connections.
- 3.2. Architecture: Conditional adversarial learning refines the autoencoder output by comparing input-conditioned fake reconstruction pairs with corresponding real reconstruction pairs.The discriminator evaluates plausibility while jointly updating the generator during training.
- 3.2. Architecture: The discriminator outputs a latent vector for real and fake reconstruction distributions instead of a binary label, helping capture geometric details in high-dimensional voxel data.The model uses WGAN-GP losses; standard binary discrimination is described as prone to GAN crashes and weak geometric feedback.
- 3.3. Objectives: The generator combines modified reconstruction loss with adversarial loss, while training alternates one discriminator update and one generator update.The reconstruction loss penalizes false positives more heavily because most voxel-grid cells are empty; training uses Adam with batch size 8.
- 3.5. Data Synthesis: Synthetic training data addresses the lack of real RGB-D datasets pairing individual depth views with complete ground-truth 3D structures.The authors note that existing real datasets contain occlusions and missing data, while 3D-EPN uses TSDF encoding unlike this network.
4. Evaluation
The evaluation compares 3D-RecGAN with alternative reconstruction methods using voxel IoU, cross-entropy, and qualitative results across per-category, multi-category, and cross-category settings. Results show accurate, detailed reconstructions and strong generalization across categories.
- 4.1. Metrics: Evaluation uses voxel IoU and cross-entropy loss for overall reconstruction, supplemented by qualitative visualizations because geometric details are not fully captured by these metrics.IoU increases with reconstruction quality, while lower cross-entropy indicates better prediction.
- 4.2. Comparison: 3D-RecGAN is compared with Poisson surface reconstruction, Varley et al.’s deep method, and an autoencoder-only variant without the GAN.The comparisons target traditional completion, a related deep-learning baseline, and the contribution of adversarial training.
- 4.2. Comparison: Per-category experiments separately train and test networks on three object categories using identical network configurations, with quantitative and qualitative comparisons.Table 1 reports IoU and cross-entropy, while Figure 6 shows reconstructed results from different approaches.
- 4.2. Comparison: Multi-category experiments train and test on multiple categories without class labels to assess model generality.Table 2 reports IoU and cross-entropy, and Figure 7 provides qualitative results.
- 4.2. Comparison: Cross-category experiments train on one category and test on five others, using three train-category groups to examine generalization to different object types.The groups train on chairs, stools, or toilets, with testing performed on the remaining listed categories.
- 4.2. Comparison: Across per-category and multi-category experiments, 3D-RecGAN completes partial views with accurate structures and fine-grained details while outperforming the state of the art by a large margin.Cross-category results also show strong performance and support learning geometric features correlated across object categories.
5. Conclusion
The paper concludes that 3D-RecGAN reconstructs complete 3D shapes from a single arbitrary depth view. It also reports satisfactory multi-category results without class labels and plausible reconstructions for unseen categories.
- 5. Conclusion: 3D-RecGAN reconstructs full 3D object structure from an arbitrary depth view and outperforms Poisson reconstruction and Varley et al.’s method for single-view category-specific completion.The framework predicts accurate structures with fine details.
- 5. Conclusion: The network predicts satisfactory 3D shapes across multiple categories without object class labels during training or testing.This extends the method beyond category-specific reconstruction.
- 5. Conclusion: The network produces plausible shapes for unseen object categories, supporting its ability to learn general 3D latent features rather than merely fit training datasets.The conclusion characterizes this setting as challenging.