Source-linked AI summary
Learning Shape Priors for Single-View 3D Completion and Reconstruction
Jiajun Wu, Chengkai Zhang, Xiuming Zhang, Zhoutong Zhang, William T. Freeman, Joshua B. Tenenbaum
TL;DR
Single-view 3D completion and reconstruction are underdetermined because multiple realistic shapes may fit one image, while supervised models tend toward blurry mean shapes. ShapeHD combines volumetric convolutional networks with adversarially learned shape priors, and experiments report state-of-the-art performance across both tasks, with limitations in evaluation data and difficult object structures.
Problem
Single-view observations can correspond to multiple realistic 3D shapes, so fully supervised models often produce blurry mean shapes instead of modeling this ambiguity.
Method
ShapeHD combines deep volumetric convolutional networks with adversarially learned shape priors that penalize unrealistic outputs rather than ground-truth deviations.
Results
ShapeHD achieves state-of-the-art results on single-view 3D shape completion and reconstruction across multiple synthetic and real datasets.
Takeaways & Limitations
Learned shape priors let ShapeHD learn from multiple hypotheses explaining a single-view observation while producing detailed, realistic shapes.
Takeaways & Limitations
Evaluation on PASCAL 3D+ can be inaccurate because each category has only around 10 CAD-model ground-truth shapes, and the model has difficulty with deformable, uncommon, and very thin parts.
Abstract
from arXiv · showhide
The problem of single-view 3D shape completion or reconstruction is challenging, because among the many possible shapes that explain an observation, most are implausible and do not correspond to natural objects. Recent research in the field has tackled this problem by exploiting the expressiveness of deep convolutional networks. In fact, there is another level of ambiguity that is often overlooked: among plausible shapes, there are still multiple shapes that fit the 2D image equally well; i.e., the ground truth shape is non-deterministic given a single-view input. Existing fully supervised approaches fail to address this issue, and often produce blurry mean shapes with smooth surfaces but no fine details. In this paper, we propose ShapeHD, pushing the limit of single-view shape completion and reconstruction by integrating deep generative models with adversarially learned shape priors. The learned priors serve as a regularizer, penalizing the model only if its output is unrealistic, not if it deviates from the ground truth. Our design thus overcomes both levels of ambiguity aforementioned. Experiments demonstrate that ShapeHD outperforms state of the art by a large margin in both shape completion and shape reconstruction on multiple real datasets.
1 Introduction
Single-view 3D completion and reconstruction remain ambiguous because multiple natural shapes can explain one observation, causing supervised models to produce mean shapes. ShapeHD addresses this with adversarially learned shape priors and reports stronger results across both tasks.
- Motivation: People’s preferences diverged on two examples, with 59% favoring airplane interpretation A and 35% favoring car interpretation A.The authors interpret the divergence as indicating that the competing reconstructions were close in quality.
- Motivation: Existing deep-network methods reduce implausible-shape ambiguity, but supervised learning does not model multiple realistic shapes that fit one view equally well.Because the ground truth is non-deterministic, supervised networks tend to generate blurry mean shapes that minimize their penalty.
- Approach: ShapeHD combines deep volumetric convolutional networks with adversarially learned shape priors that penalize unrealistic outputs rather than deviations from ground truth.This design targets ambiguity among plausible reconstructions.
- Results: Experiments on multiple synthetic and real datasets report better single-view shape completion and reconstruction than state-of-the-art systems.The analyses also report meaningful object-part attention and progressive recovery of shape details through the naturalness module.
2 Related Work
Prior work uses geometric methods, shape databases, deep networks, 2.5D sketches, and adversarial models for 3D completion and reconstruction. ShapeHD combines 2.5D sketch modeling with a naturalness loss to address ambiguity and improve detail quality.
- 3D shape completion: Traditional shape completion methods use local surface primitives or optimization, while newer methods exploit shape structures, regularities, and database priors.Database-based methods can require exact shape parts, limiting generalization.
- 3D shape completion: Data-driven approaches apply deep models to voxel completion, generative shape modeling, mesh repair, point clouds, octave trees, and tactile signals.Examples include Voxlets, 3D ShapeNets, and related extensions.
- 3D shape completion: 3D-EPN combines 3D convolutions with nonparametric patch synthesis for partial-depth completion, whereas ShapeHD uses naturalness losses to select among plausible detailed hypotheses.The paper also contrasts ShapeHD’s single-pass completion with 3D-EPN’s post-processing.
- Single-image 3D reconstruction: Single-image reconstruction methods require both recognition and shape priors, including approaches based on CAD parts, large repositories, 2.5D sketches, and adversarial models.The paper positions its approach as inspired by 2.5D sketch reconstruction while adding a naturalness loss.
3 Approach
ShapeHD estimates 2.5D sketches, completes a voxelized 3D shape, and regularizes it with an adversarially pretrained naturalness model. The completion network is fine-tuned using supervised voxel and naturalness losses after separate pretraining.
- 3 Approach: A deep naturalness model penalizes predicted shapes judged unnatural, complementing the supervised reconstruction loss.This regularizer is intended to avoid unrealistic mean shapes caused by single-view uncertainty.
- 3 Approach: The 2.5D sketch estimator predicts depth, surface normals, and silhouette from an RGB image, which are masked and passed to the 3D completion network.The completion network predicts a canonical-view 3D shape from these sketches.
- Shape Naturalness Network: The naturalness network is pretrained as a 3D GAN whose discriminator models the real shape distribution and becomes the naturalness loss.Its generator is not used during later completion-network training.
- Shape Naturalness Network: Wasserstein GAN loss with gradient penalty is used because high-dimensional 3D-shape GAN training is highly unstable.The loss compares generated and real shape distributions and includes a gradient-penalty term.
- Training Paradigm: The model is trained in two stages: separate component pretraining followed by completion-network fine-tuning with voxel and naturalness losses.The combined objective is L = Lvoxel + αLnatural.
- Training Paradigm: Jointly training the naturalness and completion networks from scratch produces unstable outputs, so the authors use pretraining followed by fine-tuning.The paper attributes the instability to large Wasserstein GAN losses and gradients.
4 Single-View Shape Completion
ShapeHD completes single-view depth inputs using a 3D shape estimator and deep naturalness network, producing detailed and plausible shapes. It improves over 3D-EPN qualitatively and quantitatively, including on real scans where camera parameters are unavailable.
- 4 Single-View Shape Completion: ShapeHD uses the 3D shape estimator and deep naturalness network for single-view depth completion.The completion setting uses only the model’s last two modules.
- 4 Single-View Shape Completion: The model predicts high-quality, plausible shapes with fine details and can infer parts absent from strongly occluded depth inputs.Qualitative results show large shape variance rather than the few details typical of mean-shape predictions.
- 4 Single-View Shape Completion: Adversarially trained naturalness losses fix errors, add object details, and smooth planar surfaces compared with the model without those losses.Examples include plane wings, car seats, chair arms, and a sofa back; the results also contain more details than 3D-EPN.
- 4 Single-View Shape Completion: ShapeHD outperforms the state of the art by a large margin on ShapeNet completion, while naturalness losses consistently improve Chamfer Distance.The comparison reports average IoU and Chamfer Distance at 32^3 resolution, with ShapeHD predictions downsampled for fairness.
- 4 Single-View Shape Completion: On physical-scanner chair depth maps, ShapeHD completes shapes well from a single view without requiring camera intrinsics or extrinsics.The real scans use depth-derived normal maps; 3D-EPN cannot be applied when the camera parameters are unknown.
5 3D Shape Reconstruction
ShapeHD is evaluated for single-view 3D reconstruction on synthetic and real datasets, where it generally produces detailed shapes and outperforms competing methods quantitatively and behaviorally.
- ShapeNet and novel-category results: 0.108 average CD is reported for ShapeHD, compared with 0.126 for DRC (3D) and 0.139 for AtlasNet.
- ShapeNet and novel-category results: ShapeHD predicts detailed 3D reconstructions that closely resemble ground truth on ShapeNet, with stronger quantitative performance than competing methods.On novel categories, it outperforms DRC and is comparable to AtlasNet, although AtlasNet receives ground-truth silhouettes while ShapeHD uses raw images.
- Real-data results: ShapeHD recovers reasonable shapes under strong self-occlusions on PASCAL 3D+ and Pix3D, including fine details.The evaluations use models trained with full 3D supervision on rendered ShapeNet images.
- Quantitative comparison: ShapeHD performs significantly better than other methods in almost all reported metrics, except CD for PASCAL 3D+ cars, where OGN performs best.The paper cautions that sparse CAD-model ground truth may make PASCAL 3D+ labels and scores inaccurate.
- Real-data results: 64.5% of user judgments preferred ShapeHD reconstructions to OGN’s on PASCAL 3D+.Users viewed the input image and two reconstructions, each shown from two views.
6 Analyses
Analyses show that ShapeHD learns detectors for meaningful object-part and generic shape patterns, while naturalness losses add details during training; thin or unusual structures remain difficult.
- Network visualization: ShapeHD learns detectors responding to meaningful object parts and generic shape patterns across categories.Visualized examples include wheels, chair parts, airplane engines, curved surfaces, and strided patterns.
- Training with naturalness loss over time: Naturalness-loss training adds fine details over time, even as voxel loss decreases slowly and consistently.The paper attributes this to details occupying small voxel-grid regions that supervised loss alone is unlikely to recover.
- Failure cases: The model sometimes confuses deformable parts, misses uncommon parts, and substitutes other patterns for very thin structures.Examples include wheels, a ring above wheels, and thin structures shown in the failure-case visualization.
- Failure cases: Voxel representations enable naturalness losses but encourage attention toward thicker parts because they contribute more loss weight.
7 Conclusion
The paper concludes that learned shape priors address single-view 2D–3D ambiguity and allow learning from multiple hypotheses, yielding state-of-the-art completion and reconstruction results.
- Learned shape priors are used to overcome 2D–3D ambiguity and learn from multiple hypotheses explaining a single-view observation.
- ShapeHD achieves state-of-the-art results on 3D shape completion and reconstruction.The authors identify ambiguity behind partial observations as a direction for further research.