Source-linked AI summary
Learning to Reconstruct Shapes from Unseen Classes
Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Joshua B. Tenenbaum, William T. Freeman, Jiajun Wu
TL;DR
Single-image 3D reconstruction often relies on shape priors biased toward training classes, limiting generalization to unseen objects. GenRe separates geometric projections from learned reconstruction using depth, spherical maps, and voxel refinement. It achieves state-of-the-art reconstruction on both seen and unseen classes, while failing to exploit global symmetry for highly regular shapes.
Problem
Existing single-image reconstruction methods often use priors biased toward training classes, limiting generalization to unseen categories.
Method
GenRe combines depth estimation, spherical-map inpainting, and voxel refinement through fixed geometric projections and learnable reconstruction modules.
Results
GenRe achieves state-of-the-art performance on shape reconstruction for both seen and unseen classes.
Takeaways & Limitations
The integrated use of 2.5D sketches, spherical maps, and voxels supports generalizable, high-quality reconstruction across seen and unseen classes.
Takeaways & Limitations
GenRe fails to exploit global shape symmetry for highly regular shapes because its network does not explicitly model that regularity.
Abstract
from arXiv · showhide
From a single image, humans are able to perceive the full 3D shape of an object by exploiting learned shape priors from everyday life. Contemporary single-image 3D reconstruction algorithms aim to solve this task in a similar fashion, but often end up with priors that are highly biased by training classes. Here we present an algorithm, Generalizable Reconstruction (GenRe), designed to capture more generic, class-agnostic shape priors. We achieve this with an inference network and training procedure that combine 2.5D representations of visible surfaces (depth and silhouette), spherical shape representations of both visible and non-visible surfaces, and 3D voxel-based representations, in a principled manner that exploits the causal structure of how 3D shapes give rise to 2D images. Experiments demonstrate that GenRe performs well on single-view shape reconstruction, and generalizes to diverse novel objects from categories not seen during training.
1 Introduction
GenRe targets single-image 3D reconstruction beyond training classes by separating geometric projections from shape completion. Its modular representation combines depth, spherical maps, and voxel refinement, achieving state-of-the-art reconstruction within and outside training classes.
- Existing single-image methods often learn priors tied to training classes, limiting reconstruction of unseen categories.
- GenRe factorizes geometric projections from learnable shape reconstruction to improve generalization to unseen shape categories.The approach uses deterministic projections alongside learned completion modules.
- Spherical maps represent object surfaces in a 2D format that supports neural inpainting while retaining semantics for 3D shape recovery.
- The model cascades depth estimation, spherical-map inpainting, and voxel refinement through fixed geometric projections.Each module predicts within its input domain and uses features from the preceding module.
- GenRe achieves state-of-the-art performance on reconstructing shapes both within and outside training classes.Examples include tables and beds reconstructed after training on cars, chairs, and airplanes.
2 Related Work
Related work covers single-image reconstruction, spherical projections, 2.5D sketch recovery, and zero- or few-shot recognition. GenRe differs by studying 3D reconstruction from single RGB images for objects from unseen classes.
- Single-image 3D reconstruction: Single-image reconstruction methods recover 3D shape from one image using learned priors, but those priors generally apply mainly to training classes.
- Spherical projections: Prior spherical-projection research addressed retrieval, classification, and symmetry, while other work studied spherical convolution and rotational equivariance.
- 2.5D sketch recovery: 2.5D sketch recovery has been explored from texture, shading, color images, depth sensors, and RGB-D datasets.
- Zero- and few-shot recognition: Zero- and few-shot recognition methods share features across categories to recognize new objects from limited examples.
- Unseen-class reconstruction: Unlike recognition and earlier special-camera systems, this paper reconstructs 3D shapes from single RGB images of unseen classes.
3 Approach
GenRe reconstructs 3D shapes through depth estimation, spherical-map inpainting, and voxel refinement connected by geometric projections. This design uses class-agnostic geometric structure and spherical representations to complete unseen surfaces while addressing self-occlusion.
- GenRe factorizes the image-to-shape mapping into geometric projections and learnable reconstruction modules.
- Single-View Depth Estimator: The depth estimator predicts a visible-surface depth map from a color image, providing an intermediate geometric representation.
- Spherical Map Inpainting Network: A projection converts the depth map into a partial spherical map, which an inpainting network completes into an object-surface representation.
- Voxel Refinement Network: Voxel refinement integrates shapes projected from the inpainted spherical map and estimated depth to recover information lost through self-occlusion.
- Single-View Depth Estimator: Depth estimation is class-agnostic because different object classes can share geometric structures such as perpendicular planes.
- Spherical Map Inpainting Network: Spherical maps cast 3D surface completion as 2D inpainting, are efficient for sparse surfaces, and empirically improve performance.
- Technical Details: The implementation trains the depth estimator, spherical inpainting network, and voxel refinement network with their corresponding rendered or projected supervision.
4 Experiments
Experiments evaluate GenRe against voxel, mesh, point-cloud, multi-view, and spherical-map baselines on synthetic and real images. Results show strong generalization to unseen classes, with two-step geometric modeling outperforming the one-step variant.
- Baselines and evaluation: GenRe is compared with DRC, MarrNet, AtlasNet, Shin et al., and spherical-map and voxel-completion baselines.The evaluation uses Chamfer distance because predicted surface voxels are not guaranteed watertight, making IoU unsuitable.
- Depth estimation: Depth prediction does not significantly degrade as novel test classes become increasingly dissimilar from cars, chairs, and airplanes.The reported comparison covers buses, trains, and tables, with p > 0.05 for degradation.
- Novel-class reconstruction: GenRe outperforms the state of the art across 9 of 10 novel ShapeNet classes while matching object-centered AtlasNet on seen classes.The test classes are ordered from most to least similar to the training classes; rifles are the exception among novel classes.
- Novel-class reconstruction: On unseen classes, GenRe reconstructions remain consistent with input images, whereas competitors may omit structural details or retrieve shapes resembling training classes.The qualitative comparisons are shown in Figure 5 using the same training dataset for all models.
- Ablation: Two-step approaches, GenRe-2step and GenRe, outperform the one-step approach across all novel categories.This comparison supports separating geometric projections from shape reconstruction in the modeling strategy.
- Real-image reconstruction: On real Pix3D images, GenRe outperforms both competitors across all unseen classes except beds, while AtlasNet performs best on seen chairs.GenRe also preserves input details such as hollow structures in qualitative reconstructions.
5 Analyses
GenRe generalizes from training classes to unseen shapes, but reconstruction quality depends on viewpoint and the model struggles with global symmetry in highly regular objects.
- 5.1 The Effect of Viewpoints on Generalization: Figure 7 maps reconstruction error by elevation and azimuth, with accidental views showing larger errors than generic views.Errors are measured using Chamfer distance for 100 tables; the displayed tables are for visualization.
- 5.1 The Effect of Viewpoints on Generalization: Generic viewpoints produce better reconstructions than accidental viewpoints when GenRe is tested on unseen ShapeNet tables.The experiment uses a model trained on chairs, cars, and airplanes, evaluated on 100 tables across 200 spherical views.
- 5.2 Reconstructing Non-Rigid Shapes: GenRe retains visible details and completes unseen surfaces of non-rigid shapes such as horses and humans under ground-truth depth input.This experiment focuses on spherical-map inpainting and voxel refinement using GenRe-Oracle.
- 5.3 Reconstructing Highly Regular Shapes: On highly regular shapes such as cones and cubes, GenRe hallucinates unseen parts but fails to exploit global shape symmetry for correct predictions.The model was trained only on cars, chairs, and airplanes and used ground-truth single-view depth.
6 Conclusion
The paper studies generalizable single-image 3D reconstruction and proposes GenRe, which integrates multiple image and shape representations for reconstruction across seen and unseen classes. Experiments report state-of-the-art performance on both seen and unseen classes.
- 6 Conclusion: GenRe integrates 2.5D sketches, spherical maps, and voxels in a viewer-centered model for generalizable 3D shape reconstruction.The conclusion presents these representations as the core components of the proposed system.
- 6 Conclusion: GenRe achieves state-of-the-art shape reconstruction performance on both seen and unseen classes.This is the paper’s overall reported experimental conclusion.