Source-linked AI summary
Category-Specific Object Reconstruction from a Single Image
Abhishek Kar, Shubham Tulsiani, João Carreira, Jitendra Malik
TL;DR
Single-image object reconstruction in realistic, unconstrained scenes requires recovering 3D shape from limited 2D evidence. The paper learns deformable category-specific 3D models from annotated images, estimates viewpoints, fits silhouettes and keypoints, and adds bottom-up detail refinement. It reports fully automatic reconstructions on PASCAL VOC and competitive results on PASCAL 3D+ evaluations.
Problem
The paper addresses reconstructing category-level 3D object surfaces from single images in realistic scenes, including images with noisy automatic segmentations.
Method
The method learns deformable 3D shape models from silhouettes and keypoints, estimates viewpoints with NRSfM augmented by silhouette information, and combines top-down fitting with bottom-up refinement.
Results
The system performs fully automatic single-image reconstruction on PASCAL VOC and achieves competitive results against specialized and class-agnostic reconstruction methods.
Takeaways & Limitations
2D annotations from existing detection datasets can bootstrap deformable 3D models for reconstruction on large real-world image collections without manual mesh design or 3D scanning.
Takeaways & Limitations
The approach still requires training annotations, and reconstruction quality and robustness at test time remain areas for improvement.
Abstract
from arXiv · showhide
Object reconstruction from a single image -- in the wild -- is a problem where we can make progress and get meaningful results today. This is the main message of this paper, which introduces an automated pipeline with pixels as inputs and 3D surfaces of various rigid categories as outputs in images of realistic scenes. At the core of our approach are deformable 3D models that can be learned from 2D annotations available in existing object detection datasets, that can be driven by noisy automatic object segmentations and which we complement with a bottom-up module for recovering high-frequency shape details. We perform a comprehensive quantitative analysis and ablation study of our approach using the recently introduced PASCAL 3D+ dataset and show very encouraging automatic reconstructions on PASCAL VOC.
1. Introduction
The paper introduces category-specific deformable 3D shape models learned from 2D annotations, enabling automatic reconstruction of objects from single images in realistic scenes. It combines top-down shape inference with bottom-up instance refinement and targets unconstrained images.
- 1. Introduction: Category-specific shape models are learned from image datasets containing segmentation masks and a small set of keypoints.The approach uses past visual experience to model 3D shape variation within object categories.
- 1. Introduction: The learned models guide top-down 3D reconstruction of novel images, while a bottom-up module refines shape details for each instance.The pipeline combines class-level inference with instance-specific refinement.
- 1. Introduction: Fully automatic reconstruction uses image pixels as input by relying on automatic object detection, segmentation, and pose-estimation modules.The method is designed to operate on images in the wild without requiring test-time annotations.
- 1. Introduction: The paper positions its approach as more suitable for unconstrained reconstruction than methods that assume high-quality segmentations or reconstruct fully annotated image sets.Explicit 3D shape models help address noisy segmentation settings.
- 1. Introduction: The training pipeline estimates camera viewpoints from annotated images and combines them with object silhouettes to learn deformable 3D shape models.The learned models represent intra-class shape variation rather than a single rigid template.
2. Learning Deformable 3D Models
The paper learns deformable, category-specific 3D shape models from 2D silhouettes and keypoints, using NRSfM to estimate viewpoints and silhouette-aware optimization to recover shape bases. The resulting models combine silhouette, keypoint, smoothness, and deformation regularization for reconstruction under noisy visual inputs.
- 2.1. Viewpoint Estimation: NRSfM jointly estimates rotation, translation, and scale for training instances while modeling intra-class shape variation.The method extends an EM-PPCA formulation to incorporate silhouette information alongside keypoint correspondences.
- 2.1. Viewpoint Estimation: Silhouette information is added to keypoint correspondences in NRSfM to robustly recover cameras and shape bases.The authors identify this combination with NRSfM as their key algorithmic modification.
- 2.1. Viewpoint Estimation: The mask constraint requires every training keypoint to lie inside its instance’s binary silhouette, improving viewpoint accuracy and learned shape bases.The constraint uses the instance mask’s Chamfer distance field.
- 2.2. 3D Basis Shape Model Learning: Category shapes are modeled as deformable point clouds with separate linear shape models for different subcategories.Each model uses a mean shape and deformation bases learned from silhouettes, avoiding the need for point correspondences between scanned 3D exemplars.
- 2.2. 3D Basis Shape Model Learning: The reconstruction objective combines silhouette containment, silhouette coverage, and keypoint consistency with local smoothness and deformation penalties.Silhouette containment prevents projected points from leaving the mask, while coverage reduces overcarving and keypoint consistency fits lifted 3D correspondences.
- 2.2. 3D Basis Shape Model Learning: Local regularizers encourage neighboring points to have similar spacing, locally smooth normals, and bounded deformation magnitudes.The deformation penalty is used to prevent unnaturally large deformations.
3. Reconstruction in the Wild
The reconstruction pipeline first fits category-level 3D shape models to detected objects and noisy segmentation masks, then refines coarse estimates using shading cues. It combines top-down inference with bottom-up recovery of high-frequency shape details.
- Initialization: The pipeline detects and segments objects, predicts viewpoint and subcategory, then initializes a scaled, rotated mean shape in the predicted bounding box.Initialization uses a CNN-based system for viewpoint and subcategory prediction before fitting the learned model.
- Learned Shape Models: Mean shapes learned with the basis formulation represent rigid object classes and are visualized with frontal depth coloring.The figure reports learned mean shapes for rigid classes in PASCAL VOC.
- Shape Inference: Top-down inference optimizes deformation weights and camera projection parameters against the learned shape model and inferred silhouette.Test-time optimization omits keypoint consistency because annotated keypoints are unavailable.
- Bottom-up Shape Refinement: Bottom-up refinement adapts SIRFS to recover high-frequency shape information from statistical regularities among shape, reflectance, and illumination.The refinement retains the recovered shape map from the reflectance, shape, and illumination outputs.
- Bottom-up Shape Refinement: The coarse fitted shape is incorporated into SIRFS through an additional loss term that guides optimization using a rendered depth map.The initial coarse shape and a differentiability parameter are used in this added term.
- Implementation: Reconstructing a novel instance takes about 2 sec on a single CPU core.The implementation uses approximate nearest neighbors and Chamfer distance fields to compute optimization gradients efficiently.
4. Experiments
Experiments evaluate learned 3D model expressiveness, robustness to noisy test inputs, and fully automatic reconstruction on PASCAL VOC, using PASCAL 3D+ comparisons and ablations. The method performs competitively with ground-truth annotations, degrades gracefully as annotations are relaxed, and produces plausible automatic reconstructions despite documented alignment and data-sparsity failures.
- Experimental setup: The evaluation measures learned-model expressiveness against PASCAL 3D+ CAD shapes and sensitivity to noisy automatic segmentations and pose predictions.The experiments use PASCAL VOC 2012 images and PASCAL 3D+ models because ground-truth 3D shapes are unavailable for PASCAL VOC.
- Expressiveness of learned 3D models: Table 1 compares the method with Vicente et al. and category-agnostic shape inflation methods using ground-truth keypoints and masks on PASCAL VOC.The comparison follows the shared whole-dataset setup used by Vicente et al.
- Expressiveness of learned 3D models: The models achieve competitive benchmark performance and show greater robustness to perspective foreshortening on trains and buses than the compared approaches.Category-agnostic Puffball and SIRFS methods consistently perform worse by themselves, while boats and tvmonitors remain especially difficult.
- Sensitivity analysis: Table 2 shows graceful degradation from fully annotated to fully automatic test settings, with robustness to some mis-segmentation and only slight degradation from imperfect pose initialization.The shape model prevents unnatural bending to explain noisy silhouettes, while bad predicted alignment can still produce high metric errors.
- Fully automatic reconstruction: Fully automatic reconstructions on detected instances with 0.5 IoU segmentation overlap produce plausible results, but failures include incorrect scale or pose and wrong subtype predictions.Data sparsity causes a significant drop for sofas with only 34 training instances, and the bottom-up component is not evaluated against PASCAL 3D+ meshes lacking matching high-frequency details.
5. Conclusion
The paper presents a fully automatic single-image reconstruction approach whose deformable 3D models can be learned from 2D annotations and applied to realistic datasets. It reports competitive results and automatic reconstructions in the wild, while identifying robustness, benchmarking, annotation, and segmentation integration as future work.
- Contribution: The approach performs fully automatic object reconstruction from a single image on a large, realistic dataset.The conclusion characterizes this as potentially the first approach of its kind.
- Results: The method achieves competitive results against models specialized for shape reconstruction from ground-truth segmentation inputs while also operating with automatic detectors in the wild.This conclusion combines benchmark evaluation with the reported ability to reconstruct from automatically detected objects.
- Future work: Future work includes improving test-time reconstruction quality and robustness, developing joint recognition-reconstruction benchmarks, and reducing training annotation requirements.The authors also identify more expressive nonlinear shape models and tighter segmentation-reconstruction integration as promising directions.