Source-linked AI summary
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling
Xingyuan Sun, Jiajun Wu, Xiuming Zhang, Zhoutong Zhang, Chengkai Zhang, Tianfan Xue, Joshua B. Tenenbaum, William T. Freeman
TL;DR
Single-image 3D modeling lacks datasets combining real images, ground-truth shapes, scale, and precise alignment. The paper introduces Pix3D, calibrates reconstruction metrics against human perception, and proposes a model that jointly estimates shape and pose using 2.5D sketches. Pix3D supports benchmarking, and the model achieves state-of-the-art performance in reconstruction, retrieval, and pose estimation.
Problem
Existing datasets either lack real images, have rough image-shape alignment, or contain too few images and shapes for single-image 3D modeling.
Method
The paper constructs Pix3D, calibrates IoU, Chamfer distance, and earth mover’s distance against human perception, and jointly predicts shape and pose from 2.5D sketches.
Results
The proposed model achieves state-of-the-art performance on 3D reconstruction, shape retrieval, and pose estimation.
Takeaways & Limitations
Pix3D provides a large-scale, well-aligned benchmark for single-image 3D shape modeling and related tasks.
Takeaways & Limitations
The image-shape alignment assumes central projection with zero skew, square pixels, and the optical center at the frame center.
Abstract
from arXiv · showhide
We study 3D shape modeling from a single image and make contributions to it in three aspects. First, we present Pix3D, a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Building such a large-scale dataset, however, is highly challenging; existing datasets either contain only synthetic data, or lack precise alignment between 2D images and 3D shapes, or only have a small number of images. Second, we calibrate the evaluation criteria for 3D shape reconstruction through behavioral studies, and use them to objectively and systematically benchmark cutting-edge reconstruction algorithms on Pix3D. Third, we design a novel model that simultaneously performs 3D reconstruction and pose estimation; our multi-task learning approach achieves state-of-the-art performance on both tasks.
1. Introduction
Pix3D addresses limitations in existing single-image 3D modeling datasets by providing diverse real image-shape pairs with precise pixel-level alignment. The paper also calibrates reconstruction metrics and introduces a joint shape-and-pose model that achieves state-of-the-art performance on both tasks.
- Motivation: Existing datasets trade off real images, precise image-shape alignment, and scale, limiting their suitability for single-image 3D object modeling.ShapeNet lacks real images, Pascal 3D+ and ObjectNet3D have rough alignment, and IKEA contains only 90 models and 759 images.
- Dataset: Pix3D contains 395 3D shapes and 10,069 real image-shape pairs with precise 3D annotations and pixel-level silhouette alignment.
- Dataset Construction: The dataset is constructed by collecting web images and scans, annotating 2D keypoints, optimizing aligned poses, and filtering poorly aligned pairs.Crowdsourcing also records truncation and occlusion attributes for each instance.
- Evaluation: The paper calibrates IoU, Chamfer distance, and earth mover’s distance against human perception, then benchmarks state-of-the-art 3D modeling algorithms on Pix3D.
- Model: The proposed model jointly estimates shape and pose from a single RGB image using intermediate 2.5D sketches, with multi-task learning improving performance.The model predicts depth, surface normals, and silhouette before regressing 3D shape and camera parameters.
2. Related Work
Related datasets provide CAD repositories, pose-annotated images, scans, or accurate alignment, but differ in realism, scale, context, or suitability for single-image reconstruction. Pix3D combines real-world variation with aligned image-shape pairs for this task.
- 3D Shape Datasets: ShapeNet and Elastic2D3D lack real images, while Pascal 3D+ and ObjectNet3D match images to predefined CAD models rather than the actual shapes.
- RGB-D Datasets: RGB-D and depth-only datasets often contain small sets of hand-held objects in clean backgrounds, whereas Pix3D targets single-image reconstruction with more real-world objects and images.
- Object Scan Datasets: The Choi et al. scan dataset may keep large objects truncated and associates each shape with only one scan, limiting contextual variation.Pix3D addresses these issues through web search and crowdsourcing.
- Pose-Annotated Benchmarks: IKEA offers accurate image-3D alignment but contains 759 images and 90 shapes, while Pix3D contains 10,069 images and 395 shapes with greater variation.
- Single-Image Reconstruction: Single-image reconstruction requires both recognition systems and prior shape knowledge; Pix3D is proposed as a benchmark for evaluating such algorithms and joint shape-pose estimation.
3. Building Pix3D
Pix3D combines web images, scanned shapes, and crowdsourced annotations to create aligned image-shape pairs. Alignment is optimized from visible 2D–3D keypoints by estimating camera and pose parameters and minimizing reprojection error.
- Collecting Image-Shape Pairs: Pix3D collects image-shape pairs from IKEA web images and from objects that the authors scan and photograph themselves.
- Image-Shape Alignment: Alignment estimates a projection matrix from corresponding 2D and 3D keypoints by minimizing their reprojection error.
- Image-Shape Alignment: The projection model contains seven estimated parameters: three rotations, three translations, and focal length.
- Image-Shape Alignment: The optimization first obtains a rough pose with Efficient PnP, then refines it with Levenberg-Marquardt and retains the solution with minimum projection error.
- Image-Shape Alignment: The dataset uses crowdsourced keypoint visibility and locations to support precise alignment between each image and its corresponding 3D shape.
- Image-Shape Alignment: To address noisy annotations, the method evaluates RANSAC or subsets of annotator responses and keeps the result with the lowest projection error.
4. Exploring Pix3D
Pix3D is explored through category distributions, image availability per shape, and quantitative comparisons of projected silhouette alignment. It achieves higher IoUs than PASCAL 3D+ and ObjectNet3D and slightly higher IoUs than IKEA.
- Dataset statistics: Figures 4 and 5 summarize the distributions of images and 3D shapes across Pix3D’s categories.
- Dataset statistics: Figure 6 shows how the number of available images varies across individual shapes.
- Quantitative evaluation: Pix3D has much higher silhouette IoUs than PASCAL 3D+ and ObjectNet3D, and slightly higher IoUs than IKEA.
- Quantitative evaluation: Pix3D contains 1,092 unoccluded and untruncated sofa images, compared with only 15 such IKEA images.
5. Metrics
The paper calibrates reconstruction metrics against human perception and benchmarks their use for evaluating 3D shape quality. CD and EMD align more closely with human judgments than IoU in the reported studies.
- Designing evaluation metrics helps encourage algorithms to reconstruct high-quality geometry rather than overfit a single metric.
- Metric calibration: 0.371, 0.544, and 0.518 are the Pearson correlations of IoU, CD, and EMD, respectively, with human ratings.
- Metric calibration: The paper calibrates IoU, Chamfer distance, and Earth Mover’s distance against human perception of shape similarity.Voxels are converted into point clouds before CD and EMD are computed.
- Metric calibration: 0.49, 0.43, and 0.32 are the Spearman correlations of CD, EMD, and IoU, respectively, with human judgments.
- User studies: The behavioral studies compare metric rankings with pairwise preferences and ratings of reconstructions against ground-truth shapes.
6. Approach
The approach jointly reconstructs 3D shape and estimates camera pose from a single RGB image through an intermediate 2.5D sketch representation. Multi-task prediction extends the reconstruction pipeline with pose estimation.
- The model uses Pix3D to perform shape reconstruction and pose estimation simultaneously.
- 2.5D representation: Given an RGB image, a 2.5D sketch estimator predicts depth, surface normals, and silhouette.
- Joint prediction: Separate decoders produce the 3D shape and camera parameters from the latent representation.
- 2.5D representation: A 2.5D sketch encoder maps the predicted sketches to a 200-D latent vector for downstream decoding.
- Training: The estimator is trained first, after which the encoder, shape decoder, and view estimator are trained jointly before concatenated prediction.
7. Experiments
Experiments on Pix3D evaluate reconstruction, image-based shape retrieval, and pose estimation. The model leads reconstruction and pose estimation, while its retrieval performance depends on whether pose estimation is included.
- Experiments use 2,894 untruncated and unoccluded chair images to evaluate reconstruction, retrieval, and pose estimation.
- Shape retrieval: The model without pose estimation achieves the highest Recall@K values for image-based shape retrieval.
- Shape retrieval: The pose-enabled retrieval variant can favor images with the same pose but not exactly the same shape.
- 3D reconstruction: The full model achieves the highest IoU, EMD, and CD for 3D shape reconstruction.It also outperforms the variant without the view estimator, supporting a benefit from multi-task learning.
- Pose estimation: The model outperforms Render for CNN in both azimuth and elevation classification accuracy.Azimuth uses 24 bins and elevation uses 12 bins.
8. Conclusion
Pix3D combines well-aligned 2D images and 3D shapes with metric calibration and a joint shape-and-pose model. Experiments report state-of-the-art performance across reconstruction, retrieval, and pose estimation.
- Pix3D is a large-scale dataset of well-aligned 2D images and 3D shapes.
- Results: Experiments report state-of-the-art performance on 3D reconstruction, shape retrieval, and pose estimation.
- Model: The model simultaneously performs 3D shape reconstruction and camera view estimation using four network components.
- Model: The 2.5D sketch encoder, 3D shape decoder, and view estimator respectively encode sketches, decode voxels, and estimate camera view.
- Model: The 2.5D sketch estimator maps a 256×256 RGB image to surface normals, depth, and silhouette outputs of size 256×256.
B. Training Paradigms
The method uses staged training for 2.5D sketch estimation, followed by joint shape and optional view estimation. Training combines reconstruction and pose losses, with specified optimization settings and metric-based retrieval comparisons.
- Training first fits the 2.5D sketch estimator, then jointly trains the sketch encoder, 3D shape decoder, and optional view estimator.
- 2.5D sketch estimation minimizes mean squared error between predicted and ground-truth sketches, including silhouette error.
- Shape and view estimation use a weighted sum of 3D reconstruction and pose estimation losses, with binary cross-entropy terms for pose predictions.
- The stated training configuration uses batch size 4, Adam at learning rate 2 × 10^-4 for one stage, and selects the minimum-validation-loss model.
- For nearest-neighbor retrieval, EMD and CD perform slightly better than IoU.
C. Evaluation Metrics
The evaluation protocol normalizes voxel and point-cloud representations before computing IoU, CD, and EMD. Retrieval comparisons indicate that EMD and CD perform slightly better than IoU.
- All voxel and point-cloud representations are preprocessed before calculating IoU, CD, and EMD to support fair comparison across voxelization methods.
- IoU evaluation thresholds objects at 0.1, pads their bounding boxes into cubes, and resamples them to resolution 32^3 using trilinear interpolation.
- For 128^3 reconstructions, 4× max pooling precedes interpolation so sparse sampling does not omit thin structures.
- CD and EMD evaluation samples 1,024 points from each reconstruction’s isosurface, then centers and scales the point clouds before comparison.
- EMD and CD perform slightly better than IoU when retrieving three nearest neighbors for a shape from Pix3D.
E. Sample Data Points in Pix3D
Pix3D samples span diverse object categories and show pixel-level 2D–3D alignment. Each 3D shape is also associated with a rich set of 2D images.
- Figures 13 and 14 demonstrate diverse Pix3D shapes and high-quality 2D–3D alignment across beds, chairs, desks, sofas, tables, tools, and wardrobes.
- Each shape in Pix3D is matched with a rich set of 2D images, as illustrated by the paired examples in Figure 15.
- The samples include scanned chairs and chairs whose 3D shapes come from IKEA.