Source-linked AI summary
Single Image 3D Interpreter Network
Jiajun Wu, Tianfan Xue, Joseph J. Lim, Yuandong Tian, Joshua B. Tenenbaum, Antonio Torralba, William T. Freeman
TL;DR
Recovering 3D object structure from one image is difficult because real images rarely have accurate 3D annotations. 3D-INN addresses this with a projection-supervised, heatmap-mediated network trained on real 2D and synthetic 3D data, achieving strong results across 2D and 3D tasks and supporting applications such as rendering and retrieval.
Problem
Real images rarely provide accurate 3D geometry annotations, while prior approaches rely on noisy 2D-keypoint optimization or synthetic-image training that may not generalize to real images.
Method
3D-INN sequentially predicts 2D keypoint heatmaps and 3D skeleton parameters, using a differentiable Projection Layer and heatmaps to combine real 2D labels with synthetic 3D data.
Results
3D-INN performs well on 2D keypoint estimation and 3D structure and viewpoint recovery, with results comparable to or better than state-of-the-art methods.
Takeaways & Limitations
The learned 3D skeleton representation supports applications including 3D rendering and image retrieval.
Takeaways & Limitations
When detection inputs are noisy, errors can arise from differences between training and test categories and viewpoints.
Abstract
from arXiv · showhide
Understanding 3D object structure from a single image is an important but difficult task in computer vision, mostly due to the lack of 3D object annotations in real images. Previous work tackles this problem by either solving an optimization task given 2D keypoint positions, or training on synthetic data with ground truth 3D information. In this work, we propose 3D INterpreter Network (3D-INN), an end-to-end framework which sequentially estimates 2D keypoint heatmaps and 3D object structure, trained on both real 2D-annotated images and synthetic 3D data. This is made possible mainly by two technical innovations. First, we propose a Projection Layer, which projects estimated 3D structure to 2D space, so that 3D-INN can be trained to predict 3D structural parameters supervised by 2D annotations on real images. Second, heatmaps of keypoints serve as an intermediate representation connecting real and synthetic data, enabling 3D-INN to benefit from the variation and abundance of synthetic 3D objects, without suffering from the difference between the statistics of real and synthesized images due to imperfect rendering. The network achieves state-of-the-art performance on both 2D keypoint estimation and 3D structure recovery. We also show that the recovered 3D information can be used in other vision applications, such as 3D rendering and image retrieval.
1 Introduction
3D-INN recovers object structure from a single image by combining 2D keypoint estimation with 3D skeleton prediction. A projection layer enables learning from real 2D annotations, while heatmaps connect real and synthetic training data.
- 3D-INN represents objects as category-specific 3D skeletons rather than meshes or depth maps, preserving structural properties through keypoints and connections.
- Single-image 3D structure recovery is limited by scarce, labor-intensive, and often inaccurate real-image 3D annotations.
- The Projection Layer converts estimated 3D skeleton parameters into 2D keypoint projections, allowing real 2D-labeled images to supervise 3D prediction.
- Keypoint heatmaps connect real images with synthetic 3D objects, helping encode plausible shapes while reducing sensitivity to differences from imperfect rendering.
- 3D-INN reports state-of-the-art performance in 2D keypoint localization and strong results in 3D structure and viewpoint estimation.
2 Related work
Prior work reconstructs single-image 3D structure using depth, meshes, skeletons, or inverse-graphics formulations, often relying on optimization or simplifying assumptions. 3D-INN instead uses neural prediction with 2D keypoint heatmaps and mixed real-synthetic training.
- Single-image reconstruction methods use depth, meshes, skeletons, or pictorial structures to represent 3D objects.
- Skeleton-based methods capture geometric changes in articulated objects and traditionally recover 3D structure by minimizing projection error.
- 3D-INN predicts 3D object skeletons from 2D keypoints with neural networks, aiming for greater robustness to imperfect detection than optimization-based recovery.
- Unlike approaches requiring representation supervision or simplified backgrounds and occlusion, 3D-INN learns 3D representations without 3D supervision and generalizes to real images.
- The method uses heatmaps rather than rendered synthetic images as the intermediate representation, minimizing effects of imperfect rendering during mixed-data training.
3 Methods
3D-INN represents category-specific objects with constrained 3D skeletons and estimates their structure from a single image through staged keypoint, interpretation, and projection components. Training combines real 2D supervision with synthetic 3D data to address missing real-image 3D annotations and projection ambiguity.
- The network takes a centered single image and outputs 2D keypoint locations, 3D structural parameters, and 3D poses.
- 3.1 3D Skeleton Representation: Each object category uses a manually designed skeleton composed of keypoints and their connections to preserve relevant structural properties.
- 3.1 3D Skeleton Representation: 3D keypoint locations are modeled as weighted sums of category-specific base shapes, with weights representing internal deformation parameters.The mean shape is one base, while others encode intra-class variations such as chair back bending and leg length.
- 3.1 3D Skeleton Representation: Recovering 3D structure requires estimating internal shape parameters and external camera parameters: rotation R, translation T, and focal length f.
- 3.2 Architecture of 3D-INN: The architecture sequentially estimates 2D keypoint heatmaps, refines them using structural constraints, infers 3D structure and viewpoint, and projects the result back to 2D.The keypoint estimator uses multi-scale inputs; refinement employs an information bottleneck to produce cleaner heatmaps.
- 3.2 Architecture of 3D-INN: The differentiable projection layer computes 2D keypoint coordinates from estimated 3D parameters, allowing real 2D-annotated images to supervise the full network.
- 3.3 Training Strategy: On FLIC, 3D-INN performs consistently better than other methods, while removing keypoint refinement makes it comparable to Tompson et al. [44].
- 3.3 Training Strategy: Training proceeds in three stages: real-image heatmap estimation, synthetic-data 3D interpretation, and end-to-end fine-tuning with real-image projection supervision.Direct end-to-end training is difficult because of the high-dimensional search space and ambiguity in 3D-to-2D projection.
4 Evaluation
3D-INN is evaluated on 2D keypoint estimation, 3D structure and viewpoint recovery, and qualitative robustness across datasets and input conditions. The experiments show strong keypoint performance, advantages over optimization-based recovery under noise and distortion, and limitations for heavily cropped objects.
- Evaluation setup: 3D-INN is evaluated on FLIC, CUB-200-2011, and Keypoint-5 for 2D keypoint estimation, with additional tests on IKEA and qualitative results on SUN.The evaluation includes PCK, PCP, AE, 3D structure recovery, and viewpoint estimation.
- 2D keypoint estimation: About 2% improvement at normalized distance ≥0.15 comes from keypoint refinement on FLIC, where 3D-INN outperforms prior methods at all precisions.Refinement also produces less noisy heatmaps.
- 3D structural parameter estimation: 3D-INN outperforms the optimization-based method for 3D structure and viewpoint estimation when synthetic heatmaps contain noise and perspective distortion.The analytic method is designed for parallel projection and comparatively clean heatmaps.
- 3D structural parameter estimation: Both fine-tuned and non-fine-tuned 3D-INN versions significantly outperform Zhou-perp, while fine-tuning improves keypoint estimation by about 5% below RMSE 0.15.The comparison uses RMSE–Recall curves on 3D structure recovery.
- 3D structural parameter estimation: 3D-INN outperforms Su et al. by about 40% relative average recall for viewpoint estimation on heavily occluded IKEA objects.Qualitative results show accurate recovery for clean or partially occluded objects, but heavily cropped objects are a major failure case.
- Robustness and limitations: Cross-category tests usually produce reasonable chair structures, but bed-trained models perform poorly on backward-facing chairs because similar training viewpoints are scarce.This test examines noisy input caused by imperfect detection results.
5 Applications
3D-INN’s learned 3D structure and viewpoint representations support retrieval by 3D configuration and visualization of object relationships. These applications illustrate how compact inferred parameters can organize and compare objects beyond pixel-level appearance.
- Qualitative Results: Qualitative evaluations include reconstruction examples across Keypoint-5, IKEA, and SUN, plus cross-category tests using networks trained on sofas or beds.The figures also display failure cases and report that most cross-category outputs are reasonable, with errors often tied to training–test differences.
- Image Retrieval: Estimated 3D structural and viewpoint parameters support image retrieval based on objects’ 3D configurations.Retrieval uses internal structural parameters for structure-based matching and external viewpoint parameters for viewpoint-based matching.
- Image Retrieval: 3D-INN retrieves objects with similar 3D structures or similar viewpoints, alongside comparisons using AlexNet FC7 features.
- Object Graph: The object graph visualizes Keypoint-5 chair images using t-SNE applied to estimated 3D viewpoints.
- Object Graph: The visualization shows a smooth transition from chairs facing left to chairs facing right.
6 Conclusion
3D-INN recovers 2D keypoints and 3D structure from a single image using skeletons, a projection layer, and heatmaps linking real and synthetic data. It performs well across keypoint, structure, and viewpoint recovery, while supporting applications of the learned representation.
- Conclusion: 3D-INN recovers 2D keypoints and 3D structure for possibly deformable objects from a single image.
- Conclusion: The framework combines 3D skeletons, a projection layer, and keypoint heatmaps to learn from 2D labels and connect real with synthetic data.
- Conclusion: 3D-INN performs comparably to or better than state-of-the-art methods on 2D keypoint estimation and 3D structure and viewpoint recovery.
- Conclusion: Applications demonstrate the potential of the skeleton representation learned by 3D-INN.