Source-linked AI summary
Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, Honglak Lee
TL;DR
Single-view 3D reconstruction must recover object shape from 2D images that mix intrinsic properties with viewpoint and other extrinsic factors, despite limited 3D training data. The paper introduces an encoder-decoder with perspective projection loss to learn volumetric shape from 2D observations without explicit 3D supervision. Experiments report strong reconstruction and generalization potential, including unseen categories.
Problem
Single-view 3D reconstruction remains difficult because 2D images entangle intrinsic object properties with extrinsic factors such as orientation, position, and illumination.
Method
An encoder-decoder predicts volumetric shape and uses a differentiable perspective transformer to match projected shapes with 2D observations.
Results
The model achieves excellent reconstruction without ground-truth 3D volumes and shows generalization potential to unseen categories.
Takeaways & Limitations
Perspective projection regularization supports learning 3D shape from single-view 2D data while extending the model toward multi-class and unseen-category reconstruction.
Takeaways & Limitations
The formulation ignores color and texture, assumes a clean white background and constant natural lighting, and requires the viewpoint transformation matrix as input.
Abstract
from arXiv · showhide
Understanding the 3D world is a fundamental problem in computer vision. However, learning a good representation of 3D objects is still an open problem due to the high dimensionality of the data and many factors of variation involved. In this work, we investigate the task of single-view 3D object reconstruction from a learning agent's perspective. We formulate the learning process as an interaction between 3D and 2D representations and propose an encoder-decoder network with a novel projection loss defined by the perspective transformation. More importantly, the projection loss enables the unsupervised learning using 2D observation without explicit 3D supervision. We demonstrate the ability of the model in generating 3D volume from a single 2D image with three sets of experiments: (1) learning from single-class objects; (2) learning from multi-class objects and (3) testing on novel object classes. Results show superior performance and better generalization ability for 3D object reconstruction when the projection loss is involved.
1 Introduction
The paper proposes single-view volumetric 3D reconstruction through an encoder-decoder that models perspective projection between predicted shapes and 2D observations. This projection-based training avoids explicit 3D supervision and supports reconstruction across object categories and viewpoints.
- The network predicts an object's volumetric 3D shape from an arbitrary single image while accounting for intrinsic shape and extrinsic viewing factors.The formulation distinguishes geometry-related properties from orientation, position, and illumination in the image observation.
- A built-in camera system projects predicted 3D shapes into 2D images, allowing the network to match perspective-transformed predictions with corresponding observations.The architecture combines an image encoder, volume decoder, and perspective transformer.
- Experiments on ShapeNetCore evaluate single-class and multi-class volumetric reconstruction, with results described as excellent.The model is trained and evaluated using subsets of 3D models from ShapeNetCore.
- 2D silhouette loss enables neural networks to learn single-view 3D shape without ground-truth volumetric data.The loss is based on perspective transformations and uses 2D observations as training feedback.
- The proposed approach also reports generalization potential to unseen categories and comparatively similar results when trained with partial rather than full azimuth angles.These comparisons are stated among the paper's main contributions.
2 Related Work
Prior work learns 3D object representations for recognition or models transformations in image data, while this paper emphasizes a differentiable, parameter-free perspective projection mechanism for reconstruction from 2D observations.
- Earlier 3D representation methods primarily model volumetric shapes or pool multiple image views for 3D shape categorization rather than reconstruction.The cited approaches focus more on recognizing 3D shapes than reconstructing them.
- Compared with concurrent 3D-from-2D work, this paper's perspective projection network is differentiable and parameter-free.The concurrent mechanism either uses learnable parameters or a non-differentiable MCMC component.
- Transformation-based representation learning has been used to model image-pair transformations and disentangle identity from viewpoint.Examples include multiplicative interactions, transformations on data manifolds, and learned out-of-plane rotations.
3 Problem Formulation
The formulation treats single-view 3D reconstruction as learning a volumetric shape from 2D observations under viewpoint-dependent perspective projection. It combines an encoder-decoder with projection-based training, view-invariant representations, and differentiable perspective transformations.
- 3 Problem Formulation: 3D object understanding is framed as disentangling intrinsic shape properties from viewpoint and illumination using 2D images.
- 3 Problem Formulation: The task focuses on binary voxel volumes while assuming a clean white background, constant natural lighting, and ignored color and texture.
- 3.1 Learning to Reconstruct Volumetric 3D Shape from Single-View: A convolutional encoder-decoder maps a single image to a predicted volume through a viewpoint-invariant latent representation.
- 3.1 Learning to Reconstruct Volumetric 3D Shape from Single-View: The projection loss compares silhouettes rendered from the predicted volume at multiple viewpoints with corresponding observed silhouettes, enabling training without ground-truth volumes.
- 3.1 Learning to Reconstruct Volumetric 3D Shape from Single-View: The combined objective weights projection and volumetric losses using λproj and λvol to control their tradeoff.
- 3.2 Perspective Transformer Networks: The perspective transformer densely samples a volume into camera coordinates and uses max projection across disparity as a ray-tracing approximation.
- 3.2 Perspective Transformer Networks: Max projection is appropriate for binary volumes because empty voxels do not create foreground pixels, while solid voxels contribute only when visible from the viewpoint.
- 3.3 Training: Training first learns a view-invariant encoder and then trains the volume decoder with perspective-transformer regularization using neighboring viewpoints in each mini-batch.
4 Experiments
Experiments evaluate the perspective-transformer network on ShapeNetCore under single-category, multicategory, partial-view, and unseen-category settings. Across these settings, projection-based training performs comparably to volume-supervised training and supports stronger generalization when trained across categories.
- Experimental setup: ShapeNetCore provides about 51,300 models across 55 categories, rendered from 24 azimuth angles and represented as 32 × 32 × 32 voxel volumes.Images are cropped and rescaled to 64 × 64 × 3 pixels under fixed elevation, camera, and lighting.
- Experimental setup: The experiments compare PTN-Proj, CNN-Vol, and PTN-Comb, trained with projection loss only, volume loss only, and combined loss, respectively.The study tests reconstruction performance with and without ground-truth volumes and evaluates generalization to unseen categories.
- Single-category training: On chairs, projection-loss training performs as well as volume-supervised training, while PTN-Proj and PTN-Comb produce faithful shapes and CNN-Vol sometimes fails to form a solid chair.The single-category evaluation computes average intersection-over-union across 24 generated volumes per test instance.
- Partial-view training: With only partial azimuth views, all models lose a little performance, but the conclusion remains that projection regularization improves 3D shape learning from 2D observations.The partial-view setting uses narrow ranges such as eight of the 24 azimuth views.
- Multicategory training: In multicategory training, combined loss is superior to volume loss in most cases, while projection loss performs as well as volume or combined loss.The multicategory visualization shows generally reasonable volume predictions, with only subtle differences in parts such as airplane wings.
- Out-of-category tests: Training on multiple categories significantly improves unseen-category generalization, and projection loss generalizes better than combined or volume loss on train, motorbike, and cabinet.The out-of-category evaluation includes bed, bookshelf, cabinet, motorbike, and train; bed, bookshelf, and motorbike are described as completely novel in shape.
5 Conclusions
The paper frames single-view 3D reconstruction as interaction between 3D shape and 2D observation, using projection transformation as regularization. It reports strong reconstruction without ground-truth volumes and generalization to unseen categories.
- The proposed encoder-decoder learns single-view 3D shape reconstruction by exploiting interactions between 3D shape and 2D observation.
- Projection transformation is used as regularization in the learning procedure.
- The model reconstructs objects without ground-truth 3D volume supervision and generalizes to unseen categories.
A Details regarding perspective transformer network
The perspective transformer converts predicted 3D volumes into viewpoint-specific 2D silhouettes through differentiable perspective sampling. Binary-volume projection uses max pooling across disparity rather than summation.
- Perspective Projection: Perspective projection uses a 4-by-4 transformation matrix combining camera calibration and extrinsic parameters.
- Perspective Projection: The transformer maps points from the 3D world frame into the camera frame using an inverse perspective transformation and disparity.
- Differentiable Volume Sampling: Differentiable volume sampling applies a sampling kernel to transformed spatial locations, enabling gradient-based projection from volumes to silhouettes.
- Differentiable Volume Sampling: The implementation uses a bilinear kernel k(x) = max(0, 1 −|x|) for interpolation.
B Details regarding learning from partial views
The partial-view experiments train on either a narrow azimuth range or sparse views spanning 360°. All three models lose a little performance, while projection regularization remains beneficial for learning 3D shape from 2D observations.
- Partial-view training: The narrow-range setting uses 8 of 24 views covering an azimuth range of 105°.
- Results: Performance drops a little for all three models under partial-view training, while projection regularization supports better 3D shape learning from 2D observations.
C Additional visualization results on 3D volumetric shape reconstruction
The additional visualizations compare ground-truth volumes with outputs from projection-loss, combined-loss, and volume-loss models across single-class and multiclass results. Each row represents an independent comparison from a 2D input image.
- Visualization layout: Each visualization row is an independent comparison beginning with the model’s input 2D image.
- Visualization layout: The second and third columns show the ground-truth 3D volume rendered from two views.
- Visualization layout: Columns four through nine compare PTN-Proj, PTN-Comb, and CNN-Vol outputs.
- Results: Figures 8, 9, 10, and 11 provide single-class and multiclass side-by-side reconstruction results, with viewing angles shown in parentheses.