Source-linked AI summary
Multi-view 3D Models from Single Images with a Convolutional Network
Maxim Tatarchenko, Alexey Dosovitskiy, Thomas Brox
TL;DR
The paper tackles single-image 3D inference, proposing a convolutional encoder-decoder that predicts arbitrary-view RGB images and depth maps for unseen objects. Fusing those predictions produces point clouds and meshes, while synthetic training supports cluttered backgrounds and reasonable real-car predictions.
Problem
Deep networks had not yet focused on learning 3D representations, despite single-image 3D inference being necessary for human-level scene understanding.
Method
A convolutional encoder-decoder infers a multi-view 3D representation from one RGB image, predicts requested-view RGB and depth maps, and fuses multiple views into a point cloud and mesh.
Results
The network improves visual quality over previous work, handles non-homogeneous backgrounds, and produces reasonable predictions for real images of cars without adaptation.
Takeaways & Limitations
A feed-forward network trained only on synthetic data can render arbitrary views and generate point clouds and meshes from single images, including natural inputs.
Takeaways & Limitations
Natural images remain harder because the synthetic training data does not fully model their variations, and dense fusion removes noise while destroying some details.
Abstract
from arXiv · showhide
We present a convolutional network capable of inferring a 3D representation of a previously unseen object given a single image of this object. Concretely, the network can predict an RGB image and a depth map of the object as seen from an arbitrary view. Several of these depth maps fused together give a full point cloud of the object. The point cloud can in turn be transformed into a surface mesh. The network is trained on renderings of synthetic 3D models of cars and chairs. It successfully deals with objects on cluttered background and generates reasonable predictions for real images of cars.
1 Introduction
The paper asks whether deep networks can infer useful 3D representations from a single image and proposes an encoder-decoder that predicts arbitrary views and depth maps. Fusing these predictions yields a point cloud and mesh, with training on diverse synthetic renderings supporting cluttered-background and real-image inputs.
- The paper addresses whether deep networks can learn 3D representations from single images, a capability described as necessary for human-level scene understanding.
- The proposed multi-view 3D model represents an object through all views and corresponding depth maps, generated for arbitrary viewpoints from one image.
- Fusing several predicted views produces a full 3D point cloud that includes parts invisible in the original image, which can be optimized into a mesh.
- When the input omits object parts, the network infers a probable shape consistent with the visible data, such as a standard sedan from a car’s front view.
- Training uses on-the-fly ShapeNet renderings with random viewpoints, lighting, and pasted backgrounds, producing diverse data and teaching object segmentation.
- The authors report improved visual quality with a simpler architecture and application to non-homogeneous and natural images, including reasonable predictions for real-world images without adaptation.
2 Related work
Related work spans neural image transformation, retrieval-based reconstruction, depth and volumetric prediction, and single-image 3D inference. The proposed approach is distinguished by direct arbitrary-view generation, feed-forward inference, and an implicit learned representation rather than fixed-dataset retrieval.
- Unseen view prediction: Earlier neural transformation methods use multiplicative interactions but generally handle small images or only small transformations.
- Unseen view prediction: Other approaches generate views through random sampling, factored latent variables, or interpolation, while this work directly specifies the desired viewpoint.
- Unseen view prediction: Prior generative models may be restricted to training objects or require retraining for new test images, whereas the encoder infers a representation from the input image at test time.
- Unseen view prediction: Compared with recurrent rotation, the proposed network varies the angle continuously and generates an arbitrary output view in one forward pass, while also producing point clouds and handling non-homogeneous backgrounds.
- 3D from single image: Single-image 3D reconstruction commonly exploits models of lighting, reflectance, object properties, or category-specific shape priors to recover explicit models.
- 3D from single image: Retrieval-based methods align or interpolate among fixed collections of 3D models, limiting generalization to novel test images and increasing cost with collection size.
3 Model description
The model encodes a single RGB image into an implicit object representation and combines it with a requested viewpoint to predict RGB and depth outputs. Multiple predictions are reprojected and fused into a point cloud, then converted into a mesh using synthetic ShapeNet training data and clustered evaluation splits.
- Model formulation: The network takes an input image and desired viewpoint, then estimates the same object’s RGB projection and depth map from that viewpoint.
- Model formulation: Because the input viewpoint is withheld, the network must infer it implicitly from the image; viewpoint angles are encoded with sine and cosine for periodicity.
- Model formulation: Training minimizes a weighted sum of squared Euclidean RGB loss and L1 depth loss, using λ = 0.1 in the experiments.
- Architecture: The encoder maps the image to a hidden 3D representation, while the decoder combines that representation with the desired viewpoint and renders the output image.
- Multi-view 3D to point cloud and mesh: For reconstruction, depth maps from multiple viewpoints are reprojected using known camera parameters into a common 3D space to form one point cloud.
- Multi-view 3D to point cloud and mesh: The point cloud is post-processed into a dense surface mesh using depth and approximate camera-directed normals.
4 Experimental evaluation
The network generates unseen RGB views and depth maps from single images, outperforming nearest-neighbor and prior methods qualitatively while reconstructing coherent 3D shapes. Predictions remain somewhat blurry or lose fine detail, especially under ambiguity, but realistic synthetic rendering supports reasonable real-image generalization.
- Unseen view prediction: The network predicts desired views for normal and difficult cars, with correct shape and color, although difficult-car outputs are blurrier.It succeeds with and without backgrounds; the difficult car is dissimilar to training models.
- Unseen view prediction: Predictions are slightly blurry and omit details, likely because squared Euclidean loss averages uncertain image predictions.Adversarial training sharpens images slightly but introduces noise, artifacts, and sensitive training.
- Unseen view prediction: The network outperforms nearest-neighbor baselines on normal and difficult cars for both image and depth prediction.Unlike nearest-neighbor retrieval, it infers subtle 3D cues and can recombine training information to create new images.
- Unseen view prediction: Our network generates more accurate chair views than IGN, capturing details such as bent legs and armrests.The comparison uses similar chair models and the same input viewpoint.
- Unseen view prediction: Realistic rendering is important for real-image generalization; simpler rendering leads to complete failure on real data.Training uses synthetic renderings with varied viewpoints, lighting, and pasted random backgrounds.
- 3D model prediction: Fusing 6 predicted depth maps yields point clouds that represent object shape and color, including for a difficult model.The views use 20° elevation and azimuths spaced every 60°.
- 3D model prediction: Dense depth fusion smooths noise but removes some details because of its regularizer.The resulting surface quality reflects the trade-off between smoothing and detail preservation.
- Analysis of the network: Prediction quality depends on viewpoint informativeness, yet weakly informative inputs still produce views consistent with one 3D shape.Side views match ground truth better, while front views require more guessing and tend toward probable shapes.
5 Conclusions
The paper presents a feed-forward network that learns implicit 3D representations from single images, generating new views, point clouds, and surface meshes, including for natural images.
- The network learns implicit 3D representations by generating new object views from a single input image.
- It can render any desired object view and generate a point cloud and surface mesh.
- Although trained only on synthetic data, the network can also process natural images.
Supplementary Material
The experiments examine realistic rendering, adversarial training, and how the network’s internal representation changes across input views of the same object.
- The supplementary experiments evaluate realistic rendering and adversarial training.
- They also analyze changes in the network’s internal representation when different views of the same object are provided.
A Realistic rendering
Training-data realism is tested by comparing networks trained with realistic versus basic renderings, showing that realistic rendering is crucial for generalization to real images.
- Realistic rendering varies light sources, intensities, and locations, while basic rendering uses two fixed-intensity light sources.
- Realistic rendering also uses alpha compositing and Gaussian smoothing, unlike the basic setup.
- The network trained on basic data fails to estimate car shape in all but one case, whereas realistic-data training performs much better.
- Figure 15 compares predictions from networks trained on realistic and basic data for each model.
B Adversarial training
The paper investigates adversarial training to reduce blur from Euclidean losses, but finds sharper predictions accompanied by noise, worse shape estimates, and greater hyperparameter sensitivity.
- Squared Euclidean objectives can produce blurred images because uncertainty leads the network to predict averages of possible views.
- Adversarial training introduces a generator-discriminator setup intended to produce more realistic images.
- The conditional generator is trained using the input image rather than random noise, with α = 0.01 in the experiments.
- Adversarial training produces sharper predictions but increases image noise, worsens car-shape estimates, and heightens sensitivity to hyperparameters.
- Because of these trade-offs, the experiments focus on standard non-adversarial losses.
C Intermediate representation
The network’s hidden representation is similar across different views of the same car and reflects similarity in car shape. Adversarial and squared Euclidean losses are also compared through prediction visualizations.
- Figure 16 compares predictions from networks trained with adversarial loss and squared Euclidean loss against ground truth.For each model, the top row uses adversarial loss, the second row omits it, and the bottom row shows ground truth.
- Different input views of the same car produce similar hidden representations.This appears as 3 × 3 diagonal blocks in the pairwise-distance matrix for five cars and three views.
- Cars with similar shapes can have close representations in the hidden space.The second and fifth cars are close in the matrix, indicated by off-diagonal blue elements.