PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning single-image 3D reconstruction by generative modelling of shape, pose and shading

Paul Henderson, Vittorio Ferrari

arXiv:1901.06447v2cs.CVcs.LG

TL;DR

Single-image 3D reconstruction is ambiguous, while many existing approaches rely on 3D supervision, annotations, or multiple views. This paper learns a mesh-based generative model of shape, pose, and lighting from 2D images; shading improves reconstruction over silhouettes, and coloured-light performance approaches that of 3D-supervised methods.

  • Problem

    Single-image reconstruction is ambiguous, and many existing methods require 3D supervision, 2D annotations, or multiple views per object instance.

    Source: §1 · pp. 1–2

  • Method

    A probabilistic mesh-based generative model jointly represents shape, pose, and lighting and is trained to reconstruct images using 2D supervision.

    Source: §1 · pp. 2–3

  • Results

    Shading improves reconstruction over silhouettes; white-light models outperform 2D-supervised methods, while coloured-light models approach 3D-supervised performance, reaching 66.8% iou versus 62.5% for Kato et al. (2018).

    Source: §1 · p. 3, §5.6 · p. 16

  • Takeaways & Limitations

    The approach supports concave classes that silhouette-based methods cannot learn and produces smoother, more natural samples than voxel-based methods.

    Source: §1 · p. 3

    Matching rendered reconstructions to input images with an L2 pixel loss would require more realistic materials and lighting for successful operation on natural images.

    Source: §6 · p. 17

Abstract

from arXiv · show

We present a unified framework tackling two problems: class-specific 3D reconstruction from a single image, and generation of new 3D shape samples. These tasks have received considerable attention recently; however, most existing approaches rely on 3D supervision, annotation of 2D images with keypoints or poses, and/or training with multiple views of each object instance. Our framework is very general: it can be trained in similar settings to existing approaches, while also supporting weaker supervision. Importantly, it can be trained purely from 2D images, without pose annotations, and with only a single view per instance. We employ meshes as an output representation, instead of voxels used in most prior work. This allows us to reason over lighting parameters and exploit shading information during training, which previous 2D-supervised methods cannot. Thus, our method can learn to generate and reconstruct concave object classes. We evaluate our approach in various settings, showing that: (i) it learns to disentangle shape from pose and lighting; (ii) using shading in the loss improves performance compared to just silhouettes; (iii) when using a standard single white light, our model outperforms state-of-the-art 2D-supervised methods, both with and without pose supervision, thanks to exploiting shading cues; (iv) performance improves further when using multiple coloured lights, even approaching that of state-of-the-art 3D-supervised methods; (v) shapes produced by our model capture smooth surfaces and fine details better than voxel-based approaches; and (vi) our approach supports concave classes such as bathtubs and sofas, which methods based on silhouettes cannot learn.

1 Introduction

The paper unifies single-image 3D reconstruction and shape generation in a mesh-based probabilistic framework that learns from weaker supervision, including unannotated 2D images. It models shape, pose, and lighting to exploit shading cues, reporting stronger reconstruction performance and support for concave classes.

1 Introduction

  • Because many shapes can explain the same image, the model learns an explicit shape prior whose low-dimensional latent representation favors regular shapes consistent with the training images.The paper notes that a single image cannot guarantee recovery of the true shape, but generation yields a plausible class instance.Source: §1 · pp. 1, 3
  • Training can use unannotated 2D images with a single view per instance and no pose, keypoint, or 3D-shape supervision.The paper describes this as a more challenging setting than approaches requiring stronger supervision.Source: §1 · pp. 2–3
  • Meshes represent arbitrarily oriented surfaces and fine detail, allowing the model to use shading cues from white or coloured lighting and learn concave classes that silhouette-based methods cannot.The paper also reports smoother, more natural mesh samples than voxel-based methods.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • Different mesh parameterisations support modelling priors such as smoothness or composition from primitives, while the expanded framework reasons over lighting rather than assuming it is known.A preliminary version assumed fixed, known lighting parameters.Source: §1 · p. 3
  • The framework models shape, pose, and lighting independently, then renders meshes into images; its inference network predicts these factors and disentangles canonical shape from pose.The model is trained end-to-end to reconstruct images and can also generate new mesh samples.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • With white light, the method outperforms state-of-the-art 2D-supervised methods; coloured lights improve performance further, approaching state-of-the-art 3D-supervised methods.The authors report this pattern across their evaluations.Source: §1 · p. 3

2 Related Work

Prior work spans single-image reconstruction and 3D-shape generation, but commonly relies on 3D data or stronger image supervision. The closest 2D-only methods use silhouettes, limiting the shapes and poses they can learn.

2.1 Learning single-image 3D reconstruction

  • Single-image reconstruction methods vary from voxel, point-cloud, and mesh predictors to fused depth maps and cuboidal primitives.Some methods use 3D supervision; weaker-supervision approaches still impose requirements such as silhouettes, keypoints, or pose labels.Source: §2.1 · p. 4
  • Silhouette-based 2D-supervised reconstruction cannot exploit shading or learn concave classes, and requires pose or keypoint labels and/or multiple views per instance.Related approaches also use known poses, paired views, keypoints, or silhouettes for training.Source: §2.1 · p. 4
  • Earlier single-image reconstruction work includes a conditional mesh model demonstrated only on simple shapes such as cubes and cylinders.Other approaches learn from videos or jointly reconstruct instances without learning a model for unseen images.Source: §2.1 · p. 4

2.2 Generative models of 3D shape

  • Most deep generative models of 3D shapes train on large collections of 3D shapes, using voxel, point-cloud, primitive, or mesh representations.One mesh-parameter model requires a separate energy-based optimization and is not directly applicable to datasets with varying mesh topology.Source: §2.2 · pp. 4–5
  • The closest prior 2D-only generative model uses a voxel GAN trained from silhouettes, but restricts poses to eight viewpoints and cannot learn concavities or smooth, arbitrarily oriented surfaces.Its use as a prior for reconstruction requires multiple views per instance during training.Source: §2.2 · p. 5

3 Generative Model

The model generates an image from independent shape, pose, and lighting variables. A decoder maps shape latents to mesh parameters, which are transformed and rendered under the selected pose and lighting.

3 Generative Model

  • Shape, camera pose, and lighting are independent latent variables in a directed model of image formation.The joint distribution factors as P(z)P(θ)P(λ)P(x|z, θ, λ).Source: §3 · p. 5
  • A decoder maps a Gaussian shape embedding to mesh parameters, which a fixed parameterization converts into a canonical mesh before pose transformation and shaded rendering.The renderer uses Gouraud shading and Lambertian reflectance; the lighting angle rotates a fixed set of directional lights.Source: §3 · p. 5
  • Observed pixels are modeled as independent Gaussian variables whose means come from a Gaussian image pyramid built from the rendered image.The multiscale pyramid supplies long-distance gradient forces during training to avoid poor local minima when reconstructions are far from the input.Source: §3 · pp. 5–6
  • Ortho-block and full-block meshes assemble cuboidal primitives, while subdivision deforms a subdivided cube for complex continuous surfaces.Ortho-block primitives are axis-aligned; full-block primitives add rotations. The subdivision example uses 98 vertices and 294 parameters.Source: §3 · p. 6, Figure 3 · p. 6

4 Variational Training

Training uses amortized variational inference to infer shape, pose, and lighting from images, including images with unknown pose and lighting. Coarse-and-fine latent representations and aggregated-posterior regularization help address pose and lighting ambiguity.

4 Variational Training

  • Training requires only single images with unknown shape, pose, and lighting, not multiple views per instance or pose annotations.Source: §4 · p. 6
  • Amortized variational inference approximates the intractable latent-variable likelihood with an encoder-produced posterior and an efficiently evaluated evidence lower bound.The expectation can be approximated using a single sample.Source: §4 · p. 6
  • The variational posterior factorizes across shape, pose, and lighting, with the encoder predicting their distributions from the input image.When multiple views are available, per-view shape embeddings are combined by dimension-wise max-pooling.Source: §4 · pp. 6–7
  • Because object symmetry can make pose posteriors multimodal, pose is represented by a categorical coarse rotation and a small Gaussian fine offset.The coarse angle is integrated over its bins, while the encoder constrains the fine offset to a small range.Source: §4 · p. 7, Figure 4 · p. 7
  • Minimizing per-image KL alone can leave coarse pose predictions collapsed, so training matches the aggregated coarse-pose posterior to its uniform prior.Lighting uses the same coarse-and-fine decomposition and aggregated-posterior regularization toward a uniform prior.Source: §4 · p. 7
  • The minibatch objective weights KL and pose-and-lighting prior-matching terms, and gradients are propagated through an approximate differentiable renderer.The method uses ADAM with gradient clipping and the reparameterization trick for Gaussian variables.Source: §4 · p. 8

5 Experiments

The experiments assess shape generation and reconstruction across object classes, mesh representations, lighting and loss settings, latent dimensions, views, and comparisons with prior methods. They find that shading—especially coloured lighting—improves reconstruction over silhouette-based training, while silhouette-only results vary across prior-method comparisons.

5.1 Generating and reconstructing diverse object classes

  • Models trained for 13 classes generate realistic, diverse shapes, including smooth surfaces and concave classes that voxel-based methods cannot faithfully represent.Examples include different sofa designs, aircraft styles, and boats with or without sails.Source: §5 · p. 8, §5.1 · p. 9, Figure 5 · p. 10
  • Reconstruction accuracy varies by class; pose supervision improves it, while shading enables results on concave classes comparable to or better than several non-concave classes.Without pose supervision, the model also disentangles pose from shape; apparent pose errors for rotationally symmetric classes reflect their ambiguous canonical frame.Source: §5.1 · pp. 9, 13, Table 1 · p. 9

5.2 Comparing mesh parameterisations

  • The best mesh parameterisation depends on the class: subdivision suits curved cars, full-block chairs, and both full-block and subdivision aircraft, while ortho-block works for sofas.Subdivision achieves the best average reconstruction across the four classes, with and without pose supervision.Source: Figure 7 · p. 12, §5.2 · p. 13, Table 2 · p. 14

5.3 Lighting

  • Coloured lighting and shading loss improve reconstruction, and the model can exploit shading even when lighting varies and its angle is unknown.With varying lighting, pose-supervised reconstruction is only slightly worse on average than with fixed, known lighting; without pose supervision, the model must disentangle shape, pose, and lighting.Source: §5.3 · p. 13, Table 3 · p. 14, Table 4 · p. 14

5.4 Latent space structure

  • Increasing latent dimensionality from 6 to 12 improves test reconstruction across classes, but further increases help aircraft and sofas while reducing performance for cars and chairs.Linear interpolation between reconstructed shapes produces gradual, smooth deformations whose intermediate shapes remain realistic.Source: Figure 9 · p. 13, §5.4 · pp. 14–15, Figure 10 · p. 14

5.6 Comparison to previous and concurrent works

  • Our colour reconstruction results are comparable to methods with full 3D supervision, despite using only 2D images.The comparison uses three training views per object and fixed lighting rotation; the compared 2D-supervised methods use pose supervision and multiple views.Source: Table 6 · p. 15, §5.6 · p. 15

5.5 Multi-view training/testing

  • Training with multiple views improves results, including when the model is tested with a single view, because the additional training information strengthens its model of valid shapes.The experiments compare multiple views at training only and at both training and test time.Source: Table 5 · p. 14, §5.5 · p. 15

5.6 Comparison to previous and concurrent works

  • Compared with voxel-based generation, mesh outputs are cleaner and capture narrow chair legs, diagonal aircraft-wing edges, and smoothly curved car and vase surfaces.The compared model is trained on single views without pose annotations; unlike it, this method also learns concave classes by using shading.Source: §5.6 · p. 15, Figure 11 · p. 16
  • With silhouettes alone, the method matches Kato et al. (2018) at 62.5% average IoU, doing worse on aeroplanes and chairs, better on cars, and identically on sofas.Adding shading to the loss significantly improves results; voxel-based methods by Yan et al. (2016), Tulsiani et al. (2017b), and Yang et al. (2018) cannot exploit shading.Source: §5.6 · p. 16
  • With coloured lighting, the method reaches 66.8% IoU versus 62.5% for Kato et al. (2018), leading on cars and sofas and matching the best method on chairs and aeroplanes.The coloured-lighting results also approach Fan et al. (2017) at 67.0% and Richter and Roth (2018) at 68.2%, both trained with full 3D supervision.Source: §5.6 · p. 16
  • On the same coloured-lighting images, the method outperforms Yan et al. (2016) with their silhouette loss, and it beats Tulsiani et al. (2017b) even when they use depth supervision.With colour, its 66.8% average IoU approaches the full-3D-supervised results of Fan et al. (2017) at 67.0% and Richter and Roth (2018) at 68.2%.Source: §5.6 · p. 16
  • With three views per instance, white-light shading-loss training beats Tulsiani et al. (2018) on IoU and IoU|θ, while coloured lighting improves results further through stronger surface-normal information.Tulsiani et al. (2018) requires multiple views during training and varies elevation as well as azimuth, making its task somewhat harder.Source: §5.6 · p. 16
  • Restricted to silhouettes, the method performs significantly worse than Tulsiani et al. (2018) across all three object classes.This contrasts with its shading-loss results, which outperform Tulsiani et al. (2018) even when that method uses depth supervision.Source: §5.6 · pp. 16–17

6 Conclusion

The framework generates and reconstructs 3D meshes under varied supervision, including single-view 2D training without pose annotations. It exploits shading for stronger reconstruction results and supports concave classes, while its use has limits involving image realism, mesh parameterisation, and silhouette-only inputs.

6 Conclusion

  • Shading yields higher performance than silhouette-based methods, supports learning concave classes, and brings results close to full 3D-supervised methods.The conclusion also reports higher performance than methods with depth supervision.Source: §6 · p. 17
  • The framework supports mesh generation and reconstruction with weaker supervision, including one view per training instance and no pose annotations.Without pose supervision, it disentangles shape from pose; with unknown lighting, it disentangles lighting from surface orientation.Source: §6 · p. 17
  • Mesh outputs produce more visually pleasing results than prior voxel-based works.Source: §6 · p. 17
  • Natural-image use requires more realistic materials and lighting because reconstruction matches input images by pixel-level L2 distance.Source: §6 · p. 17
  • Best results for a given class require the user to select a suitable mesh parameterisation.Subdivision gives reasonable results across all classes, while other parameterisations work better for particular classes.Source: §6 · p. 17
  • With multiple views but only silhouettes as input, specialized discriminative methods outperform this approach.Source: §6 · p. 17

A Network architectures

The decoder maps a latent embedding to mesh parameters, while an image encoder produces features used to parameterize variational distributions for shape, pose, and lighting.

A Network architectures

  • The decoder maps latent embedding z through a 32-channel fully connected layer with ReLU, then outputs mesh parameters in a second fully connected layer.Outputs are vertex offsets for subdivision meshes, or locations, scales, and rotations for primitive-based parameterisations; primitive scales use softplus.Source: A Network architectures · p. 17
  • The encoder is a CNN operating on 128 × 96 RGB images and produces a 128-dimensional feature vector.Its layers use batch normalisation and ReLU; convolution, max-pooling, and a final fully connected layer form the feature extractor.Source: A Network architectures · p. 17
  • Further fully connected layers map image features to variational-distribution parameters, using activations suited to bounded means, positive standard deviations, and coarse-rotation probabilities.Means for θfine and λfine use scaled tanh, standard deviations use softplus, and θcoarse and λcoarse use softmax outputs.Source: A Network architectures · p. 17

B Hyperparameters

Training uses fixed learning rates, gradient clipping, and weighted KL and prior-matching losses. The appendix also specifies rotation bins and mesh-parameterisation settings.

B Hyperparameters

  • The learning rate is 10^-3 throughout training, except rotation-decoder parameters for full-block parameterisation, which use 10^-4.Source: B Hyperparameters · p. 17
  • Training clips the global Euclidean gradient norm to at most 5 and sets the KL loss weight β to 10^3.Source: B Hyperparameters · p. 17
  • The settings use 12 object azimuth bins, 3 lighting azimuth bins over [0, π), 6 ortho-blocks or 12 full-blocks, and 4 subdivision levels.The renderer uses double-sided lighting calculations.Source: B Hyperparameters · p. 17

Related papers

Loading 1901.06447v2…