Source-linked AI summary
Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
Shangzhe Wu, Christian Rupprecht, Andrea Vedaldi
TL;DR
The paper addresses unsupervised 3D reconstruction of deformable object categories from raw single-view images, where factorizing shape, appearance, lighting, and viewpoint is ill-posed. It introduces an illumination-aware, probabilistic symmetry autoencoder and reports high-fidelity reconstructions across faces, cats, and cars, outperforming a keypoint-supervised benchmark method.
Problem
The problem is learning 3D deformable object categories from unconstrained single-view images without external supervision or prior shape models.
Method
The method is an autoencoder that predicts depth, albedo, viewpoint, illumination, and confidence maps, using illumination-aware symmetry to handle probable asymmetry.
Results
The method recovers high-fidelity 3D shapes for human faces, cat faces, and cars and outperforms a recent keypoint-supervised reconstruction method on real faces.
Takeaways & Limitations
Symmetry and illumination provide strong cues for learning meaningful monocular reconstructions from raw images.
Takeaways & Limitations
The model assumes Lambertian shading and can reconstruct inaccurately under extreme lighting, highly non-Lambertian surfaces, noisy dark textures, and extreme poses.
Abstract
from arXiv · showhide
We propose a method to learn 3D deformable object categories from raw single-view images, without external supervision. The method is based on an autoencoder that factors each input image into depth, albedo, viewpoint and illumination. In order to disentangle these components without supervision, we use the fact that many object categories have, at least in principle, a symmetric structure. We show that reasoning about illumination allows us to exploit the underlying object symmetry even if the appearance is not symmetric due to shading. Furthermore, we model objects that are probably, but not certainly, symmetric by predicting a symmetry probability map, learned end-to-end with the other components of the model. Our experiments show that this method can recover very accurately the 3D shape of human faces, cat faces and cars from single-view images, without any supervision or a prior shape model. On benchmarks, we demonstrate superior accuracy compared to another method that uses supervision at the level of 2D image correspondences.
1. Introduction
The paper targets unsupervised 3D modeling of deformable object categories from unconstrained single-view images. It uses photo-geometric factorization, illumination-aware symmetry, and learned confidence maps to handle appearance and instance asymmetries.
- Problem: The task is to learn deformable-category 3D models from single-view images without 2D or 3D ground truth, keypoints, segmentation, depth maps, or prior shape models.This avoids annotation requirements and supports settings where only still images are available.
- Method: The proposed autoencoder decomposes each input image into albedo, depth, illumination, and viewpoint without direct supervision.The factors are recombined to reconstruct the input image.
- Method: Object symmetry supplies a geometric cue for constraining the otherwise ill-posed decomposition and obtaining a virtual second view by mirroring.Symmetric correspondences would enable stereo-style 3D reconstruction, motivating symmetry as a learning constraint.
- Method: The model explicitly represents illumination and predicts a dense per-pixel probability that a symmetric counterpart exists.These mechanisms address asymmetric shading, shape, and albedo rather than assuming perfect symmetry.
- Method: The end-to-end model learns all components, including confidence maps, directly from raw RGB data.Internal representations can also be flipped to enforce symmetry probabilistically.
- Results: Experiments on human faces, cat faces, and cars report high-fidelity reconstructions and higher fidelity than unsupervised alternatives, while outperforming a keypoint-supervised method on real faces.The trained face model also generalizes to paintings and cartoons without fine-tuning.
2. Related Work
Related work spans multi-view reconstruction, monocular shape cues, category-specific learning, adversarial view synthesis, and differentiable rendering. The paper positions its approach as raw-image category reconstruction with explicit photo-geometric factors and symmetry-based constraints.
- Comparison framework: Prior image-based 3D reconstruction methods differ in their required information, assumptions, and output representations.Table 1 compares selected methods across supervision, goals, and data.
- Structure from Motion: Traditional Structure from Motion uses multiple views and 2D keypoint matches, while monocular and non-rigid variants require additional training views, videos, or deformable-object formulations.These requirements contrast with learning from unconstrained single-view images.
- Shape from X: Shape-from-X methods recover geometry using cues including shading, silhouettes, texture, and symmetry.The paper is specifically inspired by shape from symmetry and shape from shading.
- Category-specific reconstruction: Category-specific monocular methods learn object priors from training data using supervision such as 3D ground truth, videos, stereo pairs, or 2D keypoints.The paper instead uses raw monocular views and a weak-symmetry assumption.
- Category-specific reconstruction: Recent raw-monocular approaches learn 2D landmarks or deformation fields, with some lifting these representations to 3D and decomposing albedo and shading afterward.These approaches provide context for the paper’s jointly learned photo-geometric decomposition.
- Adversarial learning: Adversarial methods hallucinate views or 3D representations, but some require viewpoint supervision, lack explicit 3D reconstruction, or do not quantify results.The cited methods differ in both supervision and reconstruction outputs.
- Differentiable rendering: Because the model generates images from an internal 3D representation, it requires a differentiable renderer; the implementation uses a textured mesh with one vertex per pixel.The authors report poor texture gradients from this renderer configuration because of high-frequency texture content.
3. Method
The method uses a photo-geometric autoencoder to decompose single images into depth, albedo, illumination, and viewpoint, then reconstruct them through lighting and reprojection. Symmetry is imposed through flipped reconstructions while modeling illumination and per-pixel uncertainty to accommodate asymmetric appearance.
- Photo-geometric autoencoding: The network maps an input image to depth, albedo, global light direction, and viewpoint factors that reconstruct the image.Depth is a canonical-view map, albedo is an RGB image, light direction lies on S2, and viewpoint represents a six-parameter transformation.
- Photo-geometric autoencoding: Lighting produces a shaded canonical image, and reprojection warps it using depth and viewpoint to generate the reconstructed input.The learning objective encourages the reconstructed image to match the original image.
- Photo-geometric autoencoding: The model separates illumination from albedo because albedo may remain symmetric under asymmetric shading, while shading also provides a cue about 3D shape.
- Probably symmetric objects: Symmetry is enforced indirectly by flipping predicted depth and albedo, generating a second reconstruction, and jointly matching both reconstructions to the input.The fixed flip axis is arbitrary provided it remains fixed; the two reconstruction losses are commensurate and jointly optimized.
- Probably symmetric objects: Confidence maps let the network reduce the influence of pixels likely to violate symmetry or contain uncertain reconstruction residuals.The confidence-weighted loss corresponds to negative log-likelihood under a factorized Laplacian model and supports self-calibration.
- Image formation model: The image-formation model derives surface normals from depth, applies directional and ambient lighting to albedo, and reprojects the result through a perspective camera.The assumed camera uses an approximately 10° field of view at a nominal object distance of about 1 m.
4. Experiments
Experiments evaluate the method on synthetic and real faces, cats, and cars, showing detailed reconstructions, effective symmetry handling, and strong comparisons against supervised and unsupervised baselines. Ablations and asymmetric perturbations confirm the importance of symmetry-related components, while failures arise under extreme lighting and non-Lambertian surfaces.
- Experimental setup: The method is evaluated on human-face, cat-face, and synthetic-car datasets, using scale-invariant depth error to account for projective scale ambiguity.BFM provides synthetic 3D ground truth, while CelebA, 3DFAW, cat datasets, and ShapeNet cover real or rendered category images.
- Baseline comparison: On BFM, the method largely outperforms constant-depth baselines and approaches supervised-training results, demonstrating instance-specific 3D reconstruction.The averaged ground-truth depth baseline has access to ground-truth information, making this comparison particularly informative for learned instance-specific structure.
- Ablation study: Removing albedo or depth flipping makes performance as low as the trivial baseline, while replacing computed shading with a predicted shading map also harms reconstruction.These ablations indicate that canonical symmetry and physically computed shading provide important shape-recovery cues.
- Ablation study: Disabling perceptual loss degrades image quality and reconstruction, whereas a self-supervised encoder performs similarly to the ImageNet-pretrained encoder.Removing confidence maps has little effect on highly symmetric BFM faces but increases variance.
- Asymmetry handling: With confidence estimates, the model reconstructs asymmetrically perturbed faces correctly with little accuracy loss, whereas removing confidence causes symmetric albedo predictions and frequent geometry failure.The perturbations cover 20%–50% of image area with blending strengths from 0.5 to 1.
- Qualitative and comparative results: Qualitative results recover fine facial details under extreme expressions, generalize to paintings and cartoons, and produce higher-quality facial reconstructions than the compared methods.The method also detects symmetry planes despite asymmetric texture and lighting, assigning low confidence to asymmetric regions.
- Failure cases: Despite strong performance, reconstructions become inaccurate under extreme lighting, highly non-Lambertian surfaces, noisy dark textures, and extreme poses.The training model assumes simple Lambertian shading and ignores shadows and specularity.
5. Conclusions
The method learns high-fidelity 3D models of deformable object categories from unconstrained single-view images without supervision. Symmetry and illumination provide strong cues for meaningful reconstruction, though more complex objects may require richer representations.
- The model learns deformable object categories from unconstrained single-view images using only an unsupervised reconstruction loss.
- It produces high-fidelity monocular 3D reconstructions of individual object instances.
- Symmetry and illumination act as strong cues for recovering shape and helping training converge to meaningful reconstructions.
- The model outperforms a state-of-the-art 3D reconstruction method that uses 2D keypoint supervision.
- The canonical-view depth-map representation suits roughly convex objects with natural canonical viewpoints, while complex objects may need multiple views, meshes, or voxel maps.
6. Supplementary Material
The supplementary material describes a differentiable forward-warping solution and the network and training configurations used for depth, albedo, viewpoint, lighting, and confidence prediction.
- Differentiable rendering layer: Forward warping is implemented by first warping depth with NMR, then deriving an inverse warp to bilinearly resample the canonical image.
- Differentiable rendering layer: Using NMR only for depth makes backpropagation faster and gradients more stable than directly warping the texture image.
- Network architecture: The model uses encoder networks for viewpoint and lighting, and encoder-decoder networks for depth, albedo, and confidence predictions.
- Training details: Training uses 30 epochs, batch size 64, input and output images of 64 × 64, and loss weights λf = 0.5 and λp = 1.
- Network architecture: The architecture defines separate output sizes for viewpoint and lighting, with one depth channel and three albedo channels.
7. Qualitative Results
Qualitative experiments show reconstructions across faces, cats, cars, drawings, and videos, while intrinsic decomposition enables re-lighting in a canonical view.
- Qualitative reconstructions: The supplementary results cover human faces, face paintings, drawings, cat faces, abstract cats, and synthetic cars.
- Figures: The qualitative ablation results are reported in Figure 9, while reconstruction examples are shown in Figures 11–16.
- Re-lighting: The model separates albedo and illumination, enabling objects to be re-lit under different lighting conditions in the canonical view.
- Video testing: Applied frame by frame to VoxCeleb videos, the model produces consistent, smooth reconstructions and recovers facial motion details accurately.