Source-linked AI summary
InverseRenderNet: Learning single image inverse rendering
Ye Yu, William A. P. Smith
TL;DR
Single-image inverse rendering is ill-posed and lacks reliable ground-truth labels for uncontrolled outdoor scenes. The paper trains a fully convolutional network with differentiable-renderer self-supervision, statistical illumination priors, and offline MVS supervision, reporting that inverse rendering can be learned from real uncontrolled images and that shape-from-shading in the wild is possible.
Problem
Inverse rendering from a single uncontrolled image requires estimating shape, reflectance, and illumination without reliable ground-truth labels for outdoor scenes.
Method
A fully convolutional network regresses albedo and normal maps, estimates spherical-harmonic lighting differentiably, and learns with renderer losses, illumination priors, and MVS supervision.
Results
The authors report that inverse rendering can be learned from real-world images in uncontrolled conditions and that shape-from-shading in the wild is possible.
Takeaways & Limitations
MVS supervision provides a route to learning inverse rendering from real uncontrolled images rather than relying only on unavailable ground truth.
Takeaways & Limitations
Training is constrained to outdoor web imagery and the explicit local illumination model cannot predict cast shadows.
Abstract
from arXiv · showhide
We show how to train a fully convolutional neural network to perform inverse rendering from a single, uncontrolled image. The network takes an RGB image as input, regresses albedo and normal maps from which we compute lighting coefficients. Our network is trained using large uncontrolled image collections without ground truth. By incorporating a differentiable renderer, our network can learn from self-supervision. Since the problem is ill-posed we introduce additional supervision: 1. We learn a statistical natural illumination prior, 2. Our key insight is to perform offline multiview stereo (MVS) on images containing rich illumination variation. From the MVS pose and depth maps, we can cross project between overlapping views such that Siamese training can be used to ensure consistent estimation of photometric invariants. MVS depth also provides direct coarse supervision for normal map estimation. We believe this is the first attempt to use MVS supervision for learning inverse rendering.
1. Introduction
The paper tackles inverse rendering from a single uncontrolled image by estimating shape, reflectance, and illumination without ground-truth labels. It combines differentiable-renderer self-supervision with statistical priors and offline MVS supervision.
- Inverse rendering estimates illumination, reflectance, and shape from appearance; this work predicts all three from one uncontrolled image.The outputs are a normal map, diffuse albedo map, and spherical harmonic lighting coefficients.
- A fully convolutional image-to-image network replaces classical energy minimization, while a differentiable renderer supplies self-supervision because labels are unavailable.The renderer computes an appearance loss from the estimated quantities.
- Single-image inverse rendering is ambiguous, so the method adds statistical lighting and albedo smoothness priors plus offline multiview reconstruction supervision.The data term alone permits degenerate explanations such as image-colored albedo, planar normals, and arbitrary illumination.
- MVS enables Siamese supervision by cross-projecting photometric invariants across overlapping views with different illumination.MVS depth maps also provide coarse direct supervision for normal estimation.
- The paper positions MVS supervision as a way to learn inverse rendering from real, uncontrolled images rather than synthetic data or restricted object classes.It identifies this as a first use of MVS supervision for inverse rendering.
2. Related work
Prior work addresses inverse-rendering components or restricted settings, but generally relies on multiple images, object-specific assumptions, synthetic supervision, or limited geometric and photometric models. The paper situates its approach as a more general self-supervised alternative.
- Classical methods recover geometry from multiview images or appearance from varying-illumination images, but illumination remains baked into multiview texture maps.Attempts to combine geometry and photometry typically rely on known geometry or MVS initialization.
- Deep depth and normal-estimation methods improve shape prediction, but the cited approaches use supervised or synthetic training.Synthetic training can leave real-world generalization uncertain.
- Intrinsic image decomposition separates reflectance and shading but not shape from illumination, and existing deep methods use synthetic supervision or self-supervision on time-lapse data.Time-lapse approaches use reconstruction, reflectance-consistency, or gradient-based constraints.
- A hybrid method uses a CNN-estimated depth map to constrain classical intrinsic-image optimization.
- Deep inverse-rendering studies commonly restrict object class, geometry, lighting, or rely on synthetic pretraining, while the cited full-problem approach uses a shading model requiring synthetic supervision.These restrictions limit the generality of prior settings described by the paper.
3. Preliminaries
The paper models image formation with perspective projection, surface normals, and Lambertian spherical-harmonic illumination. Its appearance model omits several high-frequency effects and applies a fixed gamma transformation to predicted intensities.
- Perspective projection maps 3D world coordinates to image coordinates using camera intrinsics and extrinsics.The camera matrix includes focal length and principal point, while pose uses rotation and translation.
- Surface orientation is preferred over depth because local reflectance and appearance depend on surface normals.Although normals can be computed from perspective depth using image derivatives and camera parameters, the network directly estimates a normal map.
- The network represents viewer-centered normals through two surface-gradient elements and converts them into a 3D normal with a fixed layer.
- Appearance is approximated with a Lambertian diffuse model under order-2 spherical-harmonic environment lighting.The lighting coefficients are represented by a 3×9 RGB lighting matrix and combined with diffuse albedo.
- The appearance model neglects high-frequency illumination effects, cast shadows, and interreflections, then applies a fixed gamma value of 2.2.The authors report that this model works well for typical outdoor scenes.
4. Architecture
The network predicts albedo and surface normals from a single image, then derives lighting implicitly through differentiable image-wide estimation. This preserves a fully convolutional architecture while allowing variable-size inference.
- The image-to-image network regresses albedo and normal maps from a single input image.
- Separate deconvolutional decoders predict RGB albedo and surface-normal representations after a shared encoder.
- Implicit lighting prediction: Lighting is inferred from the input image together with estimated albedo and normals rather than predicted by a fully connected branch.
- Implicit lighting prediction: The method linearizes the image and stacks image, albedo, and normal quantities across all pixels for lighting estimation.
- Implicit lighting prediction: A differentiable least-squares solution estimates spherical-harmonic lighting coefficients from inferred shading and normal bases.The paper gives a pseudoinverse solution, L = (I ⊘A)B(N)+, so lighting-dependent losses can backpropagate into the network.
5. Supervision
Because single-image inverse rendering is ill-posed, the paper combines differentiable rendering with statistical lighting constraints and offline MVS supervision. MVS supplies cross-view geometry for albedo consistency and direct normal guidance.
- Self-supervision via differentiable rendering: Appearance reconstruction alone is insufficient because infinitely many albedo, normal, and illumination combinations can explain one image.
- Self-supervision via differentiable rendering: The appearance loss compares rendered and observed images in LAB space, whose transformation is used for perceptually more convincing errors.
- Natural illumination model and prior: A statistical illumination model constrains lighting to natural environments using principal components learned from 79 outdoor HDR panoramas.The model uses D = 18 dimensions and applies an L2 prior to the lighting parameter vector.
- Multiview stereo supervision: Offline MVS provides camera poses and dense depth from uncontrolled image collections, while inference still uses only single images of novel scenes.The MVS outputs provide three supervision sources, including cross-view correspondence and normal guidance.
- Multiview stereo supervision: MVS depth supports coarse direct normal supervision through an angular loss between guide and predicted surface normals.
- Multiview stereo supervision: Overlapping views are trained Siamese-style so cross-projected albedo estimates remain consistent despite illumination changes.Pairs without sufficient illumination variation are discarded, and the consistency loss accounts for overall inter-image scale ambiguity.
- Multiview stereo supervision: A cross-rendering loss combines cross-projected albedo with estimated normals and albedo to measure rendered appearance error.
- Multiview stereo supervision: Additional albedo priors use chromaticity-weighted smoothness and pseudo-supervision to reduce shading–albedo ambiguity and avoid trivial solutions.
6. Training
Training minimizes a weighted combination of appearance, MVS-normal, cross-view albedo, cross-rendering, and albedo-prior losses. The strategy uses MegaDepth preprocessing and a normal-guided pretraining stage before full self-supervision.
- The total objective weights appearance, normal-map, albedo-consistency, cross-rendering, albedo-smoothness, and albedo-pseudo-supervision losses.
- Training uses MegaDepth images with estimated dense depth and camera calibration, cropped and resized to maximize pixels with valid depth.
- Mini-batches contain overlapping image pairs with illumination variation, while sky regions are masked before network input.
- Pretraining fixes MVS normals for illumination and appearance losses so the normal decoder learns from direct supervision before full self-supervision begins.
7. Evaluation
The method is evaluated on intrinsic-image and inverse-rendering benchmarks, including uncontrolled scenes, with comparisons based on WHDR, albedo, and normal-estimation performance. It achieves the second-best IIW result among methods not trained on IIW and the best normal, MSE, and DSSIM performance on MegaDepth.
- Evaluation setup: The evaluation uses IIW for intrinsic decomposition and MegaDepth for inverse rendering on unobserved scenes.IIW is primarily indoor, whereas the training data and illumination model are outdoor-focused.
- Evaluation setup: IIW results use WHDR percentage, with lower values indicating better performance, and report each network's training dataset.
- IIW results: Despite no IIW fine-tuning and outdoor-oriented training, the method achieves the second-best performance among methods not trained on IIW.The network is evaluated directly on IIW, with albedo predictions rescaled to (0.5, 1).
- MegaDepth results: MegaDepth evaluates normals against MVS geometry and albedo against a multiview inverse-rendering estimate, with both treated as approximate ground truth.The estimated albedo reference includes ambient occlusion baked in.
- MegaDepth results: On MegaDepth, the network performs best in normal prediction and achieves the best MSE and DSSIM.Figure 6 presents qualitative inverse-rendering examples.
8. Conclusions
The paper concludes that inverse rendering can be learned from real-world images in uncontrolled conditions, with shape-from-shading in the wild shown to be possible. It identifies broader reflectance and illumination models, depth integration, and additional consistency losses as extensions.
- The authors report the first demonstration of learning inverse rendering from real-world images in uncontrolled conditions.
- They attribute the result to diverse cues including shading, texture, ambient occlusion, and high-level semantic concepts.Examples include recognizing windows and man-made buildings to constrain plausible shape and plane configurations.
- Future work includes more general reflectance models and global illumination effects such as shadowing.
- The network could be combined with depth prediction, either by merging independently predicted maps or by training a unified network.
- Additional intrinsic-image losses, including reflectance consistency from timelapse data, could improve albedo estimates.