Source-linked AI summary
SfSNet: Learning Shape, Reflectance and Illuminance of Faces in the Wild
Soumyadip Sengupta, Angjoo Kanazawa, Carlos D. Castillo, David Jacobs
TL;DR
SfSNet addresses inverse rendering of unconstrained human faces, where shape, reflectance, and illuminance must be decomposed despite limited real-world ground-truth data. It combines a lambertian-inspired residual architecture with labeled synthetic and unlabeled real images, and reports stronger results than prior inverse-rendering and individual-component methods.
Problem
The central problem is decomposing unconstrained real-world faces into shape, reflectance, and illuminance despite limited ground-truth data for real images.
Method
SfSNet jointly learns normal, albedo, and lighting with a lambertian-inspired residual architecture using labeled synthetic and unlabeled real images.
Results
SfSNet significantly outperforms state-of-the-art methods for inverse rendering and independent normal or lighting estimation.
Takeaways & Limitations
The mixed-data training paradigm uses synthetic supervision for low-frequency variation and real-image reconstruction loss for high-frequency details.
Takeaways & Limitations
The approach assumes image formation under lambertian reflectance.
Abstract
from arXiv · showhide
We present SfSNet, an end-to-end learning framework for producing an accurate decomposition of an unconstrained human face image into shape, reflectance and illuminance. SfSNet is designed to reflect a physical lambertian rendering model. SfSNet learns from a mixture of labeled synthetic and unlabeled real world images. This allows the network to capture low frequency variations from synthetic and high frequency details from real images through the photometric reconstruction loss. SfSNet consists of a new decomposition architecture with residual blocks that learns a complete separation of albedo and normal. This is used along with the original image to predict lighting. SfSNet produces significantly better quantitative and qualitative results than state-of-the-art methods for inverse rendering and independent normal and illumination estimation.
1. Introduction
SfSNet decomposes unconstrained real-world faces into shape, reflectance, and illuminance using a physical lambertian model and mixed synthetic-real training. Its architecture and training strategy address limited real-world ground truth and yield stronger reconstruction and component-estimation results.
- Motivation: Inverse rendering decomposes unconstrained faces into shape, reflectance, and illuminance, supporting applications such as relighting, light transfer, and augmented reality.Real-world ground-truth reflectance and illuminance data are difficult to collect at scale, limiting supervised learning on faces in the wild.
- Approach: SfSNet learns normal, albedo, and lighting from labeled synthetic and unlabeled real images under a lambertian rendering model.The mixed data provide supervision for available components while reconstruction loss supplies cues from real images.
- Architecture: Residual-block decomposition separates normal and albedo features, which are combined with image features to estimate lighting and produce shading.The design is intended to mimic physical image formation rather than estimate all outputs directly from image features.
- Results: 47% improvement in normal estimation accuracy, from 37% to 84%, is reported against a recent independent normal-estimation approach.SfSNet is also reported to outperform Pix2Vertex by 19% and LDAN by 12.5% on their respective comparisons.
- Results: SfSNet produces significantly better reconstruction than recent inverse-rendering methods on the same showcased images.The paper evaluates the approach on real-world CelebA data and compares it with methods estimating complete decompositions or individual components.
2. Related Work
Prior work spans classical priors and learning-based approaches for inverse rendering, alongside methods that estimate facial shape or illumination independently. These approaches include optimization-based decomposition, learned priors, synthetic-data shape estimation, and skip-connection architectures.
- Classical approaches: Classical inverse rendering formulates decomposition into shape, reflectance, and illuminance through intrinsic-image, Shape from Shading, and optimization-prior approaches.SIRFS is described as decomposing objects into surface normal, albedo, and lighting under lambertian reflection using hand-designed priors.
- Learning-based inverse rendering: Early learning-based methods replaced hand-designed priors with learned representations but showed limited performance on unconstrained real-world faces.The cited approaches include Deep Belief Nets and convolutional encoder-decoder networks.
- Individual-component estimation: A separate research direction estimates facial shape or illumination independently, often learning shape from synthetic data or fitting a 3D morphable model.SfSNet is contrasted with these methods because it performs complete decomposition while training on unlabeled real images.
- Architectures: Skip-connection architectures can transfer high-frequency information effectively, but prior inverse-rendering uses may fail to produce meaningful disentangled representations.This limitation motivates architectures that reason about the separation of high-frequency variations between normal and albedo.
3. Approach
SfSNet combines labeled synthetic and unlabeled real data with photometric reconstruction under a Lambertian model to decompose faces into normal, albedo, and lighting. Its residual-block architecture separates normal and albedo features and improves lighting estimation over skip-connection designs.
- 3. Approach: The method assumes Lambertian image formation and represents lighting with 27 spherical-harmonic coefficients, using synthetic supervision for normals, albedo, and lighting alongside unlabeled real images.The coefficients comprise nine second-order spherical-harmonic coefficients for each RGB channel.
- 3.1. ‘SfS-supervision’ Training: SfS-supervision trains on synthetic labels, generates pseudo-supervision for real images, and combines these signals with photometric reconstruction loss.Pseudo-supervision helps prevent decomposition components from collapsing into trivial solutions, while reconstruction uses shading cues from real images.
- 3.1. ‘SfS-supervision’ Training: The loss combines reconstruction, normal, albedo, and lighting terms, using L1 losses for pixel-based quantities and L2 loss for 27-dimensional lighting coefficients.The weights are λrecon, λN and λA = 0.5, and λL = 0.1.
- 3.2. Proposed Architecture: Residual blocks separately transform image features into normal and albedo features, which are combined with image features to estimate lighting.The Light Estimator concatenates these features and applies 1x1 convolutions, average pooling, and a fully connected layer.
- 3.3. Implementation Details: Synthetic faces are generated from 3DMMs across viewpoints, reflectance, and illumination using 27-dimensional spherical-harmonic lighting coefficients.The lighting distribution is estimated by fitting 3DMMs to real CelebA images with classical methods.
4. Comparison with State-of-the-art Methods
SfSNet is compared with state-of-the-art methods for inverse rendering, facial normal recovery, and lighting estimation, showing stronger qualitative and quantitative results across these tasks.
- Light Transfer: SfSNet supports light transfer by transferring source-image lighting to a target image while preserving the decomposition-based editing setup.The transfer operation uses source lighting and target content to produce a transfer image.
- Inverse Rendering: SfSNet produces more realistic inverse-rendering decompositions than Neural Face and MoFA on their showcased unconstrained real-world images.For light transfer, SfSNet removes the target's orangey illumination more effectively than Neural Face.
- Facial Shape Recovery: SfSNet produces fewer artifacts and more realistic normals and shading than Pix2Vertex under expression and non-ambient illumination.Pix2Vertex can capture more details because it uses higher-resolution inputs and produces meshes, while SfSNet is around 2000× faster.
- Lighting Estimation: SfSNet improves lighting classification accuracy over LDAN by 12.6% on the 19-way MultiPIE evaluation.MultiPIE photographs 250 individuals under 19 lighting conditions, testing consistency of estimated lighting.
5. Results on CelebA
On CelebA, SfSNet performs best on frontal faces with limited expression and ambient lighting, while difficult shadows, specularity, and occlusions produce the worst reconstructions.
- Best and Worst Reconstructions: The best 5% of CelebA reconstructions are frontal faces with little or no expression and easy ambient lighting.The paper relights each test face with a directional source to expose decomposition flaws.
- Best and Worst Reconstructions: The worst 5% contain large cast shadows, specularity, and occlusions, although recovered normals and lighting remain reasonable.Light-transfer examples, including opposite source and target lighting directions, further test the decomposition.
6. Ablation Studies
Ablations show that mixed synthetic–real training and SfSNet's lighting-prediction architecture each contribute to the decomposition and lighting results.
- Role of SfS-supervision Training: SfS-supervision training improves significantly over pseudo-supervision and synthetic-only training by using shading information to add reconstruction details.Real-image reconstruction loss alone can lead to unconstrained, trivial decompositions.
- Role of SfSNet Architecture: SfSNet predicts lighting from image, normal, and albedo representations, unlike the compared skip-connection architecture that predicts lighting directly from the image.The comparison uses a fully convolutional SkipNet+ with separate normal and albedo decoders.
- Role of SfSNet Architecture: SfSNet outperforms SkipNet+ in lighting prediction even though their reconstruction errors are similar.The paper attributes this to SfSNet learning an informative latent subspace for albedo and normal before estimating lighting.
7. Conclusion
The paper concludes that SfSNet jointly decomposes faces into normal, albedo, and lighting using mixed synthetic and real data, outperforming methods for inverse rendering and individual component estimation.
- Conclusion: SfSNet learns inverse face rendering from labeled synthetic and unlabeled real images using residual blocks to separate normal and albedo subspaces.Image features are combined with these representations to estimate lighting.
- Conclusion: Detailed qualitative and quantitative evaluations show that SfSNet significantly outperforms state-of-the-art methods for inverse rendering, normal estimation, and lighting estimation.The paper's acknowledgments identify National Science Foundation support and collaborators who provided visual results and test images.
8. Appendix
The appendix details SfSNet’s residual-block architecture, compares it with SkipNet variants, and defines its spherical-harmonics rendering and reconstruction setup. It also presents qualitative comparisons on CelebA and against several inverse-rendering methods.
- SfSNet Architecture: SfSNet uses separate residual-block and convolution branches to produce normal and albedo maps, while a light estimator predicts 27 spherical-harmonics coefficients.The normal and albedo branches do not share weights; the light estimator concatenates features from the shared convolution block and both residual branches.
- SkipNet Architectures: SkipNet uses an encoder, an MLP bottleneck, separate normal and albedo decoders, and a lighting branch, whereas SkipNet+ increases capacity and removes the MLP bottleneck.SkipNet+ retains skip connections between encoder convolution layers and its decoders.
- Spherical Harmonics: Under the lambertian model, pixel intensity is rendered as albedo multiplied by spherical-harmonics shading from the surface normal and lighting coefficients.The basis is nine-dimensional per pixel, and the lighting representation uses coefficients independently for the three RGB channels.
- Spherical Harmonics: The reconstruction loss compares the original image with an image rendered from SfSNet’s inferred normal, albedo, and 27-dimensional lighting coefficients.This loss is illustrated using the relit CelebA results, where directional lighting and uniform albedo expose reconstructed-normal quality.
- Qualitative Results: On CelebA, SfSNet is comparable to Pix2Vertex under mostly ambient illumination, while showing more robust normal recovery under expressions and non-ambient lighting.Additional qualitative comparisons report more detailed shape and reflectance than MoFA and more realistic reconstructions than Neural Face.