Source-linked AI summary
Self-calibrating Deep Photometric Stereo Networks
Guanying Chen, Kai Han, Boxin Shi, Yasuyuki Matsushita, Kwan-Yee K. Wong
TL;DR
Photometric stereo must recover surface normals when light directions are unknown, a setting that existing deep methods designed for known lighting do not handle. SDPS-Net first estimates lighting from images and then estimates normals using those lighting estimates and the images. It significantly outperformed existing state-of-the-art uncalibrated methods on synthetic and real datasets.
Problem
Uncalibrated photometric stereo remains an open challenge, and deep methods developed for calibrated settings do not handle unknown lighting.
Source: §1 · p. 1Method
SDPS-Net uses one network to estimate light directions and intensities from an arbitrary number of images, followed by a network that predicts normals from the estimated lighting and images.
Source: §4 · p. 3Results
SDPS-Net significantly outperformed existing state-of-the-art uncalibrated photometric stereo methods on synthetic and real datasets.
Source: §6 · p. 8Takeaways & Limitations
The lighting-estimation stage can be integrated with existing calibrated methods to handle unknown lighting.
Source: §1 · p. 1Because the framework is trained only on uniform-material surfaces, it may not perform well with steep color changes from multi-material surfaces.
Source: §6 · p. 8
Abstract
from arXiv · showhide
This paper proposes an uncalibrated photometric stereo method for non-Lambertian scenes based on deep learning. Unlike previous approaches that heavily rely on assumptions of specific reflectances and light source distributions, our method is able to determine both shape and light directions of a scene with unknown arbitrary reflectances observed under unknown varying light directions. To achieve this goal, we propose a two-stage deep learning architecture, called SDPS-Net, which can effectively take advantage of intermediate supervision, resulting in reduced learning difficulty compared to a single-stage model. Experiments on both synthetic and real datasets show that our proposed approach significantly outperforms previous uncalibrated photometric stereo methods.
1. Introduction
Uncalibrated photometric stereo seeks surface normals when lighting is unknown, but existing methods face reflectance or light-distribution constraints. SDPS-Net addresses this setting with separate lighting-estimation and normal-estimation stages.
1. Introduction
- Uncalibrated photometric stereo seeks normals from images with unknown lighting; this work targets surfaces with general, unknown isotropic reflectance.Calibrated methods assume known light directions, which require tedious calibration.Source: §1 · p. 1
- Earlier uncalibrated methods often assume Lambertian reflectance, while general-BRDF methods rely on uniformly distributed light directions.Some approaches also focus on resolving shape-light ambiguity such as GBR.Source: §1 · p. 1
- Deep photometric-stereo methods [25, 31, 15, 5] have shown promise with known lighting, but cannot handle unknown light directions.Directly mapping images to normals without lighting input performs worse than models given both images and lighting [5].Source: §1 · p. 1
- SDPS-Net estimates light directions and intensities from an arbitrary number of images, then predicts normals from those estimates and the images.The explicit lighting stage provides intermediate supervision and can be integrated with calibrated methods for unknown lighting.Source: §1 · p. 1
2. Related Work
Related work spans calibrated learning-based photometric stereo, uncalibrated methods for resolving lighting ambiguity, and learned lighting estimation. The paper contrasts SDPS-Net with approaches that require known lighting, extra cues, or assumptions about light distributions.
Learning based photometric stereo
Uncalibrated photometric stereo
- For Lambertian objects, unknown lighting leaves a 3 × 3 ambiguity, reducible to a 3-parameter GBR ambiguity with integrability; prior methods use additional cues to resolve it.Non-Lambertian methods have used reference objects or reflectance cues, while some require lights distributed uniformly over the sphere.Source: §2 · p. 2
- Unlike UPS-FCN [5], which directly regresses normals from intensity-normalized images, SDPS-Net estimates lighting before normals and avoids extra cues or light-distribution assumptions.The paper also distinguishes its multi-image directional-light estimation for general-shape, non-Lambertian scenes from prior single-image lighting estimators.Source: §2 · p. 2
3. Image Formation Model
The image-formation model considers non-Lambertian surfaces under specified camera and lighting assumptions. The task is to infer surface normals from measured intensities by first estimating lighting and then using it with the observations.
3. Image Formation Model
- The model assumes an orthographic camera, linear radiometric response, white directional lights from the upper hemisphere, and a viewer-facing view direction.Surface appearance is described by a general isotropic BRDF.Source: §3 · p. 2
- The measured intensity model accounts for attached shadows through max(·, 0), while ε represents global illumination effects and noise.The setup describes a surface point under an incoming light direction and intensity.Source: §3 · p. 2
- Given measured intensities alone, the goal is to estimate surface normals by first estimating lighting and then solving for normals using the estimated lighting and intensities.Figure 1 depicts the Lighting Calibration Network and Normal Estimation Network as the two stages.Source: §3 · p. 2, Figure 1 · p. 3
4. Learning Uncalibrated Photometric Stereo
SDPS-Net estimates lighting with LCNet, then uses those estimates and the input images to predict surface normals with NENet. Its design combines discretized lighting estimation with local-global features and trains on synthetic scenes with varied materials and lighting.
4. Learning Uncalibrated Photometric Stereo
- LCNet estimates light directions and intensities from the input images, and NENet uses those estimates and the images to predict a normal map.The two-stage design provides intermediate lighting estimates to the normal-estimation stage.Source: §4 · p. 3
4.1. Lighting Calibration Network
- Discretizing lighting makes a joint Kd^2 direction classification costly, so LCNet predicts azimuth and elevation in separate Kd-class problems.The method also divides the possible intensity range into Ke classes.Source: §4.1 · p. 3
- LCNet concatenates each image's local feature with a max-pooled global feature for lighting prediction; including the object mask can improve performance.The global feature aggregates information across observations, while each local feature represents one observation.Source: §4.1 · p. 4
4.2. Normal Estimation Network
- NENet normalizes images with LCNet's predicted intensities and combines them with predicted light directions; training on discretized estimated lighting makes it more robust to lighting noise than PS-FCN.Its architecture otherwise resembles PS-FCN, with a shared-weight feature extractor, aggregation layer, and normal-regression sub-network.Source: §4.2 · p. 4
4.3. Training Data
- Training uses 85,212 synthetic Blobby and Sculpture samples, each rendered under 64 upper-hemisphere light directions.The scenes include diverse MERL materials and rendered cast shadows and inter-reflections; light intensities are randomly scaled over [0.2, 2.0].Source: §4.3 · p. 4
5. Experimental Results
Experiments analyze SDPS-Net on synthetic data and evaluate it on real datasets, comparing network variants and previous methods.
5.1. Network Analysis with Synthetic Data
- Discretization added only marginal normal-estimation error for PS-FCN when Kd ≥30; the implementation used Kd = 36 and Ke = 20.The authors chose relatively sparse lighting discretization partly to help NENet tolerate small lighting-estimation errors at test time.Source: §5.1 · p. 5, Figure 4 · p. 5
- Classification-based LCNet outperformed regression-based LCNetreg for lighting estimation, while object masks and local-global feature fusion improved its results.The paper associates masks with occluding-contour information and feature fusion with information from multiple observations.Source: §5.1 · p. 6, Table 1 · p. 6
- On MERLTest, SDPS-Net achieved average MAEs of 2.71 on SPHERE and 4.09 on BUNNY, comparable to fully calibrated PS-FCN.NENet also performed better than PS-FCN with potentially noisy estimated lightings at test time.Source: §5.1 · p. 6, Table 2 · p. 6
- With comparable inputs and parameter counts, SDPS-Net significantly outperformed the best single-stage model, especially on complex BUNNY geometry.Wider or deeper networks, masks, and additional lighting supervision improved single-stage performance only to some extent.Source: §5.1 · p. 6, Table 2 · p. 6
- On BUNNY, SDPS-Net handled four BRDF types better than PF14 on surfaces with specular highlights, while both methods degraded under biased lighting.PF14 also degraded on non-Lambertian surfaces and could not reliably estimate light intensities across all BRDFs.Source: §5.1 · p. 7, Figure 6 · p. 7
5.2. Evaluation on Real Datasets
- On DiLiGenT, SDPS-Net achieved state-of-the-art normal estimates on almost all objects except BEAR, with average MAE 9.51 and a 1.52 average-MAE improvement over LCNetreg+NENet†.Table 3 reports the benchmark results; the passage attributes the improvement to the classification-based LCNet.Source: §5.2 · p. 7, Table 3 · p. 8
- SDPS-Net estimated lighting accurately on Gourd&Apple and Light Stage Data Gallery and produced visually pleasing normal maps for both datasets.Lighting was evaluated quantitatively, while normal estimation was evaluated qualitatively for these datasets.Source: §5.2 · pp. 7–8
6. Conclusion and Discussion
SDPS-Net estimates lighting from an arbitrary number of images, then predicts surface normals using those estimates and the input images. The authors report that it significantly outperformed existing state-of-the-art uncalibrated photometric stereo methods on synthetic and real datasets.
6. Conclusion and Discussion
- SDPS-Net estimates light directions and intensities from an arbitrary number of images, then predicts the object's normal map using those estimates and the input images.The two-stage framework uses intermediate supervision to reduce learning difficulty and improve final normal estimation.Source: §6 · p. 8
- The reported real-dataset results include Gourd&Apple and the Light Stage Data Gallery; Table 4 covers lighting estimation on those datasets.Source: Table 4 · p. 8, §6 · p. 8
- Experiments on synthetic and real datasets showed that SDPS-Net significantly outperformed existing state-of-the-art uncalibrated photometric stereo methods.The conclusion describes lighting-estimation results for Gourd&Apple and the Light Stage Data Gallery in Table 4.Source: §6 · p. 8, Table 4 · p. 8
- Training only on uniform-material surfaces may limit performance on multi-material surfaces with steep color changes.The authors identify better training data and network architectures for spatially varying BRDFs as future work.Source: §6 · p. 8