Source-linked AI summary
PS-FCN: A Flexible Learning Framework for Photometric Stereo
Guanying Chen, Kai Han, Kwan-Yee K. Wong
TL;DR
Non-Lambertian photometric stereo needs methods that avoid restrictive reflectance models while handling flexible observations. PS-FCN directly predicts normal maps from arbitrary image-light-direction sets, generalizes from synthetic training to real datasets, and outperforms existing calibrated methods while showing promising uncalibrated performance.
Problem
Non-Lambertian photometric stereo remains challenging because analytical reflectance models support limited materials and require complex optimization.
Method
PS-FCN is a fully convolutional network that directly estimates normal maps from arbitrary numbers of images and associated light directions using flexible multi-image fusion.
Results
PS-FCN outperforms existing approaches in calibrated photometric stereo and achieves promising results in the uncalibrated scenario.
Takeaways & Limitations
The framework generalizes from synthetic training to challenging real datasets and can be extended to uncalibrated photometric stereo.
Takeaways & Limitations
Training requires ground-truth normal maps, so the paper uses synthetic datasets because obtaining real-object ground truth is difficult and time-consuming.
Abstract
from arXiv · showhide
This paper addresses the problem of photometric stereo for non-Lambertian surfaces. Existing approaches often adopt simplified reflectance models to make the problem more tractable, but this greatly hinders their applications on real-world objects. In this paper, we propose a deep fully convolutional network, called PS-FCN, that takes an arbitrary number of images of a static object captured under different light directions with a fixed camera as input, and predicts a normal map of the object in a fast feed-forward pass. Unlike the recently proposed learning based method, PS-FCN does not require a pre-defined set of light directions during training and testing, and can handle multiple images and light directions in an order-agnostic manner. Although we train PS-FCN on synthetic data, it can generalize well on real datasets. We further show that PS-FCN can be easily extended to handle the problem of uncalibrated photometric stereo.Extensive experiments on public real datasets show that PS-FCN outperforms existing approaches in calibrated photometric stereo, and promising results are achieved in uncalibrated scenario, clearly demonstrating its effectiveness.
1 Introduction
Photometric stereo must estimate surface normals for non-Lambertian objects despite the limitations of simplified reflectance models. PS-FCN addresses this with flexible inputs, order-agnostic processing, and strong results on real datasets.
- Motivation: Non-Lambertian materials make photometric stereo difficult because analytical reflectance models cover only limited material sets and require complex optimization.General BRDFs are difficult to handle directly, motivating more flexible methods.
- Proposed method: PS-FCN is a flexible fully convolutional network that estimates an object’s normal map from photometric stereo observations.Its architecture combines shared feature extraction, multi-image fusion, and normal regression.
- Flexibility: PS-FCN accepts an arbitrary number of images and associated light directions without requiring a predefined light-direction set during training or testing.Testing light directions may differ from those used during training.
- Flexibility: The model handles multiple images and light directions in an order-agnostic manner and predicts a normal map in a fast feed-forward pass.This design supports flexible image collections rather than a fixed observation arrangement.
- Results: Extensive experiments show that PS-FCN outperforms existing approaches in calibrated photometric stereo and achieves promising uncalibrated results.The paper reports generalization from synthetic training data to real datasets.
2 Related work
Existing non-Lambertian photometric stereo methods use outlier rejection, reflectance models, exemplars, or learning. PS-FCN targets their practical constraints by supporting flexible observations and uncalibrated inputs.
- Method categories: Non-Lambertian photometric stereo methods are broadly divided into outlier-rejection, reflectance-model, exemplar-based, and learning-based categories.These categories organize representative approaches reviewed in the paper.
- Outlier rejection: Outlier-rejection methods generally require many input images and struggle with dense non-Lambertian observations such as broad, soft specular highlights.They treat non-Lambertian observations as local and sparse outliers.
- Reflectance models: Reflectance-model methods solve complex optimization problems and handle only limited classes of materials.They fit analytical models to observations rather than rejecting specular measurements.
- Exemplar methods: Exemplar-based methods can handle spatially varying BRDFs without known light directions, but require reference objects with known shape and material.That requirement limits their applications.
- Learning-based methods: Unlike DPSN, PS-FCN accepts arbitrary numbers of images, permits different training and testing light directions, and processes inputs order-agnostically.PS-FCN can also be extended to uncalibrated photometric stereo by removing light directions from the input.
3 Problem formulation
The paper formulates non-Lambertian photometric stereo as estimating surface normals from images and light directions under unknown general BRDFs. PS-FCN learns this mapping directly instead of explicitly modeling the BRDF.
- Image formation: Under orthographic projection and directional-light assumptions, images, surface normals, light directions, and BRDFs are represented as matrices or tensors.The viewing direction is assumed to point toward the viewer.
- Image formation: The image formation model combines BRDF terms with repeated normal-light-direction products through element-wise multiplication.The operator repmat repeats a matrix three times along the first dimension.
- Lambertian case: For Lambertian surfaces, three or more independent observations can theoretically recover the albedo-scaled normal using linear least squares.Pure Lambertian surfaces rarely exist, motivating the non-Lambertian formulation.
- Non-Lambertian case: The non-Lambertian task estimates the normal matrix from image and light-direction matrices under unknown general BRDFs.The BRDF is not assumed to follow a specific analytical model.
- Learning formulation: PS-FCN directly learns the mapping from images and light directions to surface normals without explicitly modeling the BRDF.Images are normalized by light intensities and light directions are represented as unit 3-vectors.
4 Learning photometric stereo
PS-FCN adapts fully convolutional networks to photometric stereo with variable, unordered image-light inputs. Its shared feature extraction, max-pooling fusion, and normal regression components support flexible inference and cosine-similarity supervision.
- Variable-input CNNs require a fusion mechanism because convolutional layers expect a fixed channel count during training and testing.
- RNN fusion handles sequential inputs but is sensitive to input order, which is undesirable when illumination ordering should not constrain photometric stereo.
- Max-pooling for multi-feature fusion: Max-pooling preserves salient responses, including high intensities and specular highlights that provide clues for surface normal inference.
- Network architecture: PS-FCN is a multi-branch siamese network with a shared-weight feature extractor, fusion layer, and normal regression network.
- Network architecture: The model processes each image-light pair independently, max-pools their feature maps, and regresses a normal map from the fused representation.
- Network architecture: PS-FCN accepts arbitrary image counts, supports different image scales, is order-agnostic, and can extend to uncalibrated photometric stereo by removing light directions during training.
- Loss function: Training uses cosine similarity between predicted and ground-truth normals, with similar orientations producing a small loss.
5 Dataset
The authors construct synthetic training and testing data with diverse shapes, materials, views, and lighting, then evaluate transfer to real photometric stereo datasets. The real datasets include both quantitatively evaluated data with ground-truth normals and qualitatively evaluated data without them.
- Dataset construction: PS-FCN requires ground-truth normal maps for training, so the authors generate synthetic datasets and reserve public real datasets for generalization evaluation.
- Synthetic data for training: Synthetic data uses blobby and sculpture shapes, Mitsuba rendering, and 100 MERL BRDFs to represent diverse real-world materials.
- Synthetic data for training: The blobby training dataset contains 25,920 samples, with each sample comprising 64 images rendered under randomly sampled light directions.
- Synthetic data for testing: The synthetic test set contains 200 samples from Sphere and Bunny shapes, rendered with all 100 MERL BRDFs under 100 randomly sampled light directions.
- Real datasets: The DiLiGenT benchmark provides 10 complex-material objects, each with 96 images and ground-truth normals, enabling quantitative evaluation.
- Real datasets: Gourd&Apple and Light Stage Data Gallery lack ground-truth normals and are used for qualitative transferability evaluation.
6 Experimental evaluation
Experiments evaluate PS-FCN through network analysis, synthetic and real benchmarks, material diversity, and qualitative reconstructions. The method achieves strong calibrated and uncalibrated photometric-stereo performance and produces robust normal estimates on challenging real data.
- Evaluation setup: PS-FCN was evaluated using synthetic network analysis, DiLiGenT benchmarks, and qualitative tests on Gourd&Apple and Light Stage Data Gallery datasets.Mean angular error (MAE) measured predicted normal-map accuracy.
- Network analysis: Adding the sculpture dataset with more complex normal distributions improved performance, suggesting benefits from more realistic training data.The comparison used models trained on both blobby and sculpture datasets.
- Network analysis: PS-FCN significantly outperformed L2 Baseline on Sphere samples rendered with 100 different BRDFs.Performance was generally better on light-colored than dark-colored materials, which the authors relate to max-pooling and feature activation.
- Benchmark comparisons: 8.39° average MAE was achieved on the DiLiGenT main dataset using PS-FCN trained with 32 inputs and tested with all 96 inputs.With only 16 testing inputs, PS-FCN still outperformed previous methods in average MAE.
- Benchmark comparisons: PS-FCN produced more robust estimates than DPSN in cast-shadow regions and outperformed other methods on the DiLiGenT test dataset.The cast-shadow comparison is reported qualitatively, while the test-dataset comparison uses withheld ground-truth normals.
- Benchmark comparisons: UPS-FCN, an uncalibrated variant using images without light directions, outperformed existing uncalibrated methods in average MAE.The model was trained on synthetic data with normalized images and 32 image-light pairs as input during training.
- Testing on other real datasets: Reconstructed surfaces on Gourd&Apple and Light Stage Data Gallery convincingly reflected object shapes, supporting the accuracy of PS-FCN’s predicted normal maps.These evaluations were qualitative because ground-truth normal maps were unavailable.
7 Conclusions
The paper concludes that PS-FCN flexibly regresses normal maps from arbitrary image-light inputs without predefined light directions. It generalizes from synthetic training to challenging real datasets and extends to uncalibrated photometric stereo, where results are promising.
- Contributions: PS-FCN accepts an arbitrary number of images with associated light directions and regresses an accurate normal map.It handles multiple images and light directions in an order-agnostic manner.
- Contributions: PS-FCN does not require predefined light directions during training or testing and permits testing directions to differ from training directions.This flexibility distinguishes the framework from methods tied to fixed light-direction sets.
- Contributions: Training uses two synthetic datasets with varied realistic shapes and materials, after which PS-FCN generalizes well on challenging real datasets.The conclusion reports results across diverse real datasets.
- Contributions: PS-FCN can be easily extended to uncalibrated photometric stereo, with promising results reported in that scenario.The conclusion also reports that PS-FCN outperforms previous calibrated photometric-stereo methods.