Source-linked AI summary
CNN-PS: CNN-based Photometric Stereo for General Non-Convex Surfaces
Satoshi Ikehata
TL;DR
Conventional photometric stereo relies on BRDF-based models that struggle with global illumination on non-convex surfaces. This paper learns surface-normal relationships directly with a CNN using a fixed-shape observation map and isotropy-based rotational pseudo-invariance, and reports stronger recovery on synthetic and real non-convex scenes than conventional methods.
Problem
BRDF-based photometric stereo models generally cannot account for global illumination effects such as shadows and inter-reflections on non-convex surfaces.
Method
An end-to-end CNN uses a fixed-shape observation map for unstructured inputs, exploits isotropic rotational pseudo-invariance, and trains on synthetic data containing global illumination effects.
Results
The method most successfully recovers non-convex, non-Lambertian surfaces on the DiLiGenT benchmark among the compared conventional algorithms, with state-of-the-art performance reported on synthetic and real data.
Takeaways & Limitations
Observation-map-based CNN inference can handle unstructured photometric stereo inputs and global illumination effects in isotropic scenes, especially when surfaces are non-convex.
Abstract
from arXiv · showhide
Most conventional photometric stereo algorithms inversely solve a BRDF-based image formation model. However, the actual imaging process is often far more complex due to the global light transport on the non-convex surfaces. This paper presents a photometric stereo network that directly learns relationships between the photometric stereo input and surface normals of a scene. For handling unordered, arbitrary number of input images, we merge all the input data to the intermediate representation called {\it observation map} that has a fixed shape, is able to be fed into a CNN. To improve both training and prediction, we take into account the rotational pseudo-invariance of the observation map that is derived from the isotropic constraint. For training the network, we create a synthetic photometric stereo dataset that is generated by a physics-based renderer, therefore the global light transport is considered. Our experimental results on both synthetic and real datasets show that our method outperforms conventional BRDF-based photometric stereo algorithms especially when scenes are highly non-convex.
1 Introduction
Photometric stereo must recover surface normals from unstructured images despite global illumination effects that BRDF-based models cannot fully represent. This paper addresses the problem with an end-to-end CNN, a fixed-shape observation map, isotropy-based rotational pseudo-invariance, and synthetic training data.
- Photometric stereo recovers scene surface normals from appearance changes under different illuminations, but its image inputs are unstructured.
- BRDF-based image formation models generally cannot account for global effects such as shadows and inter-reflections on non-convex surfaces.
- The proposed end-to-end CNN learns relationships between surface-normal appearances without physically modeling image formation, using synthetic patterns corrupted by global effects.
- The method merges unstructured photometric stereo data into a fixed-shape observation map suitable for a standard CNN and exploits rotational pseudo-invariance from isotropic materials.
- The method is evaluated on the real DiLiGenT benchmark and most successfully recovers non-convex, non-Lambertian surfaces among the compared algorithms.
- The contributions include a supervised CNN for unstructured calibrated inputs and a synthetic dataset injecting cast shadows and inter-reflections.
2 Related Work
Prior photometric stereo methods address non-Lambertian appearance through image-formation modeling, robust decomposition, example-based modeling, or learning. The paper instead abandons complicated image-formation modeling while accepting unstructured inputs and targeting scenes with combined non-Lambertian and global illumination effects.
- A BRDF relates observed intensity to surface normal, lighting direction and intensity, viewing direction, and reflectance, forming the basis of conventional photometric stereo inversion.
- Photometric stereo approaches to non-Lambertian effects include robust methods, non-Lambertian BRDF modeling, example-based reflectance modeling, and learning-based methods.
- Robust methods model diffuse reflectance while treating other effects as outliers, including rank-based decompositions of Lambertian images and non-Lambertian corruptions.
- BRDF-modeling approaches use parametric or non-parametric reflectance representations, including compact or sum-of-lobes models for isotropic BRDFs.
- Example-based approaches use objects with known shape and reflectance under the same illumination environment, with later work relaxing exact material matching and calibration constraints.
- The proposed learning-based algorithm uses a deep CNN trained on synthetic data, accepts inputs invariant to image number and order, and targets real scenes combining non-Lambertian reflections with global illumination.
3 Proposed Method
The method converts unordered photometric-stereo observations into fixed-size observation maps, then uses isotropy-derived rotational pseudo-invariance and a CNN to estimate surface normals under complex appearance conditions.
- 2-D observation map: Observation maps project pixelwise observations onto a hemispherical lighting grid, producing a fixed representation for CNN input.The map is formed from known lighting directions and normalized image intensities.
- 2-D observation map: The representation is independent of input-image number and size, invariant to image ordering, and does not explicitly require lighting information as a separate network input.
- 2-D observation map: Zero-valued missing entries are retained because interpolating sparse observation maps can degrade performance and shadows carry meaningful information.For a 32 × 32 map with 100 images, only about 10% of entries are non-zero.
- Rotation pseudo-invariance: Under isotropy, rotating lighting and surface normals around the viewing axis preserves appearance, yielding rotational pseudo-invariance for the observation map and network.The relationship is also stated to hold for indirect illumination in non-convex scenes when geometry and environment illumination are rotated together.
- Architecture and training data: A DenseNet variation estimates normalized surface normals from observation maps, and the training data uses diverse materials, geometries, and illumination rendered with complex light transport.The dataset is designed to address limited real-data diversity and fixed lighting configurations.
- Rotation pseudo-invariance: The network is trained with rotated lighting and normal vectors, while prediction averages inversely rotated normal estimates from K rotated observation maps.This prediction-time aggregation further emphasizes the intended pseudo-invariance.
4 Experimental Results
The experiments evaluate the method on synthetic and real datasets, including isotropic materials and non-convex scenes with global illumination effects. Across these settings, the method generally performs strongly, with particular advantages on highly non-convex surfaces.
- Datasets: The evaluation uses two synthetic datasets and one real benchmark, covering isotropic materials, non-convex geometry, and calibrated lighting.MERLSphere excludes cast shadows and inter-reflections, CyclesPSTest includes them, and DiLiGenT contains 10 real objects under 96 known lighting directions.
- MERLSphere: On MERLSphere, the CNN-based method performs comparably to, though not better than, IA14 for most materials.The comparison uses mean angular errors of predicted surface normals across 100 measured MERL BRDF materials.
- CyclesPSTest: On non-convex CyclesPSTest scenes, competing algorithms fail under strong cast shadows and inter-reflections, whereas the proposed method remains effective.All methods work well on the convex specular sphere, but all algorithms except the proposed method fail on non-convex surfaces; rotational averaging further improves accuracy.
- DiLiGenT: The DiLiGenT comparison gives the proposed method the smallest mean angular error over 10 objects and the best score on 6 of 10 objects.Its CPU inference takes less than five seconds per dataset when K = 1, while two other top-ranked methods require substantially more computation.
- DiLiGenT: Performance varies across real non-convex objects: the method handles HARVEST’s shadowed bag region well but does not perform best on READING.The authors attribute the READING weakness to narrow, high-intensity specular inter-reflections that were rare in the training data.
5 Conclusion
The paper presents a CNN-based photometric stereo method for isotropic scenes with global illumination effects. Its observation-map representation supports unstructured inputs, and experiments show strong performance on synthetic and real data, especially for non-convex surfaces.
- Conclusion: The method targets isotropic scenes affected by global illumination effects.It is designed for photometric stereo settings where conventional modeling does not fully capture scene appearance.
- Conclusion: Projecting photometric images and lighting information onto an observation map lets unstructured inputs be fed naturally into a CNN.The observation map provides the fixed representation needed for varying and unordered photometric stereo inputs.
- Conclusion: Experiments show state-of-the-art performance on synthetic and real data, especially when surfaces are non-convex.The authors identify better training data for narrow inter-reflections as a future direction.