PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Inverse Rendering for General Reflectance Photometric Stereo

Tatsunori Taniai, Takanori Maehara

arXiv:1802.10328v2cs.CVcs.NE

TL;DR

Photometric stereo still faces challenges when surfaces have unknown general BRDFs, while deep-learning approaches are limited by training-data and input-permutation difficulties. This paper fits a physics-based network to each test scene using image reconstruction loss, and reports state-of-the-art performance on a real-world benchmark.

  • Problem

    Photometric stereo for unknown general BRDFs remains challenging, and deep-learning approaches are limited by difficulties obtaining accurate training ground truth and handling input-image permutations.

    Source: §1 · pp. 1–2

  • Method

    A network predicts surface normals and reflectance, renders the observed images, and fits its weights on the test scene by minimizing reconstruction loss without ground-truth normals or external pre-training.

    Source: §1 · p. 2

  • Results

    The method achieves state-of-the-art performance on the real-world benchmark, with the best average score and best individual scores on eight DiLiGenT scenes.

    Source: §4.1 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    The approach combines deep networks' expressibility for complicated BRDFs with physical reflectance properties in an unsupervised framework.

    Source: §6 · p. 8

    The method is slow, taking about one hour for 1,000 SGD iterations per scene, and has limited performance on complex scenes such as HARVEST.

    Source: §5 · p. 8

Abstract

from arXiv · show

We present a novel convolutional neural network architecture for photometric stereo (Woodham, 1980), a problem of recovering 3D object surface normals from multiple images observed under varying illuminations. Despite its long history in computer vision, the problem still shows fundamental challenges for surfaces with unknown general reflectance properties (BRDFs). Leveraging deep neural networks to learn complicated reflectance models is promising, but studies in this direction are very limited due to difficulties in acquiring accurate ground truth for training and also in designing networks invariant to permutation of input images. In order to address these challenges, we propose a physics based unsupervised learning framework where surface normals and BRDFs are predicted by the network and fed into the rendering equation to synthesize observed images. The network weights are optimized during testing by minimizing reconstruction loss between observed and synthesized images. Thus, our learning process does not require ground truth normals or even pre-training on external images. Our method is shown to achieve the state-of-the-art performance on a challenging real-world scene benchmark.

1. Introduction

Photometric stereo recovers surface normals from shading across images, but unknown general reflectance remains challenging beyond the simple Lambertian case. The paper proposes test-time, physics-based unsupervised learning to predict normals and reflectance, synthesize the observations, and optimize reconstruction loss.

1. Introduction

  • Photometric stereo uses shading changes across images to recover surface normals and can recover fine surface details, but unknown general BRDFs remain challenging.The simplest Lambertian model has a closed-form least-squares solution, but ideally diffuse materials are rare.Source: §1 · p. 1
  • Deep networks could learn complicated BRDFs, but progress is limited by scarce accurate training ground truth and the demands of realistic synthetic data.Real-object normals require highly accurate 3D shapes; realistic rendering must account for varied materials, cast shadows, and interreflections, while real-material BRDF measurements are difficult and databases are limited.Source: §1 · pp. 1–2
  • Pre-trained networks must be invariant to permutations of input images and corresponding illuminations, while existing methods avoid this issue by fixing illumination patterns across training and testing.That fixed-pattern assumption limits application scenarios.Source: §1 · p. 2
  • The method outperforms state-of-the-art learning-based and classical unsupervised methods on a challenging real-world scene benchmark.It predicts surface normals and spatially varying BRDFs, then optimizes network weights on the test scene using reconstruction loss without ground-truth normals or pre-training.Source: §1 · p. 2
  • The proposed network inverse-renders normals and reflectance, then uses the rendering equation to synthesize images and train from reconstruction loss directly on each test scene.Because it needs no external pre-training, the framework sidesteps the input-permutation problem; it also learns reflectance models while using prior knowledge about reflectance properties.Source: §1 · p. 2

2. Preliminaries

The preliminaries define calibrated photometric stereo and review how Lambertian least squares gives way to more difficult estimation when reflectance is non-Lambertian. Existing general-BRDF approaches either reject deviations as outliers, impose analytic models, or exploit broad BRDF properties; the proposed approach learns broad classes while incorporating physical reflectance knowledge.

2. Preliminaries

  • Photometric stereo seeks surface normals from intensities under changing illumination, typically with a fixed viewpoint and distant, spatially uniform light sources.The setup assumes known lighting; for a Lambertian surface, at least three bright-pixel measurements yield a linear system, solved with the lighting matrix pseudoinverse and followed by L2 normalization.Source: §2 · p. 2, §2.1 · pp. 2–3

2.1. Lambertian model and least squares method

  • For Lambertian reflectance, intensity is linear in the scaled normal, enabling least-squares recovery; in practice, thresholds select inlier observations amid sensor noise and interreflections.The Lambertian model assumes constant reflectance, called albedo; the simplest solution uses bright pixels and three or more measurements.Source: §2.1 · pp. 2–3

2.2. Photometric stereo for general BRDF surfaces

  • With unknown non-Lambertian reflectance, photometric stereo must assume a BRDF model or estimate the BRDF alongside the surface normal.Outlier-rejection methods treat non-Lambertian components as sparse deviations from Lambertian reflectance, but broad, soft specularity violates that sparse-outlier assumption.Source: §2.2 · p. 3
  • Analytic BRDF methods explicitly model specularity, often as diffuse plus specular components, but hand-crafted models handle only narrow material classes.Examples include Torrance-Sparrow, Ward, and Ward mixture models.Source: §2.2 · p. 3
  • Other methods exploit general BRDF properties, such as isotropy or low-frequency structure, whereas this method learns broad BRDF classes without a fixed reflectance model.Its network also incorporates physical reflectance knowledge developed in prior work.Source: §2.2 · p. 3

3. Proposed method

The method estimates surface normals with PSNet and uses IRNet to synthesize observed images through the rendering equation. It fits both networks to a test scene by minimizing reconstruction loss, with temporary weak supervision to warm up optimization.

3.1. Network architecture

  • PSNet predicts a surface-normal map from concatenated input images, while IRNet renders each observed image to define a reconstruction loss that trains both networks without ground-truth supervision.The calibrated setup also supplies lighting directions for rendering and the specular-component input.Source: §3.1 · p. 4, Figure 3 · p. 4

3.1.1. PHOTOMETRIC STEREO NETWORK

  • PSNet transforms the concatenated images into features and predicts a surface-normal map, with L2 normalization making each pixel's normal a unit vector.Its feature layers use three 3×3 convolutions with 384 channels, BatchNorm, and ReLU.Source: §3.1.1 · p. 4

3.1.2. IMAGE RECONSTRUCTION NETWORK

  • IRNet transforms each observation, blends its features with scene-wide PSNet features, and predicts a reflectance image used with the normal and lighting in the rendering equation.A specularity cue is included because it is highly correlated with the actual specular component and can promote learning complex BRDFs.Source: §3.1.2 · pp. 4–5

3.2. Learning procedures (optimization)

  • The network minimizes reconstruction loss plus a prior-normal loss that is activated only during early SGD iterations to warm up randomly initialized networks.The network is fitted directly to each test scene without pre-training on other data.Source: §3.2 · p. 5

3.2.1. RECONSTRUCTION LOSS

  • Reconstruction loss is mean absolute error between synthesized and observed images over target-object regions, making it more robust to high-intensity specular highlights.The loss is computed over the binary object mask.Source: §3.2.1 · p. 5

3.2.2. EARLY-STAGE WEAK SUPERVISION

  • For complex scenes, a least-squares prior normal map guides the early optimization even though shadows and non-Lambertian specularity can make that prior inaccurate.Its weight is 0.1c for the first 50 iterations and zero afterward.Source: §3.2.2 · p. 5

3.2.3. IMPLEMENTATION DETAILS

  • Each iteration predicts normals, reconstructs all observed images as a minibatch, computes loss, and updates both networks' parameters.The procedure uses 1,000 Adam updates per test scene, with the learning rate reduced for the last 100 iterations.Source: §3.2.3 · p. 5, §3.1.2 · p. 5
  • During optimization, 90% of reconstruction-loss elements are randomly dropped and the loss is rescaled by 10 to address poor local convergence associated with large-minibatch SGD.BatchNorm uses statistics from the given test data rather than moving-average statistics.Source: §3.2.3 · p. 6

4. Experiments

Experiments on DiLiGenT show strong benchmark performance, while architecture and supervision analyses identify components that support learning complex reflectance and stable optimization.

4.1. Real-world scene benchmark (DiLiGenT)

  • On DiLiGenT, the method achieves the best average score and the best individual scores on eight scenes, with exceptions at GOBLET and HARVEST.The benchmark uses 96 observed images per scene; the method is compared with ten existing methods by mean angular error.Source: §4.1 · p. 6, §4 · p. 6

4.2. Analysis of the network architecture

  • The full architecture performs best, and removing the specularity-channel input has the largest negative impact.The specularity channel eases learning complex BRDFs, including metallic surfaces in COW.Source: §4.2 · p. 6

4.3. Effects of early-stage weak supervision

  • Early-stage weak supervision is more stable than no supervision and less biased by inaccurate least-squares priors than all-stage supervision.After supervision ends at t = 50, accuracy continues to improve; no-supervision runs often reach bad local minima, while all-stage runs often produce worse solutions.Source: §4.3 · p. 6

5. Discussions and related work

The discussion connects the method to deep image priors and prior photometric-stereo networks, and reports benchmark comparisons, design analyses, and remaining practical limitations.

Relation to deep image priors and neural photometric stereo

  • Unlike the deep image prior’s single-image reconstruction setup, this architecture jointly estimates normals using global observation statistics and reconstructs individual observations for the loss.The architecture uses PSNet and IRNet; IRNet enables direct unsupervised learning on test data.Source: §5 · pp. 6, 8

Benchmark results

  • On DiLiGenT, the method has the best accuracies on all but GOBLET and HARVEST; Santo et al. (2017) is second best and outperforms other methods only on HARVEST.Results are median scores over 11 runs; Table 1 compares mean angular errors in degrees across ten scenes and average scores.Source: Table 1 · p. 7

Architecture and supervision analyses

  • Table 2 evaluates architecture and weak-supervision settings using median and mean scores over 11 runs; Figure 5 contrasts early-stage supervision with no- and all-stage supervision.Figure 5 shows mean angular errors and loss values for READING and COW, with colored run distributions and median curves.Source: Table 2 · p. 8, Figure 5 · p. 8

Relation to deep image priors and neural photometric stereo

  • The method’s photometric-stereo network is partly influenced by Santo et al. (2017), but uses fewer layers and channels for single-scene reflectance learning and adds IRNet.Santo et al.’s network has five fully connected and ReLU layers with 4096 and 2048 channels in its internal layers.Source: §5 · pp. 6, 8

Relation to prior photometric-stereo methods

  • Earlier neural photometric-stereo studies used constraints such as same-material calibration spheres, specialized capture setups, or Lambertian surfaces; this method does not require them.The method learns broad BRDF classes from observations while leveraging neural-network expressivity and physical reflectance knowledge.Source: §5 · p. 8

Limitations and future work

  • The method remains slow—about one hour for 1,000 SGD iterations per scene—and has limited performance on complex scenes such as HARVEST.The authors identify faster training schemes and pre-training as possible future directions; pre-training may increase accuracy on complex scenes but would lose permutation invariance.Source: §5 · p. 8

6. Conclusions

The paper concludes that its physics-based unsupervised CNN learns complicated BRDFs and achieves state-of-the-art performance for photometric stereo.

6. Conclusions

  • The proposed approach combines deep networks’ expressibility with known physical reflectance properties to learn complicated BRDFs without supervision and achieve state-of-the-art performance.The authors suggest physics-based unsupervised learning may stimulate work on tasks lacking ground-truth training data.Source: §6 · p. 8

Supplementary Material of Neural Inverse Rendering for General Reflectance Photometric Stereo

The supplementary material presents additional results for all ten benchmark scenes, including visual comparisons, image reconstructions, and weak-supervision convergence analyses.

Supplementary material

  • The supplement provides results omitted from the paper for all ten benchmark scenes.It includes visual comparisons with existing methods, image reconstruction results, and convergence analyses under different weak-supervision strategies.Source: Supplementary Material of Neural… · p. 11

A. Additional visual comparisons

Across ten DiLiGenT scenes, the appendix compares normal estimates and angular-error maps with three state-of-the-art methods and least squares; our method is among the top four on every scene and has the best result on eight.

A. Additional visual comparisons

B. Additional image reconstruction results

Figures A11–A20 show reconstruction outputs for all ten DiLiGenT scenes, with six of the 96 observed images displayed for each scene.

B. Additional image reconstruction results

C. Additional convergence analyses

Across ten DiLiGenT scenes, the convergence analyses compare early-stage weak supervision with no supervision and all-stage supervision. Training without supervision is unstable, while all-stage supervision is strongly biased by inaccurate least-squares priors.

C. Additional convergence analyses

Related papers

Loading 1802.10328v2…