Flexible SVBRDF Capture with a Multi-Image Deep Network

1. Introduction

• Real-world materials depend on viewing and lighting directions, while densely sampling both requires expensive, time-consuming hardware.Lightweight methods instead use strong priors to fill gaps from a few measurements.Source: §1 · p. 1
• Single photographs can miss surface relief and reflectance cues, leaving artifacts in predicted diffuse albedo and specular maps.Side views or lights can reveal fine geometry and reflectance details.Source: §1 · p. 2
• The method processes an arbitrary number of uncalibrated images with order-invariant pooling to combine appearance cues across inputs.With few images, it uses learned priors; additional images help explain more observations.Source: §1 · p. 2
• 1 to 10 images provide significant improvement over single-image methods while requiring fewer images and less constrained capture than traditional multi-image methods.Source: §1 · p. 2

2. Related Work

• Controlled capture methods can represent complex effects, including anisotropy, but require tens to hundreds of measurements and dedicated hardware.Recent methods instead recover plausible spatially-varying appearance from few pictures using priors on materials and lighting.Source: §2 · p. 2
• Single-image methods can miss material effects that are absent from, or ambiguous in, the input image.The proposed multi-image approach reports increasing quality with more images and uses 1 to 10 rather than around a hundred for optimization-based multi-image methods, while calibrated solutions can be more accurate.Source: §2 · p. 2
• Fixed-size networks treat inputs asymmetrically, while pooling-based approaches offer order-agnostic processing for loosely structured image sets.Prior work includes recurrent multi-view reconstruction and pooling-based methods for other set-valued tasks.Source: §2 · p. 2

3. Capture Setup

• The setup captures a material sample in a paper frame using a handheld smartphone and either its flash or a separate flash.The frame can be captured with one hand while holding a flash in the other, or with a co-located phone flash.Source: §3 · p. 2
• Frame corners provide the homography used to rectify the images and crop away the paper before processing.Source: §3 · p. 3
• 3456 × 3456 pixel captures are cropped and resized to 256×256 pixels.Source: §3 · p. 3

4. Multi-Image Material Inference

• The target SVBRDF comprises per-pixel normal, diffuse albedo, specular albedo, and specular roughness maps for a Cook-Torrance BRDF.Source: §4 · p. 3
• Each input is processed by an identical single-image network, and its output features are fused by a shared order-agnostic pooling layer.Source: §4 · p. 3

4.1. Single-image network

• The network infers light and view information from visual cues rather than receiving their positions explicitly.Its single-image network modifies the original output to produce a 64-channel feature map for later processing.Source: §4.1 · p. 3

4.2. Multi-image fusion

• Max-pooling combines feature maps into a fixed-size representation, which is decoded into four SVBRDF maps; shared weights allow arbitrary image counts at test time.Training varies copies from 1 to 5, while testing uses 1 to 10 input images.Source: §4.2 · p. 3, Figure 3 · p. 4

4.3. Loss

• Training combines differentiable rendering comparisons with four direct L1 losses between predicted maps and their ground-truth counterparts.The direct supervision was found to stabilize training.Source: §4.3 · p. 3

4.4. Training

• Training runs for 1 million iterations with batch size 2 and 256×256 pixel inputs on an Nvidia GTX 1080 TI.The training run takes 7 days and uses Adam with learning rate 0.0002 and β = 0.5.Source: §4.4 · p. 4

5. Online Generation of Training Data

• Online generation yields up to 2 million training images by rendering a new SVBRDF and its multiple views at each iteration.Rendering overhead is approximately offset by time saved on data loading, making online rendering virtually free in the reported experiments.Source: §5 · p. 4, §5.2 · p. 5

5.1. SVBRDF synthesis

• Training uses 1,850 procedural SVBRDFs and follows random pairwise convex-combination augmentation to produce varied materials.The source materials cover common classes including plastic, metal, wood, and leather.Source: §5.1 · p. 4

5.2. SVBRDF rendering

• The renderer evaluates four SVBRDF maps at each pixel and randomizes camera, flash, ambient lighting, and image degradations to simulate casual captures.It takes rectified images as input, so it does not simulate perspective projection.Source: §5.2 · pp. 4–5

6.1. Number of input images

• Re-rendering SSIM rises quickly with the first images and stabilizes around 10; normals keep improving, diffuse albedo stabilizes fastest, and roughness stays largely unchanged.The trends are averaged over the synthetic test set.Source: Figure 4 · p. 5, §6.1 · p. 5
• Overall, added pictures reduce diffuse-albedo artifacts and improve normal estimation while maintaining casual capture.The reported overall pattern draws on results in Figures 4–10 and supplemental material.Source: §6.1 · p. 5

6.2. Comparison to multi-image optimization

• The complete method achieves higher SSIM than training without randomized viewing and lighting perturbations on a test set with similar variations.Both networks were trained for 750,000 iterations.Source: §6.1 · p. 5, Figure 5 · p. 6
• On diffuse materials, optimization needs dozens of calibrated images to match our method’s 5 uncalibrated images; wood required more than 900 optimization images to match its quality.Classical optimization can recover more precise SVBRDFs when given enough carefully calibrated images.Source: §6.2 · p. 6

6.3. Comparison to alternative deep learning methods

• With five inputs, the multi-image network’s average SSIM is 0.847, versus 0.826 for a five-image extension of the Deschaintre et al. [DAD∗18] network, while allowing variable input counts.Source: §6.3 · p. 6
• Additional side views help recover relief cues and information obscured by saturated or incomplete flash highlights in single-image methods.More views can expose highlights across the sample, including regional roughness differences that became apparent with the fourth input in one real capture.Source: §6.3 · p. 6

6.4. Limitations

• The method is limited to isotropic Cook-Torrance materials; high-relief parallax can misalign rectified inputs and reduce map sharpness, and shadows are not modeled.Correlated maps in the training data can lead to albedo variations being interpreted as normal variations and dark materials as shiny.Source: §6.4 · p. 6

6.1. Number of input images

• On a measured BTF and real captures, added views recover more relief and regional roughness; real captures lack ground-truth maps.For the measured BTF, three images capture most normal and roughness structure, while later inputs further improve diffuse albedo and add subtle normal and specular detail.Source: §6.1 · p. 5, Figure 6 · p. 7

• Single-image methods offer little margin for users to correct erroneous predictions.Source: §7 · p. 7
• The network accepts a variable number of input images, letting users capture as many as needed to exhibit desired material effects.Source: §7 · p. 7
• The approach bridges single-image and many-image methods, targeting faithful capture from a handful of uncalibrated light-view images.Source: §7 · p. 7