PaperlayerView exact revision on arXiv

Source-linked AI summary

Single-Image SVBRDF Capture with a Rendering-Aware Deep Network

Valentin Deschaintre, Miika Aittala, Fredo Durand, George Drettakis, Adrien Bousseau

arXiv:1810.09718v1cs.GR

TL;DR

Single-image SVBRDF recovery is ill-posed because one photograph can correspond to multiple reflectances. The paper trains a rendering-aware deep network on augmented procedural data, combining local and global features, and reports improved capture across diverse materials, subject to resolution, dynamic-range, and architecture-related limitations.

  • Problem

    Recovering spatially varying reflectance from one image is ill-posed because different reflectances can yield the same observed photograph.

  • Method

    The paper trains a deep network on rendered procedural SVBRDFs, using a rendering loss and a secondary global-feature track alongside local convolutional features.

  • Results

    The method outperforms existing work across a wide range of materials and generalizes to real input photographs.

  • Takeaways & Limitations

    A single network can reconstruct complex spatially varying BRDFs from one flash photograph using synthetic training data alone.

  • Takeaways & Limitations

    The network is constrained to 256×256 inputs and low-dynamic-range images, and sharp saturated highlights can produce residual artifacts or failures.

Abstract

from arXiv · show

Texture, highlights, and shading are some of many visual cues that allow humans to perceive material appearance in single pictures. Yet, recovering spatially-varying bi-directional reflectance distribution functions (SVBRDFs) from a single image based on such cues has challenged researchers in computer graphics for decades. We tackle lightweight appearance capture by training a deep neural network to automatically extract and make sense of these visual cues. Once trained, our network is capable of recovering per-pixel normal, diffuse albedo, specular albedo and specular roughness from a single picture of a flat surface lit by a hand-held flash. We achieve this goal by introducing several innovations on training data acquisition and network design. For training, we leverage a large dataset of artist-created, procedural SVBRDFs which we sample and render under multiple lighting directions. We further amplify the data by material mixing to cover a wide diversity of shading effects, which allows our network to work across many material classes. Motivated by the observation that distant regions of a material sample often offer complementary visual cues, we design a network that combines an encoder-decoder convolutional track for local feature extraction with a fully-connected track for global feature extraction and propagation. Many important material effects are view-dependent, and as such ambiguous when observed in a single image. We tackle this challenge by defining the loss as a differentiable SVBRDF similarity metric that compares the renderings of the predicted maps against renderings of the ground truth from several lighting and viewing directions. Combined together, these novel ingredients bring clear improvement over state of the art methods for single-shot capture of spatially varying BRDFs.

1 INTRODUCTION

The paper addresses the ill-posed problem of recovering spatially varying material properties from one flash-lit image by combining synthetic training data, rendering-aware supervision, and local-global feature processing.

  • Motivation: Single-image SVBRDF recovery is ill-posed because different reflectances can produce the same observed image.A diffuse albedo map could reproduce a photograph by painting highlights onto the surface.
  • Approach: The method uses a deep network trained with forward-rendered simulations to estimate SVBRDFs from one flash-lit image of a flat surface.The target comprises per-pixel normal, diffuse albedo, specular albedo, and specular roughness maps.
  • Training data: A procedural SVBRDF dataset is expanded through parameter perturbations, material mixing, and varied scaling, rotation, and lighting to produce up to 200,000 realistic samples.The procedural materials are sampled and rendered under multiple lighting directions before training.
  • Network design: The network combines local encoder-decoder features with a secondary global track to exchange information across distant image regions.This addresses complementary cues such as highlights in central pixels and diffuse or normal variation in outer pixels.
  • Training objective and outcome: The approach compares predicted and ground-truth materials through a rendering loss and reports better performance than existing work across a wide range of materials.The method is intended to approximate casually captured appearance rather than recover accurate constituent-map measurements.

2 RELATED WORK

Related work addresses material-capture ambiguity through additional measurements, hand-designed priors, or learned representations, while this paper emphasizes data-driven SVBRDF estimation and rendering-based comparison.

  • Material capture: Lightweight material-capture methods seek easy recovery of spatially varying materials in the wild, within a broader spectrum of capture techniques.The paper focuses on methods requiring only limited measurements.
  • Hand-designed priors: Existing priors include spatial or angular homogeneity, texture repetition, sparse lighting, sky-light polarization, and mixtures of basis BRDFs.Many such assumptions restrict the family of materials that can be captured.
  • Data-driven capture: Aitala et al. use a single flash image but cannot handle non-repetitive material samples, whereas this work adopts a data-driven approach.The neural network learns internal assumptions from the materials used for training.
  • Deep material capture: Compared with Li et al., this work recovers per-pixel specular albedo and roughness and trains one all-purpose network instead of separate material-specific networks.Li et al. recover uniform specular parameters and use separate networks for material types.
  • Rendering losses: The paper uses a differentiable rendering loss as an SVBRDF similarity metric rather than comparing renderings directly with the input photograph.This contrasts with rendering losses used for unsupervised facial capture, material editing, or optimization-based inverse rendering.
  • Local and global features: Unlike related local-global image-transformation networks, the proposed architecture exchanges information between global and local tracks after every layer.This repeatedly transmits information across image regions.

3 NETWORK ARCHITECTURE

The network maps a single flash-lit photograph to SVBRDF parameters by combining a U-Net encoder-decoder with a global feature track. A rendering-aware loss further trains predictions to reproduce material appearance rather than merely matching parameter maps.

  • 3.1 U-Net Image-to-Image Network: The base network translates a photograph into a multi-channel SVBRDF representation using a U-Net encoder-decoder.Its implementation uses an encoder followed by a decoder, with instance normalization and dropout in the decoder.
  • 3.1 U-Net Image-to-Image Network: The network predicts per-pixel normal, diffuse albedo, specular albedo, and specular roughness from a single flash-lit image.The implementation represents these quantities as a nine-channel output image.
  • 3.2 Global Features Network: Distant regions provide complementary cues, but vanilla U-Net processing produced residual shading and detail artifacts in outputs that should be uniform.The authors demonstrate this failure both on average-color prediction and on SVBRDF maps.
  • 3.2 Global Features Network: A parallel fully connected track propagates global feature vectors and exchanges them with convolutional features at multiple scales.The track incorporates instance-normalization means into a global vector, processes it with fully connected layers and nonlinearities, and reinjects the result into the U-Net.
  • 3.3 Rendering Loss: The rendering loss compares renderings of predicted and ground-truth SVBRDFs under identical lighting and viewing conditions.Compared with an l1 loss, it produces predictions that more faithfully reproduce the target appearance after rendering, even when parameter values are not identical.

4 PROCEDURAL SYNTHESIS OF TRAINING DATA

The training set is built from artist-created procedural SVBRDFs and expanded through parameter perturbation, material mixing, and varied rendering conditions. The synthesis pipeline is designed to increase material and shading diversity while preserving plausible appearance.

  • 4 PROCEDURAL SYNTHESIS OF TRAINING DATA: The source materials come from a dataset of more than 800 artist-designed procedural SVBRDFs.The dataset was created by artists from the movie and video game industry.
  • 4 PROCEDURAL SYNTHESIS OF TRAINING DATA: The authors curate 155 high-quality procedural SVBRDFs spanning nine material classes for training.They also select 12 challenging procedural SVBRDFs as an independent testing set for comparison with prior work.
  • 4 PROCEDURAL SYNTHESIS OF TRAINING DATA: Random parameter perturbations generate around 1,850 material variants, while alpha-blending random pairs generates around 20,000 additional combinations.The authors state that mixing increases low-level shading diversity while remaining close to plausible real-world materials.
  • 4 PROCEDURAL SYNTHESIS OF TRAINING DATA: Each SVBRDF is rendered with varied lighting, scaling, orientation, and cropping to expose the network to different image conditions.Random crops are applied during training so the network sees slightly different data at each epoch.
  • 4 PROCEDURAL SYNTHESIS OF TRAINING DATA: The rendering setup uses a textured plane, a fronto-parallel camera, and a randomly offset small emitting sphere as the light source.Camera distance and light strength are held fixed because SVBRDF scale, camera distance, and light strength are generally ambiguous.

5 EVALUATION

The method is evaluated on casual real-world photographs and compared with prior single-image SVBRDF methods, reproducing varied appearance effects while exposing resolution, dynamic-range, material-model, and viewing-angle limitations.

  • 5.1 Real-world photographs: Nearly 350 materials were captured with regular cell phones and built-in flashes during casual indoor, outdoor, and public-space photography.The dataset spans materials from domestic, office, public, and outdoor environments.
  • 5.1 Real-world photographs: The method reproduces reflectance effects across metals, plastics, paint, wood, and other substances in re-renderings under novel environment illumination.Representative results include spatially varying specularity and complex normal variations.
  • 5.1 Real-world photographs: Distant highlight cues inform globally consistent specularity, including gold paint identification and differing roughness levels across black and white tiles.The method resolves low roughness from small highlights on a center tile and outer tile edges.
  • 5.1 Real-world photographs: Predicted maps remain consistent across duplicate shots and varying flash positions, while comparisons with BTF renderings and photographs show faithful appearance reproduction with somewhat blurrier highlights.The method also captures spatially varying wallpaper specularity and wood-carving normal variations.
  • 5.2 Comparisons: Compared with Aitala et al. [2016], the method captures larger-scale variations and non-repetitive materials, but sacrifices some local detail and consistency.Its feedforward evaluation is practically instant, whereas Aitala et al. [2016] require more than an hour per material sample.
  • 5.2 Comparisons: Compared with Aitala et al. [2015], the method produces similar results from a single image at lower resolution, while earlier approaches assume stationarity or require an additional no-flash image.The method reproduces ground-truth specularity more accurately than Li et al. [2017], with sharper reflections and highlights.
  • 5.3 Limitations: The 256×256-pixel input limits recovery of very fine details, and low dynamic range can leave artifacts around sharp saturated highlights.Increasing input resolution would increase memory consumption and may hinder convergence.
  • 5.3 Limitations: The four-map Cook-Torrance representation excludes effects such as anisotropic reflections and multiple scattering, while fronto-parallel input prevents accurate Fresnel recovery.Materials such as thick fabric or skin require a different spatially varying appearance model and a fast renderer for the loss.

6 CONCLUSION

The paper shows that a neural network can reconstruct complex spatially varying BRDFs from a single photograph, trained entirely on synthetic data. Its global feature track and rendering-aware treatment of SVBRDF interactions support broad real-image generalization, while failures remain on assumption-violating materials.

  • Synthetic-data training enables neural reconstruction of complex spatially varying BRDFs from a single input photograph.The conclusion attributes result quality to training-data quantity and realism, rendering loss, and global feature fusion.
  • Materials outside the training-data scope, such as anisotropic brushed metal, cannot be reproduced properly.The figure also reports struggles with structured albedo detail and missed concentrated highlights that can cause overestimated roughness or highlight-removal artifacts.
  • The method generalizes well to real input photographs and one network handles a large variety of materials.
Loading 1810.09718v1…