PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks

Xiao Li, Yue Dong, Pieter Peers, Xin Tong

arXiv:1809.00886v1cs.GR

TL;DR

Recovering plausible SVBRDFs from a single photograph is ill-posed and requires difficult-to-collect labeled training pairs. The paper proposes self-augmentation with CNN predictions and physically based rendering to expand training from unlabeled photographs. The approach is demonstrated on spatially varying wood, metals, and plastics, with self-augmentation producing more plausible results and reducing the required labeled samples.

  • Problem

    Single-photograph SVBRDF recovery under unknown natural lighting is ill-posed, while obtaining sufficiently large labeled photograph–reflectance pairs is difficult and time-consuming.

  • Method

    A CNN predicts provisional reflectance parameters for unlabeled material photographs, renders corresponding images under new lighting, and uses the resulting temporary pairs to progressively refine training.

  • Results

    The method is demonstrated on spatially varying wood, metals, and plastics, while self-augmentation produces more plausible results with fewer visual artifacts than regular training using the same 40 labeled samples.

  • Takeaways & Limitations

    Unlabeled photographs can augment CNN training for plausible SVBRDF estimation when a physically based rendering algorithm provides the corresponding forward process.

  • Takeaways & Limitations

    Each network targets one material class and planar samples under distant lighting with homogeneous specular components; estimates also depend on reflectance information present in the input photograph.

Abstract

from arXiv · show

We present a convolutional neural network (CNN) based solution for modeling physically plausible spatially varying surface reflectance functions (SVBRDF) from a single photograph of a planar material sample under unknown natural illumination. Gathering a sufficiently large set of labeled training pairs consisting of photographs of SVBRDF samples and corresponding reflectance parameters, is a difficult and arduous process. To reduce the amount of required labeled training data, we propose to leverage the appearance information embedded in unlabeled images of spatially varying materials to self-augment the training process. Starting from an initial approximative network obtained from a small set of labeled training pairs, we estimate provisional model parameters for each unlabeled training exemplar. Given this provisional reflectance estimate, we then synthesize a novel temporary labeled training pair by rendering the exact corresponding image under a new lighting condition. After refining the network using these additional training samples, we re-estimate the provisional model parameters for the unlabeled data and repeat the self-augmentation process until convergence. We demonstrate the efficacy of the proposed network structure on spatially varying wood, metals, and plastics, as well as thoroughly validate the effectiveness of the self-augmentation training process.

1 INTRODUCTION

The paper targets plausible SVBRDF recovery from a single photograph under unknown natural lighting, where accurate labeled training data is difficult to obtain. It introduces self-augmentation that uses provisional predictions and physically based rendering to expand training from a smaller labeled set.

  • Motivation: Recovering SVBRDFs from one photograph under unknown natural lighting is challenging and ill-posed, but plausible estimates can still support applications such as virtual worlds and games.The paper contrasts this goal with the time-consuming manual work of skilled artists.
  • Proposed representation: The CNN estimates a reduced SVBRDF comprising spatially varying diffuse albedo and normals with homogeneous specular albedo and roughness.The reduction addresses the difficulty of recovering fully spatially varying specular appearance from a single image.
  • Training-data challenge: Labeled photographs paired with reflectance parameters are tedious to gather, while unlabeled material photographs are typically easier to obtain.Existing databases rarely densely cover all spatial variations within a material class.
  • Self-augmentation: The self-augmentation strategy uses the target CNN to predict provisional reflectance parameters for unlabeled images, then renders corresponding images under new lighting to create temporary labeled pairs.Rendering supplies the known inverse relationship needed to generate training examples without direct reflectance labels.
  • Self-augmentation: The network is refined with these rendered pairs and the provisional parameters are repeatedly re-estimated to continue self-augmentation.The supplied introduction describes this as a progressive training strategy built around the CNN and rendering algorithm.

2 RELATED WORK

Prior work addresses single-image reflectance modeling through controlled or known illumination, strong priors, user interaction, and learned methods using synthetic or unlabeled data. This paper relates self-augmentation to generative training approaches while using unlabeled observations rather than parameter perturbations.

  • 2.1 Single Image Reflectance Modeling: Single-image reflectance methods commonly make estimation more tractable by controlling or assuming known incident lighting.Examples include light-field, near-field, flash-image, and known-directional-light settings.
  • 2.1 Single Image Reflectance Modeling: Other approaches recover reflectance under uncontrolled or unknown lighting using reflectance maps, expected lighting distributions, or strong priors.These methods cover homogeneous objects as well as spatially varying reflectance and normals in selected settings.
  • 2.1 Single Image Reflectance Modeling: AppGen accelerates reflectance-component authoring through limited user interaction but does not scale well to large-scale content generation and assumes directional lighting.Its manual interaction requirement distinguishes it from fully learned estimation.
  • 2.2 Deep Learning with Unlabeled/Synthetic Data: Deep-learning methods often depend on large datasets, motivating the use of synthetic data for tasks including detection, pose estimation, segmentation, and intrinsic decomposition.The cited work positions synthetic training data as a way to alleviate dataset-gathering difficulty.
  • 2.2 Deep Learning with Unlabeled/Synthetic Data: Self-augmentation differs from Breeder Learning by using unlabeled data to explore and refine the search space rather than perturbing estimated parameters.The forward rendering component can serve as the generative black-box in self-augmentation.

3 SVBRDF-NET

SVBRDF-net estimates plausible appearance parameters from a single naturally lit photograph using separate networks for homogeneous and spatially varying components. Its self-augmentation strategy uses provisional predictions and forward rendering to exploit unlabeled material photographs and refine the network.

  • SVBRDF-net: SVBRDF-net estimates plausible appearance parameters from a single near-field photograph of a planar material under unknown natural lighting.The model targets a reduced SVBRDF with homogeneous specular and spatially varying diffuse and normal components.
  • SVBRDF-net: The reduced representation combines homogeneous specular albedo and roughness with spatially varying diffuse albedo and per-pixel surface normals.The specular component uses the Ward BRDF, while diffuse albedo and normals vary spatially.
  • Network configuration: The network separates homogeneous and spatially varying outputs, using a fully connected synthesis network for the former and mirrored upsampling layers for the latter.The spatially varying branch concatenates analysis feature maps into corresponding synthesis layers to reintroduce high-frequency details.
  • Training data: Training requires broad coverage of material spatial variations alongside viewing and lighting conditions, but labeled pairs are costly because corresponding model parameters require acquisition or authoring.Unlabeled photographs are easier to obtain and contain material instances whose parameters are unknown.
  • Training by self-augmentation: Self-augmentation estimates provisional parameters for unlabeled images, renders exact visualizations under new conditions, and uses the resulting temporary labeled pairs to refine the network.The process alternates labeled and unlabeled training and is repeatedly applied as the network improves.
  • Training by self-augmentation: The strategy assumes unlabeled data lies within or near the search space covered by labeled data, because the CNN’s extrapolation behavior outside that region is not guaranteed.Outside the covered region, provisional parameters may be implausible or create manifold-folding ambiguities.

4 RESULTS

The proposed SA-SVBRDF-nets estimate plausible reflectance parameters from single photographs across wood, plastics, and metals, while self-augmentation improves results using unlabeled images. Evaluation shows good agreement with reference parameters, fewer artifacts than regular training, and clear material-specific scope boundaries.

  • Training and data preparation: The system is implemented by first training an initial CNN with labeled data, then self-augmenting it with unlabeled training images.The implementation uses Caffe and ADAM; the initial network is trained for 10 epochs before self-augmentation.
  • Training and data preparation: The datasets contain 50 wood, 60 plastic, and 60 metal SVBRDFs, with 10 from each material reserved for testing.The remaining 40 wood and 50 plastic and metal SVBRDFs are used for training; unlabeled sets contain 1000 wood, 1000 plastic, and 1200 metal images.
  • Reflectance estimation: Revisualizations under novel lighting exhibit visual qualities similar to the inputs, suggesting that the networks estimate plausible reflectance parameters.This qualitative observation is reported for wood, plastic, and metal SA-SVBRDF-nets.
  • Reflectance estimation: Estimated parameters match reference parameters well across wood, plastic, and metal examples, despite differences between estimates and references.The comparison uses held-out input photographs and includes diffuse albedo, normal maps, homogeneous specular components, and novel-lighting visualizations.
  • Impact of self-augmentation: With the same 40 labeled wood samples, self-augmentation produces more plausible estimates and fewer visual artifacts than regular SVBRDF-net training.Reported artifacts include specular remnants in diffuse albedo, overestimated roughness, and faint splotchy structure; evaluation takes 0.3 seconds on a GPU.
  • Scope and limitations: The networks are material-specific, and significant spatial variation in the specular component can make results unpredictable.Cross-estimating wood and metal with the other material’s network fails to produce physically plausible results, while modest specular variation remains plausible.

5 DISCUSSION

The discussion validates self-augmentation on homogeneous BRDF estimation, examines its data and smoothness assumptions, and identifies practical scope and theoretical limitations. It also relates the approach to GANs, VAEs, and rendering-assisted learning.

  • Self-augmentation validation: Self-augmentation is validated on a simpler homogeneous BRDF-net using monochrome images, monochrome lighting, and monochrome BRDFs.The experiment estimates homogeneous BRDFs from single images of smooth spherical objects under unknown natural lighting.
  • Self-augmentation validation: The BRDF-net training set samples 10 diffuse and specular albedos and 15 roughness values, yielding 1500 training samples.Diffuse and specular albedos are uniformly sampled in [0.05, 1.0], while roughness is log-uniformly sampled in [0.02, 1.0].
  • Self-augmentation validation: Reconstruction error is measured as mean squared error between visualizations of ground-truth and estimated reflectance parameters under the input view and lighting.The experiment compares a densely trained reference baseline with networks trained using reduced labeled data and self-augmentation.
  • Assumptions and limitations: Self-augmentation assumes a locally smooth search space; on discontinuous spaces it improves overall accuracy but tends to smooth the discontinuity.The method can also be driven toward an incorrect local minimum depending on local gradients.
  • Assumptions and limitations: Self-augmentation assumes unlabeled data lies within the labeled search-space region, although empirical results show accuracy can improve beyond that region.Reliable estimates are obtained in practice when unlabeled data and queries remain close to the covered search space.
  • Practical scope: Each SA-SVBRDF-net is trained for one material class, and estimates worsen when relevant reflectance features, such as specular highlights, are absent from the input.The method is restricted to planar samples under distant lighting with a homogeneous specular component.
  • Assumptions and limitations: A formal derivation of convergence conditions and labeled/unlabeled data-distribution conditions remains missing.The paper presents empirical evidence that self-augmentation can greatly reduce the required labeled data.
  • Relation to other deep learning methods: Unlike GANs and VAEs, the method uses a fixed synthesizer and randomly samples physically meaningful parameters through lighting and view variation.The comparison emphasizes the method’s use of parameters with precise physical meaning.

6 CONCLUSION

The paper concludes that SA-SVBRDF-net estimates physically plausible reflectance parameters from a single photograph while using self-augmentation to reduce labeled-data requirements. It validates the strategy on BRDF estimation and identifies theoretical analysis and non-planar generalization as future directions.

  • Conclusion: SA-SVBRDF-net estimates physically plausible reflectance parameters from a single photograph of a planar spatially varying material under unknown natural lighting.The method is a convolutional neural network for single-photograph reflectance estimation.
  • Conclusion: Self-augmentation reduces the required labeled training data by leveraging embedded information in a large collection of unlabeled photographs.The strategy progressively uses provisional estimates and synthesized training pairs.
  • Conclusion: The strategy relies on an exact inverse of the desired SVBRDF-net in the form of a rendering algorithm.The rendering component supports synthesis of additional training pairs from unlabeled images.
  • Conclusion: The paper demonstrates the trained SVBRDF-net and thoroughly validates self-augmentation on a homogeneous BRDF-net.Future work includes analyzing theoretical limits and conditions and generalizing the SVBRDF-net to non-planar samples.
Loading 1809.00886v1…