PaperlayerView exact revision on arXiv

Source-linked AI summary

Materials for Masses: SVBRDF Acquisition with a Single Mobile Phone Image

Zhengqin Li, Kalyan Sunkavalli, Manmohan Chandraker

arXiv:1804.05790v1cs.CV

TL;DR

The paper addresses ill-posed single-image recovery of spatially varying materials and normals from mobile-phone photographs. It combines a physically informed CNN, specialized SVBRDF data, rendering and classification supervision, flash illumination, and task-specific refinement. The resulting system outperforms prior work, generalizes to real images, and supports material editing and relighting, though material misclassification can cause failures.

  • Problem

    Single-image material estimation is limited by high-dimensional SVBRDFs, insufficient material datasets, and entanglement among shape, lighting, and material effects.

  • Method

    A physically informed CNN estimates normals and SVBRDF components from one flash-enabled mobile-phone image, using a large-scale dataset, differentiable rendering, material classification, and DCRF refinement.

  • Results

    The approach outperforms prior works, handles a broader class of materials, and shows strong performance on real indoor and outdoor images from multiple phone cameras.

  • Takeaways & Limitations

    Single-image mobile-phone capture can recover useful spatially varying materials and normals for material editing and relighting of novel shapes.

  • Takeaways & Limitations

    A real-data failure occurs when the material is misclassified as metal, causing the central specular highlight to be over-suppressed.

Abstract

from arXiv · show

We propose a material acquisition approach to recover the spatially-varying BRDF and normal map of a near-planar surface from a single image captured by a handheld mobile phone camera. Our method images the surface under arbitrary environment lighting with the flash turned on, thereby avoiding shadows while simultaneously capturing high-frequency specular highlights. We train a CNN to regress an SVBRDF and surface normals from this image. Our network is trained using a large-scale SVBRDF dataset and designed to incorporate physical insights for material estimation, including an in-network rendering layer to model appearance and a material classifier to provide additional supervision during training. We refine the results from the network using a dense CRF module whose terms are designed specifically for our task. The framework is trained end-to-end and produces high quality results for a variety of materials. We provide extensive ablation studies to evaluate our network on both synthetic and real data, while demonstrating significant improvements in comparisons with prior works.

1 Introduction

The paper presents a lightweight system that recovers spatially varying materials and normals from one flash-enabled mobile-phone image of a near-planar surface under unconstrained lighting. Its physically informed CNN, specialized dataset, rendering supervision, material classification, and DCRF refinement outperform prior methods across diverse materials and real-world settings.

  • Motivation: Single-image SVBRDF estimation is difficult because material properties are high-dimensional, training data are limited, and shape, lighting, and material effects are entangled.SVBRDFs depend on incident and exitant lighting directions, making estimation from one image ill-posed.
  • Approach: The proposed acquisition method uses an off-the-shelf mobile phone to capture a near-planar surface under unconstrained environment illumination.It avoids the significant equipment and expense typically associated with conventional BRDF capture setups.
  • Contributions: The method combines a lightweight architecture, a perceptually oriented large-scale SVBRDF dataset, and DCRF post-processing designed around the microfacet BRDF model.These components are presented as contributions for joint reconstruction, material classification, and refinement.
  • Approach: The CNN jointly predicts surface normals, diffuse texture, and specular roughness while incorporating a differentiable rendering layer and material classifier.Rendered views under arbitrary lighting provide additional training supervision, while classification constrains the latent representation.
  • Approach: Flash illumination improves acquisition by minimizing occlusion shadows, revealing high-frequency specular highlights, and simplifying estimation and post-processing.The flash is combined with unknown environment illumination during capture.
  • Results: The approach outperforms the state of the art, handles a broader class of materials than stochastic-texture methods, and improves over Li et al. despite using a more compact model.Experiments also show strong performance on real indoor and outdoor images from multiple phone cameras, with applications including material editing and relighting.

2 Related Work

BRDF acquisition has progressed from calibrated, expensive measurements toward learning-based single-image methods, but existing approaches retain constraints in capture requirements, materials, or optimization speed.

  • Traditional BRDF acquisition densely samples a four-dimensional reflection space using expensive, calibrated systems.
  • Photometric stereo methods recover shape and BRDFs, but prior approaches generally require calibrated acquisition or known illumination.
  • Mobile-device methods reconstruct SVBRDFs and normals from 30+ calibrated images, whereas the proposed goal is a single image.
  • A flash/no-flash optimization method reconstructs stochastic SVBRDFs and normals, while the proposed method targets more materials with substantially greater speed.
  • Material-recognition networks can be driven more by object context than material appearance, while homogeneous-BRDF methods address a narrower representation.
  • Earlier single-image approaches include neural style-transfer optimization limited to stationary textures and supervised CNN estimation trained on a small dataset.

3 Acquisition Setup and SVBRDF Dataset

The acquisition setup estimates a spatially varying microfacet BRDF for a near-planar surface from one flash-enabled mobile-phone image, using a dataset of high-resolution SVBRDF materials.

  • Acquisition Setup: The setup reconstructs a near-planar surface’s SVBRDF from one mobile-phone image with the flash enabled under largely uncontrolled illumination.
  • Acquisition Setup: The camera’s approximate perpendicularity to the planar surface is an explicit acquisition assumption evaluated in experiments.
  • Acquisition Setup: The mobile flash and camera provide additional constraints that can improve BRDF reconstruction over environment illumination alone.
  • SVBRDF Model: The surface appearance uses a physically based microfacet BRDF model with per-pixel diffuse color, normal, and roughness parameters.
  • SVBRDF Model: The observed image is modeled as I(d_i, n_i, r_i, L), and the task is to recover d_i, n_i, and r_i for every pixel under unknown illumination L.
  • SVBRDF Dataset: The dataset contains 688 high-resolution 4096 × 4096 spatially varying BRDF materials, split into 588 training and 100 testing materials.

4 Network Design for SVBRDF Estimation

The network combines a shared encoder, parameter-specific decoders, physically motivated rendering and classification supervision, and end-to-end DCRF refinement for single-image SVBRDF estimation. Its losses and refinement terms address parameter balance, normal oversmoothing, diffuse artifacts, and roughness ambiguity.

  • Network architecture: A shared encoder feeds three decoders that reconstruct diffuse color, surface normals, and roughness from an RGB image plus pixel coordinates.The shared representation exploits correlations among BRDF parameters while reducing network size; skip links help retrieve SVBRDF components.
  • Loss functions: The overall training objective combines diffuse, normal, roughness, rendered-image, and classification losses with coefficients balancing their contributions.The classification coefficient is set to 0.0005 to limit its gradient magnitude, while the other coefficients are set to 1 in the experiments.
  • Rendering supervision: A differentiable rendering layer compares predicted-parameter renderings with ground-truth renderings and adds supervision under randomly sampled novel lights.Rendering enables backpropagation through appearance and discourages overfitting to collocated illumination.
  • Loss functions: Normal-loss weighting groups directions into three bins to reduce oversmoothing caused by the dataset’s predominance of flat normals.The bins are defined by the angle between each normal and the z axis, with bin-dependent weights.
  • Material classification: The classifier predicts material type and softmax-weights material-specific BRDF predictions, avoiding separate networks while adding material-aware supervision.Only the final decoder layer expands to K × N outputs; the implementation uses eight material types.
  • DCRF refinement: A task-specific DCRF refines network outputs by smoothing diffuse color, normals, and roughness while preserving relevant image and prediction structure.Diffuse refinement lowers unary influence near specular-highlight artifacts; normal refinement uses position and diffuse-gradient kernels, while roughness refinement combines network estimates with grid search.

5 Experiments

Experiments evaluate the method on synthetic and real data, testing reconstruction quality, architectural components, illumination choices, and generalization across mobile devices. Results show accurate relighting, benefits from material classification and point illumination, and strong real-image performance, with a material-classification failure case.

  • Synthetic results: Synthetic reconstructions recover spatially varying normals, diffuse albedo, and roughness, enabling relighting under substantially different environment and point lights.Rendered results closely match ground truth under novel lighting conditions.
  • Ablations: Adding the material classifier reduces SVBRDF, normal-estimation, and rendering L2 errors by exploiting correlations between material parameters and material type.Joint BRDF-reconstruction and classification training achieves 73.65% material classification error versus 54.96% for a pure classification network.
  • Ablations: Point-plus-environment illumination outperforms both the prior environment-lighting model and the authors’ environment-only network.The authors attribute this to more consistent illumination and better capture of high-frequency appearance.
  • Ablations: Lower relative flash intensity increases errors, supporting flash acquisition; paired flash and no-flash images instead require alignment and reduce applicability.The network works across tested flash-to-environment intensity ratios.
  • Real data: On real images from Huawei P9, Google Tango, and iPhone 6s, one network predicts SVBRDFs and normals whose renderings closely resemble inputs.Captures used handheld phones with the optical axis only approximately perpendicular to the surface.
  • Real data: The real-data evaluation includes a failure case where incorrect metal classification over-smooths a central specular highlight.The limitation is tied specifically to material misclassification.
  • Comparisons: The method is more accurate than the two-shot method on stationary textures while supporting arbitrarily varying SVBRDFs.The comparison reports similar normals but a diffuse reconstruction closer to ground truth for the proposed method.

Appendix

Appendix materials provide additional visual comparisons, rendering examples, error visualizations, and qualitative results. They extend evaluation across normal estimation, relighting, and material editing.

  • Additional results: Figure 10 documents a metal-classification failure that over-smooths specularity.The error is attributed to incorrect material classification.
  • Additional results: Figure 11 demonstrates material editing by swapping reconstructed geometry and material properties before relighting under novel illumination.The example uses two reconstructed samples.
  • Additional results: Compared with [8], which requires two images and over six hours with GPU acceleration, the proposed result is more accurate.The comparison concerns a method with stationary-texture assumptions.
  • Additional results: Figure 12 compares predicted normal maps with [10] using photometric stereo as reference, showing high-quality predictions from a lightweight acquisition system.The figure presents the proposed method alongside the comparison and reference.
  • Additional results: Additional appendix material includes spherical renderings of estimated real SVBRDFs and further qualitative results on synthetic and real data.The appendix also provides error visualizations for SVBRDF estimation.

6 Discussion

The discussion presents a single-mobile-phone framework for SVBRDF acquisition, combining a physically informed CNN, a purpose-built dataset, and simple acquisition settings. It emphasizes high-quality generalization to unconstrained real environments.

  • Discussion: The framework recovers spatially varying BRDFs from a single mobile-phone image using a CNN designed around physical insights into BRDF estimation.The approach targets near-planar surfaces and uses simple acquisition settings.
  • Discussion: The proposed dataset is larger and better suited to material estimation than prior datasets, while the network generalizes well to unconstrained real data.The stated scope includes high-quality results in unconstrained test environments.

A Further Experimental Analysis

Error distributions reveal physically meaningful structure in the test results. Lower-error materials tend to be simpler in normals, diffuse color, and specular shape, whereas higher-error materials exhibit stronger spatial and highlight complexity.

  • Error analysis: Figure 14 reports error distributions for diffuse albedo, normals, roughness, and relighting, then examines materials across combined-error percentiles.Materials are sorted using Ld + Ln + Lrec.
  • Error analysis: Lower-error materials tend to have flat normals, uniform diffuse color, and wide specular lobes.These trends are observed among materials at lower error percentiles.
  • Error analysis: Higher-error materials tend to have more complex normals, stronger local diffuse variation and roughness, or more prominent highlights.The passage links these patterns to physically meaningful trends in the network’s errors.

B Further Results on Real Data

The method estimates normals and SVBRDFs from single mobile-phone images, including unconstrained real captures, and produces qualitatively reasonable results under novel lighting.

  • Normal-map comparison: 52 directional point-light observations support photometric-stereo normal maps used as the reference comparison.The comparison discards the five brightest and five darkest observations before estimating normals.
  • Prediction-error results: 1.7 × 10−4 is the lowest reported SVBRDF error, while 5.2 × 10−2 is the highest.The displayed error is Ld + Ln + Lrec, with corresponding test-set exceedance percentages of 0.0% and 87.5%.
  • Real data: Real captures from wood, tiles, carpets, and other materials yield qualitatively correct normals and spatially-varying BRDFs.The camera is handheld, approximately parallel to the surface, and the flash is enabled.
  • Real data: The network generalizes across iPhone 6s, Huawei P9, and Lenovo Phab 2 devices without mobile-phone calibration.The examples include tight specular lobes that remain well estimated in relighted images.
  • Novel-lighting relighting: Rendering the estimated material on a sphere under a substantially different oblique light direction produces a reasonably matching appearance.The input remains an approximately planar material patch.

C Microfacet BRDF Model

The method uses a microfacet BRDF to model diffuse color, normals, roughness, and intensity, with Fresnel reflectance parameterized differently for conductors and dielectrics. Rendering uses F0 = 0.5 for metals and F0 = 0.05 for other materials, producing more realistic metallic highlights.

  • BRDF formulation: The microfacet BRDF models pixel intensity from diffuse color, normal, roughness, view direction, and light direction.The formulation combines distribution, Fresnel, and geometric terms.
  • BRDF formulation: The geometric term uses view and light directions, surface normals, and a roughness-related parameter through masking functions G1(v, n) and G1(l, n).These functions define the geometric component used in the BRDF.
  • Fresnel reflectance: For conductors, F0 is determined by the index of refraction η and absorption coefficient κ, while dielectric F0 is determined by η.F0 denotes specular reflectance at normal incidence.
  • Rendering settings: F0 = 0.5 for metals and F0 = 0.05 for other materials during dataset rendering.The authors use these values as material-specific rendering settings.
  • Rendering settings: F0 = 0.5 produces a much larger specular-highlight area on smooth aluminum and better matches metallic appearances than F0 = 0.05.The comparison is shown in Figure 19.

D Details of Continuous DCRFs

The refinement stage uses continuous densely connected conditional random fields customized for spatially varying BRDF reconstruction. Its spatially varying weights encode confidence in network predictions and image-dependent cues, while normal prediction uses uniform weighting.

  • DCRF design: The DCRFs remove neural-network artifacts and incorporate task-specific spatially varying weight maps into BRDF inference.The customization targets artifacts caused by saturated highlights and prediction noise.
  • Diffuse albedo: Diffuse-albedo weights reduce confidence near the image center and specular highlights, where colocated flash illumination commonly causes white saturation.The minimum RGB channel value is used as a cue for saturated pixels.
  • Normal prediction: Normal prediction uses a uniform pixel weight because prediction error shows no strong correlation with image position or intensity.The normal-prediction energy function is defined with this uniform weighting choice.
  • DCRF parameters: The DCRF combines unary confidence, pairwise smoothness, and a prior correlating normal and albedo boundaries, with learnable trade-off parameters.Different material types use separate BRDF-parameter sets, averaged according to the material classifier output.
  • Roughness prediction: Roughness refinement gives larger weight to grid-search predictions near glossy regions, where those predictions are usually accurate.The network prediction is denoted ˆri and the grid-search prediction ˜ri.
  • Training and inference: Training uses positive clipped DCRF coefficients optimized with Adam at learning rate 2 × 10^-4, halved every 2000 iterations, with batch size 32.Diffuse albedo, and roughness and normal DCRFs train for 4000 and 3000 iterations, respectively.

E Details of Data Augmentation

Data augmentation scales BRDF parameters in addition to rotating and cropping high-resolution materials. This augmentation improves generalization, with roughness scaling reducing validation error by 15%.

  • Augmentation strategy: Augmentation rotates and crops high-resolution spatially varying materials before rendering image patches.BRDF parameters are also scaled independently for each patch.
  • Parameter scaling: Diffuse albedo and normal-map scale coefficients are sampled uniformly from 0.8 to 1.4, while roughness scales follow a Gaussian distribution centered at 1 with standard deviation 0.2.Normal-map x and y components are scaled and then renormalized to unit length.
  • Effect on generalization: 15% lower validation error for roughness prediction results from simply scaling the roughness parameter for each patch.The authors report that this augmentation greatly improves generalization ability.
Loading 1804.05790v1…