Deep Reflectance Maps

• Reflectance-map estimation remains difficult because identical appearances can arise from different combinations of shape, material, and illumination.Source: §1 · p. 1
• The method estimates orientation-dependent appearance for specular materials under complex natural illumination without factoring material and illuminant separately.A reflectance map represents a fixed material’s appearance across surface orientations under fixed illumination.Source: §1 · p. 1
• The system directly predicts a reflectance map from a segmented object image using an end-to-end convolutional-deconvolutional architecture.Source: §1 · p. 2
• An indirect variant first predicts surface normals, then uses learned sparse-data interpolation to complete the reflectance map.This provides additional supervision through object surface normals during training.Source: §1 · p. 2
• The paper introduces a synthetic-and-real dataset for studying this task and demonstrates applications including material transfer, inpainting, augmented reality, and image-based editing.Source: §1 · p. 2
• The architecture includes an image-to-spherical-domain mapping and addresses sparse unstructured data interpolation.Source: §1 · p. 2

• Classic intrinsic-image and shape-from-shading methods factor images into reflectance, illumination, shading, orientation, or shape, while larger-scale measurements support statistical modeling.Source: §2 · p. 2
• Prior optimization methods exploit relations among shape, reflectance, and natural illumination but often require one component to be known.This paper instead assumes a known object class and segmentation while retaining reflectance and illumination together.Source: §2 · p. 2
• Earlier decomposition work commonly targets scalar diffuse reflectance or limited illumination, whereas this paper addresses specular materials under general illumination.Source: §2 · pp. 2–3
• Spherical-harmonics approaches are limited to diffuse materials, while the paper compares against them for reflectance-map reconstruction.Source: §2 · p. 2
• Computer-graphics reflectance maps, lit spheres, and MatCaps capture orientation-dependent appearance, but this approach automates matching image appearance to orientation.Source: §2 · p. 2
• Deep convolutional encoder-decoder models support per-pixel prediction and image synthesis, whereas this work maps images to the intrinsic reflectance-map domain.Source: §2 · pp. 2–3

3.1. Reflectance Map Representation

• The model targets reflectance maps: orientation-indexed RGB radiance that combines material and illumination under a fixed viewing setup.The representation uses normalized surface-normal x and y coordinates over the visible half-sphere.Source: §3.1 · pp. 3–4

3.2. Direct approach: End-to-end model for prediction of reflectance maps

• The direct approach learns a convolutional-deconvolutional mapping between an object image and its reflectance map, reducing feature maps to 1×1 and upsampling them to 32 × 32 pixels.The final layer uses an Euclidean loss between RGB values for predicted and ground-truth reflectance maps.Source: §3.2 · p. 4
• The direct model must learn both the image-to-directional-domain transformation and appearance interpolation for normals absent from the image.This differs from typical CNN regression tasks with spatial correspondence between input and output.Source: §3.2 · p. 4

3.3. Indirect approach: Reflectance maps from inferred normals and sparse interpolation

• The indirect approach predicts per-pixel orientations, upsamples them, maps image appearances into a sparse directional reflectance map, and predicts a dense map.The first and fourth stages are learned CNNs, whereas upsampling and domain conversion are prescribed transformations.Source: §3.3 · p. 4, Figure 4 · p. 4
• To reduce shadow contamination, the indirect method takes the maximum among samples within ϵ = cos(5°) rather than averaging them.The resulting map remains sparse when orientations are unobserved, so learned interpolation is needed for densification.Source: §3.3 · p. 5
• SparseNet learns dense reflectance maps from sparse, noisy maps created from predicted orientations and image appearances, using synthetic sparse–dense training pairs.The network concatenates encoder features with decoder outputs and uses an L2 loss against dense reference maps.Source: §3.3 · p. 5, Figure 5 · p. 5

4. The SMASHINg Challenge

• SMASHINg combines images, ground-truth reflectance maps and normals where available, reconstruction methods, and metrics for evaluating specular-material reflectance estimation.The challenge includes both synthetic and real imagery and was intended to release data, baselines, methods, and metrics.Source: §4 · p. 5

4.1. Dataset

• The dataset contains segmented synthetic images, photographs, and Internet images of cars.All images are segmented into foreground and background.Source: §4.1 · p. 5

4.1.1 Synthetic images

• The synthetic set contains 60 k images generated from 140 shapes, 100 MERL materials, 40 HDR environments, randomized views, and varied exposure, with disjoint shape, material, and illumination splits.These factors are randomized to create challenging appearance variation.Source: §4.1.1 · p. 5, Figure 6 · p. 5

4.1.2 Photographs

• The real photograph set contains 120 images of six single-lacquer toy cars across four lighting conditions and five views.The photographs were manually segmented and used for qualitative evaluation.Source: §4.1.2 · p. 6

4.1.3 Internet Images

• The Internet test set adds 32 segmented car images without ground-truth normals or reflectance maps for realistic image-based editing evaluation.It supports testing single-material normal and reflectance-map prediction in a realistic setting.Source: §4.1.3 · p. 6

4.2. Methods

• Six reconstruction methods are compared using RGB L2 error and SSIM structural difference.The methods include the direct and indirect approaches, RBF interpolation, spherical harmonics, and variants using ground-truth reflectance maps or normals.Source: §4.2 · p. 6, §4.3 · p. 6

5.1. Quantitative Results

• Synthetic results show accurate reflectance-map estimates for both the direct method and the best indirect methods, while real-image errors increase by roughly one order of magnitude but retain rich specular structure.The two metrics rank approaches consistently, and visual results support the quantitative findings.Source: §5.1 · p. 6
• Additional normal supervision benefits real-image performance, although RBF interpolation can achieve the best metrics while often mislocalizing high-light features.The direct and best indirect approaches perform similarly on synthetic data, whereas indirect supervision helps more on real examples.Source: §5.1 · p. 6
• Ground-truth normals indicate potential to double indirect-method performance, while the diffuse spherical-harmonics baseline trails the best methods by a large DSSIM margin.These comparisons highlight both the value of better normal estimation and the importance of modeling general reflectance maps beyond diffuse assumptions.Source: §5.1 · p. 7
• Dual losses and joint upsampling improve first-stage normal estimation on synthetic data, with upsampling slightly increasing accuracy rather than degrading it.The analysis uses synthetic data because the rendering pipeline provides ground-truth normals.Source: §5.1 · p. 7, Table 2 · p. 7

5.2. Qualitative Results

• Estimated reflectance maps capture overall appearance, highlights, and schematic environmental structure across synthetic, real, and web images.Web images lack reference reflectance maps, while the qualitative examples include predicted normals and reflectance maps across all three sources.Source: §5.2 · p. 7, Figure 7 · p. 8
• Acquired reflectance maps transfer material, specularity, and illumination from real objects to virtual objects of the same or other shapes.Figure 8 shows rendered virtual objects using automatically acquired maps, while Figure 9 illustrates material transfer between scene objects.Source: §5.2 · p. 7, Figure 8 · p. 7, Figure 9 · p. 8
• Changing estimated surface orientations and resampling the reflectance map enables shape manipulation while preserving details, shadows, and consistent material appearance.The application uses painting-based orientation changes and restores appearance differences during re-synthesis.Source: §5.2 · p. 7

• Experiments are limited to segmented cars or objects on uniform backgrounds, although the learning-based approach may adapt to other classes with dedicated training data.Source: §6 · p. 8
• The model assumes a single material, distant illumination, and baked-in illumination, preventing accurate handling of multiple textured parts, nearby light interactions, or separate material and illumination editing.Source: §6 · p. 8
• Quantitative evaluation is limited by the absence of reliable perceptual metrics for reflectance maps.Source: §6 · p. 8

• The SMASHINg benchmark combines synthetic, real, and web images of increasing difficulty with reflectance maps, normals where available, reconstruction baselines, and evaluation metrics.Source: §7 · p. 8
• The approach provides end-to-end image-to-directional-domain learning, learning-based sparse interpolation, and normal supervision that improves accuracy while producing normals as a byproduct.Source: §7 · pp. 8–9
• Results produce truthful reflectance maps across all three investigated scenarios and support several image-based editing tasks.Source: §7 · p. 9