PaperlayerView exact revision on arXiv

Source-linked AI summary

Intrinsic Image Diffusion for Indoor Single-view Material Estimation

Peter Kocsis, Vincent Sitzmann, Matthias Nießner

arXiv:2312.12274v2cs.CVcs.AIcs.GR

TL;DR

Single-view appearance decomposition is ambiguous because lighting and material jointly shape an image, and existing methods struggle with high-frequency material detail. Intrinsic Image Diffusion samples material explanations with a conditional diffusion model adapted from a pretrained image prior. It reports improved albedo prediction and cleaner, more consistent materials that enable optimization-based emissive-light fitting.

  • Problem

    Single-image decomposition is ambiguous because lighting and material jointly determine appearance, and recent data-driven methods still struggle to capture high-frequency detail.

    Source: §1 · p. 1

  • Method

    The method conditions a diffusion-based generative model on one input image to sample material-property solutions, fine-tuning a pretrained model on photorealistic synthetic data.

    Source: §1 · p. 2

  • Results

    77.6% FID and 4.04dB PSNR improvement are reported for albedo prediction, alongside cleaner and more consistent materials than prior work.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The material predictions enable direct optimization-based emissive lighting fitting.

    Source: §5 · p. 8

    Training currently uses paired synthetic samples, and material estimation is the focus while lighting is optimized independently.

    Source: §4.4 · p. 8

Abstract

from arXiv · show

We present Intrinsic Image Diffusion, a generative model for appearance decomposition of indoor scenes. Given a single input view, we sample multiple possible material explanations represented as albedo, roughness, and metallic maps. Appearance decomposition poses a considerable challenge in computer vision due to the inherent ambiguity between lighting and material properties and the lack of real datasets. To address this issue, we advocate for a probabilistic formulation, where instead of attempting to directly predict the true material properties, we employ a conditional generative model to sample from the solution space. Furthermore, we show that utilizing the strong learned prior of recent diffusion models trained on large-scale real-world images can be adapted to material estimation and highly improves the generalization to real images. Our method produces significantly sharper, more consistent, and more detailed materials, outperforming state-of-the-art methods by $1.5dB$ on PSNR and by $45\%$ better FID score on albedo prediction. We demonstrate the effectiveness of our approach through experiments on both synthetic and real-world datasets.

MIT EECS

Intrinsic Image Diffusion samples multiple material explanations for an indoor scene from one view, using a conditional generative model and a learned diffusion prior.

MIT EECS

  • The method estimates indoor-scene materials from a single view by sampling multiple possible explanations.The approach uses a conditional generative model and the learned prior of recent diffusion models [36].Source: Figure 1 · p. 1
  • Its material estimates are described as detailed and consistent on complex indoor scenes.Source: Figure 1 · p. 1
  • The approach is reported to outperform recent state-of-the-art methods and support high-quality controllable lighting optimization.Source: Figure 1 · p. 1

1. Introduction

Single-view appearance decomposition is ambiguous because lighting and material jointly determine observed appearance, while existing methods struggle with high-frequency material detail. The introduction proposes probabilistic diffusion sampling and adaptation of a pretrained image prior as an alternative to deterministic prediction.

1. Introduction

  • Appearance decomposition from one image is inherently ambiguous because observed appearance combines lighting and material properties.The decomposition seeks geometric, material, and lighting properties from a single image.Source: §1 · p. 1
  • Recent data-driven methods predict overall material properties but continue to struggle with accurately capturing high-frequency details.Source: §1 · p. 1
  • The introduction attributes these limitations to deterministic models averaging a solution space in which lighting and material can explain one another.Source: §1 · p. 1
  • The proposed conditional diffusion model samples material-property solutions conditioned on the input image.Source: §1 · p. 2
  • The method fine-tunes pretrained Stable Diffusion V2 on photorealistic synthetic InteriorVerse data because no large-scale real-world intrinsic-image dataset is available.The stated aim is to adapt the model's learned image prior to material estimation.Source: §1 · p. 2
  • 77.6% FID and 4.04dB PSNR improvement are reported for albedo prediction using a pretrained diffusion-model prior.The approach also optimizes lighting as multiple point lights and a global environment map.Source: §1 · p. 2
  • The introduction presents sharp, non-averaged material alternatives and a real-image prior as intended ways to reduce the synthetic-to-real domain gap.It also identifies semantic and perceptual cues in the prior as potentially useful for material estimation.Source: §1 · p. 2

2. Related work

Related work spans intrinsic-image methods, learned inverse-rendering systems, single-object material acquisition, and diffusion-model applications. The paper distinguishes its approach by sampling multiple material-space solutions for multi-object estimation from one image and transferring a pretrained image prior.

2. Related work

  • Earlier intrinsic-image methods decomposed single-object images into reflectance and shading using heuristic cues or priors, while indoor-scene datasets introduced manually annotated relative material comparisons.The datasets provide a benchmark for evaluation [21].Source: §2 · p. 2
  • Recent inverse-rendering methods use feed-forward models, including UNets, cascaded networks, and Transformers, whereas this work samples multiple possible decompositions.Source: §2 · p. 2
  • Probabilistic material acquisition has addressed diffuse–specular ambiguity for a single scanned object, while this method targets multi-object estimation from one image.It additionally uses a pretrained image prior to improve generalization.Source: §2 · p. 2
  • The method fine-tunes a diffusion model to generate samples in material space, transferring learned image cues while aiming to mitigate the domain gap.Source: §2 · p. 2

3. Method

The method samples albedo, roughness, and metallic explanations from a single image using a conditional latent diffusion model, adapting a pretrained image prior on synthetic material data. It also uses the predicted materials to optimize scene lighting.

3. Method

  • The model generates possible material properties, including albedo, roughness, and metallic maps, from a single input image using a latent diffusion model conditioned on the input image.Training uses InteriorVerse, which contains more than 50,000 rendered images from over 4,000 scenes.Source: §3 · pp. 2–3

3.2. Training

  • Training encodes material maps, adds noise to their latent features, and conditions noise prediction on the input image using a squared-error objective.The model adapts Stable Diffusion’s learned image prior; a trainable image encoder and CLIP image embedding provide conditioning.Source: Figure 2 · p. 3, §3.2 · pp. 3–4

3.1. Material representation

  • The material representation uses a GGX microfacet BRDF with albedo, roughness, and metallic properties, separately encoded with albedo into latent material features.The BRDF image stores roughness in R, metallic in G, and zero in B.Source: §3.1 · p. 3

3.2. Training

  • Compared with baselines [27, 48], predictions on InteriorVerse retain sharp, detailed albedo textures where baseline results are smoothed and may bake in lighting, specularities, or shadows.Source: Figure 3 · p. 4
  • For a single view, the model samples multiple material explanations; variance across 100 samples is higher for specular and emissive objects, whose properties are highly ambiguous.Source: Figure 4 · p. 4
  • The method adapts a prior trained on large-scale real images to material estimation by fine-tuning Stable Diffusion on synthetic data, rather than fine-tuning a synthetic-trained prior on a smaller real-world dataset.Training uses the photo-realistic InteriorVerse dataset.Source: §3 · p. 3, §3.2 · p. 4

3.3. Inference

  • At inference, sampled material features are split into albedo and BRDF features and decoded separately to produce a possible material explanation.Source: §3.3 · p. 4

3.4. Lighting optimization

  • For relighting, the method estimates geometry with OmniData and optimizes global Spherical Gaussian incident lighting together with point lights that use Spherical Gaussian emission profiles.The resulting lighting representation is described as expressive for detailed emissions and controllable for relighting.Source: §3.4 · p. 4, Figure 5 · p. 4

4. Experiments

Experiments evaluate sampled material predictions on synthetic and real-world data, compare them with prior methods, and examine pre-training and current scope. Results cover albedo fidelity and consistency, uncertainty, roughness and metallic prediction, real-image evaluation, and image reconstruction.

4.1. Synthetic results

  • Our method improves albedo FID over [48] by 44.99% on synthetic InteriorVerse data.Table 1 evaluates predicted albedos using 10 sampled solutions per view.Source: Table 1 · p. 5, §4.1 · p. 5
  • Averaging the solution space yields +1.28dB PSNR and the best SSIM; the best sample is +1.01dB PSNR closer to ground truth.Evaluating samples independently gives +0.22dB PSNR but similar results to [48].Source: §4.1 · pp. 5–6
  • Across 100 samples, large diffuse objects have low variance, while specular, emissive, and small objects are more uncertain; albedo-variance/metallic Pearson correlations range from 0.47 to 0.68.The authors interpret the pixel-wise variance map as an uncertainty map.Source: §4.1 · pp. 6–7
  • For roughness and metallic prediction, our method improves FID over [48] by 39.90% and 53.56%, respectively.[48] improves roughness FID over [27] by 60.21%; [27] does not estimate metallic properties.Source: §4.1 · p. 7

4.2. Real-world results

  • On real images, our method produces detailed materials, while baselines struggle with baked-in lighting; WHDR is similar, and the metric favors smoother results over high-frequency detail.The table caption also reports sharper predictions and higher Perceptual Quality in the user study.Source: §4.2 · p. 7, Table 3 · p. 7
  • Our full lighting optimization reconstructs all light sources faithfully and outperforms the baseline by a high margin; the baseline lighting-and-geometry module misses small sources.The table compares rerenderings using [48]’s and our material predictions with the full pipeline.Source: §4.2 · p. 7, Table 4 · p. 8, Figure 8 · p. 8

4.3. Ablations

  • Pre-training significantly improves predictions on both InteriorVerse and IIW; the real-image prior also benefits synthetic results and reduces the domain gap.Without pre-training, the model struggles with under- and over-saturated regions.Source: §4.3 · p. 8, Table 5 · p. 8, Figure 9 · p. 8

4.4. Limitations and Future Work

  • The current method trains on paired synthetic samples and optimizes lighting independently; weak supervision with real-world samples and fused lighting estimation remain future directions.The authors also identify text-guided material editing as a possible use of generative models.Source: §4.4 · p. 8

5. Conclusion

Intrinsic Image Diffusion treats ambiguous appearance decomposition probabilistically, producing cleaner, more consistent material explanations. The predicted materials and optimized lighting support material and lighting edits.

5. Conclusion

  • The method formulates ambiguous appearance decomposition probabilistically and uses a diffusion-model image prior to improve material prediction quality and reduce the domain gap.Its generative model samples possible explanations rather than predicting a single solution.Source: §5 · p. 8

A. Applications

  • Predicted materials and optimized lighting enable edits that change specific image aspects, such as materials or lighting.The materials are sharp and have minimal or no baked-in lighting, supporting editing in material space.Source: A. Applications · p. 12, §A.1 · p. 12

A.1. Material Editing

  • Changing a beige wall to cyan also changes its reflections, which turn greenish because the lamp emission is mostly red and green while the wall reflects green and blue most.The sharp material predictions have smooth but sharp features for each object.Source: §A.1 · p. 12

A.2. Lighting Editing

  • The lighting representation allows independent editing of fitted light-source emissions and physically realistic relighting.The applications include fitting small light sources, even close to objects, and editing them independently.Source: §A.2 · p. 12

B. Method Details

Lighting optimization combines global environment illumination with spatially varying point lights. Reconstruction fitting and regularization constrain the representation, with pruning used to remove weak sources.

B.1. Lighting Optimization

  • A hybrid representation combines Spherical Gaussian environment lighting with Nlight point lights to represent global and out-of-view illumination and nearby lights with varying emission profiles and colors.Each point light also uses an SG emission profile for expressivity.Source: §B.1 · p. 12
  • Point-light positions are initialized on an image-space grid and backprojected with a 1e−2 offset from the surface along the normal in normalized depth space.Emission profiles start with minimal uniform emission.Source: §B.1 · p. 12
  • Lighting parameters are optimized using L2 reconstruction loss with predicted materials and OmniData normals, considering every light per pixel without occlusions.Taking the absolute value of the geometry term stabilizes optimization by preserving gradients when a light moves behind an object and out of the scene.Source: §B.1 · p. 12
  • Regularization penalizes SG emission weights and inverse distance to the nearest surface, while adaptive pruning encourages using fewer point lights.Without regularization, one source could be represented by multiple point lights distributed over a sphere around it.Source: §B.1 · p. 12
  • The loss combines reconstruction, position, and emission-weight regularization terms: L = L_rec + λ_pos L_pos + λ_val L_val.The position term sums inverse nearest-surface distances; the value term sums SG weights.Source: §B.1 · p. 12
  • Optimization uses Adam with Nlight=6×8 and Nsg=2×6, halves the learning rate on stagnation, prunes sources below 5% of the strongest source, and typically takes 5−10 minutes on one A6000 GPU.It stops after performance stagnates longer.Source: §B.1 · p. 13

C. Experiment Details

The experiment details evaluate both baselines at 320 × 240 resolution because their reported performance degrades at higher resolutions.

C. Experiment Details

  • Both baselines are evaluated at 320 × 240 resolution because the original papers report performance degradation at higher resolution.This matches the resolution at which both baselines were trained.Source: C. Experiment Details · p. 13

D.1. Synthetic Results

The appendix provides additional InteriorVerse material-estimation results and examines uncertainty in single-view albedo estimation, including its relationship to metallic variance.

D.1. Synthetic Results

  • Additional material-estimation results are provided on the InteriorVerse dataset.They are shown in Fig. 14.Source: §D.1 · p. 13
  • Glossy objects tend to have higher albedo uncertainty, and metallic and albedo variance maps are correlated, though not perfectly because specularity is not the only ambiguity source.Specularity is identified as one major source of ambiguity in single-view albedo estimation.Source: §D.1 · p. 13

D.2. Real Results

The real-world results include additional material estimates on IIW, full-pipeline image rerenderings, and a comparison with S-AWARE. The S-AWARE comparison reports estimates for roughness and metallic properties alongside improved albedo without baked-in lighting or shadows.

D.2. Real Results

  • Additional IIW material-estimation results are provided in Fig. 15.Source: §D.2 · p. 13
  • A user study was conducted to evaluate the real-world predictions perceptually.Source: §D.2 · p. 13
  • The authors provide additional image rerenderings using the full pipeline in Fig. 12.Source: §D.2 · p. 13
  • Against S-AWARE, the method predicts roughness and metallic properties and improves albedo by avoiding baked-in lighting or shadows.The comparison is shown in Fig. 13.Source: §D.2 · p. 13

E. Additional Ablations

The ablation reports that depth and normal conditioning can improve appearance decomposition, though the image-conditioned variant remains the main model for baseline consistency. Additional comparisons show material-estimation behavior on synthetic and real-world images, while variance is associated with ambiguity in glossy objects.

Variance evaluation

  • Glossy objects tend to have higher albedo-prediction variance because of specular ambiguity; emissive, small, or under- and over-exposed objects add ambiguity.The variance comparison is between albedo predictions and the true metallic map.Source: Figure 11 · p. 13

Effect of depth-conditioning

  • Geometry information provides helpful cues and can improve performance, but the image-conditioned variant remains the main model for consistency with baselines.The comparison uses predicted depth and normals at test time; training variants used ground-truth or predicted geometry.Source: Table 6 · p. 13, E. Additional Ablations · p. 13

Image reconstruction

  • Additional image-reconstruction results using the full pipeline are shown in Fig. 12.Source: Figure 12 · p. 14

Synthetic material estimation

  • In synthetic comparisons, both baselines produce generally good albedo colors but tend to bake in lighting and specularities, while this method yields sharp, detailed materials.The comparison is against [27, 48] in Fig. 14.Source: Figure 14 · p. 16

Real-world material estimation

  • In challenging real-world settings, baselines often bake lighting and shadows into albedo, while this method produces sharp, detailed materials.The comparison is against [27, 48] in Fig. 15.Source: Figure 15 · p. 18
Loading 2312.12274v2…