PaperlayerView exact revision on arXiv

Source-linked AI summary

Casual Flash Lighting for Gaussian Splat Inverse Rendering

Jiamin Xu, Dongheng Wei, Jiarong Zhao, Qi Wang, James Tompkin, Weiwei Xu, Gang Xu

arXiv:2610.06035v1cs.CV

TL;DR

Inverse rendering from photographs is ambiguous, while active-light methods often require specialized capture conditions. This paper combines casual flash and static lighting with a GS-anchored diffuse field, and reports improved material decomposition and relighting over recent baselines.

  • Problem

    Recovering geometry, materials, and lighting from images is ambiguous because these scene properties are entangled.

    Source: §I · p. 1

  • Method

    The method combines casual flash and static-light observations with deferred rendering and a GS-anchored diffuse field for view-consistent diffuse color.

    Source: §I · p. 1, C. GS-anchored Diffuse Field for… · p. 4

  • Results

    Across synthetic and real-world scenes, the method improves material decomposition and relighting quality over recent baselines.

    Source: §I · p. 1, §VII · p. 12

  • Takeaways & Limitations

    Flash and non-flash images provide complementary physical cues for material decomposition in casually captured indoor scenes.

    Source: §I · p. 1, §VII · p. 12

    The method is less sensitive to subtle roughness changes than one-light-at-a-time methods and may produce noisy material estimates in regions overexposed even without flash.

    Source: §VI · p. 12

Abstract

from arXiv · show

Recovering geometry, materials, and lighting from photographs is highly ambiguous when only static illumination is available. Active-lighting setups reduce the ambiguity but require dark rooms or specialized hardware. Instead, we synergize both static and flash lighting from casual indoor capture, with the flash on or off, each from independent viewpoints. The flash residual constrains albedo and the BRDF, while static lighting captures grazing-angle specular highlights that flash misses. With a 2DGS reconstruction framing, our key contribution is a GS-anchored diffuse field: a hash-encoded MLP is queried at the rasterized 2DGS depth. As it depends only on world position, it is view consistent in 3D and allows the flash residual to drive material decomposition instead of being absorbed by alpha-blending drift across views. At the same time, we render static lighting with deferred shading such that it can also supervise material decomposition. On five synthetic and three real indoor scenes, our method outperforms six recent baselines on diffuse color, albedo and roughness material parameters, and in relighting where PSNR improves by 4.17 dB over the next-best baseline.

I. INTRODUCTION

Inverse rendering is ambiguous because geometry, materials, and lighting are entangled. The paper combines casual flash and static-light capture with a GS-anchored diffuse field to improve material decomposition across views.

I. INTRODUCTION

  • Geometry, materials, and lighting are difficult to disentangle because inverse rendering has many degrees of freedom and inherent ambiguities.Source: §I · p. 1
  • Casual flash and static lighting provide complementary material cues: flash reaches statically occluded regions, while static lighting captures grazing reflections flash misses.Flash also aids albedo estimation when static lighting is complex or non-white.Source: §I · p. 1
  • Using rasterized 2DGS depth to query a hash-encoded MLP for diffuse color, the GS-anchored field improves diffuse/specular decomposition and retains 2DGS primitives for interactive rendering.Source: §I · p. 1

II. RELATED WORK

  • Static-light inverse-rendering methods use Gaussian representations and physically based shading, but simplified or partially observed illumination can make material estimation challenging.Scene-level methods such as FIPT and IRIS use explicit physically based rendering formulations, yet complex illumination can be absorbed into material appearance when only static-light observations are used.Source: §II · p. 2
  • Active-light methods add material-estimation constraints through controlled illumination, but often require dark rooms, calibrated devices, or dense measurements.Source: §II · p. 2
  • WildLight physically constrains the BRDF through flash differentials, whereas this method physically models ambient shading so both lighting conditions inform BRDF recovery.In WildLight, strong ambient cues such as window sunlight producing specular highlights can be absorbed by its ambient network rather than constrain the BRDF.Source: §II · p. 2
  • Feed-forward methods offer efficient material predictions, but may produce lighting-dependent albedo casts and unreliable roughness without scene-specific rendering constraints.The paper therefore uses feed-forward predictions as auxiliary priors rather than final results.Source: §II · pp. 2–3

III. METHOD

The method uses rasterized Gaussian primitives to represent scene geometry and appearance, with 2DGS disks supporting geometric reconstruction. A microfacet BRDF models material appearance through diffuse and specular components.

A. Preliminaries

  • Gaussian Splatting represents scenes through rasterized primitives with geometric attributes and opacity, plus appearance attributes.Source: A. Preliminaries · p. 3
  • 2DGS represents surfaces with oriented disks evaluated where explicit rays intersect them, enabling high-quality geometric reconstruction.Source: A. Preliminaries · p. 3
  • Front-to-back alpha blending composites Gaussians sorted by their centers, using opacity multiplied by Gaussian weight.Source: A. Preliminaries · p. 3
  • The microfacet BRDF uses albedo, roughness, and metallicity to combine a diffuse term with an isotropic specular GGX lobe.Its normal distribution depends on roughness; Fresnel and microfacet shadowing are also modeled.Source: A. Preliminaries · pp. 3–4

B. Overview

The method combines static-light and flash images captured from independent viewpoints, using a 2DGS-based representation and deferred shading. A GS-anchored diffuse field enforces view consistency for diffuse color, helping improve diffuse-specular decomposition.

B. Overview

  • The method reconstructs geometry and spatially varying BRDF materials from static-light and flash images captured at different viewpoints and in potentially different numbers.The capture uses a commercial camera with a standard flashlight mounted on top.Source: B. Overview · p. 4

C. GS-anchored Diffuse Field for Diffuse-Specular Decomposition

  • A hash-encoded MLP queried at rasterized 2DGS depth provides world-position-dependent diffuse color, enforcing multi-view consistency.Per-Gaussian material attributes are rasterized into shading buffers; the field is used for diffuse color, while other material properties remain per-Gaussian.Source: B. Overview · p. 4, C. GS-anchored Diffuse Field for… · pp. 4–5
  • The GS-anchored diffuse field reduces view-dependent diffuse-color effects and improves diffuse-specular decomposition.The figure identifies highlights as an example of the view-dependent effects reduced.Source: Figure 3 · p. 4, C. GS-anchored Diffuse Field for… · p. 4
  • Per-Gaussian diffuse colors can become inconsistent across views because alpha blending mixes different primitives with different weights.Removing spherical harmonics from diffuse color does not resolve this inconsistency, which complicates illumination decomposition.Source: C. GS-anchored Diffuse Field for… · p. 4, B. Overview · p. 4
  • The field uses rasterized depth to query world-space positions, and depth-distortion and normal-prior losses support multi-view-consistent rendered depth.The authors attribute remaining diffuse-color inconsistency to alpha blending rather than virtual geometry.Source: C. GS-anchored Diffuse Field for… · p. 4
  • Predicting all view-independent material properties with the field degraded fine-scale detail, so the method applies it only to diffuse color.The authors attribute the degradation to small cross-view inconsistencies in queried world-space positions, compounded across correlated material channels.Source: C. GS-anchored Diffuse Field for… · p. 5

D. Deferred Shading under Static and Flash Lighting

Deferred shading models static illumination with spatially varying environment maps and models flash images as static illumination plus a flash contribution. Separate diffuse and specular representations and flash-residual constraints support material decomposition.

D. Deferred Shading under Static and Flash Lighting

  • Static incident radiance is interpolated from environment maps at the eight neighboring grid cells around each shading point.The scene bounding box is partitioned into cubic cells sized using estimated scene height.Source: D. Deferred Shading under Static and… · p. 5
  • Static-light shading comprises diffuse and specular components, with separate environment-map representations optimized independently for each.The diffuse map is queried by surface normal; the mipmapped specular map is queried by reflected direction and roughness.Source: D. Deferred Shading under Static and… · p. 5
  • Diffuse color is factored into albedo, diffuse irradiance, and shadow, with the flash-induced residual serving as the primary cue for separating albedo from shadow.The image-space shadow map is formed by alpha-blending learnable per-Gaussian shadow attributes.Source: D. Deferred Shading under Static and… · p. 5
  • The specular component uses a split-sum approximation that factors the integral into a specular lobe term and a BRDF integral term.The specular environment map is mipmapped and queried using reflected direction and roughness.Source: D. Deferred Shading under Static and… · p. 5
  • The flash is approximated as a camera-colocated point light, with direct shading evaluated explicitly and indirect shading precomputed for efficient optimization.The model omits indirect flash radiance outside the corresponding image’s field of view and assumes indirect flash radiance is purely diffuse.Source: D. Deferred Shading under Static and… · pp. 5–6
  • The flash residual is the captured flash radiance minus rendered non-flash radiance, and estimates the flash-induced contribution.Sampled rays intersect the reconstructed mesh and project onto flash images to retrieve this residual.Source: D. Deferred Shading under Static and… · p. 6

E. Optimization

Optimization combines image losses with priors and decomposition losses to supervise geometry and materials across flash and non-flash images. Training first uses non-flash images, then adds flash images to the batches.

E. Optimization

  • Training proceeds in two stages, adding flash images after an initial non-flash-only optimization stage.The later batches contain one non-flash and one flash image.Source: E. Optimization · p. 6
  • The total loss combines rendering and prior terms with diffuse, specular, semantic, and 2DGS regularization losses.The loss is a weighted sum, with λ denoting each term’s weight.Source: E. Optimization · p. 6
  • The rendering loss compares captured pixel colors with rendered colors for non-flash or flash images.The indicator m specifies whether the image contains flash lighting.Source: E. Optimization · p. 6
  • Normal priors from MVInverse are downweighted near normal discontinuities, while albedo and its gradients receive additional prior supervision.The edge-derived weight map reduces normal regularization near geometric boundaries; the albedo prior addresses ill-posed albedo and shadow optimization.Source: E. Optimization · p. 6
  • The diffuse decomposition loss matches rendered diffuse color to the GS-anchored diffuse field and decomposes it into material and shading maps.The decomposition uses an L1 loss over metallic, albedo, diffuse irradiance, and shadow maps.Source: E. Optimization · p. 6
  • The specular residual identifies low-roughness regions for regularization, and the method also uses a semantic loss and 2DGS regularizers.The residual is formed from the captured image and the sum of non-flash color and flash diffuse shading.Source: E. Optimization · p. 6

IV. IMPLEMENTATION DETAILS

Implementation uses a two-stage, 25,000-iteration training schedule and fixed capture settings for real scenes. Flash position and direction are assumed to coincide with the camera center and optical axis.

IV. IMPLEMENTATION DETAILS

  • Each scene is trained for 25,000 iterations, with static illumination used for the first 18,000 and flash enabled for the remaining 7,000.During the flash stage, each batch pairs a non-flash image with the nearest flash image by camera pose.Source: §IV · p. 7
  • The GS-anchored diffuse field uses an Instant-NGP network, while Adam optimizes both 2DGS and material parameters.The albedo-gradient prior weight is 2.0 for synthetic scenes and 10.0 for real scenes to compensate for greater capture noise and reconstruction uncertainty.Source: §IV · p. 7
  • Real-scene captures keep exposure, white balance, and flash power fixed, and rendered linear RGB is converted to sRGB before photometric loss.The captures use fixed 5500K white balance; camera response is not explicitly estimated and saturated pixels are not masked.Source: §IV · p. 7
  • Flash is assumed collocated with the camera, with its centerline aligned to the optical axis; robustness to violations is evaluated in ablation.The camera-to-flash extrinsic transformation is not explicitly calibrated.Source: §IV · p. 7

V. EXPERIMENTS

Experiments use five synthetic and three real indoor scenes with independently captured flash and non-flash views. Material evaluation is quantitative on synthetic scenes with ground-truth maps, while real-scene assessment is mainly qualitative.

V. EXPERIMENTS

  • The evaluation covers five Blender-rendered synthetic scenes and three real indoor scenes captured with a flash-equipped camera.Synthetic scenes have 100–150 flash and non-flash images each; real scenes have 80–100 of each.Source: §V · p. 7
  • Synthetic material predictions are compared with ground-truth maps, while real-world material quality is assessed qualitatively.Image reconstruction uses PSNR, SSIM, and LPIPS; relighting uses five sample images under five point lights and an environment map.Source: §V · p. 7
  • WildLight is included as a baseline because it also uses active illumination, and Gaussian reflective reconstruction methods are evaluated with segmentation support.Ref-Gaussian and MaterialRefGS receive ground-truth masks and segmentation loss; Ref-Gaussian also receives the normal prior used by the proposed method.Source: §V · pp. 7–8
  • FIPT and IRIS receive ground-truth meshes and masks, with about 50 extra views added to meet their input and coverage needs.FIPT additionally requires HDR input; MVInverse is included as a feed-forward multiview baseline, and SAM2 masks are tested as an alternative to ground-truth segmentation.Source: §V · p. 8

A. Comparisons

On synthetic scenes, the method reports improved material decomposition and relighting against the evaluated baselines. Qualitative real-scene comparisons show plausible material estimates and effective relighting, while generated segmentation masks preserve most reported advantages.

A. Comparisons

  • The method reports more accurate diffuse color, albedo, and roughness, and significantly better relighting quality than all other approaches on the synthetic dataset.The quantitative comparison is presented in Table I; the authors also report competitive runtime efficiency.Source: A. Comparisons · pp. 8–9
  • With SAM2-generated masks, the method still outperforms other methods in diffuse color, albedo, and relighting, but its roughness is slightly worse than IRIS.IRIS uses ground-truth segmentation masks in this comparison.Source: A. Comparisons · p. 9
  • Synthetic qualitative comparisons show albedo maps with less shading or specular contamination and relighting highlights closer to ground truth in distribution and intensity.The examples include cabinet colors in the Kitchen and Market scenes.Source: A. Comparisons · p. 9, Figure 5 · p. 9
  • WildLight’s albedo and roughness are less accurate under sparse views, while FIPT and IRIS roughness is less accurate than in their original reports.The authors suggest limited input-view coverage as a possible explanation for the FIPT and IRIS difference.Source: A. Comparisons · p. 9
  • Real-scene examples show recovered diffuse and specular materials, coherent roughness, and effective relighting despite the absence of ground-truth material annotations.Reported examples include chair and floor materials in Coffee Bar, podium-floor albedo in Classroom, and plastic identification in Car.Source: A. Comparisons · pp. 9–11, Figure 6 · p. 10

B. Ablations

The ablations show that flash input improves albedo and roughness estimation, while other representation and loss choices affect material estimates. Results also test multiple environment maps and a camera–flash offset.

B. Ablations

  • Flashlight images significantly improve albedo and roughness estimation, although removing them raises diffuse-color PSNR by 0.07 dB.The authors suggest flash primarily adds material-estimation cues rather than diffuse-color reconstruction cues.Source: B. Ablations · p. 11
  • Using multiple environment maps improves material estimation, and the method remains robust with a 0.5 camera–flash offset simulating real-world capture.Source: B. Ablations · p. 11
  • Replacing explicit albedo and roughness fields with NGP-based fields decreases albedo PSNR and worsens roughness estimation.The authors speculate that NGP fields optimize fine-scale details less effectively than the explicit 2DGS representation.Source: B. Ablations · pp. 11–12
  • The normal prior supports normal quality and material estimation; albedo loss improves albedo, specular decomposition loss substantially improves roughness, and semantic loss adds a smaller consistent gain.Source: B. Ablations · p. 12

VI. LIMITATIONS AND FUTURE WORK

The method has reduced sensitivity to subtle roughness changes relative to one-light-at-a-time methods, and overexposure can weaken material constraints.

VI. LIMITATIONS AND FUTURE WORK

  • Compared with one-light-at-a-time methods, the approach is not sufficiently sensitive to subtle roughness changes.Source: §VI · p. 12
  • Regions overexposed even without flash illumination may lose constraints and produce noisy material estimates.Source: §VI · p. 12
  • The authors plan to explore RAW image formats and camera response function modeling to better handle overexposed regions.Source: §VI · p. 12

VII. CONCLUSION

The paper presents an inverse-rendering framework for casual indoor capture that combines flash and non-flash cues with physically based rendering. Across synthetic and real-world scenes, it reports better material decomposition and relighting than recent baselines.

VII. CONCLUSION

  • The framework targets casual indoor scenes where complex structure, varied reflectance, occlusions, and indirect light transport challenge material and geometry estimation.Source: §VII · p. 12
  • Combining flash and non-flash images provides complementary physical cues for material decomposition, integrated with physically based static and flash rendering and a view-consistent diffuse representation.Source: §VII · p. 12
  • Across synthetic and real-world scenes, the approach improves material decomposition and relighting quality over recent baselines.Source: §VII · p. 12

Related papers

Loading 2610.06035v1…