PaperlayerView exact revision on arXiv

Source-linked AI summary

MatSpray: Fusing 2D Material World Knowledge on 3D Geometry

Philipp Langsteiner, Jan-Niklas Dihlmann, Hendrik P. A. Lensch

arXiv:2512.18314v1cs.CVcs.GR

TL;DR

Neural scene reconstruction can leave materials unsuitable for relighting, while 2D material predictions are not directly consistent across views or attached to 3D. MatSpray lifts diffusion predictions onto reconstructed Gaussian geometry and merges them across views. The authors report improved material and relighting quality, particularly for specular objects, while noting dependence on the chosen diffusion model.

  • Problem

    Neural reconstruction can entangle illumination with appearance, and 2D material predictions are not directly consistent across views or attached to 3D representations.

    Source: §1 · p. 2

  • Method

    MatSpray projects diffusion-predicted material maps onto 3D Gaussians by ray tracing, refines per-Gaussian estimates with Neural Merger, and supervises rendered maps and PBR images.

    Source: §1 · p. 2, §3.4 · pp. 4–5

  • Results

    MatSpray improves multi-view consistency and material-map and relit-appearance quality, with greater gains for highly specular objects.

    Source: §4.2 · p. 7, §5 · p. 8

  • Takeaways & Limitations

    MatSpray produces relightable 3D assets with spatially varying materials by combining pretrained 2D diffusion estimators with 3D Gaussian geometry.

    Source: §1 · p. 2, §5 · p. 8

    Material quality remains dependent on the chosen diffusion model, although the PBR-to-image loss partially corrects small prediction deviations.

    Source: §5 · p. 8

Abstract

from arXiv · show

Manual modeling of material parameters and 3D geometry is a time consuming yet essential task in the gaming and film industries. While recent advances in 3D reconstruction have enabled accurate approximations of scene geometry and appearance, these methods often fall short in relighting scenarios due to the lack of precise, spatially varying material parameters. At the same time, diffusion models operating on 2D images have shown strong performance in predicting physically based rendering (PBR) properties such as albedo, roughness, and metallicity. However, transferring these 2D material maps onto reconstructed 3D geometry remains a significant challenge. We propose a framework for fusing 2D material data into 3D geometry using a combination of novel learning-based and projection-based approaches. We begin by reconstructing scene geometry via Gaussian Splatting. From the input images, a diffusion model generates 2D maps for albedo, roughness, and metallic parameters. Any existing diffusion model that can convert images or videos to PBR materials can be applied. The predictions are further integrated into the 3D representation either by optimizing an image-based loss or by directly projecting the material parameters onto the Gaussians using Gaussian ray tracing. To enhance fine-scale accuracy and multi-view consistency, we further introduce a light-weight neural refinement step (Neural Merger), which takes ray-traced material features as input and produces detailed adjustments. Our results demonstrate that the proposed methods outperform existing techniques in both quantitative metrics and perceived visual realism. This enables more accurate, relightable, and photorealistic renderings from reconstructed scenes, significantly improving the realism and efficiency of asset creation workflows in content production pipelines.

1. Introduction

The introduction identifies a gap between plausible 2D material predictions and 3D representations that are consistent across views and suitable for relighting. MatSpray transfers those predictions to 3D Gaussians and refines them for consistency and relightable material estimates.

1. Introduction

  • Neural reconstruction can entangle illumination with appearance, while classical inverse rendering relies on strong lighting and exposure assumptions and is fragile with spatially varying materials.2D material predictors offer learned priors but are not directly consistent across views or attached to a 3D representation.Source: §1 · p. 2
  • MatSpray fuses swappable diffusion-based 2D PBR priors with 3D Gaussian material optimization through ray-traced assignment and PBR-consistent supervision.It targets spatially varying base color, roughness, and metallic parameters, and reports 3.5× less per-scene optimization time than IRGS [9].Source: §1 · p. 2
  • Neural Merger aggregates per-Gaussian, multi-view estimates to improve cross-view consistency, suppress baked-in lighting, and stabilize joint environment-map optimization.The introduction describes it as a softmax neural merger.Source: §1 · p. 2

2. Related Work

Related work spans spatially varying material models, 2D diffusion-based material prediction, Gaussian ray tracing, and neural scene reconstruction. MatSpray combines these directions by lifting diffusion material estimates into 3D Gaussian representations rather than optimizing material and geometry parameters per scene.

2. Related Work

  • Earlier spatially varying BRDF work established high-resolution texel-based material parameterization, while later work explored richer parameterizations and more expressive models.This work adopts a Cook–Torrance variant based on the Disney principled BRDF and real-time engine formulations.Source: §2 · p. 2
  • Diffusion models have been applied to infer PBR maps from 2D images, with DiffusionRenderer [16] particularly motivating this work.Related diffusion research also covers HDR prediction, texture estimation, and relighting [30].Source: §2 · p. 2
  • Gaussian ray-tracing methods provide a mechanism for transferring 2D material data to 3D, and MatSpray extends prior formulations toward material-aware reconstruction.The formulation builds on Mai et al. [34] and Moenne-Loccoz et al. [39].Source: §2 · p. 2
  • Unlike R3DGS and IRGS, which optimize scene parameters per scene, MatSpray lifts diffusion material estimates into 3D Gaussians for faster, more consistent reconstruction.Its pipeline projects the 2D materials by ray tracing and refines them with Neural Merger; the authors describe broader material reasoning across scenes.Source: §2 · pp. 2–3, Figure 2 · p. 3

3. Method

MatSpray combines per-view diffusion material maps with reconstructed 3D Gaussians, then lifts and merges the estimates across views. Two complementary losses refine the material maps and rendered appearance while preserving 2D priors and consistency.

3. Method

  • The method combines per-view base color, roughness, and metallic predictions with Gaussian Splatting geometry and normals, then jointly refines materials and an optimizable environment map.It transfers estimates to 3D, applies Neural Merger for multi-view consistency, and uses deferred-rendering loss for further refinement.Source: §3 · p. 3

3.1. Diffusion Material Prediction

  • The method accepts per-view material maps from diffusion predictors; experiments select DiffusionRenderer [27], which achieves about thirty percent higher PSNR than the other tested methods.Predicted maps can still vary across views or batches, making direct projection unreliable and often producing blurry, washed-out maps.Source: §3.1 · p. 3

3.2. 2D-to-3D Material Lifting

  • Gaussian ray tracing assigns pixel material values to intersected Gaussians, and per-view estimates are aggregated by their median to reduce distortion from outliers and overlapping footprints.Gaussians not intersected in any view are removed; grid-based supersampling is used to ensure stable hits.Source: §3.2 · pp. 3–4

3.3. Neural Merger

  • For each Gaussian, Neural Merger uses projected per-view material values and encoded position to predict view weights and fuse estimates by weighted interpolation.Softmax weighting constrains outputs to interpolation among predictions, supporting plausible estimates and reliable environment-map convergence; separate mergers are used for each material channel.Source: §3.3 · p. 4

3.4. Refinement, Supervision and Loss Functions

  • Refinement uses an L1 loss between rendered and diffusion-predicted material maps, plus deferred-shaded PBR rendering loss against ground-truth images.The material-map loss optimizes only Neural Merger, while rendering loss supervises both the merger and environment-map estimation. Figure 3 compares relighting under shared environment maps; reconstructed geometry may partially occlude the environment map in IRGS.Source: §3.4 · pp. 4–5, Figure 3 · p. 5

4. Experiments

The experiments compare the method with extended R3DGS and IRGS on synthetic and real-world scenes, assessing material maps, relighting, runtime, and ablations. The reported results favor the method in relighting and basecolor quality, with the Neural Merger central to the ablation gains.

4.1. Implementation Details

  • Evaluation covers 17 synthetic objects with 100 training and 200 evaluation images per object, plus real-world objects averaging 27 images; metrics compare material maps and relit renderings against ground truth.Real-world scenes use COLMAP reconstruction for initialization and camera inference; highly specular objects are trained with DiffusionRenderer normals as RGB to guide geometry.Source: §4.1 · p. 5, §4 · p. 5

4.3. Quantitative Results

  • Across 17 synthetic objects, our method consistently outperforms all baselines in relighting quality and basecolor estimation.The comparison includes IRGS and an extended R3DGS that supports metallic materials.Source: Table 1 · p. 7, §4.3 · p. 7

4.2. Qualitative Results

  • On real-world scenes, our method gives the most consistent relighting; R3DGS appears too bright and shiny, while IRGS produces too-dark colors despite smoother geometry.In the broader relighting comparison, our results more closely resemble ground truth, particularly for specular objects.Source: §4.2 · pp. 5, 7
  • Our material maps reduce baked-in lighting and view-dependent inconsistencies; the method also improves normals over extended R3DGS and preserves fine detail on metallic and specular surfaces.IRGS can have normals closer to ground truth in some regions, but exhibits floaters and loses fine detail; it cannot predict metallic materials.Source: §4.2 · p. 7

4.3. Quantitative Results

  • For roughness, IRGS scores higher PSNR (16.182 vs. 15.331), while our method scores better SSIM (0.820 vs. 0.744) and LPIPS (0.181 vs. 0.192).For metallic maps, the method produces stable non-metallic predictions; infinite-PSNR objects are excluded from the calculation.Source: §4.3 · p. 7

4.4. Computation Time

  • Our method takes 1,488 seconds (about 25 minutes) on average, approximately 3.5× faster than IRGS at 5,347 seconds (about 89 minutes).Timings are reported on the Navi dataset; material optimization averages 975 seconds and can reach 3,631 seconds for complex objects.Source: Table 2 · p. 7, §4.4 · p. 8

4.5. Ablation Study

  • The full model achieves PSNR 29.164, SSIM 0.9105, and LPIPS 0.0626, outperforming the ablated variants; the Neural Merger enforces multi-view consistency while preserving diffusion-predicted material properties.The Supervised variant performs worse than Proj. Average, which directly projects and averages predictions without further optimization.Source: Table 3 · p. 8, §4.5 · p. 8

5. Conclusion

MatSpray produces relightable 3D assets with spatially varying materials, while its material quality depends on the diffusion model and its performance can be constrained by reconstruction geometry.

5. Conclusion

  • MatSpray reconstructs relightable 3D assets with spatially varying materials and reports stronger results for highly specular objects and metallic maps across synthetic and real-world inputs.The approach uses pretrained 2D diffusion material estimators without additional expensive training on large-scale 3D PBR datasets.Source: §5 · p. 8
  • The Neural Merger improves multi-view consistency, including in comparison with video-based material predictors that still struggle with consistency.Source: §5 · p. 8
  • Material quality remains dependent on the chosen diffusion model, although the PBR-to-image loss partially corrects small prediction deviations.Source: §5 · p. 8
  • Inconsistent geometry or normals from R3DGS can challenge the method, and very small or flat Gaussians may be missed during ray tracing.Photometric loss may partially mitigate inconsistent geometry and normals; future work could address missed Gaussians with a projection-transformer combination assignment similar to [41].Source: §5 · p. 8
  • The geometry-material association could support accurate 3D object-part segmentation, potentially paired with language features for object-specific reconstruction constraints or natural manipulation interfaces.Source: §5 · p. 9

Overview

The supplementary material collects extended results, analyses, and implementation details, organized into four components.

Overview

  • Additional videos and real-world object reconstructions compare the method with earlier approaches.Source: Overview · p. 12
  • The Neural Merger ablation examines the final Softmax layer using qualitative and quantitative evidence.Source: Overview · p. 12
  • The tone-mapping analysis discusses DiffusionRenderer’s effects on predicted material maps and their mismatch with linear ground truth.Source: Overview · p. 12
  • Implementation details cover training, supersampling, Neural Merger inputs, and practical considerations for stable optimization.Source: Overview · p. 12

A. Additional Videos and Real-World Objects

The supplementary videos and real-object examples compare MatSpray with other methods across viewpoints, material predictions, and relighting conditions.

A. Additional Videos and Real-World Objects

  • The supplementary video collection compares MatSpray with Extended R3DGS [9], IRGS [11], and DiffusionRenderer’s forward renderer [27] to provide a broader visual view of consistency and stability.Source: A. Additional Videos and Real-World… · p. 12, Figure 7 · p. 12
  • Across the videos, MatSpray maintains stable reconstructions across viewpoints, while other methods can flicker or structurally collapse, especially on complex or specular objects.Extended R3DGS may lose surface smoothness, while IRGS may oversmooth surfaces and bake metallic reflections into base color.Source: A. Additional Videos and Real-World… · p. 12
  • DiffusionRenderer’s material maps vary with camera angle and lighting, whereas MatSpray aligns predictions across views more reliably.The examples include three relighting videos and three material-prediction videos.Source: A. Additional Videos and Real-World… · p. 12
  • On additional real objects, MatSpray’s base color is locally sharp and coherent, while Extended R3DGS and IRGS show noise, distortions, or view-dependent artifacts.Under relighting, the baselines can retain baked-in lighting effects or appear washed out.Source: A. Additional Videos and Real-World… · p. 12

B. Neural Merger Ablation

The ablation shows that the Neural Merger’s Softmax normalization is important for consistent material maps and relighting.

B. Neural Merger Ablation

  • Softmax normalizes the Neural Merger’s outputs into weights for averaging inputs, helping keep Gaussian material parameters stable across viewpoints.The layer has a sizable impact on final reconstruction quality despite being a small architectural detail.Source: B. Neural Merger Ablation · pp. 12, 14
  • Without Softmax, the merger can absorb lighting and shadows into material maps, producing view-varying materials and unrealistic relighting.The comparison also reports shadows in base color and blur in high-detail regions, including the birdhouse example.Source: Figure 9 · p. 13, B. Neural Merger Ablation · p. 14
  • The full model outperforms the version without Softmax across metrics, with especially large gains in LPIPS.The passage reports results across all dataset scenes but provides no metric values here.Source: B. Neural Merger Ablation · p. 14

C. DiffusionRenderer Tone Mapping Analysis

DiffusionRenderer’s predicted base color was darker than the linear ground truth in nearly all tested objects, which the authors associate with tone-mapped training supervision. Tone mapping affects material channels differently and cannot be fully reversed with approximate inverse curves.

C. DiffusionRenderer Tone Mapping Analysis

  • DiffusionRenderer’s predicted base color appeared darker than the linear ground truth in almost all tested objects, suggesting a systemic issue.The authors attribute this discoloration to DiffusionRenderer being supervised with tone-mapped ground-truth images.Source: C. DiffusionRenderer Tone Mapping… · p. 14
  • Removing the Softmax layer causes the network to encode lighting and degrades all metrics in the all-object ablation.Source: Table 4 · p. 14
  • Tone mapping compresses display-range values and complicates recovery of physically meaningful material parameters; it is not analytically invertible, and approximate inverses introduce errors near shadows and highlights.Source: C. DiffusionRenderer Tone Mapping… · pp. 14–15
  • Base color is partially recovered by an inverse gamma of roughly 1.8, but roughness collapses more severely under tone mapping while metallic values remain closer to zero or one.The authors connect these channel-specific effects to differences between predicted maps and linear ground truth.Source: C. DiffusionRenderer Tone Mapping… · p. 15

D. Implementation Details

Implementation choices include matching DiffusionRenderer’s 512×512 resolution, using dense ray sampling for material projection, and refining projected materials with a Neural Merger. Geometry learning is disabled for scenes with strong specular highlights because it can destabilize the representation.

D. Implementation Details

  • Scenes with strong specular highlights keep Gaussian positions fixed because additional geometry optimization tends to destabilize the representation.Training views are rendered at 512×512 to match DiffusionRenderer’s internal resolution.Source: D. Implementation Details · p. 15
  • The Neural Merger uses a 0.001 learning rate and L1 material supervision with weight 1.0; random view sampling avoids viewpoint bias.The stated loss balance prevents shadow overfitting while maintaining material-map fidelity.Source: D. Implementation Details · p. 15
  • A 16×16 grid of rays per pixel reduces missed Gaussians during material projection; lower sampling can produce holes and unstable geometry.Figure 11 illustrates reconstruction defects at lower sampling rates.Source: D. Implementation Details · p. 15, Figure 11 · p. 15
  • The Neural Merger receives Gaussian positional encodings and projected base color, roughness, and metallic values to balance local detail with global consistency.Figure 12 shows these inputs.Source: D. Implementation Details · p. 15, Figure 12 · p. 15
Loading 2512.18314v1…