GraftRay: Physically Based Mesh Insertion into Gaussian Splatting Scenes via Path Tracing

1. Introduction

• Gaussian Splatting supports high-quality real-time scene rendering, but inserted objects must also match scene illumination, cast consistent shadows, and contribute indirect light.These effects require explicit light transport without sacrificing the original reconstruction’s visual fidelity.Source: §1 · p. 1
• Environment maps omit spatial emitter positions and interactions with scene geometry, while image-space harmonization does not explicitly model scene-level light transport.Joint inverse rendering is physically grounded but introduces additional unknowns, and LDR observations weaken constraints for distinguishing emissive sources from reflective surfaces.Source: §1 · pp. 1–2
• GraftRay estimates Gaussian-level materials and normals from rendered views, then uses them with explicit area emitters and path tracing for physically consistent insertion.The pipeline avoids joint inverse rendering and uses differential rendering to transfer object-induced shadows and indirect illumination while preserving the background reconstruction.Source: §1 · p. 2

Lighting estimation for object insertion

• Unlike methods centered on environment maps or distributed emissive radiance, GraftRay parameterizes local area emitters so their extent and scene occlusion affect illumination and shadows.Related approaches include learned HDR environment maps, spatially varying lighting, lighting volumes, and editable illumination components.Source: §2 · p. 2

Compositing and harmonization in 3D

• Generative harmonization methods target perceptual consistency or scene editing, whereas GraftRay explicitly models light transport for view-consistent shadows and indirect illumination.The cited approaches include multi-view lighting and shadows, diffusion-based harmonization, object composition, text-specified insertion, and direct 3DGS editing.Source: §2 · p. 2

Inverse rendering on Gaussian splats

• Gaussian inverse-rendering methods recover materials or global illumination, while GraftRay fixes geometry and prior-initialized materials and optimizes emitter radiances.This constrained fitting reduces coupled unknowns and supports explicit treatment of saturated LDR observations.Source: §2 · p. 3

Ray tracing Gaussian primitives

• Gaussian ray-tracing work provides approaches for tracing primitives, and GraftRay extends an OptiX path tracer to estimate physical illumination with analytic area emitters and stochastic transparency.Figure 2 summarizes its workflow from reconstructed scenes and lifted materials to emitter-radiance fitting and differential compositing.Source: §2 · p. 3, Figure 2 · p. 3

3. Method

• GraftRay estimates per-Gaussian materials and normals, fits user-marked emitter regions as analytic rectangles, and path-traces inserted objects with the scene.Its pipeline then transfers object-induced lighting changes to the original 3DGS appearance through differential compositing.Source: §3 · p. 3

3.1. Material and Normal Estimation

• Materials and normals come from diffusion predictions across virtual views, filtered by visible-surface depth and aggregated across views and random seeds.The resulting attributes remain fixed during illumination estimation, avoiding joint lighting-and-material optimization.Source: §3.1 · pp. 3–4

3.2. Emitter Initialization

• Users mark emitter regions in one or a few views; the method back-projects them, fits one-sided analytic rectangles, and refines candidates during radiance estimation.Plane fitting is weighted toward bright Gaussians, and candidates with little contribution to observed illumination can be suppressed.Source: §3.2 · p. 4

3.3. Path Tracing on Gaussian Scenes

• Gaussian ray intersections use 3σ bounds and stochastic acceptance, whose first-hit probability matches the alpha-compositing weight without sorting all intersections.A separate shadow-ray program accumulates transmittance deterministically to support smooth soft shadows.Source: §3.3 · p. 4

3.4. Light Transport

• The renderer combines direct illumination from analytic rectangles with BSDF-sampled paths for indirect illumination, using multiple importance sampling.It uses a Cook–Torrance BRDF with a GGX microfacet model.Source: §3.4 · p. 5

3.5. Emitter Radiance Estimation

• Emitter radiances are fitted from unit-radiance basis renders, with clipped image channels imposing lower-bound constraints rather than equality targets.Unsaturated room surfaces constrain candidate radiance, allowing candidates that explain little room illumination to receive near-zero radiance.Source: §3.5 · p. 5

3.6. Differential Compositing

• Differential compositing modulates the original 3DGS rendering by the path-traced radiance ratio, transferring cast-shadow attenuation and bounce-light increases while retaining scene appearance details.The ratio is exposure-adjusted and capped; on synthetic scenes, the method reproduces object brightness and contact shadows that baselines miss or misjudge.Source: §3.6 · pp. 5–6, Figure 3 · p. 6

4. Experiments

• Evaluation spans synthetic and real scenes plus 3DGS scenes, with object-focused and shadow metrics used when ground truth is available.Objects, placements, and lights are held identical across methods; each scene has three views.Source: §4 · pp. 6–7

4.1. Comparison

• GraftRay achieves the best overall performance on FIPT and NFINITE, particularly in shadow reconstruction.On synthetic scenes, it reproduces object brightness and contact shadows while baselines miss shadows or misjudge brightness.Source: §4.1 · pp. 7–8
• Recovered albedo scores 19.9 dB for GraftRay and 20.2 dB for EAG-PT, while GS-IR and GI-GS bake shading and window light into albedo.GraftRay also recovers roughness and metallic, although metallic spills onto diffuse surfaces.Source: Figure 5 · p. 7, §4.1 · p. 8

4.2. Ablation

• Point lights and environment maps lose 8–9 dB ΔPSNR and a third of shadow IoU; direct-only lighting yields 25.7 dB and overly dark objects and shadows.Point lights ignore a window’s extent, while environment maps miss its falloff.Source: §4.2 · p. 8
• Without object-to-room bounce, the object remains plausible but shadow IoU falls to 0.58 or 0; gray albedo halves mask PSNR to 15.6 dB.Both fitted lights and shading depend on albedo.Source: §4.2 · p. 8
• Using a full path-traced frame instead of differential compositing drops ΔPSNR to 24.8 dB; emitter size and shifts cost at most 1.3 dB while shadow IoU stays 0.80.The re-rendered room can inherit reconstruction errors, such as a missing mirror reflection, that differential compositing cancels.Source: §4.2 · p. 8

4.3. Limitations

• Emitters must be marked manually, and errors in estimated albedo, roughness, metallic, or normals carry over to fitted lights and insertion.The method uses 2D estimators for these priors.Source: §4.3 · p. 8

• GraftRay recovers emitter geometry, orientation, and absolute radiance from ordinary LDR multi-view captures by treating saturated pixels as censored observations.Source: §5 · p. 8
• A stochastic-transparency path tracer over the Gaussians gives inserted objects spatially varying shading, contact shadows, and bounce light without degrading the reconstruction.Source: §5 · p. 8
• With geometry and materials fixed, emitter radiance is estimated through a low-dimensional fit over path-traced basis images, and the paper gives a protocol to isolate insertion effects from full-image metrics.Source: §5 · p. 8