Source-linked AI summary
LightFuse: Relightable Interactive Gaussian Scene Reconstruction via Multi-Scan Fusion and 2D Gaussian Ray Tracing
Haonan Zhou, Gaoxiang Linghu, Youlin Jia, Hongyu Cui, Kewei Wei, Kaiyue Zhou, Bruce X. B. Yu, Gaoang Wang
TL;DR
Relightable interactive reconstruction must avoid baked lighting and fixed-scene assumptions so edited layouts retain consistent appearance under new illumination. LightFuse fuses multi-state scans, refines geometry for ray tracing, and separates shared material from state-specific illumination. Across synthetic scenes, it reports 25.38 dB PSNR and 0.852 SSIM, exceeding ReCap (Adapted) by +9.74 dB PSNR and +0.121 SSIM on average.
Problem
Existing methods either bake material and lighting into appearance or assume fixed geometry, preventing consistent relighting of rearranged layouts.
Method
LightFuse fuses multi-state scans into shared background and movable objects, refines geometry for ray tracing, and uses staged inverse rendering to separate shared material from state-specific illumination.
Results
25.38 dB PSNR and 0.852 SSIM, outperforming ReCap (Adapted) by +9.74 dB PSNR and +0.121 SSIM on average.
Takeaways & Limitations
The resulting scene supports object rearrangement, material editing, and relighting with updated visibility and light transport.
Takeaways & Limitations
Geometry decomposition depends on multi-view masks and GaussianCut, while one-bounce transport does not faithfully recover long-path global illumination, caustics, transmission, or sharp mirror chains.
Abstract
from arXiv · showhide
Relightable interactive scene reconstruction aims to build an editable 3D model from scans of different object arrangements and render new layouts under novel illumination. Existing methods either bake lighting into appearance or recover material and illumination only for fixed scenes, leaving edited layouts with inconsistent shadows and indirect lighting. We present LightFuse, a 2D Gaussian framework that extends interactive scene reconstruction with explicit material-illumination decomposition and physically based relighting. LightFuse first fuses observations across states to reconstruct a shared background and movable objects. It then conducts ray-tracing-oriented geometry refinement to produce more complete and consistent surfaces. On the refined geometry, staged training with differentiable one-bounce ray tracing separates shared metallic--roughness material from state-specific environment lighting. The resulting scene supports object rearrangement, material editing, and relighting, while ray tracing recomputes appearance after each interaction. Experiments across synthetic scenes demonstrate state-of-the-art relighting quality, outperforming the strongest baseline by +9.74\,dB PSNR and +0.121 SSIM on average. Project page: https://zhn202.github.io/LightFuse/
Introduction
LightFuse addresses the challenge of relightable interactive reconstruction by separating shared scene properties from state-specific illumination. It combines multi-scan fusion, geometry refinement, and staged inverse rendering to support edited layouts under changing illumination.
- Interactive reconstruction builds editable 3D models from scans with rearranged objects, enabling new layouts for embodied simulation, robotic learning, virtual staging, and mixed reality.
- Existing methods bake material and lighting into appearance, leaving shadows, highlights, and indirect lighting attached to objects after rearrangement.
- Relightable interactive reconstruction requires completed movable components, material–illumination separation, and recomputed shadows and indirect lighting after rearrangement.
- LightFuse shares intrinsic geometry and material across states while modeling illumination separately for each state.
- LightFuse refines geometry with coverage, scale, depth, and normal regularization, then uses staged inverse rendering for physically consistent relighting.
- +9.74 dB PSNR and +0.121 SSIM over the strongest baseline are reported on average across synthetic scenes.
Related Work
Prior work addresses object manipulation, multi-scan fusion, inverse rendering, and cross-condition appearance recovery separately. LightFuse targets their combination for discrete multi-state scans with shared materials and state-specific illumination.
- Gaussian editing methods support object selection, appearance editing, and geometric manipulation but do not fuse scene rearrangements.
- Multi-scan methods recover manipulable geometry by fusing states, yet do not recover shared material and state-specific illumination for consistent shadows and inter-reflections.
- Inverse-rendering methods decompose appearance into geometry, reflectance, and illumination, using neural fields or explicit Gaussian primitives with physically based rendering.
- ReCap shares Gaussian material across environments while jointly fitting illumination, and GauUpdate recovers shared material under distinct lighting for object insertion.
- Other methods address temporal dynamics or object–scene composition, whereas LightFuse focuses on fusing discrete scene states for interactive relighting.
Method
LightFuse constructs a componentized 2D Gaussian scene from multi-state scans, representing shared background and movable objects before ray-tracing-oriented refinement and appearance optimization.
- LightFuse fuses observations across states into a shared background and movable objects, then refines geometry and separates shared materials from state-specific illumination.
- Given K scans with different arrangements, the method constructs a 2D Gaussian representation decomposed into a shared background and M movable objects.
- Coverage and scale regularization constrain reconstructed geometry before subsequent ray-tracing-oriented geometry optimization.
- Each scene component is composed of oriented 2D Gaussian surfels representing the shared background or movable objects.
- Each surfel stores center, orientation, in-plane scale, opacity, reconstruction features, base color, metallic, and roughness attributes.
- Object transforms act only on surfel centers and orientations, allowing movable components to be rearranged across scene states.
- Multi-view masks and GaussianCut decompose a reference reconstruction, while rearrangements expose surfaces occluded in other states for completion.
1 Interactive Scene Reconstruction
The supplied passage identifies the ray-tracing-oriented geometry refinement stage but provides no substantive description of its procedure or results.
- Ray-Tracing-Oriented Geometry Refinement is presented as a distinct method stage.
4 Novel-State Relighting
LightFuse refines a shared multi-state Gaussian scene for ray tracing, then uses staged inverse rendering to separate shared material from state-specific illumination. The resulting representation supports physically consistent editing and relighting by recomputing visibility and light transport.
- Cross-state geometry refinement: Coverage and scale regularization improve ray-tracing geometry by discouraging opacity holes, oversized surfels, and false occlusions.The opacity term encourages complete coverage, while the scale term limits surfel extent during reconstruction and refinement.
- Ray-tracing-oriented rendering: 2D Gaussian ray tracing intersects oriented surfel planes, filters unsupported intersections, sorts hits by depth, and composites attributes front to back.The composited quantities can represent position, depth, normals, or material attributes for subsequent shading.
- Cross-state geometry refinement: Multi-state supervision jointly refines shared background and movable-object surfels, using depth and normal constraints to exploit complementary surfaces across arrangements.Each state is assembled with object transforms and rendered from observed views; residuals update shared surfel parameters.
- Multi-state inverse rendering: Staged inverse rendering optimizes shared material and independently optimized state-specific illumination over refined geometry using differentiable one-bounce path tracing.The rendering equation uses a metallic–roughness BRDF with shared base color, metallic, and roughness parameters; the estimator models direct illumination and first-order indirect transport.
- Scene editing and relighting: LightFuse supports object rearrangement, base-color and metallic–roughness edits, and relighting under recovered or user-provided HDR environments with updated visibility and light transport.After layout changes, retracing improves occlusions and cast shadows while allowing highlights, reflections, and indirect illumination to respond to the new configuration.
Experiments
Experiments evaluate LightFuse on synthetic and real-world novel-state relighting, including changes in object layout and illumination, with quantitative comparisons and ablations. LightFuse achieves strong relighting quality, while geometry and transport components each address distinct reconstruction or rendering errors.
- Experimental Setup: The evaluation uses synthetic and real-world datasets, with synthetic scenes providing held-out layouts under eleven unseen environment maps.The synthetic benchmark contains four scenes, three training states, and one held-out state; real-world scenes include Captured 1, Captured 2, Scene 000, and Scene 001.
- Experimental Setup: PSNR and SSIM measure pixel-wise color and intensity fidelity and local structural consistency against ground-truth images.
- Qualitative Relighting: LightFuse improves occlusions, cast shadows, reflections, indirect illumination, and low-texture appearance after layout or illumination changes.Light retracing reevaluates visibility on recomposed geometry, while multi-state refinement supplies more complete object and background geometry.
- Real-World Synthesis: 24.20 dB PSNR on Captured 1 and Captured 2 exceeds ReCap (Adapted) by +1.40 dB under joint layout and illumination changes.On IGFuse scenes, fixed illumination limits the benefit of multiple illumination conditions for disentangling material and lighting.
- Ablations: Removing any geometry-refinement regularizer decreases PSNR and produces a distinct artifact, while removing the transport estimator causes the largest quality degradation.The artifacts include light leakage, false occlusions, low-texture surface errors, and non-smooth surfaces; progressive 8-to-16-to-32 spp achieves the best performance.
Limitations
LightFuse remains sensitive to segmentation quality and computationally expensive transport-aware rendering. Its one-bounce formulation does not faithfully recover several long-path or complex light-transport effects.
- Segmentation: Geometry decomposition depends on multi-view masks and GaussianCut, making object-component quality sensitive to segmentation errors.Because geometry remains fixed during material–lighting optimization, segmentation errors may propagate to recovered material and illumination estimates.
- Light Transport: One indirect bounce does not faithfully recover long-path global illumination, caustics, transmission, or sharp mirror chains.The authors identify faster rendering and more expressive light transport as future work.
Conclusion
LightFuse combines multi-state component reconstruction, geometry refinement, and staged inverse rendering in a relightable 2D Gaussian scene model. It supports edited layouts with updated visibility and achieves strong synthetic relighting results against the strongest baseline.
- Conclusion: LightFuse combines shared geometry and material, state-specific illumination, and retraced edited geometry to support rearrangement, material editing, and relighting.The unified representation integrates multi-state component reconstruction, ray-tracing-oriented geometry refinement, and staged inverse rendering.
- Conclusion: 25.38 dB PSNR and 0.852 SSIM surpass the strongest baseline by +9.74 dB PSNR and +0.121 SSIM across four synthetic scenes and eleven unseen environment maps.