RRG-SLAM: Real-time Reflection-aware Gaussian SLAM for Indoor Scenes

• Diffuse-only Gaussian SLAM struggles to represent reflected content, while reflections can corrupt image features and photometric consistency, contributing to drifting errors.The paper identifies planar reflective indoor surfaces as a particular challenge for real-time reconstruction.Source: §1 · p. 2
• The proposed hybrid representation separates diffuse base appearance from plane-associated reflection components while remaining compatible with online SLAM.The base uses a TSDF volume and base Gaussians; reflection groups model planar reflective effects.Source: §1 · p. 2
• Reflective planes are identified by combining geometry, semantic categories, and temporal color variance, with repeated detections aggregated across frames.Candidate planes are checked against raycast color variance, and only planes repeatedly identified in a sliding window are accepted.Source: §1 · p. 2
• Tracking initializes pose with point-to-plane ICP, then refines it using image features outside reflection-dominated regions.The reflection mask is obtained by rendering reflection color and thresholding its intensity.Source: §1 · p. 2
• The system fuses observations into the TSDF and optimizes and prunes base and reflection Gaussians online; experiments report better reconstruction, tracking, and novel-view rendering while maintaining real-time performance.The evaluation includes self-captured reflective scenes and a synthetic reflective dataset, with public RGB-D benchmarks also used to assess compatibility with general indoor scenes.Source: §1 · pp. 2–3

• In reflective scenes, appearance variation can destabilize photometric alignment and feature matching, while standard RGB-D SLAM methods do not explicitly model reflections.Gaussian-based SLAM methods also assume diffuse materials, leaving reflected content difficult to represent accurately.Source: §2 · p. 3
• NeRF-based RGB-D SLAM can improve rendering quality but repeated MLP evaluation and volumetric rendering limit online efficiency; Gaussian methods offer faster rendering.GPS-SLAM combines a TSDF with sparse Gaussians and sorting-free two-pass rendering.Source: §2 · p. 3
• Existing radiance-field SLAM representations are insufficient for strongly reflective scenes, motivating plane-associated reflection Gaussians alongside TSDF geometry and base Gaussians.The paper presents this extension as better suited to reflective indoor scenes and as benefiting tracking, mapping, and novel-view synthesis.Source: §2 · p. 3
• Offline reflection methods include environment-map, ray-tracing, and geometry-aware approaches, but their assumptions, computational costs, or reliance on dense multi-view optimization limit direct use in real-time SLAM.The paper positions online reflective-plane detection and tailored reflection-Gaussian initialization as its way to address the online reconstruction gap.Source: §2 · pp. 3–4

3.1 Modeling

• The base representation uses a TSDF for geometry and coarse diffuse appearance, while base Gaussians refine diffuse appearance and TSDF voxels store reflection-aware attributes.The voxel attributes include temporal color variance, reflection strength, and plane ID.Source: §3.1 · p. 4
• Each detected reflective plane defines a reflection group, and its Gaussians represent reflected appearance as virtual content behind that plane.Reflection Gaussians carry the ID of their associated reflective plane.Source: §3.1 · p. 4
• Plane-level reflection groups separate reflective appearance from the base scene and allow each group to contribute only to pixels associated with its plane.This organization is intended to prevent reflection-induced artifacts from contaminating diffuse reconstruction.Source: §3.1 · p. 4

3.2 Rendering

• TSDF raycasting produces color, depth, plane IDs, and a reflection mask; the mask is formed by thresholding reflection strength at 0.5.The plane ID map associates visible pixels with their planar surfaces.Source: §3.2 · p. 4
• Base Gaussians are rendered with depth culling against the TSDF surface and order-independent accumulation before their color is combined with TSDF color.The accumulated Gaussian attributes can yield color, weight, or depth maps.Source: §3.2 · p. 4
• Reflection Gaussians are rasterized in one pass, with each group filtered to pixels whose raycast plane ID matches its associated plane, then composited using the reflection mask.This produces a reflection image that is combined with the base image.Source: §3.2 · pp. 4, 6

4 Online Reconstruction Process

• The pipeline estimates poses, identifies reflective planes, and fuses observations into an augmented TSDF before incrementally adding, optimizing, and pruning base and reflection Gaussians.Tracking and TSDF fusion run on each incoming frame, while plane identification and Gaussian optimization run periodically at lower frequencies.Source: §4 · p. 6

4.1.2 Reflection-aware tracking.

• A two-stage tracker initializes pose with point-to-plane ICP against the TSDF raycast surface, then refines it with ORB features after masking reflection-dominated pixels.The combined geometric initialization and reflection-aware feature refinement is described as enabling robust tracking in strongly reflective indoor scenes.Source: §4.1.2 · p. 6

4.1.3 Reflective plane identification.

• Reflective-plane candidates combine depth-based plane segmentation with semantic categories, then temporal color variance and repeated detections validate planes.Confirmed planes contribute semantic masks that form a predicted reflection mask fused into the TSDF.Source: §4.1.3 · p. 6

4.1.4 Augmented TSDF fusion.

• Augmented TSDF fusion tracks voxel color variance, global plane IDs, and reflection strength to support later reflection detection, tracking, and Gaussian initialization.Reflection strength is updated with a sliding average over recent predicted reflection masks.Source: §4.1.4 · p. 7

4.2.1 Gaussian densification.

• Base Gaussian candidates are pixels with high color error and low existing coverage; their 3D positions are initialized by backprojecting observed depth.Reflection candidates additionally require reflective-region membership, high temporal variance and reconstruction error, and low reflection-Gaussian coverage; their positions are sampled along viewing rays behind the plane.Source: §4.2.1 · p. 7

4.2.2 Gaussian optimization.

• Gaussian optimization periodically samples recent frames and historical keyframes, jointly refining base and reflection groups with color, depth, and reflection-smoothness losses.The depth weight is 0.2 and reflection smoothness weight is 0.05 in all experiments.Source: §4.2.2 · p. 7

4.2.3 Gaussian pruning.

• After each optimization round, the system prunes Gaussians by scale and opacity to remove oversmoothing, low-contribution, unstable, or redundant primitives.The pruning is intended to maintain a compact and stable representation during long-term online reconstruction.Source: §4.2.3 · p. 7

5.2.1 Rendering quality.

• RRG-SLAM achieves the best novel-view rendering on RIRD-Syn and ScanNet++ and the best input-view rendering on RIRD-Real; Replica results are comparable.Qualitative comparisons show more coherent reflective appearance across surfaces with varying roughness, while decomposing diffuse base appearance from planar reflections.Source: §5.2.1 · pp. 9–10, Table 1 · p. 10

5.2.2 Tracking accuracy.

• RRG-SLAM attains the lowest ATE on reflective RIRD-Syn and performs competitively on Replica and TUM-RGBD.On Meeting Room, masked ORB refinement with ICP initialization suppresses reflection interference and yields more stable tracking than the compared methods.Source: §5.2.2 · pp. 10, 12, Table 2 · p. 12

5.2.3 Geometry quality.

• On RIRD-Syn, RRG-SLAM outperforms other methods in geometry quality; on ScanNet++, its geometry is on par with Gaussian-based SLAM methods.ScanNet++ mapping uses ground-truth poses because consecutive frames can change abruptly.Source: Table 3 · p. 12, §5.2.3 · p. 12

5.2.4 Efficiency analysis.

• On reflective Meeting Room, RRG-SLAM uses 0.64M Gaussians and reaches 70.5 FPS, while GPS-SLAM uses 2.37M and runs at 51.0 FPS.On non-reflective Replica Office 0, RRG-SLAM adds only negligible overhead compared with GPS-SLAM while achieving comparable rendering quality.Source: Table 4 · p. 12, §5.2.4 · p. 12
• Semantic segmentation takes 7.04 ms per frame, and pre-mapping totals 10.12 ms versus 14.18 ms for Gaussian mapping; three-pass rendering takes 3.22 ms per frame (310.0 FPS).Pre-mapping can run in parallel with Gaussian mapping, limiting its impact on system speed.Source: §5.2.4 · pp. 12–13, Table 5 · p. 13, Table 6 · p. 13

5.2.5 Reflective plane identification.

• Combining geometric, semantic, and temporal cues enables the system to localize reflective regions on planar surfaces.The visualized inputs include plane and semantic segmentation maps, TSDF color-variance maps, and predicted reflection masks.Source: §5.2.5 · p. 13, Figure 8 · p. 13

5.3.1 Reflection grouping and rendering.

• Plane-associated reflection groups improve reflection fidelity, while single-pass plane filtering preserves quality with higher rendering efficiency than separate masked passes.Without plane grouping, reflections appear dimmer and blurrier.Source: §5.3.1 · pp. 13–14, Table 7 · p. 14

5.3.2 Reflection-aware tracking.

• On RIRD-Syn, unmasked ORB refinement raises ATE from 0.56 to 1.21, whereas reflection-aware masking reduces it to 0.31; on Meeting Room, removing masking leaves visible drift.The full model uses masked ORB-based refinement after ICP initialization.Source: §5.3.2 · p. 14, Table 8 · p. 14, Figure 10 · p. 14

• The system separates base scene geometry and diffuse appearance from plane-associated reflection components using a reflection-aware TSDF-Gaussian hybrid representation.Source: §6 · pp. 14–15
• The temporal color variance threshold ablation on RIRD-Syn Room 2 reports PSNR, SSIM, and LPIPS across threshold values.Source: Table 10 · p. 15
• Across self-captured reflective scenes, a synthetic reflective dataset, and public RGB-D benchmarks, the method reports better reconstruction, tracking robustness, and novel-view rendering while maintaining real-time performance.Source: §6 · p. 15
• Planar reflections are the method’s focus; it cannot explicitly model curved-surface reflections, and transparent objects without reliable depth remain challenging.The authors identify curved reflective surfaces, transparent materials, and more general non-Lambertian effects as future directions.Source: §6 · p. 15

• RIRD-Real is a self-collected real-world indoor RGB-D dataset with reflective materials and objects, including glossy floors, tables, and TVs, at 1280×720 resolution.Source: A Dataset details · p. 17
• RIRD-Syn is a Blender-rendered synthetic RGB-D dataset at 1280×720 for quantitative novel-view synthesis and tracking evaluation; two reflective ScanNet++ scenes are also included at 876×584.Source: A Dataset details · p. 17

B Implementation Details

• Reflective-plane identification and Gaussian optimization run every 10 frames, using two recent frames and seven sampled keyframes; densification uses color-error thresholds of 0.05 for both Gaussian types.Base and reflection Gaussian weight thresholds are 3 and 0.5, respectively; TSDF voxels are 1 cm in the Meeting Room and 0.5 cm elsewhere.Source: B Implementation Details · p. 17
• Gaussian pruning removes elements below the opacity threshold or outside the scale range; base thresholds are 0.005, 0.003, and 0.1, versus 0.005, 0.005, and 0.2 for reflections.Source: B Implementation Details · p. 17

C TSDF Color Variance Update

• Per-voxel color intensity statistics use a Welford-style online update to maintain temporal variance alongside the fused RGB color.New RGB observations are normalized to [0, 1] and converted to scalar intensity.Source: C TSDF Color Variance Update · p. 17
• Raycast variance is interpolated from neighboring voxels, and pixels above the predefined variance threshold mark unstable regions used for reflective-plane verification and reflection Gaussian initialization.Source: C TSDF Color Variance Update · p. 17

• Per-scene tracking tables report ATE for synthetic RIRD-Syn scenes, TUM-RGBD sequences, and Replica scenes, with averages reported for each dataset.Source: Table 3 · p. 18, Table 4 · p. 18, Table 5 · p. 18

D Global Plane Maintenance

• Local planes are registered into a global table; duplicate instances can be merged, and canonical IDs keep associations consistent for reflection rendering and initialization.Source: D Global Plane Maintenance · p. 18
• Global plane association combines normal-angle and point-to-plane tests with spatial footprints and overlap, IoU, or gap criteria to distinguish separate coplanar instances.Source: D Global Plane Maintenance · p. 18

E Grounded SAM2 Settings

• Grounded SAM2 uses predefined semantic prompts to produce masks that generate reflective-plane candidates only, before temporal reflection verification.The prompts include floor, table, vending machine, closet, wall art, TV, and whiteboard.Source: E Grounded SAM2 Settings · p. 18

• The per-scene tracking comparison includes ORB-SLAM2, BundleFusion, SplaTAM, RTG-SLAM, GauS-SLAM, GS-ICP SLAM, and GPS-SLAM.Source: §F.1 · p. 18
• ATE results are reported for each evaluated sequence in RIRD-Syn, TUM-RGBD, and Replica, alongside averaged tracking results from the main paper.Source: §F.1 · p. 18
• The detailed comparisons cover reflective and general indoor scenes to provide a per-scene view of tracking robustness.Source: §F.1 · p. 18

• Tables 6–9 report per-scene rendering quality for ScanNet++, RIRD-Syn, RIRD-Real, and Replica, complementing the main paper’s averaged results.Source: §F.2 · p. 18
• The ScanNet++ table compares novel-view rendering quality on selected reflective scenes using PSNR, SSIM, and LPIPS.Source: Table 6 · p. 19
• The ScanNet++ results include scores for each of two scenes and an average across both.Source: Table 6 · p. 19

F.3 Tracking Error Visualization on RIRD-Syn

• GPS-SLAM’s TSDF-based ICP is unaffected by reflection artifacts, while reflections disturb GS-ICP SLAM’s Gaussian-map optimization and GauS-SLAM’s color-and-depth pose estimation.Source: §F.3 · p. 19
• Per-scene tables compare PSNR, SSIM, and LPIPS for RIRD-Syn, RIRD-Real, and Replica; the RIRD-Syn table also reports averages across three scenes.Source: Table 7 · p. 19, Table 8 · p. 20, Table 9 · p. 20
• RRG-SLAM achieves more accurate tracking and higher-quality mesh reconstruction in the RIRD-Syn reflective scenes.Source: §F.3 · p. 20