PaperlayerView exact revision on arXiv

Source-linked AI summary

Dual Covariance Gaussian Splatting SLAM: Decoupling Rendering and Registration for Robust Real-Time Tracking

Edward Beng Wai Tan, Siew-Kei Lam

arXiv:2609.25746v1cs.ROcs.CV

TL;DR

ICP-based 3DGS SLAM reuses covariances optimized for rendering, although tracking needs sensor uncertainty and can remain underconstrained in weak-geometry directions. The paper separates mapping and tracking covariances and uses KLT image anchors; it reports robust tracking, reduced drift, and real-time performance, with limits in large-scale scenes and at high depth error.

  • Problem

    A shared, mapper-optimized Gaussian covariance may not represent sensor uncertainty accurately, while weak scene geometry can leave tracking underconstrained.

    Source: §I · p. 1

  • Method

    The method gives each Gaussian a mapping covariance and a sensor-derived tracking covariance, then uses tracking covariances with KLT-tracked image corners as anchors.

    Source: §I · pp. 1–2, D. Image residual · p. 3

  • Results

    The method is reported to improve tracking across diverse scenes, including difficult conditions, reduce odometry drift, and maintain real-time performance.

    Source: §I · pp. 1–2, §VI · p. 6

  • Takeaways & Limitations

    The authors report that real-time tracking and mapping performance enables deployment on robot platforms for online tracking and mapping.

    Source: §VI · p. 6

    Large-scale scenes lack bundle adjustment for long-term consistency, and high depth error—often increasing with distance—can reduce tracking accuracy.

    Source: §V · p. 6

Abstract

from arXiv · show

ICP-based 3D Gaussian Splatting (3DGS) SLAM tracks in real time by registering incoming frames against map Gaussians, using each primitive's covariance for both rendering and registration. These two uses place conflicting demands on one covariance. The mapper shapes it to minimize photometric error, often flattening it against surfaces, while robust registration typically benefits from measurement uncertainty. We propose a dual-covariance parameterization. Each Gaussian keeps a single mean but holds two covariances: a rendering covariance optimized by the mapper, and a tracking covariance derived from an RGB-D sensor noise model. We further use the tracking covariances as Gaussian anchors for image corners, providing constraints in directions where depth geometry is weak. We evaluate on TUM RGB-D, ScanNet, Replica, and two outdoor sequences recorded with a RealSense D435i on wheeled and handheld platforms. We achieve robust tracking performance across multiple scenes and reduced odometry drift, while tracking at $\sim$ 60 FPS.

I. INTRODUCTION

Photometric 3DGS SLAM can be computationally heavy, while ICP-based methods reuse mapping covariances that may not represent sensor uncertainty. The paper separates tracking and mapping covariances while sharing each Gaussian’s mean, and combines sensor-derived tracking covariances with KLT anchors.

I. INTRODUCTION

  • Earlier photometric 3DGS pose methods, were effective but usually ran at single-digit FPS; hybrid approaches included sparse feature matching and ICP point-cloud alignment.Source: §I · p. 1
  • Mapper-optimized covariances can misrepresent sensor uncertainty, recording confidence in the wrong direction and contributing to tracking failures in geometrically ill-conditioned scenes.Earlier ICP methods reused the same covariance for mapping and tracking.Source: §I · p. 1
  • Each Gaussian keeps one shared mean but receives separate tracking and mapping covariances.The tracking covariance stores sensor uncertainty measurements, while the other covariance is used for mapping.Source: §I · pp. 1–2
  • The joint tracking strategy uses sensor-derived covariances for ICP and stable landmark Gaussians as KLT anchors to improve observability.Source: §I · p. 2
  • The method is reported as robust to challenging conditions with reduced odometry drift.Source: §I · pp. 1–2

II. RELATED WORK

Prior work moved 3DGS SLAM from photometric pose optimization toward faster classical tracking, but ICP methods still face geometry, observability, and RGB-D noise issues. This section frames the problem as both incorrect uncertainty weighting and missing geometric constraints.

II. RELATED WORK

  • Photometric pose optimization using rasterized Gaussian appearance produced high-quality maps but often had relatively low FPS; Photo-SLAM and GS-ICP SLAM introduced classical tracking for higher speed.Source: §II · p. 2
  • ICP-based methods still face issues with geometry [8], observability [6], and RGB-D sensor noise, while uncertainty-weighted G-ICP allows arbitrary covariances derived from sensor estimates,.For 3DGS, the unresolved question is how to model a primitive also optimized by photometric error.Source: §II · p. 2
  • In self-similar environments, underconstrained pose directions can cause degeneracy and tracking failure; prior work [6] also reported failures in structure-less 3DGS scenes.Registration-Hessian diagnostics are commonly used for this degeneracy.Source: §II · p. 2
  • The section distinguishes spurious precision from flattened mapper covariances from genuinely missing geometric constraints, addressing them with sensor-derived tracking covariances and KLT anchors, respectively.A single plane may lack a direction that depth-residual reweighting can recover.Source: §II · p. 2

A. Preliminaries

The preliminaries formulate point-cloud registration using camera-frame points, a camera-to-world pose, nearest-neighbour associations, and metric covariances. For 3DGS, map Gaussians supply registration points, but mapper-flattened rendering covariances can assign misleading precision.

A. Preliminaries

  • Frame points are represented in camera coordinates as a_i, and T=(R,t) denotes the camera-to-world pose.Source: A. Preliminaries · p. 2
  • Generalized-ICP uses the residual d_i=b_π(i)−(Ra_i+t), with metric covariances for point positions and π denoting nearest-neighbour association.Source: A. Preliminaries · p. 2
  • GS-ICP SLAM registers map Gaussian means using KNN frame-point scatter covariances regularized as Z(Σ)=Σ/λ2(Σ).The mapper can flatten rendering covariances against surfaces, making the target term assign small scale along the surface normal despite lacking corresponding depth precision.Source: A. Preliminaries · p. 2

B. Tracking covariance

The method gives each Gaussian a mapper-optimized rendering covariance and a sensor-derived tracking covariance. Depth residuals use observation uncertainty together with the map covariance.

B. Tracking covariance

  • Each Gaussian keeps a rendering covariance optimized by the mapper and a tracking covariance derived from sensor uncertainty.The tracking covariance uses the RGB-D sensor model, including structured-light range noise σz(z) = A+Bz^2.Source: B. Tracking covariance · p. 2
  • At keyframes or spawn, in-view Gaussians receive tracking covariance anchored to the most recent observation's noise, with p = 0.5 px.Source: B. Tracking covariance · p. 2

C. Depth residual

  • Depth-pair uncertainty combines the map Gaussian's tracking covariance with transformed observation uncertainty that includes local tangent-plane association ambiguity.The observation uncertainty also reflects measurement noise and the maximum nearest-neighbor search distance.Source: C. Depth residual · p. 2

D. Image residual

KLT corners provide image-based tangent-plane constraints where depth observations lack constraints. Each landmark combines image tangential position with the map's normal coordinate and anchor covariance.

  • A nearby Gaussian anchors each back-projected KLT corner to its tangent plane, supplying the constraint missing from depth observations.Source: D. Image residual · p. 3
  • Each landmark keeps image-derived tangential position, takes its normal coordinate from the map, and inherits the anchor's tracking covariance.Source: D. Image residual · p. 3
  • Accumulated KLT landmarks act as a best-effort frame-to-map tracker until viewpoint, illumination, or other image artifacts lose corners.Their purpose is to improve observability rather than serve as the primary tracker; ablations show a general robustness increase when combined with tracking covariance.Source: D. Image residual · p. 3

E. Estimation

Pose estimation jointly optimizes ICP and image residuals with a Huber kernel using Levenberg–Marquardt on SE(3). Experiments evaluate tracking, map quality, and sequence-level FPS.

E. Estimation

  • Pose estimation jointly minimizes ICP and image residuals with a Huber kernel, solved by Levenberg–Marquardt on SE(3).Source: E. Estimation · p. 3

A. Implementation and Experimental Setup

  • Evaluation uses TUM RGB-D, ScanNet, and Replica, measuring tracking with aligned ATE and RPE and map quality with PSNR, SSIM, and LPIPS.FPS is the average input frames processed over the sequence during tracking, excluding input processing and post-hoc refinement.Source: A. Implementation and Experimental Setup · p. 3
  • Experiments ran on an i9-13900KF CPU and RTX 4090 GPU; comparison methods used publicly released implementations and default configurations.Source: A. Implementation and Experimental Setup · p. 3

B. Tracking Performance

Across TUM RGB-D, challenging low-texture or low-structure scenes, ScanNet, Replica, and outdoor sequences, the method generally tracks robustly with low drift. Results vary by setting: it performs strongly on multiple benchmarks, while Replica’s near-ideal depth conditions expose a small disadvantage against other ICP methods.

B. Tracking Performance

  • On standard TUM RGB-D scenes, the method has the lowest average ATE among coupled methods and higher average FPS than FeatureSLAM and SGAD-SLAM.Its ATE is lower than other methods on the fast-motion desk scene; ATE evaluation caps this method and GS-ICP SLAM at 30 FPS.Source: B. Tracking Performance · p. 3, Table I · p. 4
  • Across no-texture and no-structure scenes, the method maintains robust tracking without a detection mechanism like [6].The table separates depth-only tracking methods from methods using image-level signals; “X” marks failed or partial tracking due to late initialization.Source: B. Tracking Performance · p. 4, Table III · p. 4
  • On ScanNet, tracking is stable and on par with ORB-SLAM3 using BA and loop closure, and dense photometric methods SplaTAM and Gaussian-SLAM.Some scenes, including 0169, offer loop-closure opportunities.Source: B. Tracking Performance · p. 4, Table IV · p. 4
  • On Replica, the method is marginally worse than other ICP methods but still outperforms non-ICP 3DGS methods.Replica has no depth sensor noise and is described as a near-ideal case for ICP; the covariance model requires a small non-zero depth noise floor for numerical stability.Source: B. Tracking Performance · p. 4, Table V · p. 4
  • On extended TUM RGB-D sequences, the method has the lowest overall relative pose error among ICP-based methods, with drift characteristics more like ORB-SLAM3.The comparison covers translation and rotation RMSE over a 1-second window without trajectory alignment.Source: B. Tracking Performance · p. 4, Table VI · p. 4
  • On outdoor trajectories longer than 70 m, the method has the lowest drift among evaluated 3DGS SLAM methods on both wheeled and handheld platforms, behind only ORB-SLAM3.The outdoor sequences were recorded with a D435i, and the method constructs a 3DGS map in real time without bundle adjustment.Source: B. Tracking Performance · p. 4, Figure 3 · p. 5

C. Mapping Performance

The method produces mapping quality similar to Photo-SLAM in PSNR and leads evaluated real-time methods in SSIM and LPIPS. Qualitative renders compare scenes from TUM RGB-D, ScanNet, and custom sequences.

C. Mapping Performance

  • The method performs similarly to Photo-SLAM in PSNR and has the best SSIM and LPIPS among evaluated real-time methods.The authors attribute part of this result to robust tracking allowing mapping from accurate viewpoints.Source: C. Mapping Performance · p. 5
  • Qualitative renders compare the method with selected baselines on TUM RGB-D, ScanNet, and custom sequences.Source: Figure 4 · p. 5

D. Observability analysis

The analysis uses the Hessian condition number as a proxy for scene-level observability. Tracking covariances improve conditioning, while image constraints add complementary information when scene geometry is weak.

D. Observability analysis

  • Replacing rendering covariance with tracking covariance substantially improves Hessian conditioning even without image constraints, so the gain is not solely attributable to KLT.The condition number is used as a proxy for observability; Figure 5 reports the comparison across ICP methods.Source: D. Observability analysis · p. 5, Figure 5 · p. 5
  • In the str notex scene, adding KLT changes the condition number little because perpendicular walls and floors already provide strong geometric constraints.Source: D. Observability analysis · p. 5
  • Where geometric constraints are weaker, KLT further improves conditioning by contributing complementary image-based information.Source: D. Observability analysis · p. 5

E. Ablation studies.

The full method attains the lowest ATE, while the section identifies large-scale consistency and depth-error constraints and reiterates real-time tracking across difficult scenes.

E. Ablation studies.

V. LIMITATIONS

  • Large-scale scenes lack bundle adjustment for long-term consistency, and high depth error can reduce accuracy in outdoor or far-dominated views.Source: §V · p. 6

VI. CONCLUSION

  • The method improves tracking in difficult scenes while maintaining real-time performance.Source: §VI · p. 6
Loading 2609.25746v1…