PaperlayerView exact revision on arXiv

Source-linked AI summary

RIDE: Relocalization-Informed Depth Estimation with 3D Gaussian Splatting

Jiarong Lian, Zhe Xiao, Zhaoyang Zhang, Wei Li, Ruizhi Chen

arXiv:2609.11079v1cs.RO

TL;DR

RIDE addresses the underused role of relocalization correspondences in dense metric depth estimation. It calibrates a frozen video-depth prior with sparse PnP-RANSAC-derived metric observations using global, spatial, and temporal corrections. The method improves depth accuracy and temporal consistency across public RGB-D and robot evaluations without fine tuning on robot sequences.

  • Problem

    Pose recovery and dense metric depth estimation use complementary visual information but are typically addressed separately, leaving relocalization correspondences underused for dense depth.

  • Method

    RIDE calibrates a frozen VDA-S prior with sparse metric depth observations from PnP-RANSAC inliers using robust global scale, local spatial correction, and flow-guided temporal memory.

  • Results

    RIDE achieves the highest δ1 and lowest held-out error and TGE among evaluated methods on public RGB-D data, and the best reported results on all three metrics across 27 robot routes.

  • Takeaways & Limitations

    Localization geometry can support both camera pose recovery and dense robot depth perception, including short periods without reliable metric anchors after scale initialization.

Abstract

from arXiv · show

Render--match--PnP relocalization establishes correspondences between query image pixels and 3D map points for camera pose recovery, but their potential to support dense depth estimation is often overlooked. To exploit this geometric information, we present RIDE, which estimates dense metric depth from a robot's RGB stream. Given a metrically scaled 3D Gaussian Splatting (3DGS) model, RIDE combines sparse metric depth observations derived from PnP-RANSAC inlier correspondences with the geometric prior of a pretrained video-depth model. To handle uneven and intermittent observations, it integrates global and local depth correction with temporal memory, supporting depth estimation through short observation gaps after metric scale initialization. Trained on public RGB-D videos, RIDE is evaluated on robot sequences without fine tuning. Experiments show improved depth accuracy and temporal consistency over scale-only calibration, demonstrating how localization geometry can support both pose recovery and dense robot perception.

I. INTRODUCTION

RIDE treats relocalization correspondences as metric depth evidence, combining them with video-depth priors to estimate dense depth despite sparse and intermittent observations. It links pose recovery and depth estimation through calibration modules designed for spatial errors and temporal gaps.

  • I. INTRODUCTION: Pose estimation and dense metric depth provide complementary information for navigation, obstacle avoidance, mapping, and interaction, yet are typically handled separately.RIDE connects these tasks through render–match–PnP geometry.
  • I. INTRODUCTION: Global-scale memory, local spatial correction, and temporal calibration address spatially varying prior errors, unstable scales, and short anchor outages.Accumulated calibration information supports depth estimation after metric scale initialization when reliable anchors temporarily disappear.
  • I. INTRODUCTION: RIDE combines a frozen VDA-S prior with sparse metric anchors to obtain dense metric depth from a robot’s RGB stream.The prior supplies temporally coherent relative structure, while anchors connect predictions to metric scale.
  • I. INTRODUCTION: RIDE derives sparse metric depth anchors from PnP-RANSAC inliers, extending metric 3DGS relocalization beyond pose recovery without additional deployment sensors.The retained correspondences are geometrically verified and less vulnerable than dense rendered depth to dynamic, unseen, or incomplete scene content.
  • I. INTRODUCTION: Across scene-disjoint public RGB-D sequences and 27 robot routes in five scenes, RIDE achieves the highest δ1 and lowest held-out error and temporal inconsistency on the public benchmark.Among evaluated methods, it also achieves the best dense depth estimation for all three real-robot metrics.

B. Video Depth Priors

RIDE calibrates a frozen VDA-S relative-depth prior using sparse metric observations obtained during 3DGS relocalization. Its design combines learned anchor reliability, global scale estimation, spatial correction, and recurrent temporal information.

  • B. Video Depth Priors: RIDE uses PnP-RANSAC inlier correspondences as sparse, nonuniform metric observations for calibrating dense relative depth.Unlike methods requiring externally supplied sparse maps, these observations arise from image matching and relocalization and may be temporarily unavailable.
  • B. Video Depth Priors: RIDE begins with RGB frames, a metric 3DGS model, camera intrinsics, and a coarse pose, then retains PnP-refined inliers as depth observations.Rendered RGB and depth support matching, map-point lifting, pose refinement, and anchor formation.
  • B. Video Depth Priors: Learned anchor reliability supports robust global-scale estimation and sparse spatial prompting, while recurrent memory and flow-guided correction carry calibration across frames.The corrections are bounded in spatial and temporal log scale.
  • B. Video Depth Priors: Frozen VDA-S supplies dense scene structure, while global-scale, spatial, and temporal calibration transforms its relative predictions into dense metric depth.RIDE uses VDA-S as a frozen prior rather than fine-tuning the video-depth model.
  • B. Video Depth Priors: The prior is filtered to finite positive values using a per-frame floor, producing a positive prior and a mask for replaced pixels before anchor-based calibration.Scale estimation uses valid anchors whose prior values lie outside the floor mask.

B. Spatial and Temporal Metric Calibration

RIDE calibrates a frozen video-depth prior using robust global-scale estimation, bounded local correction, and flow-guided temporal memory. These components preserve metric depth through unreliable anchors and short observation gaps after initialization.

  • Global-scale calibration: RIDE estimates a robust global scale from encoded anchors and updates it recurrently using support, reliability, residual dispersion, prior statistics, flow confidence, and innovation.The instantaneous observation is a normalized weighted mean of anchor residuals and is valid only with at least eight scale-valid anchors.
  • Global-scale calibration: The global-scale memory retains the predicted state when observations are invalid, while valid observations resume updates without resetting accumulated scale.The first valid observation initializes the state directly, and metric output remains invalid until initialization.
  • Spatial correction: A top-down decoder rasterizes anchor prompts and combines them with the log prior and multiscale VDA features to predict a bounded spatial log-scale residual.The bound restricts local multiplicative calibration while the frozen prior supplies the underlying scene structure.
  • Temporal correction: Flow aligns earlier latent states, after which a ConvGRU combines history with current features, anchor context, scale, spatial logits, and flow cues for temporal correction.Missing flow zeros the warped history while preserving global memory, and inference remains causal.
  • Temporal correction: The final metric depth combines the prior with global, spatial, and temporal calibration while using only images, anchors, and recurrent states available up to the current time.This design supports causal operation under changing anchor availability.

C. Staged Training and Supervision

RIDE is trained in stages to learn anchor reliability, spatial calibration, and recurrent global and temporal correction. Supervision uses held-out anchors, dense depth, temporal correspondences, and scale targets while preserving the frozen prior and core components during later stages.

  • Training data and targets: RIDE addresses irregular relocalization geometry by sampling registered metric depth at detected feature locations rather than relying on conventional scan-line or structured-light patterns.The sampled observations approximate the sparse, nonuniform inputs produced by PnP-RANSAC correspondences.
  • Staged optimization: Stage 1 trains anchor reliability with clean held-out anchors, while Stage 2 freezes VDA-S and reliability and trains spatial correction using the global log-scale observation.Spatial training includes dense-depth, edge-aware smoothness, and residual-magnitude objectives, with depth terms evaluated only when scale is valid.
  • Staged optimization: The spatial residual is trained with auxiliary dense-depth, edge-aware smoothness, and regularization terms outside a four-pixel neighborhood of each input anchor.These objectives constrain correction beyond directly observed anchor locations.
  • Staged optimization: Stage 3 freezes VDA-S, SEA-RAFT, reliability, and spatial decoding, then trains global and temporal recurrent modules on five-frame chunks with truncated backpropagation through time.The loss aligns log-depth errors across flow correspondences and supervises global scale with a median dense log-scale target.
  • Training data and targets: Held-out anchors, dense depth, outlier labels, and utility targets are used only as supervision or evaluation references rather than deployment inputs.This separates training references from the sparse observations available during inference.

A. Experimental Setup

The evaluation combines scene-disjoint public RGB-D clips with continuous real-robot routes, using sparse observations and shared masks to measure depth, held-out-anchor, and temporal performance. Baselines include current-frame scale calibration, spatial correction, and alternative depth models.

  • Public RGB-D data: Public RGB-D data use the RobbyReal subset of LingBot-Depth, split by scene into 147/18/18 training, validation, and test scenes with 134,787/14,815/15,451 causal clips.Each clip contains five consecutive frames ending at the target frame.
  • Robot sequences: The robot evaluation covers 3,012 frames from 27 routes across five scenes, using continuous processing with state resets only at route boundaries and no robot-data fine tuning.Sparse metric inputs come from PnP-RANSAC inliers against a metric 3DGS model, while registered RGB-D serves as evaluation reference.
  • Metrics and masks: Evaluation reports depth metrics on valid pixels, held-out error on clean withheld observations, and temporal geometric error over eligible consecutive target-frame pairs.Held-out error is reported only on public RGB-D data, whereas robot observations are not split into input and held-out sets.
  • Metrics and masks: All robot methods process the same 3,012 frames but are scored on each method’s valid-output intersection with the required reference mask.The temporal metric uses shared precomputed flow and reference-validity masks.
  • Baselines: Baselines range from uniform-median and robust current-frame scale calibration to a spatial variant and alternative sparse-guided or relative-depth models.The spatial baseline adds the local decoder without temporal correction, while “+ LS” fits scale using current-frame anchors.

B. Relocalization-Derived Metric Anchors

RIDE extracts sparse metric depth anchors from refined render–match–PnP correspondences, then checks their agreement with registered RGB-D depth. In the example frame, most retained inliers produce anchors with sub-meter depth differences.

  • Anchor extraction: After PnP-RANSAC inlier selection and pose refinement, retained map points provide optical-axis depth observations at matched query pixels.The workflow transforms retained map points into the query camera frame to obtain metric anchors.
  • Anchor validation: The workflow compares metric anchor extraction from query–render correspondences with registered RGB-D reference depth in one robot frame.
  • Anchor validation: 50 of 63 correspondences are retained as PnP-RANSAC inliers, with a mean inlier reprojection error of 1.56 pixels.This frame includes a person absent from the map.
  • Anchor validation: 47 anchors have valid RGB-D references, with median and 90th-percentile absolute depth differences of 0.138 m and 0.352 m.Reprojection consistency and reference-depth agreement provide complementary checks of the extraction workflow.

C. Comparison with Baselines

RIDE outperforms the evaluated baselines on the public and robot-sequence comparisons, combining strong frame-wise accuracy with lower temporal inconsistency. Qualitative results show coherent scene structure while exposing local artifacts in competing predictions.

  • Public RGB-D data: On public RGB-D data, RIDE has the highest δ1 and lowest held-out error and TGE, while PriorDA v1.1 has the lowest AbsRel.RIDE reduces TGE from 0.0674 for the best competing value to 0.0394, a 41.5% reduction.
  • Robot sequences: On robot sequences, RIDE achieves the best reported values on all three metrics, including AbsRel 0.0927, δ1 0.9361, and TGE 0.0298.Relative to the VDA-S spatial variant, AbsRel decreases from 0.1056 to 0.0927, δ1 increases from 0.8919 to 0.9361, and TGE falls from 0.0530 to 0.0298.
  • Qualitative comparison: The robot-scene qualitative comparison uses one metric color scale across depth maps and marks representative local artifacts or discrepancies.RIDE recovers coherent lobby geometry and clear structural boundaries.

D. Ablation Studies

Ablations show that local spatial correction improves over global scaling, while recurrent scale memory, temporal adaptation, flow alignment, and confidence weighting contribute to the full model’s performance.

  • Progressive corrections: Adding the spatial decoder reduces AbsRel from 0.2173 to 0.1178 and held-out error from 0.1515 to 0.0676 beyond global-only calibration.The spatial addition improves all four evaluated metrics and extends correction to unsupplied feature locations.
  • Temporal components: Adding global scale memory and the temporal adapter jointly reduces TGE from 0.0513 to 0.0354 across the four ablation metrics.The ablations use identical validation clips, anchor realizations, and evaluation masks.
  • Temporal components: Resetting recurrent state each frame raises TGE to 0.0438, while shuffling history causes the largest degradation across all four metrics.These interventions test whether calibration benefits from carrying relevant temporal context.
  • Flow interventions: Shuffling flow raises TGE from 0.0354 to 0.0492, and removing flow-confidence weighting increases held-out error from 0.0576 to 0.0693.The interventions support distinct roles for motion alignment and confidence weighting.

E. Qualitative Temporal Analysis

RIDE maintains depth predictions through short anchor shortages by carrying recurrent states and preserving temporal structure. Its memory trajectory smooths instantaneous scale observations while still incorporating new valid anchors.

  • Static sequence: After scale initialization, RIDE maintains coherent depth through intervals where insufficient anchor support interrupts frame-wise estimators.The space–time slices place image position horizontally and time vertically, with unavailable predictions shown as hatched bands.
  • Dynamic sequence: In a dynamic sequence, RIDE preserves the moving foreground boundary while maintaining continuous predictions through anchor shortages.The result illustrates temporal continuity alongside sensitivity to foreground motion.
  • Scale dynamics: The memory trajectory varies more smoothly than instantaneous anchor-based scale observations and resumes observation-based updates when valid anchors return.The accumulated scale state is not reset during these updates.

V. CONCLUSION

RIDE combines relocalization and dense metric video-depth estimation by using PnP-RANSAC inlier correspondences to calibrate a frozen video-depth prior. It achieves the best reported results across all three metrics on 27 robot routes and maintains depth through short anchor outages after scale initialization.

  • RIDE achieves the best reported results on all three metrics across 27 robot routes under the stated valid-output protocol.The robot sequences use relocalization-derived observations without fine tuning.
  • The system derives sparse metric depth from PnP-RANSAC inliers and calibrates a frozen VDA-S prior using global scale, spatial correction, and temporal memory.
  • Ablations support spatial correction and reliable temporal context, while qualitative examples show continuous output through short anchor outages and preservation of moving foreground boundaries.
  • RIDE demonstrates the value of reusing relocalization correspondences for dense robot perception.
Loading 2609.11079v1…