PaperlayerView exact revision on arXiv

Source-linked AI summary

RADIO-ViPE: Online Tightly Coupled Multi-Modal Fusion for Open-Vocabulary Semantic SLAM in Dynamic Environments

Zaid Nasser, Mikhail Iumanov, Tianhao Li, Maxim Popov, Jaafar Mahmoud, Sergey Kolyubin

arXiv:2604.26067v1cs.CV

TL;DR

Existing systems lack a unified way to perform online, open-vocabulary semantic grounding with robust geometry from uncalibrated monocular video in dynamic scenes. RADIO-ViPE tightly couples multimodal embeddings with geometric SLAM and adaptive robust optimization, achieving competitive semantic and dynamic-SLAM performance while operating online without calibration.

  • Problem

    Existing approaches lack simultaneous real-time operation, open-vocabulary grounding, robust odometry, accurate mapping, and dynamic-scene robustness, while internet video lacks depth, pose, and semantic annotations.

  • Method

    RADIO-ViPE combines RADSeg/RADIO multimodal embeddings with monocular flow, depth, intrinsics, and geometry in online factor-graph optimization wrapped by adaptive robust kernels.

  • Results

    RADIO-ViPE achieves best average ATE on dynamic TUM-RGBD sequences, ranks top-3 on Replica, and retains most accuracy with PCA-based compression.

  • Takeaways & Limitations

    The system provides calibration-free online semantic SLAM for open-vocabulary 3D grounding in dynamic environments and unconstrained video.

Abstract

from arXiv · show

We present RADIO-ViPE (Reduce All Domains Into One -- Video Pose Engine), an online semantic SLAM system that enables geometry-aware open-vocabulary grounding, associating arbitrary natural language queries with localized 3D regions and objects in dynamic environments. Unlike existing approaches that require calibrated, posed RGB-D input, RADIO-ViPE operates directly on raw monocular RGB video streams, requiring no prior camera intrinsics, depth sensors, or pose initialization. The system tightly couples multi-modal embeddings -- spanning vision and language -- derived from agglomerative foundation models (e.g., RADIO) with geometric scene information. This coupling takes place in initialization, optimization and factor graph connections to improve the consistency of the map from multiple modalities. The optimization is wrapped within adaptive robust kernels, designed to handle both actively moving objects and agent-displaced scene elements (e.g., furniture rearranged during ego-centric session). Experiments demonstrate that RADIO-ViPE achieves state-of-the-art results on the dynamic TUM-RGBD benchmark while maintaining competitive performance against offline open-vocabulary methods that rely on calibrated data and static scene assumptions. RADIO-ViPE bridges a critical gap in real-world deployment, enabling robust open-vocabulary semantic grounding for autonomous robotics and unconstrained in-the-wild video streams. Project page: https://be2rlab.github.io/radio_vipe

I. INTRODUCTION

RADIO-ViPE targets knowledge mapping from uncalibrated monocular video by combining open-vocabulary semantics with geometric SLAM for dynamic, unconstrained environments.

  • Knowledge mapping grounds free-form linguistic concepts onto 3D geometric representations, extending robots beyond purely geometric maps toward semantic reasoning.
  • Internet-scale video offers affordable, scalable, diverse supervision, but is unstructured and lacks the depth, pose, and semantic annotations used by many 3D methods.
  • Dynamic agents and displaced furniture can corrupt data associations and degrade localization and mapping when unaddressed.
  • RADIO-ViPE ingests raw unconstrained monocular RGB video without calibration and produces semantically and geometrically consistent maps supporting open-vocabulary language grounding.
  • Its contributions include tightly coupled vision-language-geometric fusion, adaptive robust optimization, and online open-vocabulary semantic SLAM.

A. Geometric SLAM Systems

Prior SLAM and 3D scene-understanding systems typically trade semantic richness, online operation, calibration flexibility, or dynamic-scene robustness, leaving these capabilities insufficiently integrated.

  • ORB-SLAM3 provides robust visual-inertial odometry but lacks semantic interpretation and dynamic object handling, while RVWO handles dynamic objects without semantic awareness.
  • Kimera and RGBDS-SLAM add instance semantics or RGB-D fusion but remain confined to closed-set, predefined object categories.
  • Offline open-vocabulary methods build semantically rich 3D representations or zero-shot segmentations, but lack integrated odometry and assume static scenes.
  • Recent real-time systems reconcile language understanding with mapping, but the surveyed approaches provide different combinations of clustering, Gaussian representations, or dense voxel embeddings.
  • Feed-forward reconstruction removes rigid camera assumptions or estimates video geometry, yet broadly assumes rigidity and lacks high-level representations for open-vocabulary grounding.
  • The stated integration gap concerns simultaneous real-time operation, open-vocabulary grounding, robust odometry, accurate mapping, and dynamic-scene robustness.

III. METHODOLOGY

RADIO-ViPE builds an online sliding-window factor graph that jointly estimates camera, depth, semantics, and geometry from uncalibrated monocular video, with adaptive kernels for dynamic scenes.

  • The system combines flow-based pose estimation, monocular foundation depth, aligned 3D points, and dense RADIO embeddings in sliding-window factor-graph optimization.The pipeline operates online at approximately 8–10 FPS.
  • Camera intrinsics are bootstrapped from sampled frames with GeoCalib and then co-optimized within bundle adjustment without calibration targets or known camera models.
  • Keyframes are selected using weighted dense optical-flow motion, then added to a graph whose vertices are keyframes and edges are pairwise connections.
  • RADSeg embeddings are extracted per keyframe, upsampled to H/8 × W/8, and compressed to D=256 dimensions for scalable memory use.
  • Semantic correspondence from RADIO features augments geometric flow priors, and bundle adjustment jointly refines intrinsics, poses, and 3D structure.

B. Dense Visual Features Extraction

The feature pipeline adapts RADSeg to produce spatially discriminative, language-aligned embeddings while integrating them with geometric flow and joint bundle adjustment.

  • RADSeg supplies dense embeddings, using RADIO-based agglomerative capabilities to generate language-aligned features in the SigLIP embedding space.
  • A sliding-window inference scheme and self-attention refinement balance spatial discriminability with semantic language alignment.
  • PCA is applied in encoder feature space rather than language-aligned space to preserve structural information needed for spatial reasoning.
  • PCA components are computed after a sufficient keyframe buffer accumulates during bundle-adjustment initialization, balancing efficiency and feature expressivity.
  • The factor-graph formulation jointly refines poses, dense disparity maps, and camera intrinsics using a weighted vision-language-geometric energy.
  • Dense optical-flow constraints use learned correspondences and confidence weights to penalize discrepancies between geometric reprojection and predicted flow.

2) RADIO Embedding Similarity Term:

RADIO-ViPE introduces a cross-view embedding similarity term that aligns dense RADIO features under geometric projection, integrating semantic consistency into bundle adjustment.

  • Semantic consistency: The embedding similarity term enforces cross-view feature alignment under geometric constraints within bundle adjustment.It directly incorporates RADIO’s multimodal representations into the optimization.
  • Semantic consistency: Each source pixel is projected into a target frame using current pose and depth estimates before recovering the target embedding by bilinear interpolation.The embedding maps are PCA-compressed dense feature maps with dimensions K×H×W.
  • Similarity computation: The method computes cosine similarity after ℓ2-normalizing the source and projected target embeddings.
  • Residual formulation: The embedding residual is expressed in photometric form to remain compatible with the flow term.The scaling λ_embed=2 produces residuals comparable in magnitude to classical photometric SLAM intensity differences.
  • Residual formulation: The full embedding similarity term incorporates photometric flow confidence through the per-pixel weight w(u).

D. Temporally Consistent Adaptive Robust Kernel

RADIO-ViPE uses temporally aggregated semantic evidence to distinguish static, movable, and actively moving scene elements more reliably than single-edge similarity.

  • Motivation: A kernel based only on cosine similarity from one edge cannot distinguish static surfaces, movable objects, and actively moving agents.Reliable classification therefore aggregates evidence across multiple connected edges over time.
  • Motivation: The adaptive robust kernel improves SLAM robustness in dynamic environments by using temporally consistent semantic evidence.
  • Temporal aggregation: For each pixel, RADIO-ViPE computes the temporal mean and variance of per-edge cosine similarities over all keyframes connected to keyframe i.

1) Temporal Stability Field:

The temporal stability field summarizes cross-view semantic consistency and maps it to differentiated Barron loss regimes for static, displaced, and actively moving regions.

  • Temporal Stability Field: The temporal stability field S_i(u) identifies genuinely static surfaces through consistently high similarity and low temporal variance.Values near 1 indicate this stable regime.
  • Temporal Stability Field: Values near 0 identify either active motion through low mean similarity or displaced objects through high temporal variance.Both categories are treated as unreliable for mapping.
  • Three-Regime Barron Shape Mapping: RADIO-ViPE maps temporal stability to Barron’s shape parameter α using a differentiable piecewise-linear mapping.The mapping separates three physically distinct categories: static surfaces, agent-moved objects, and actively moving agents.
  • Three-Regime Barron Shape Mapping: The three loss regimes recover ℓ2 for static surfaces, Huber for movable objects, and Cauchy for actively moving agents.These correspond to α=2, α=1, and α→0, respectively.
  • Three-Regime Barron Shape Mapping: The adaptive kernel jointly down-weights low-confidence and semantically dynamic regions while preserving full ℓ2 influence on static structure.The weight w handles confidence, while α_i j(u) controls the dynamic-region loss shape.

E. Objective for Tightly-Coupled Optimization

RADIO-ViPE jointly optimizes camera, depth, and intrinsic variables in a tightly coupled objective, using foundation-depth regularization to stabilize disparity estimates.

  • Tightly-Coupled Optimization: Gauss–Newton optimization jointly estimates camera poses, disparities, and camera intrinsics by minimizing the complete objective function.
  • Tightly-Coupled Optimization: The regularization term uses disparity priors from a foundation depth model with regularization weight α_disp set to 1.0 in experiments.
  • Tightly-Coupled Optimization: Foundation-depth regularization prevents drift while allowing depth refinement through multi-view consistency.

IV. EXPERIMENTS

RADIO-ViPE is evaluated on open-vocabulary semantic segmentation and dynamic-environment robustness, with experiments conducted on Replica and TUM-RGBD.

  • Evaluation covers open-vocabulary semantic segmentation on Replica and robustness in dynamic environments on TUM-RGBD.

A. SLAM Performance

RADIO-ViPE combines embedding-based and adaptive robust-kernel mechanisms for dynamic SLAM, while providing calibration-free semantic mapping with competitive benchmark performance and a memory-efficiency trade-off.

  • Dynamic SLAM: Embedding errors and adaptive robust kernels attenuate incorrect data associations caused by object motion in dynamic SLAM.The approach targets both actively moving and quasi-static scene elements through geometric and semantic consistency.
  • 3D Semantic Segmentation: RADIO-ViPE operates without known camera parameters, poses, point clouds, or depth for Replica semantic mapping.
  • 3D Semantic Segmentation: D=256 closely matches the full-dimensional baseline with ∆mIoU < 1% on the Replica semantic-mapping ablation.
  • 3D Semantic Segmentation: RADIO-ViPE produces quantitative results on Replica for different text queries.
  • 3D Semantic Segmentation: ∼1–2% in f-mIoU and Acc separates RADIO-ViPE from RADIO-ViPEGT in the without-background setting.The gap reflects removal of ground-truth depth, pose, and calibration; performance degrades more noticeably with background.
  • Overall Performance: RADIO-ViPE ranks top-3 on Replica and achieves competitive performance on TUM-RGBD despite PCA-based memory compression.The reported trade-off is framed as suitable for real-time robotics, AR/VR, and unconstrained in-the-wild video.
Loading 2604.26067v1…