PaperlayerView exact revision on arXiv

Source-linked AI summary

VOIM: Training-Free Open-Vocabulary 3D Instance Mapping for RGB-D and Monocular SLAM

Sangmin Song, Sarath Kodagoda, Marc G. Carmichael, Karthick Thiyagarajan, Amal Gunatilake, Kelly Prentice, Jodi Martin

arXiv:2609.00775v1cs.CVcs.AI

TL;DR

Training-free open-vocabulary 3D instance mapping has largely assumed RGB-D and externally provided poses, despite the practical need for ordinary-language interaction with robots and the limitations of depth sensing. VOIM accumulates soft voxel-level evidence across views before assigning labels and instances, and outperforms OVO-SLAM on ScanNet++ while extending unchanged to monocular RGB with pooled parity on Replica.

  • Problem

    Training-free open-vocabulary 3D instance mapping has predominantly assumed RGB-D input and externally provided poses, although depth sensors add cost and can fail outdoors or on reflective surfaces.

  • Method

    VOIM accumulates per-voxel soft open-vocabulary label distributions and region features across views, then assigns labels and separates instances after aggregation using a cross-model veto.

  • Results

    VOIM exceeds OVO-SLAM by +11.7 mIoU on ScanNet++ and wins all ten scenes, while the same system runs fully monocular on Replica at pooled parity.

  • Takeaways & Limitations

    The aggregation stage is agnostic to geometry source, enabling instance-level open-vocabulary maps from RGB-D or monocular RGB and exporting occupancy grids with queryable instance indexes.

  • Takeaways & Limitations

    VOIM inherits detection-driven blind spots for stuff categories, and its advantage is regime-dependent under Replica’s all-classes scoring; monocular operation is drift-bound at building scale.

Abstract

from arXiv · show

We present Voxel-Grounded Online Instance Manager (VOIM), a training-free voxel-grounded instance manager that builds open-vocabulary 3D instance maps from RGB-D or from monocular RGB alone, a regime no prior training-free system addresses. Online systems typically segment object instances and label them at first detection, committing when evidence is weakest. VOIM instead defers label and instance decisions until soft evidence from unmodified, off-the-shelf perception has accumulated per voxel across views. We show that the mapping stage, rather than the particular perception models, carries the result: across four perception configurations on ScanNet++, varying the region descriptor, the detector label prior and the mask source, the map exceeds the strongest online RGB-D system, OVO-SLAM, by between 4.8 and 11.7 mIoU. Perception is not neutral, and substituting that baseline's own descriptor family costs 4.1 of the margin, yet the baseline carries the marginally better 2D descriptor (33.7 vs. 31.5 mIoU over three scenes) and still realizes the weaker map. Under a like-for-like protocol VOIM reaches 44.07 mIoU on ScanNet++ against 32.37, winning all ten scenes and both aggregations (pooled 33.31 vs. 25.97), and the same system runs unchanged to fully monocular RGB, matching that baseline pooled on Replica (27.80 vs. 27.50). The advantage is regime-specific: under Replica's all-classes scoring, matched inputs give a split result, 28.60 vs. 27.50 pooled against 24.59 vs. 30.11 on the per-scene mean. Room scale is label-limited and building scale drift-limited. Labeling does not run in real time, dominated by per-class detection over the full vocabulary. The maps export occupancy grids and resolve free-form queries to object instances.

I. INTRODUCTION

VOIM addresses the need for open-vocabulary 3D instance maps without depth by deferring semantic and instance decisions until multi-view voxel evidence accumulates. Its mapping-stage design supports monocular operation and fair gains over OVO-SLAM.

  • Motivation and approach: VOIM accumulates soft label distributions from off-the-shelf perception as per-voxel evidence across keyframes before deciding labels and instances.This shifts uncertainty handling from individual detections to the mapping stage.
  • Contributions: Monocular RGB operation is supported unchanged because the aggregation stage is agnostic to the geometry source.Feed-forward SLAM backbones supply poses and geometry in the monocular setting.
  • Contributions: The method includes a fair comparison protocol, analyzes room-scale label limits and building-scale drift, and exports navigation-ready occupancy and instance indexes.The exported map resolves free-form queries to goal poses.

II. RELATED WORK

Related work spans offline open-vocabulary 3D understanding, online RGB-D mapping, and monocular semantic SLAM. VOIM targets the combined setting of training-free, online, instance-level, monocular mapping.

  • Open-vocabulary 3D scene understanding: Offline systems lift vision-language features or label 3D proposals, but the reviewed approaches assume depth and operate offline.VOIM inverts the segment-then-label pattern by aggregating voxel evidence before instance separation.
  • Online mapping and SLAM: Online systems such as ConceptGraphs, Open-Fusion, HOV-SG, and OVO-SLAM build queryable maps during exploration, predominantly from RGB-D inputs.OVO-SLAM labels online segments at query time but commits segment identities at first detection.
  • Concurrent work: Concurrent training-free instance mappers consume RGB-D with externally supplied poses, while monocular semantic systems generally produce semantic-level rather than instance-level output.The reviewed concurrent setting lacks a system simultaneously satisfying training-free, online, instance-level, and monocular requirements.
  • Feed-forward monocular SLAM: Feed-forward monocular SLAM backbones provide dense pointmaps and odometry that enable semantic fusion from RGB streams.VOIM uses MASt3R-SLAM as its primary monocular front-end.

III. METHOD

VOIM converts RGB-D or monocular RGB into a dense 3D point map whose points carry open-vocabulary distributions and are organized into object instances. Its stages combine SLAM geometry, 2D perception, and deferred 3D organization.

  • System overview: VOIM produces a dense 3D point map with an open-vocabulary semantic distribution attached to every point and 3D object-instance organization.In monocular mode, the map is defined up to a similarity transform.
  • Qualitative behavior: The monocular reconstruction recovers furniture and dominant surfaces comparably to the RGB-D baseline in Replica room0, with remaining errors primarily semantic.The RGB-D baseline shows fragmentation and eye-level holes in the qualitative comparison.
  • System overview: The pipeline uses a dense SLAM backend to provide camera poses and per-keyframe pointmaps, then applies open-vocabulary detection and mask generation.MASt3R-SLAM is used for monocular input, while VGGT is used where noted.

B. Perception

VOIM combines detector labels with frozen region-descriptor distributions over the full vocabulary, then uses a cross-model veto to suppress incompatible detector hallucinations. The resulting instance embeddings support open-vocabulary retrieval.

  • Detection: Every eligible SLAM keyframe is processed asynchronously, with Grounding-DINO prompted using every class in the full benchmark vocabulary and SAM2 generating instance masks.On Replica, semantic processing labels 14–22 of approximately 66 SLAM keyframes per scene.
  • Region descriptor: TextRegion independently scores each mask against the full vocabulary and returns a softmax distribution from frozen image-text embeddings.The PE-Core-L14-336 encoder uses mask-region pooling, with temperature τ=0.03.
  • Label fusion: The detector and descriptor are fused as complementary label sources because detector outputs are precise but may hallucinate absent classes, while descriptor distributions cover every mask but are less peaked.The fusion is a convex combination with α=0.5.
  • Cross-model veto: The cross-model veto accepts a detector label only when it appears among the descriptor’s top-5 classes; otherwise the pure descriptor distribution is retained.The veto is active for Replica all-classes scoring and disabled for ScanNet++ present-classes scoring.
  • Open-vocabulary retrieval: Aggregated vision-language embeddings resolve free-form and compositional language queries to highlighted 3D instances by cosine similarity.The retrieval demonstration covers both a fully monocular benchmark map and a real RGB-D lab map.

D. VOIM: Voxel-Grounded Online Instance Manager

VOIM delays hard labeling and instance formation until voxel-level evidence has been aggregated across views. Incremental soft voting supports open-vocabulary features, while final classwise clustering produces 3D instances.

  • Voxel-grounded aggregation: VOIM postpones discrete labels and instance associations until soft evidence is aggregated across all observations in 3D.This contrasts with early-commitment pipelines that associate 2D and 3D instances at first detection.
  • Voxel-grounded aggregation: Each retained voxel preserves its complete keyframe–pixel observation history, allowing every mapped point to access all masks that observed it.
  • Soft label voting: VOIM accumulates confidence-weighted mask distributions per voxel, with total vote mass serving as per-point confidence.The same incremental accumulator averages normalized region embeddings for open-vocabulary querying.
  • Instance extraction: After voting, low-confidence points are discarded, labels are smoothed spatially, and HDBSCAN clusters points independently within each predicted class.Each surviving cluster becomes a 3D object instance carrying its class, aggregated feature, and point set.
  • Instance extraction: Mapping remains incremental during tracking, but final argmax labeling and per-class instance clustering are a lightweight batch post-pass, so end-to-end throughput is offline-rate.

IV. EXPERIMENTS

The experiments evaluate open-vocabulary 3D segmentation on ScanNet++ and Replica under matched protocols, reporting both pooled and per-scene aggregation. Monocular evaluation additionally aligns reconstruction geometry before scoring.

  • Datasets and metrics: The evaluation covers ScanNet++ with 10 scenes and top-100 classes and Replica with 8 scenes and 51 classes, using OVO-SLAM’s scoring code.
  • Protocol: On ScanNet++, both methods receive identical frame lists, with OVO’s keyframe schedule rerun at every frame rather than its default every tenth frame.
  • Datasets and metrics: Both pooled mIoU and per-scene mean are reported on both datasets, including cases where per-scene aggregation favors the baseline.
  • Monocular evaluation: Before monocular scoring, a single global Sim(3) alignment maps the estimated trajectory and map into the ground-truth gauge without using ground-truth labels.
  • Implementation: The perception stack is frozen across datasets: Grounding-DINO detection, SAM2 segmentation, and PE-Core region descriptors.

B. ScanNet++: input-parity head-to-head

Under matched ScanNet++ inputs and vocabulary, VOIM outperforms OVO-SLAM under both aggregations and every scene. Decomposition shows the realized 3D lift, rather than the stronger 2D descriptor, drives the comparison.

  • Head-to-head result: 44.07 mIoU versus 32.37 gives VOIM a +11.70 per-scene-mean advantage over OVO-SLAM on ScanNet++.VOIM also scores 33.31 versus 25.97 pooled, winning all 10 scenes with per-scene margins from +1.9 to +17.7.
  • Head-to-head result: VOIM wins under both aggregations and all 10 scenes when both systems use the same frames, ground-truth poses, and vocabulary.The baseline retains a depth-source advantage under the stated protocol.
  • Decomposition: 33.7 versus 31.5 mean mIoU gives OVO-SLAM the marginally stronger 2D descriptor on three scenes, yet VOIM’s realized map gains 5.9 over its vote.OVO-SLAM’s realized map instead loses 5.0 relative to its vote.
  • Ablations: 38.82 mIoU remains above the baseline with class-agnostic SAM2 masks and descriptor-only labels, retaining 55% of the original margin across all ten scenes.
  • Ablations: 37.13 mIoU results without the detector, and the detector’s label prior accounts for 6.94 points while its mask source contributes −1.69 points.

C. Replica: monocular capability

Replica tests VOIM under privileged input parity and fully monocular RGB. Results favor VOIM on pooled mIoU but favor OVO-SLAM on the per-scene mean, defining a regime-specific advantage.

  • Input parity: 28.60 pooled mIoU versus 27.50 gives VOIM an input-parity advantage, while OVO-SLAM leads the per-scene mean 30.11 versus 24.59.
  • Monocular capability: 27.80 pooled mIoU versus 27.50 matches the RGB-D baseline under fully monocular input, while VOIM scores 23.55 on the per-scene mean.The per-scene deficit concentrates in small office scenes, worst on office1 at 15.4.
  • Metric interpretation: Replica’s all-classes, 51-class scoring charges scenes for absent classes and weights small offices equally in the per-scene mean.
  • Context: OVI-MAP reports 26.5 under RGB-D and ground-truth poses with 200-frame subsampling, while Ov3R reports 30.4 per-scene mean from its own reconstruction.

D. Instance-level evaluation

VOIM’s instance-level evaluation measures object detection quality directly and isolates the costs of soft evidence, geometry, and coverage.

  • VOIM evaluates 3D instances with greedy AP at IoU 0.25 and 0.50 over Replica’s 51-class vocabulary.Predicted points are transferred to ground-truth mesh vertices using k=5 nearest neighbors within a 10 cm radius.
  • Diagnostic costs: 65.1 to 52.4 purity and a widening AP@25-to-AP@50 gap identify estimated geometry as a boundary-quality cost.The monocular system reaches within 5.8 AP@25 of the RGB-D baseline but trails by 11.3 AP@50.
  • Diagnostic costs: 30.1% versus 40.8% coverage at IoU≥0.25 reveals a separate recall deficit, with missed instances rather than only poorer boundaries.The systems are not claimed to have instance parity on either axis.
  • Soft multi-view voting: 3.17 mean mIoU is lost when soft per-mask distributions are replaced by one-hot argmax labels, falling from 23.55 to 20.38 across eight monocular scenes.All eight scenes degrade, with sign-test p ≈0.008.
  • Hyperparameter sensitivity: Fusion hyperparameters have small effects: α changes mean mIoU by at most 0.33, while veto depth k changes it by at most 0.63.These changes are far below the reported mapping-stage deltas.

F. Analysis

The analysis separates room-scale labeling limits from building-scale geometric drift and documents navigation-oriented map outputs and runtime behavior.

  • Room scale: 28.60 versus 27.80 pooled mIoU shows that privileged depth and poses add only 0.8 at room scale, where 2D labeling caps performance.The detector misses small classes that contribute zero IoU regardless of reconstruction.
  • Building scale: 4.6 mIoU on a larger ScanNet++ scene demonstrates building-scale monocular collapse from pose drift rather than tracking failure.On another scene, continuous tracking still accumulates 3.4× the ground-truth path length; ground-truth poses roughly double mIoU from 7.9 to 16.9.
  • Runtime: 290 s end-to-end runtime on one A10G and 13.2 GiB peak VRAM characterize the measured room0 deployment.Semantic labeling runs asynchronously at 15.3 s per labeled keyframe.
  • Navigation output: The exported map projects labels into a 2D occupancy grid and resolves free-form text queries to highlighted instances and nearest goal poses.Each 3D instance carries an open-vocabulary index for descriptor-similarity retrieval.
  • Navigation output: A real single-sweep building-floor map contains 773 object instances and 158 m2 of connected navigable free space.Replacing visual-inertial odometry with the monocular backend yields comparable loop error: 2.3 versus 1.9 m.

VI. LIMITATIONS

VOIM’s limitations are tied to detector blind spots, regime-dependent evaluation, monocular drift, metric scale, offline labeling, and memory growth.

  • Detection scope: Detector-unfriendly “stuff” categories such as wall and floor can remain entirely unlabeled.Addressing this requires a dense open-vocabulary segmenter instead of a detector.
  • Evaluation regime: Under Replica’s all-classes scoring, VOIM wins the pooled metric but trails OVO-SLAM on per-scene mean: 24.59 against 30.11 at matched inputs.Fully monocular per-scene mean is 23.55.
  • Geometry: Monocular operation is drift-bound at building scale, while estimated depth degrades on wide-baseline sequences.The output is defined only up to a global similarity, so metric deployment needs an external scale reference.
  • Systems constraints: Offline-rate labeling and feature-buffer growth constrain deployment: approximately 4M points at 1024-D exhaust a 23 GiB GPU.The buffer therefore bounds map density on current hardware.
Loading 2609.00775v1…