PaperlayerView exact revision on arXiv

Source-linked AI summary

TRACKGRAPH: Online Open-Vocabulary 3D Scene Graphs via Image-Space Tracking

Peder Borge Hellesylt, Albert Gassol Puigjaner, Kostas Alexis, Annette Stahl

arXiv:2609.31005v1cs.CVcs.RO

TL;DR

Natural-language object search requires 3D maps that combine geometry, persistent queryable objects, and semantic information. TRACKGRAPH propagates mask identities in image space before 3D fusion, reporting competitive segmentation and retrieval with efficient online operation, alongside quadruped deployments and recorded aerial-view data.

  • Problem

    Natural-language object search needs 3D maps that combine metric geometry, persistent queryable object identities, and semantic information.

    Source: §I · p. 1

  • Method

    TRACKGRAPH propagates sparse FastSAM masks with dense DINOv3 features between keyframes, fuses source tracks into persistent class-agnostic 3D segments, and stores multi-view CLIP features for retrieval.

    Source: §I · pp. 1–2

  • Results

    TRACKGRAPH achieves competitive open-vocabulary segmentation and retrieval with efficient online operation, and experiments demonstrate quadruped scene-graph construction and search plus mapping and retrieval from recorded drone viewpoints.

    Source: §VI · p. 8

  • Takeaways & Limitations

    The demonstrations cover onboard scene-graph construction and object search on a quadruped, with recorded drone data testing mapping and retrieval from aerial viewpoints.

    Source: §VI · p. 8

Abstract

from arXiv · show

Open-vocabulary 3D maps enable robots to reason about previously unknown environments using natural language. However, existing systems typically segment every incoming image, associate detections with persistent 3D segments, and frequently perform costly Vision-Language (VL) inference. We present TRACKGRAPH, an online open-vocabulary system that maintains short-term 2D mask identity directly in the image stream before fusing segments into 3D. FastSAM masks and CLIP features are computed at sparse keyframes, while dense DINOv3 features are used to propagate masks at a high rate in between. The resulting tracked masks are fused into a class-agnostic 3D segment layer within a hierarchical scene graph, with 3D association handling tracking interruptions and long-term revisits. Compact multi-view CLIP embeddings enable open-vocabulary retrieval. Across Replica, ScanNet++, and HM3D, TRACKGRAPH achieves competitive open-vocabulary segmentation and retrieval against state-of-the-art mapping methods, including the highest synonym frequency on Replica (0.50). On the same NVIDIA A100, it is 1.7x faster and uses 3.3x less GPU memory than ViT-H OVI-MAP. Real-world quadruped deployments demonstrate onboard scene graph construction and object search at 7.5Hz, while recorded drone data is used to test the method under aerial viewpoints.

I. INTRODUCTION

Natural-language object search requires maps that combine geometry, persistent queryable objects, and semantic information, while existing approaches have limits in vocabulary or identity preservation. TRACKGRAPH tracks mask identities in image space before 3D fusion, reducing repeated segmentation and VL inference.

II. RELATED WORK

  • Hydra builds scene graphs but uses closed-set object labels, while ConceptGraphs and HOV-SG lift frame-local masks into 3D and consolidate observations across views.Source: §I · p. 1

I. INTRODUCTION

  • TRACKGRAPH propagates sparse FastSAM masks with dense DINOv3 features, then fuses tracked masks into persistent class-agnostic 3D segments with multi-view CLIP galleries.3D association reconciles tracking interruptions and long-term revisits.Source: §I · pp. 1–2
  • TRACKGRAPH demonstrates competitive open-vocabulary segmentation and retrieval with efficient operation, including quadruped scene-graph construction and object search and drone-data evaluation from aerial viewpoints.Source: §I · p. 2

II. RELATED WORK

  • Unlike approaches that segment each frame and associate masks with the 3D map, TRACKGRAPH maintains short-term mask identity in image space and uses 3D association after interruptions or revisits.CLIP features are computed at keyframes and aggregated per source track rather than inferred at every frame.Source: §II · p. 2

III. PROBLEM STATEMENT

The problem is to incrementally build an open-vocabulary 3D segment map in a hierarchical scene graph while preserving identities across observations and revisits. TRACKGRAPH uses image-space tracking to drive a class-agnostic segment layer within that graph.

III. PROBLEM STATEMENT

  • The map must combine metric geometry, persistent 3D segments, and open-vocabulary features while preserving segment identities across consecutive observations and revisits.Source: §III · p. 2

IV. METHOD

  • TRACKGRAPH propagates FastSAM masks and identities between keyframes with DINOv3 features, fuses source tracks into the TSDF and mesh, and uses 3D association to consolidate tracks after interruptions or revisits.Source: §IV · p. 2

A. Scene Graph Representation

  • TRACKGRAPH retains Hydra’s hierarchical graph but replaces its object layer with class-agnostic segments; segments connect to place nodes and are active while a connected mesh vertex remains within the 8 m spatial window.Each segment stores its source tracks, aggregated DINOv3 feature, CLIP feature gallery, and active or historical status.Source: A. Scene Graph Representation · pp. 2–3

B. Instance Tracking

TRACKGRAPH maintains persistent 2D source-track identities by propagating masks between keyframes and correcting them against new FastSAM detections. CLIP features are computed asynchronously at keyframes for downstream retrieval.

B. Instance Tracking

  • Dense DINOv3 features are extracted per frame, while FastSAM masks are detected at keyframes separated by Kf frames.The tracker maintains an active source-track set with persistent identifiers and aggregated DINOv3 features.Source: B. Instance Tracking · p. 3
  • DINOv3 patch similarities propagate source-track identity distributions from keyframes and recent frames, producing intermediate masks while retaining existing identities.Keyframe masks initialize one-hot track assignments; propagated distributions are upsampled and converted to masks by pixel-wise argmax.Source: B. Instance Tracking · pp. 3–4
  • At keyframes, FastSAM detections are matched to predicted tracks using overlap and visual similarity, then greedy one-to-one assignment continues, initializes, or retires tracks.Minimum overlap and combined-score thresholds prevent matching by appearance without spatial overlap or by overlap between visually dissimilar masks.Source: B. Instance Tracking · p. 4
  • Padded keyframe-mask crops receive asynchronous CLIP features that are fused into track averages for downstream retrieval.Computing CLIP features asynchronously avoids delaying mask propagation.Source: B. Instance Tracking · p. 4

C. Fusing Source Tracks into 3D

Source-track masks are fused into TSDF voxels and mesh attributes to build 3D segments. Propagated observations receive lower, time-decaying confidence than keyframe masks.

C. Fusing Source Tracks into 3D

  • Each TSDF voxel retains at most K source-track hypotheses, updating a track’s weight when the projected voxel falls inside its mask.The update uses the TSDF’s geometric observation weight.Source: C. Fusing Source Tracks into 3D · p. 4
  • Propagated-mask observations receive lower confidence that decays with frames since the track’s latest FastSAM keyframe detection.Keyframe masks use confidence 1; propagation confidence is controlled by αp and βp.Source: C. Fusing Source Tracks into 3D · p. 4
  • When all K voxel slots are occupied, an unrepresented track replaces the lowest-weight hypothesis, while displaced hypotheses can still contribute to 3D segments.A new track may enter without initially outcompeting the existing hypotheses.Source: C. Fusing Source Tracks into 3D · p. 4
  • Mesh nodes inherit voxel track hypotheses; winning identities assign vertices to 3D segments, while non-winning hypotheses preserve track-overlap information.The inherited hypothesis set comes from each supporting voxel.Source: C. Fusing Source Tracks into 3D · p. 4

D. 3D Segment Updater

The segment updater groups active-window mesh vertices by their winning source-track identity and associates tracks with persistent segment nodes. Overlap and DINOv3 similarity support consolidation when tracking is interrupted.

D. 3D Segment Updater

  • Active-window mesh vertices are grouped by their winning source-track identity, and an unassigned active track is matched to an existing segment or starts a new node.The winning identity is the track with the highest inherited weight at a vertex.Source: D. 3D Segment Updater · p. 4
  • Candidate segments are evaluated by bidirectional mesh-vertex overlap, and a match also requires DINOv3 feature similarity above its threshold.Either directional overlap ratio can meet the overlap threshold.Source: D. 3D Segment Updater · p. 5
  • Mutual overlap is prioritized over containment, then candidates are ranked by shared vertex count and DINOv3 similarity, allowing multiple tracks to consolidate after interruptions.If no candidate matches, the updater creates a new node and updates the segment’s geometry and features.Source: D. 3D Segment Updater · p. 5

E. Long-term Re-identification

Active-window association may fail on revisits because newly reconstructed surfaces lack shared source-track weights with historical geometry. Long-term re-identification searches historical candidates and merges a match back into an active node.

E. Long-term Re-identification

  • When a segment leaves the active window, newly reconstructed mesh vertices may share no source-track weights with its historical geometry, preventing active-window association.Without re-identification, this can create duplicate nodes for the same physical object.Source: E. Long-term Re-identification · p. 5
  • A centroid k-d tree narrows historical candidates, which are then evaluated using geometric overlap, DINOv3 similarity, and a thresholded combined score.The highest-scoring candidate is selected among those passing the respective thresholds.Source: E. Long-term Re-identification · p. 5
  • An accepted association merges the active node into the historical node, transfers its tracks and open-vocabulary features, and redirects mesh-to-segment edges; otherwise, the active segment remains distinct.Source: E. Long-term Re-identification · p. 5

F. Open-Vocabulary Galleries and Querying

TRACKGRAPH stores separate CLIP features for each source track in a segment’s open-vocabulary gallery and ranks segment nodes by their best gallery match to a query.

F. Open-Vocabulary Galleries and Querying

  • Each segment node stores a CLIP feature gallery keyed by its assigned source tracks, with entries synchronized to the tracks’ updated average features.Features from different tracks remain separate rather than being averaged into one node-level representation; distinct views can carry useful alternatives.Source: F. Open-Vocabulary Galleries and… · p. 5
  • A text or image query is embedded in the same CLIP space as the stored gallery features.Source: F. Open-Vocabulary Galleries and… · p. 5
  • Each segment node’s score is its maximum cosine similarity to any feature in its gallery, producing a ranked list of nodes.The maximum selects the view that best matches the query instead of diluting the score with less informative views.Source: F. Open-Vocabulary Galleries and… · p. 5

V. EXPERIMENTAL RESULTS

The experiments evaluate open-vocabulary segmentation and retrieval across three datasets, with runtime and resource comparisons on Replica and real-world robot-data demonstrations.

V. EXPERIMENTAL RESULTS

  • TRACKGRAPH and strong baselines are evaluated for open-vocabulary segmentation and object retrieval on Replica, ScanNet++, and HM3D following OpenLex3D.Source: §V · p. 5
  • Runtime and resource use are evaluated on Replica, where scenes have comparable durations and memory footprints and baseline measurements are available.Source: §V · p. 5
  • Real-world experiments demonstrate onboard quadruped deployment and mapping and retrieval using recorded drone data.Source: §V · p. 5

A. Experimental Setup

The setup specifies TRACKGRAPH’s implementation and evaluation protocols, including OpenLex3D segmentation and retrieval measures and the scope of its reported efficiency comparison.

A. Experimental Setup

  • TRACKGRAPH computes FastSAM-x masks and OpenCLIP ViT-H/14 features every 24 frames, propagating masks between keyframes with DINOv3 ViT-S+ features.DINOv3 inputs are resized to 720 pixels on the short side, and the selected model has 29M parameters.Source: A. Experimental Setup · p. 5
  • The Replica room0 visualizations mark ignored, background, or non-evaluated areas in white and describe broad synonym coverage comparable to HOV-SG [3] and OVI-MAP [4].The caption reports larger contiguous clutter, missing, or incorrect regions for ConceptGraphs [2] and FindAnything [5].Source: Figure 3 · p. 6
  • OpenLex3D Top-5 segmentation frequencies are reported across datasets, with the table caption describing TRACKGRAPH’s synonym score as best on Replica and second-best on ScanNet++.Source: Table I · p. 6
  • Segmentation compares each point’s vision-language feature with dataset-vocabulary text embeddings, while category frequencies are averaged per ground-truth object with equal object weighting.Walls, floors, and ceilings are excluded, and output point clouds are downsampled to 0.05 m.Source: A. Experimental Setup · p. 6
  • Retrieval is evaluated with AP averaged over 3D IoU thresholds 0.50–0.95, plus AP50 and AP25, and the table reports results in percentages.The table caption notes that encoder variants are named in parentheses and † denotes processing every tenth frame.Source: A. Experimental Setup · pp. 6–7, Table II · p. 6
  • The Replica efficiency table reports sequence-averaged timing, peak RAM, and VRAM alongside AP and synonym frequency; ConceptGraphs and HOV-SG efficiency figures come from RTX 3090 measurements.The caption identifies AP as percentage points and S as unit-scale synonym frequency.Source: Table III · p. 7

B. Open-Vocabulary Semantic Segmentation and Retrieval

TRACKGRAPH is competitive across open-vocabulary segmentation and retrieval, with results varying by dataset and encoder. Its reported efficiency advantages come with hardware and input-setting comparability caveats, and voxel resolution trades memory use against retrieval AP.

B. Open-Vocabulary Semantic Segmentation and Retrieval

  • TRACKGRAPH ViT-H has the highest synonym frequency on Replica, ranks second to HOV-SG on ScanNet++, and remains competitive on HM3D, while using 2.4× fewer segmentation and open-vocabulary inference steps.On HM3D, TRACKGRAPH matches OVI-MAP’s ViT-H variant in synonym score; OVI-MAP with SigLIP-L performs best there.Source: B. Open-Vocabulary Semantic… · p. 7
  • On Replica and ScanNet++, TRACKGRAPH trails only OVI-MAP variants on retrieval metrics except AP25 on ScanNet++, where HOV-SG is strongest; on HM3D it trails matched-encoder baselines on AP and AP50 but is competitive on AP25.The HM3D pattern is consistent with correct-region retrieval despite segments that may not fully cover object extent in larger, diverse scenes.Source: B. Open-Vocabulary Semantic… · p. 7

C. Mapping Quality and Computational Efficiency

  • On Replica, TRACKGRAPH has lower reported sequence processing time than ConceptGraphs, HOV-SG, and OVI-MAP, and the lowest reported GPU memory usage.The efficiency comparisons are indicative because hardware and input-frame settings are not identical across all methods.Source: C. Mapping Quality and Computational… · p. 7
  • Increasing voxel size from 0.03 m to 0.05 m reduces peak RAM by more than one-third, with little change in synonym frequency but lower retrieval AP.Source: C. Mapping Quality and Computational… · p. 7
  • With ViT-L, TRACKGRAPH performs better than FindAnything on Replica retrieval but worse on HM3D AP and AP50, while using lower GPU but higher CPU memory at 5 cm.Source: C. Mapping Quality and Computational… · p. 7

D. Ablations

The tracker ablation shows faster processing with nearly unchanged or better reported mapping metrics, while real-world tests demonstrate object retrieval on a quadruped and recorded drone data. The conclusion characterizes TRACKGRAPH as competitive and efficient, with onboard and aerial deployment demonstrations.

D. Ablations

  • 15.0 Hz versus 6.0 Hz, TRACKGRAPH’s tracker sustains 2.5× the rate with nearly identical AP and higher AP50, AP25, and synonym frequency on Replica.The comparison replaces tracking with per-frame FastSAM detections associated by BoT-SORT while keeping the rest of the pipeline fixed.Source: D. Ablations · p. 7, Table IV · p. 7
  • The gallery ablation supports retaining feature galleries rather than scoring each segment by similarity to its normalized mean CLIP feature.Table V covers Replica with ViT-H, averaged across scenes; AP is in percentage points and S is unit-scale synonym frequency.Source: Table IV · p. 7, Table V · p. 7

E. Real-world Experiments

  • Across three indoor quadruped environments, all queried targets were retrieved, and recorded drone data demonstrated mapping and retrieval from aerial viewpoints.Quadruped trials used autonomous exploration followed by text- or reference-image-based object search; the aerial demonstration reports successful retrieval of all image and text queries.Source: Figure 5 · p. 8, Figure 4 · p. 8

VI. CONCLUSIONS

  • TRACKGRAPH is reported as competitive in segmentation and retrieval with efficient online operation and low memory use, alongside quadruped and aerial deployment demonstrations.The conclusion specifies onboard scene-graph construction, retrieval, and object search on a quadruped, plus mapping and retrieval from aerial viewpoints.Source: §VI · p. 8

Related papers

Loading 2609.31005v1…