PaperlayerView exact revision on arXiv

Source-linked AI summary

SuperMap: A Spatio-Temporal SLAM System for Visual-Language Navigation

Shibo Zhao, Guofei Chen, Honghao Zhu, Zhiheng Li, Changwei Yao, Nader Zantout, Seungchan Kim, Wenshan Wang, Ji Zhang, Sebastian Scherer

arXiv:2608.22896v1cs.RO

TL;DR

SuperMap addresses the challenge of maintaining object identities, semantics, and spatial relations for open-vocabulary navigation in changing environments. It builds an online, queryable 4D scene memory by combining geometric SLAM with semantic perception, achieving competitive segmentation, stronger instance-level results, and robust map maintenance across dynamic scenes.

  • Problem

    Open-vocabulary navigation requires object-centric maps that preserve identities, attributes, and relations over time despite occlusion, relocation, removal, and newly introduced objects.

  • Method

    SuperMap integrates geometric SLAM with semantic features from 2D foundation models for online 2D-3D object tracking, instance validation, change detection, and persistent 4D scene-graph construction.

  • Results

    SuperMap achieves 55.48% accuracy on ScanNet class-level segmentation, outperforms established object-centric baselines, and outperforms comparative baselines on instance-level segmentation and full-system ablations.

  • Takeaways & Limitations

    The resulting queryable 4D scene memory supports long-term spatio-temporal reasoning and language-guided navigation in dynamic indoor environments.

  • Takeaways & Limitations

    Tracking highly dynamic objects remains limited, and open-vocabulary detection currently relies on a predefined list of object prompts.

Abstract

from arXiv · show

Robotic navigation in human environments requires a spatio-temporal semantic representation that can rec- oncile open-vocabulary perception with long-term environmental changes. While foundation models provide strong zero-shot recognition, their predictions are intermittent and view-dependent, and naively integrating them into mapping pipelines leads to identity drift and stale semantics over time. We present SuperMap, a 4D spatio-temporal mapping framework for language-guided navigation that integrates high-frequency geometric SLAM with asynchronous open-vocabulary perception. Our core contribution is a consistency-driven mapping engine that combines 3D-aware instance association/re-activation with a principled existence-and-label confidence update to maintain stable object identities and prune outdated map content under occlusions and scene changes. SuperMap produces a queryable 4D scene-graph representation that interfaces naturally with Vision-Language Models by supporting compositional queries over object semantics, relations, We demonstrate SuperMap on benchmarks and real robots, including dynamic scenes with appearance/disappearance and relocation, and provide ablations and runtime analysis. We release the full system as open-source to provide the community with a deployable baseline for open-vocabulary spatio-temporal mapping. Project website: superodometry.com/supermap.

I. INTRODUCTION

SuperMap addresses the difficulty of maintaining open-vocabulary, instance-level spatio-temporal maps for language-guided navigation in changing environments. It combines online geometric reconstruction, change-aware object mapping, and a queryable 4D scene graph for robust long-term reasoning.

  • Motivation: Dynamic environments require object-centric maps that preserve identities, semantics, and spatial relations despite occlusion, relocation, removal, and newly introduced objects.Per-frame recognition and static semantic maps can produce fragmented identities, stale content, and inconsistent relational structure over long horizons.
  • Motivation: Existing approaches often address either dynamic change detection or static object recognition, while few handle both object changes and their spatial-temporal evolution.This dual challenge is important for robots operating in dynamic real-world environments.
  • Contributions: SuperMap builds a persistent, queryable open-vocabulary 4D scene memory online for open-vocabulary semantic and instance mapping.The system is evaluated on public benchmarks and real-robot deployments involving appearance, disappearance, and long-horizon changes.
  • Contributions: Its spatio-temporal object tracker combines 2D–3D association, validation, and change-aware updates to maintain instance consistency under occlusions, partial observations, label variability, and scene change.These mechanisms target stable instance identities rather than only class-level recognition.
  • Contributions: An instance-level 4D scene graph integrates spatial and temporal information, supporting queries such as locating moved objects and recalling past scenes.The representation is designed for instance-level spatio-temporal reasoning and language-conditioned tasks.
  • Contributions: The framework includes a real-robot visual-language navigation pipeline, ablations, runtime profiling, and an open-source release for reproducible research.The navigation demonstration compares structured 4D scene memory with raw detections or video-only inputs for robust grounding.

B. Semantic SLAM

Semantic SLAM combines geometric mapping with semantic and temporal object representations, but existing systems often lack real-time, open-vocabulary, instance-level modeling of scene evolution.

  • Existing semantic SLAM systems combine state estimation, reconstruction, tracking, and open-vocabulary perception, but many remain closed-vocabulary or class-level.
  • Offline scene-graph methods are unsuitable for online or real-time dynamic tasks, while CLIO lacks spatio-temporal capabilities.
  • Khronos models short- and long-term dynamics but is closed-set, lacks instance-level tracking, and struggles to operate in real time.
  • The spatio-temporal SLAM problem represents objects as scene-graph nodes connected by relationships that evolve across space and time.
  • SuperMap processes RGB-D or point-cloud video, estimating poses and producing a consistent global map of semantic instance-level objects at each time.
  • Given current observations and the existing map, the system estimates pose, assigns instance identities, and updates the map.

IV. METHOD

SuperMap runs onboard by combining online geometric reconstruction with spatio-temporal object updates and a queryable scene graph. Metric pose and depth information anchor semantic observations to a physically consistent map.

  • SuperMap jointly estimates robot state and maintains an instance-level spatio-temporal map as the environment evolves.
  • Its three components are online 3D reconstruction, spatio-temporal object updates, and scene-graph construction for a queryable VLN interface.
  • SuperOdometry estimates robot pose and produces a colorized dense 3D reconstruction from image, depth or LiDAR, and IMU streams.
  • Estimated pose and synchronized depth support 3D back-projection, motion compensation, and global map consistency.
  • For each frame, 2D instance detections are assigned unique IDs using 3D spatial consistency and the preceding map.
  • The tracklet state contains image centroid, bounding-box size, and image-plane translational velocity, with an optional scale-velocity extension.

2) 3D-to-2D Motion-Compensated State:

SuperMap compensates for rapid egomotion by predicting instance locations from 3D geometry, then combines geometric depth evidence with probabilistic updates to maintain consistent objects and prune changes.

  • Motion compensation: 3D-to-2D motion compensation projects each instance’s 3D centroid into the current camera frame to predict its image location.This projection replaces a purely linear motion prior and supplies the tracker with geometry-aware prediction.
  • Motion compensation: The Kalman-filter state transition uses the projected centroid as its prior, preserving identity through occlusions and aggressive robot maneuvers.The state transition matrix and process-noise term remain part of the update model.
  • Geometric consistency: SuperMap compares projected depth with raw sensor depth to classify map points as observable, behind a surface, or in front of a surface.The residual is ∆d = dproj − D(u), with dproj representing expected depth and D(u) the measured depth at the projected pixel.
  • Geometric consistency: Gaussian depth-noise likelihood and recursive log-odds updates provide numerically stable occupancy estimation from successive observations.The inverse sensor model uses thresholded geometric consistency of the depth residual.
  • Geometric consistency: Dynamic points are penalized in the log-odds update, pruning transient or moved objects while supplying geometric cues for semantic fusion across frames.This couples change detection with maintenance of a consistent global map.

4) Bayesian Semantic Fusion:

SuperMap fuses uncertain open-vocabulary labels over time and exposes maintained object instances as a spatio-temporal graph for language-grounded reasoning and action.

  • Bayesian semantic fusion: Each instance maintains a categorical label distribution that is updated from sequential open-vocabulary detector observations.The posterior combines the new detector likelihood with the previous label belief.
  • Bayesian semantic fusion: The detector likelihood is represented by a confusion matrix, making semantic updates explicitly account for recognition uncertainty.This likelihood enters the Bayesian posterior for each candidate label.
  • Bayesian semantic fusion: Semantic labels are updated for observable points, and points with sufficiently small posterior belief are removed to suppress transient misclassifications.Geometric observability gates when semantic evidence is fused.
  • 4D scene graph: The map becomes a graph whose nodes represent object instances, with spatial edges encoding class-dependent geometric relations and temporal edges tracing identities across time.Centroid-distance clustering supports real-time spatial-edge construction, while object association supplies temporal links.
  • 4D scene graph: The 4D scene graph gives Vision-Language Models a compact interface for zero-shot reasoning over long-term dynamics without processing raw point clouds or video.Serialized subgraphs contain instance IDs, labels, centroids, and symbolic spatial and temporal relations.
  • Grounded actuation: Parsed target instance IDs are mapped to 3D centroids, converting language-model outputs into deterministic navigation waypoints.The parser requires target IDs inside <answer></answer> tags before retrieving their map coordinates.

V. EXPERIMENTS

SuperMap is evaluated onboard for semantic quality, long-horizon change handling, grounded reasoning, and online language-guided navigation. Results emphasize competitive class-level segmentation, stronger instance-level performance, and persistent identities through dynamic indoor changes.

  • Evaluation setup: SuperMap runs its odometry, instance tracking, 4D graph maintenance, and VLM interface onboard using a LiDAR, panoramic camera, CPU, and GPU.The platform is a mecanum-wheeled mobile robot designed for agile indoor navigation.
  • Evaluation goals: The experiments assess semantic quality, spatio-temporal consistency, grounded VLM reasoning, and end-to-end online visual-language navigation.The evaluation includes public benchmarks and real-robot deployment questions spanning these four capabilities.
  • Semantic quality: 55.48% class-level segmentation accuracy on ScanNet outperforms ConceptGraphs and ConceptFusion, while trailing RayFronts by 1.32%.The authors report a lower computational footprint than RayFronts, supporting high-rate robotic feedback loops.
  • Semantic quality: SuperMap significantly outperforms HOV-SG and ConceptGraphs on instance-level segmentation, particularly for discrete objects within the camera’s instantaneous field of view.The reported gain is attributed to 3D-aware tracking-by-detection rather than point-feature clustering or over-segmented geometries.
  • Spatio-temporal consistency: In a 10-minute indoor experiment, three objects were removed and three added to test long-duration map integrity and environmental-change detection.The 30m × 20m scene included a plant, trash can, and chair removals, plus a bucket, cart, and safety-sign additions.
  • Spatio-temporal consistency: Instance IDs remained consistent during disappearance and appearance events, indicating identity preservation over long horizons.Examples include a newly inserted bucket and a chair that disappeared while retaining its association in the visualization.

D. Spatial-Temporal Consistency on Quantitative Evaluation

SuperMap evaluates spatio-temporal change detection and mapping maintenance using object-level recall criteria, ablations, and VLM reasoning comparisons. The results emphasize consistent identity handling, stale-content removal, and improved structured spatial-temporal reasoning.

  • Object Detection Recall requires 3D IoU > 0.1, centroid distance < 0.3m, and the correct semantic label during object appearance.
  • Change Detection Recall requires detecting an object during appearance and producing zero false positives after its disappearance.
  • The full system outperforms all baseline configurations across precision, recall, and F1 in the 41-object ablation scene.Its tracker prevents fragmentation, while geometric and semantic fusion helps recover from momentary detection failures.
  • Appearance examples assign new IDs to introduced objects while retaining nearby identities, and disappearance examples preserve instance IDs after objects leave view.
  • The 4D scene graph provides more reliable spatial reasoning, clearer temporal inference, and more accurate instance-level retrieval than raw video.

G. Visual Language Navigation

SuperMap uses its real-time 4D scene graph to support language-guided navigation through spatial relationships and temporal object information. Demonstrations include ambiguous-target selection and relational retrieval, with scene-graph reasoning outperforming raw video on structured queries.

  • The robot navigates in unseen environments using natural-language commands grounded in spatial relationships among 3D objects.
  • Scene-graph input yields more reliable spatial reasoning, clearer temporal inference, and more accurate instance-level retrieval than raw video.
  • The robot selects the correct target among four visually identical whiteboards by reasoning over their relative spatial arrangement.
  • The robot retrieves the fridge–painting relationship to reach the intended adjacent destination.
  • Pose estimation runs at 10 Hz, while 2D segmentation, 3D mapping, and 4D scene-graph updates run at 1 Hz, 3 Hz, and 5 Hz.

1) 2D-Only Tracking vs. 2D–3D Tracking:

The 2D–3D tracker uses geometric cues to preserve object identities when viewpoint changes or objects move, whereas 2D-only tracking fragments identities and repeatedly loses targets.

  • The 2D–3D tracker refines 2D box states using estimated visibility and projections of 3D object centroids.
  • Under large viewpoint changes, 2D-only tracking fragments nearby-chair and table identities, while 2D–3D tracking preserves temporally coherent IDs.
  • For a moving person, the 2D-only baseline repeatedly initializes new tracks, whereas 2D–3D tracking maintains one continuous identity.
  • Joint geometric and semantic checks remove stale points and projection outliers, producing an up-to-date map in a dynamic scene.

B. Qualitative Instance-Level Segmentation Results

Qualitative evaluations show that SuperMap preserves instance identities, handles long-term appearance and disappearance events, and produces cleaner maps through joint consistency checks. Its persistent object-level representation supports richer temporal scene understanding at moderate computational cost.

  • SuperMap better preserves instance identities over time than ConceptGraphs and HOV-SG, which more frequently fragment physical objects into multiple instances.The compared CLIP-based instance features can become less discriminative over long sequences, increasing foreground-background confusion.
  • The 2D–3D tracker preserves consistent tracks across significant viewpoint changes and dynamic-object motion, unlike the 2D-only baseline.
  • SuperMap identifies newly appeared and disappeared objects and updates the spatio-temporal map accordingly across repeated environment traversals.
  • The system updates pose estimation at 10 Hz, dense 3D mapping at 3 Hz, 2D segmentation at 1 Hz, and the 4D scene graph at 5 Hz.
  • Persistent object representations, temporal associations, and a 4D scene graph trade minimal memory usage for temporally consistent mapping and richer scene understanding.
  • Combining geometric and semantic consistency checks removes stale points and projection outliers more completely than either no check or geometric-only checking.

B. 3D-aware Instance Tracking

SuperMap augments 2D tracking with 3D geometric cues to associate observations with existing map instances and maintain identities through viewpoint changes and scene evolution. Its instance layer combines local map context, depth consistency, track updates, mask-based 3D reconstruction, and semantic consistency checks.

  • The tracker preserves ByteTrack’s 2D state representation and motion model while incorporating geometric cues into tracklet updates.
  • A 3D-aware instance tracking mechanism improves matching between new bounding boxes and existing map objects.
  • The system incorporates newly appeared buckets, carts, and safety signs into the object-level map, while removing plants, trash cans, and chairs once absent.
  • The instance-association pipeline gathers nearby map objects, checks geometric consistency using depth, updates tracks from 3D map cues, and associates detections.
  • After association, SuperMap generates instance masks, unprojects them into 3D using depth and odometry, and performs semantic consistency checks before updating the map.
Loading 2608.22896v1…