PaperlayerView exact revision on arXiv

Source-linked AI summary

Point-Level Refinable Semantic Mapping for Spatial Understanding

Taimeng Fu, Yi Du, Zitong Zhan, Qiwei Du, Shaoshu Su, Zhipeng Zhao, Chen Wang

arXiv:2610.31476v1cs.RO

TL;DR

Existing semantic maps often append points without revising noisy or outdated points within object instances. This paper uses probabilistic point-level updates with tracking and re-identification to refine maps over time. Experiments report improved mapping accuracy and robustness, alongside navigation and question-answering evaluations.

  • Problem

    Existing maps commonly append points without revisiting them, and dynamic methods update object-level representations without explicitly revising noisy or outdated points within each instance.

    Source: §I · p. 1

  • Method

    The framework updates point confidence from temporal evidence and combines object tracking with feature-based re-identification to refine maps and preserve identities under motion.

    Source: §I · pp. 1–2

  • Results

    Experiments report improved semantic mapping accuracy and robustness to perception errors, while navigation and embodied question answering evaluate the map as online spatial memory.

    Source: §I · p. 2, §V · p. 8

  • Takeaways & Limitations

    The map supports object navigation and embodied question answering as an online spatial memory for robotic agents.

    Source: §I · p. 2

Abstract

from arXiv · show

Open-vocabulary 3D semantic maps have emerged as a key spatial memory representation for long-horizon robot autonomy, enabling robots to reason about objects, their spatial relationships, and past observations across time. However, existing object-centric semantic maps are primarily designed for static environments and rely on an append-only point integration strategy, where newly observed points are accumulated but previously integrated points are rarely reevaluated. Consequently, perception errors accumulate over time, and outdated points persist after objects move or disappear. Although recent dynamic mapping methods update object-level representations in changing scenes, they do not explicitly revise noisy or obsolete points within each object. We propose an online refinable semantic mapping framework that continuously refines object representations through probabilistic point-level confidence updating. Each object point maintains a confidence score that is updated online using multi-view temporal evidence: consistent observations reinforce reliable points, while inconsistent segmentation masks, depth measurements, and visibility cues suppress spurious and outdated points. Combined with object tracking and re-identification to preserve instance identities under motion and relocation, our framework enables object-centric semantic maps to be continuously refined rather than monotonically accumulated. Experiments demonstrate improved mapping accuracy over the previous method by 3.46% on Replica and 2.55% on ScanNet, show robustness in dynamic environments and under perception noise, and provide an effective spatial memory for object navigation and embodied question-answering with success rates of 82.4% and 76.7%.

I. INTRODUCTION

Open-vocabulary semantic maps support robotic spatial reasoning, but existing systems often accumulate points without revising them. The proposed framework refines object points from temporal evidence and maintains identities as objects move or are re-observed.

A. Open-Vocabulary Semantic Mapping

  • The introduction situates open-vocabulary semantic maps as spatial memory for locating task-relevant objects, reasoning about relationships, and retaining context for embodied tasks, [2], [3], [4].Recent systems also connect these maps to large-language-model reasoning and planning,,.Source: §I · p. 1, A. Open-Vocabulary Semantic Mapping · p. 2

I. INTRODUCTION

  • Existing open-vocabulary maps typically append newly observed points and rarely revisit integrated points, despite objects moving, disappearing, or being rearranged.Dynamic mapping methods update objects or relations but may leave noisy or outdated points embedded within an instance.Source: §I · p. 1
  • The framework updates each point’s probabilistic confidence from multi-view temporal evidence, reinforcing consistent points and down-weighting points contradicted by masks, depth, or visibility cues.Points can eventually be removed when evidence indicates they are spurious or outdated.Source: §I · p. 1
  • Object tracking and feature-based re-identification preserve instance identities during in-view motion and after objects leave and re-enter the field of view.The framework is evaluated as online spatial memory for vision-language navigation and embodied question answering.Source: §I · p. 2

B. Dynamic Semantic Map Updating

Dynamic mapping methods address scene changes at object or relation level, while this framework also revises individual points. It represents object points with probabilistic confidence scores for online updating.

B. Dynamic Semantic Map Updating

  • Prior dynamic maps reason about object changes or movable-object relations but do not explicitly revise noisy or outdated points within an otherwise tracked instance.Relation-based updating also depends on a suitable stable anchor and sufficiently complete re-observation; this method maintains instances independently of predefined anchors.Source: B. Dynamic Semantic Map Updating · p. 2

III. METHOD

  • The mapping objective is to use each RGB-D observation both to add object points and revise existing points through detection and visibility cues.Source: §III · p. 2

A. Probabilistic Map Representation

  • The map represents each object instance with 3D points and probabilistic confidence scores that can be revised online.Instance metadata includes label histograms, bounding boxes, observation counts and timestamps, and descriptors for retrieval and re-identification.Source: A. Probabilistic Map Representation · p. 2
  • Log-odds confidence scores allow observation increments to be added directly, providing Bayesian probability updating more efficiently.The point score represents the confidence of that point and is updated probabilistically from observations.Source: A. Probabilistic Map Representation · p. 2

B. Online Semantic Mapping

The online mapping process lifts open-vocabulary detections from posed RGB-D observations into 3D. It then reasons about point visibility to update confidence while leaving occluded or unobserved points unchanged.

1) Open-vocabulary Detection:

  • Vision foundation models generate open-vocabulary detections and segmentation masks from RGB-D observations, with region features and an LLM-generated class prompt.Each detection is lifted to a world-frame point cloud, then DBSCAN retains its largest connected component to filter depth noise and reconstruction artifacts.Source: 1) Open-vocabulary Detection · p. 2

2) Point-level Visibility Reasoning:

  • For associated detections, current-mask-supported points gain confidence, inconsistent points are down-weighted, and points from removed objects can eventually be pruned.Points unobserved in the current frame remain unchanged until further evidence arrives.Source: Figure 2 · p. 3, 2) Point-level Visibility Reasoning · p. 3
  • Visibility reasoning labels projected points as visible, missed, or occluded by comparing their depth with observed pixel depth.Visible points can be revised using detection evidence; missed points may be removed, while occluded points remain unchanged in the current frame.Source: 2) Point-level Visibility Reasoning · p. 3
  • Visibility reasoning and score updates are restricted to nearby map instances, limiting computation to a local subset rather than the global map.Source: 2) Point-level Visibility Reasoning · p. 3

3) Detection-to-Instance Association:

Detections are matched to object instances using semantic and geometric evidence, then point confidence is revised from visibility and consistency with current observations.

3) Detection-to-Instance Association:

  • The matching score sums semantic cosine similarity and 3D IoU, accepting the best instance only when its score exceeds a threshold.If no valid match exists, a new instance is initialized; matched detections update the instance descriptors and label histogram, and its point cloud is voxel-downsampled.Source: 3) Detection-to-Instance Association · p. 3

4) Confidence Score Updating:

  • Visible object points reprojecting inside the associated segmentation mask receive positive evidence and increased confidence.Their confidence increases by the log-odds of the detection confidence.Source: 4) Confidence Score Updating · p. 3
  • Unobserved points remain unchanged, while detectable-but-unmatched instances and depth-contradicted points are penalized, and low-confidence points are pruned.Visibility is used to avoid penalizing instances that may be occluded or weakly visible; depth-contradicted points can reflect object removal or spurious depth points.Source: 4) Confidence Score Updating · pp. 3–4
  • Visible points outside the associated mask receive negative evidence and have their confidence decreased.Source: 4) Confidence Score Updating · pp. 3–4

C. Object Fragment Merging

Periodic instance merging addresses fragmentation when disconnected partial observations of one physical object are initialized as separate instances.

C. Object Fragment Merging

  • Disconnected partial observations of the same physical object can be initialized as separate instances, motivating periodic merging when geometric and visual cues agree.Source: C. Object Fragment Merging · p. 4
  • The rule uses a 3D point-cloud overlap ratio, with Overlap defined following, and applies thresholds.Source: C. Object Fragment Merging · p. 4
  • The merge rule requires agreement in geometry, semantics, and appearance, aiming to correct fragmentation while avoiding false merges.Source: C. Object Fragment Merging · p. 4

D. Dynamic Object Tracking and Re-Identification

The system preserves instance identities during in-view motion with tracking and handles out-of-view relocation through feature-based re-identification.

D. Dynamic Object Tracking and Re-Identification

1) Tracking Moving Objects in the Camera View:

  • BoT-SORT associates detections across frames so a moving object is fused into its existing instance rather than repeatedly initialized as new.Points left at previous locations are removed by confidence updates within the next 1–2 frames when later depth observations contradict them.Source: 1) Tracking Moving Objects in the… · p. 4

2) Re-Identification of Relocated Objects:

  • Removed instances are stored with their ID, bounding box, and visual features in a buffer that prioritizes recently and frequently observed objects.When the buffer exceeds capacity, its lowest-scored entries are discarded.Source: 2) Re-Identification of Relocated… · p. 4
  • Re-identification checks semantic, appearance, and geometric-size consistency; ambiguous matches are skipped, while a unique match across three consecutive attempts restores the stored ID.Source: 2) Re-Identification of Relocated… · p. 4
  • Re-identification is independent of observation order: an object may be rediscovered after disappearing or matched after first appearing as a new instance.Source: 2) Re-Identification of Relocated… · p. 4

E. Integration with VLN framework

The method is integrated into VL-Nav as symbolic-spatial memory for an LLM-based robot agent that queries the map and invokes navigation or question-answering skills.

E. Integration with VLN framework

IV. EXPERIMENTS

The experiments evaluate mapping accuracy and related system behavior, with tests on Replica and ScanNet showing improvements over the previous state of the art. Setup and qualitative examples also examine perception errors and object fragmentation.

IV. EXPERIMENTS

  • Evaluation covers mapping accuracy, component contributions, perception-error robustness, dynamic-object handling, and downstream real-world VLN.The mapping benchmarks use Replica and ScanNet and compare against ConceptGraphs, HOV-SG, and DualMap.Source: §IV · p. 5, B. Mapping Accuracy · p. 5

A. Experimental Setup

  • The perception pipeline combines open-vocabulary detection and mask generation, with refinement parameters calibrated on held-out scenes and then fixed across experiments.It uses YOLO-World, MobileSAM, FastSAM, MobileCLIP, and DINO features.Source: A. Experimental Setup · p. 5

B. Mapping Accuracy

  • Qualitative examples show cross-view suppression of chair-mask outliers, removal of false detections, and merging of disconnected object fragments.The examples concern a thin, hollow chair, a plastic bag mistakenly detected as a pillow, and a table observed in disconnected partial views.Source: Figure 3 · p. 5, Figure 4 · p. 6, Figure 5 · p. 6
  • 3.46% and 2.55% mIoU improvements over DualMap on Replica and ScanNet, respectively, accompany higher F-mIoU and mAcc on both datasets.The reported F-mIoU gains are 5.84% and 3.44%, and mAcc gains are 13.85% and 1.45%, respectively.Source: B. Mapping Accuracy · p. 5

C. Computational Efficiency

The method uses less CPU and GPU memory and achieves lower average time per frame despite adding visibility reasoning and confidence updates. Ablations and error tests examine the roles of point refinement, fragment merging, and temporal consistency.

C. Computational Efficiency

  • The method uses less CPU and GPU memory and achieves lower average time per frame despite visibility reasoning and confidence updates.The reported implementation optimizations include batched object processing, parallel feature extraction, and pipelined perception and mapping.Source: C. Computational Efficiency · p. 6

D. Controlled Ablations

  • Disabling point-level confidence updating raises Duplication from 1.1690 to 1.2568, while disabling fragment merging raises it to 1.3426.Duplication is the average number of mapped instances matched to each ground-truth object.Source: D. Controlled Ablations · p. 6

F. Qualitative Analysis of Common Mapping Errors

E. Robustness to Segmentation Errors

  • Under Gaussian mask-boundary noise with σ = 3 px, the method degrades less than DualMap, with the advantage more pronounced in mIoU and mAcc than F-mIoU.The reported explanation is that temporal and geometric consistency suppress points introduced by segmentation errors; F-mIoU weights large objects more heavily.Source: E. Robustness to Segmentation Errors · p. 6

1) Inaccurate Masks:

Qualitative cases illustrate how refinement addresses mask and detection errors, object fragmentation, and under-segmentation. In dynamic scenes, point-level visibility reasoning removes stale instances, while re-identification recovers many relocated objects' identities.

1) Inaccurate Masks:

  • Cross-view confidence updates suppress chair points introduced when inaccurate masks fill thin, hollow regions.DualMap retains these mask-induced artifacts after fusion.Source: 1) Inaccurate Masks · p. 6

2) Incorrect Detections:

  • Confidence updates remove a false pillow instance created by a plastic-bag detection while preserving the true pillow.The example shows positive evidence accumulating for consistently detected objects and negative evidence for unsupported instances.Source: 2) Incorrect Detections · p. 6

1) Out-of-View Relocation and Re-Identification:

  • After objects are moved, in seven sequential object-navigation tasks, the full system travels 76.41 m, compared with 103.22 m for the ablation without map refinement, which revisits stale object locations.Source: Figure 8 · p. 7

3) Object Fragmentation:

  • When limited-overlap views of a table’s two sides are observed at different times, it is initially reconstructed as two instances; more complete observations later let the method merge them into one.Source: 3) Object Fragmentation · p. 7

4) Under-segmentation:

  • After under-segmentation initially fuses a pillow into a chair, later correct segmentation creates a separate pillow instance and removes erroneous chair points.The change is made through confidence updates.Source: 4) Under-segmentation · p. 7

1) Out-of-View Relocation and Re-Identification:

  • Across five HSSD scenes, the method retains 100% of original objects and 92.3% after relocation, removes all 26 stale instances, and re-identifies 20 relocated objects.Four ambiguous identity matches are conservatively rejected; two objects are too small to detect at their new locations.Source: 1) Out-of-View Relocation and… · p. 7, G. Dynamic Object Handling · p. 7

2) In-View Tracking:

In-view tracking updates moving objects while preserving their instance IDs. Real-world tests show shorter travel after map refinement and successful object navigation and embodied question answering in unseen environments.

2) In-View Tracking:

  • The method tracks and updates 27 of 32 moved objects, an 84.4% success rate, while preserving their instance identities.The remaining five cases fail because objects become too small to detect when moved farther away.Source: 2) In-View Tracking · p. 7

H. Real-World Evaluation on VLN Tasks

1) Benefit of Map Refinement:

  • After environmental rearrangement, the full system completes seven sequential navigation tasks while traveling 76.41 m, compared with 103.22 m without map refinement.Continuous updates remove stale objects during execution, preventing repeated selection and revisits to obsolete locations.Source: 1) Benefit of Map Refinement · p. 7

2) Broader VLN Evaluation:

  • 82.4% object-navigation and 76.7% EQA average success rates demonstrate the map's use as online spatial memory in three unseen environments.The environments are Hallway, Office, and Outdoor.Source: 2) Broader VLN Evaluation · pp. 7–8

V. CONCLUSION

  • The conclusion characterizes the map as an online spatial memory for language-guided robotic agents, supported by mapping, dynamic-object, navigation, and question-answering experiments.The framework represents object points with log-odds confidence updated from multi-view observation evidence.Source: §V · p. 8
Loading 2610.31476v1…