PaperlayerView exact revision on arXiv

Source-linked AI summary

ActiveLang: Active Open-Vocabulary 3D Mapping with Semantic-Uncertainty-Guided Exploration

Liyan Chen, Hairong Yin, Huangying Zhan, Yi Xu, Raymond A. Yeh, Philippos Mordohai

arXiv:2610.09518v1cs.CVcs.RO

TL;DR

Robots in unfamiliar environments need maps that capture both scene geometry and semantics, while closed-vocabulary mapping may not cover concepts needed for new tasks. ActiveLang combines online semantic adaptation with autonomous viewpoint selection, and outperforms online and offline passive mapping baselines in 2D and 3D open-vocabulary segmentation.

  • Problem

    Closed-vocabulary mapping relies on predefined labels that may not cover concepts needed in unfamiliar environments or new tasks, motivating language-queryable 3D maps.

    Source: §1 · pp. 1–2

  • Method

    ActiveLang jointly updates geometry, appearance, language features, and semantic uncertainty in a dual-Gaussian map, adapting language features online and using uncertainty to guide viewpoint selection.

    Source: §1 · p. 2, §3.1 · p. 5, §3.2 · p. 6

  • Results

    ActiveLang outperforms existing online and offline passive mapping baselines in 2D and 3D open-vocabulary segmentation while matching geometry-focused mappers in visual quality at lower computational cost.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The paper concludes that actively exploring scenes can build language-annotated 3D maps more efficiently.

    Source: §5 · p. 9

    The system assumes ground-truth poses and has minor surface-detail trade-offs from camera-orientation constraints.

    Source: §5 · p. 9

Abstract

from arXiv · show

As robots increasingly assist humans with diverse tasks, they need both geometric and semantic understanding of their surroundings. Moreover, robots often operate in unfamiliar environments and take on new tasks without knowing the relevant concepts ahead of time. This motivates language-annotated 3D maps that support open-vocabulary scene understanding and human-robot interaction. We introduce ActiveLang, an autonomous system for active open-vocabulary 3D mapping with semantic-uncertainty-guided exploration. ActiveLang performs online language-feature adaptation on a compact dual-Gaussian representation to jointly reconstruct scene geometry, appearance, and open-vocabulary semantics with modest memory overhead. Its planner efficiently selects informative viewpoints, enabling effective mapping with fewer observations and lower computational cost. Experiments on Replica and ScanNet++ demonstrate substantial improvements in 2D and 3D open-vocabulary segmentation over both online and offline baselines, highlighting that actively exploring scenes builds language-annotated 3D maps more efficiently.

1 INTRODUCTION

Robots need maps that connect scene geometry and semantics to natural-language concepts, especially in unfamiliar environments and new tasks. ActiveLang addresses the challenges of adapting dense open-vocabulary semantics online while selecting useful viewpoints.

1 INTRODUCTION

  • Closed-vocabulary maps rely on predefined labels that may not cover unfamiliar environments, new tasks, or the terms people use, motivating language-annotated 3D maps.Such maps associate spatial scene representations with language features, supporting natural-language queries.Source: §1 · pp. 1–2
  • Active mapping lets the robot determine its sensor trajectory, unlike passive mapping, where that trajectory is imposed externally.Effective active exploration can limit acquisition effort while producing a map of the same quality as passive mapping.Source: §1 · p. 2
  • Open-vocabulary mapping must preserve distinctions across ambiguous language and visual embeddings while compressing high-dimensional features from an evolving, unevenly observed stream.Online compression can favor frequently observed content and blur semantic distinctions.Source: §1 · p. 2
  • Viewpoint selection must balance newly revealed surfaces against revisits that may resolve semantic ambiguity, rather than optimizing geometric coverage alone.These objectives can conflict because newly visible surfaces may have weak or inconsistent language features.Source: §1 · p. 2
  • ActiveLang couples incremental reconstruction of geometry, appearance, and language features with autonomous viewpoint selection to build dense, language-queryable maps.Its compact semantic representation uses online adaptation and prototype-based balancing to preserve feature modes while reducing bias and confusion.Source: §1 · p. 2

2 RELATED WORK

The related work is organized by whether mapping is offline, online, or active, and by the scope of its semantic vocabulary. Prior approaches cover parts of this space, while active open-vocabulary methods often focus on objects or frontier-based exploration.

2 RELATED WORK

  • Prior work is classified by mapping mode—offline, online, or active—and semantic scope, from none or object-level to closed or open vocabulary.The comparison covers methods that address appearance and geometric mapping.Source: §2 · p. 3, Table 1 · p. 3
  • Active-mapping work uses NeRF, hybrid, and Gaussian Splatting representations; GS uncertainty predictors are described as faster and more scalable, while AREA3D maps only RGB-D.Many cited approaches combine uncertainty predictors with additional representations.Source: §2 · p. 3
  • Active closed-vocabulary approaches include methods limited to 8 or 15 classes, 2D motion, or object-only mapping, while ActiveSGM handles over 100 labels across foreground and background.These methods avoid the cross-modal language-to-image label mapping challenge of open-vocabulary settings.Source: §2 · p. 3
  • Offline open-vocabulary methods include LangSplat, which trains its scene-specific autoencoder on all frames, whereas online methods adapt semantics sequentially.Online examples include moving-average latent updates and trainable autoencoders that compress CLIP features for language Gaussians.Source: §2 · pp. 3–4
  • The cited active open-vocabulary systems often use frontier exploration or map objects rather than dense object-and-surface semantics guided by a semantic-uncertainty criterion.OpenFrontier supports language-driven tasks without dense 3D reconstruction; RayFronts ignores background categories, and RoboAtlas considers object instances.Source: §2 · p. 4

3 METHOD

ActiveLang couples incremental open-vocabulary 3D mapping with autonomous viewpoint selection. It uses paired Gaussian representations, online semantic adaptation, and exploration guided by geometric coverage and semantic uncertainty.

3 METHOD

  • ActiveLang couples incremental reconstruction of geometry, appearance, and dense open-vocabulary semantics with autonomous selection of the next observation based on the evolving map.New observations update the map, which then guides subsequent viewpoint selection.Source: §3 · p. 4

3.1 3DGS REPRESENTATION

  • Paired RGB-D and semantic Gaussians share centers but retain branch-specific parameters; RGB-D reconstruction updates geometry, while semantic optimization also learns uncertainty.Online 3D smoothing and 2D Mip filtering address scale-dependent rendering artifacts.Source: §3.1 · p. 5

3.2 ONLINE OPEN-VOCABULARY SEMANTIC MAPPING

  • OLAE compresses frozen 768D features through 32D codes into stored 15D features, while keyframes and supervision gating focus updates on informative views.Views with insufficient feature diversity or extreme close-ups are excluded from semantic adaptation but remain usable for RGB-D reconstruction.Source: §3.2 · pp. 5–6
  • A class-agnostic prototype bank balances recurring feature modes, while a separation penalty discourages initially dissimilar modes from becoming overly similar after compression.Equal prototype weighting reduces dominance by large image regions, and current pixel samples preserve within-mode variation.Source: §3.2 · pp. 5–6

3.3 SEMANTIC-UNCERTAINTY-GUIDED EXPLORATION

  • Viewpoint scoring considers travel cost, novel-pixel coverage, and semantic uncertainty, adding uncertainty only after mean candidate coverage reaches the sufficient-map threshold.Below a mean coverage of 0.5, selection uses travel cost and novel-pixel count; the condition is re-evaluated at each planning step.Source: §3.3 · pp. 6–7
  • The planner normalizes candidate criteria and uses an RRT-based planner for the selected viewpoint; camera motion fixes roll and pitch to keep observations upright.The allowed 4-DoF motion comprises translation and yaw.Source: §3.3 · p. 7

4 EXPERIMENTS

Across Replica and ScanNet++, ActiveLang improves open-vocabulary mapping over passive baselines while remaining competitive in reconstruction quality. Ablations show that online semantic adaptation, semantic-aware viewpoint selection, and keyframe management each affect different aspects of map quality.

4 EXPERIMENTS

  • Evaluations use eight Replica and eight ScanNet++ scenes, held-out SLAM views, zero-shot novel views, and reconstructed 3D maps; no active open-vocabulary baseline is available.LangSplat and Online Language Splatting are the primary passive open-vocabulary baselines.Source: §4 · pp. 7–8, §4.1 · p. 8

4.1 ACTIVE OPEN-VOCABULARY MAPPING

  • ActiveLang consistently improves 2D segmentation and localization over passive baselines across both datasets, and achieves the highest 3D mIoU across both datasets.The 3D improvement over OnlineLangSplat is generally smaller than the 2D improvement.Source: §4.1 · p. 8

4.2 RECONSTRUCTION QUALITY ON REPLICA

  • On Replica, ActiveLang improves PSNR by more than 4 dB and reduces depth error by approximately 6 cm versus LangSplat and OnlineLangSplat.It surpasses ActiveSGM and performs comparably to ActiveGAMER; adding dense semantics and semantic-aware exploration does not substantially compromise appearance or depth reconstruction.Source: §4.2 · pp. 8–9
  • ActiveLang has the best geometric accuracy among open-vocabulary baselines and improves completeness over OnlineLangSplat, while geometry-focused active mappers achieve higher completeness.Removing the 4-DoF constraint yields modest geometric gains but lowers 2D mIoU by 8.6% and 3D mIoU by 5.0%.Source: §4.2 · p. 9

4.3 ABLATION STUDY

  • Removing prototype-based adaptation lowers 2D novel-view mIoU from 40.91% to 38.44% and 3D mIoU from 27.34% to 21.12%.Removing semantic supervision gating causes similar reductions; the prototype ablation evaluates the full mechanism, not the separation term alone.Source: §4.3 · p. 9, Table 4 · p. 9
  • Removing semantic uncertainty from viewpoint selection lowers 2D mIoU from 40.91% to 39.88% and 3D mIoU from 27.34% to 27.09%.The results indicate that semantic uncertainty adds guidance beyond geometric coverage.Source: §4.3 · p. 9
  • Using only local keyframes raises depth error from 0.85 to 1.18 cm and lowers PSNR from 33.07 to 32.19 dB, while semantic metrics remain nearly unchanged.The results suggest that global keyframes mainly support ongoing geometric and appearance refinement.Source: §4.3 · p. 9, Table 4 · p. 9

5 CONCLUSION

ActiveLang combines active exploration with online semantic adaptation in a compact dual-Gaussian representation. Across Replica and ScanNet++, it outperforms online and offline passive mapping baselines in 2D and 3D segmentation while matching geometry-focused mappers in visual quality at lower computational cost.

5 CONCLUSION

  • ActiveLang assumes ground-truth poses and has minor surface-detail trade-offs from camera orientation constraints; future work targets unknown poses and the geometric gap.The conclusion reports results on Replica and ScanNet++ and compares against online and offline passive mapping baselines.Source: §5 · p. 9

A APPENDIX

The appendix adds experimental results, analyses of image–text feature misalignment and compression-induced mode collapse, and reproducibility and implementation details.

A APPENDIX

  • The appendix supplements experiments, investigates image–text feature misalignment and feature-mode imbalance with compression-induced mode collapse, and documents datasets, hardware, software, and module hyperparameters.Source: A APPENDIX · p. 14

A.1 ADDITIONAL RESULTS

Additional experiments report per-scene semantic and rendering results, qualitative comparisons, and exploration trajectories for Replica and ScanNet++. In Replica office1, uncertainty declines overall as reconstruction and semantic measures improve, though semantic progress is uneven.

A.1 ADDITIONAL RESULTS

  • ActiveLang outperforms baselines in most Replica scenes, particularly on 2D semantic metrics.The appendix reports per-scene mapping and novel-view rendering results using ten scene-specific query categories.Source: §A.1 · p. 14, Table A2 · p. 15
  • In Replica office1, exploration reaches 37.30 dB PSNR, 97.80% geometric completeness, and 40.38% 2D mIoU; semantic quality temporarily dips at intermediate checkpoints.Median rendered language uncertainty decreases overall, but its distribution remains broad, indicating uneven resolution across evaluation views.Source: §A.1 · pp. 14–15, Figure A1 · p. 14
  • On Replica room0, ActiveLang reduces spurious “sofa” responses and produces more consistent responses to “window” than the compared methods.The PCA colors are fitted independently for each method and are not directly comparable.Source: §A.1 · p. 15, Figure A2 · p. 16
  • On ScanNet++, ActiveLang achieves the highest 2D and 3D mIoU in all eight scenes and the highest localization accuracy on SLAM views.It has the best PSNR, SSIM, and LPIPS in three of eight scenes; novel-view synthesis remains competitive overall.Source: §A.1 · p. 16, Table A4 · p. 16
  • ActiveLang trajectories extend beyond predefined SLAM and novel-view evaluation paths, covering a broader range of viewpoints throughout each room.The trajectory panels include a 1 m scale bar.Source: Figure A3 · p. 17, §A.1 · p. 17

A.2 ANALYSIS OF IMAGE–TEXT ALIGNMENT IN SEMANTIC FEATURES

The analysis tests whether category-level OpenCLIP image features align with one another and with category-name text embeddings. Even features drawn from ground-truth semantic regions remain ambiguous, motivating care when constructing language-queryable maps.

Experimental setup

  • The analysis averages OpenCLIP image embeddings over labeled sub-images by category and encodes each corresponding category name with the text encoder.It retains the ten most frequent categories in Replica office0.Source: §A.2 · pp. 17–18

Similarity matrices

  • Across ten frequent Replica office0 categories, the smallest off-diagonal image-embedding similarities exceed 0.7, and rug, blinds, table, and floor match wall text better than their own labels.The analysis averages features from ground-truth semantic regions across 2,000 frames; Figure A4(a) and (b) show the image–image and image–text matrices.Source: §A.2 · p. 18, Figure A4 · p. 18

Interpretation and scope

  • Ground-truth region labels do not ensure that nearest-text matching recovers the category name, but the analysis excludes individual-region variation, compression, and online adaptation.It evaluates category-averaged features before compression.Source: §A.2 · p. 18

A.3 SEMANTIC PRESERVATION IN ONLINE OPEN-VOCABULARY MAPPING

Online compression can weaken semantic distinctions in an evolving, uneven stream of observations. ActiveLang’s adaptation and supervision mechanisms target this issue; diagnostics show selective category recovery and higher purity in later-created map portions, with evaluation limited to spatially matched map regions.

Definition and evaluation protocol

  • Purity measures semantic agreement only on spatially matched map centers; centers farther than 5 cm from the reference cloud are excluded rather than counted as semantic errors.It therefore does not measure scene coverage.Source: §A.3 · pp. 18–19

Feature-mode imbalance and compression-induced collapse

  • The frozen autoencoder reduces lamp accuracy from 55.3% to 0.7%, table from 84.9% to 21.5%, and picture from 96.7% to 27.3%.Pixel-averaged reconstruction loss can shift underrepresented categories toward more frequently observed categories; here, underrepresentation refers to the online training distribution.Source: §A.3 · p. 19

Prototype-balanced and separation-preserving adaptation

  • A bounded prototype bank weights retained feature modes equally, while a separation objective discourages excessive latent cosine similarity between sufficiently dissimilar frozen-space prototypes.Current-observation pixels preserve within-mode variation; prototypes represent feature modes, not ground-truth categories, and encoded targets are detached when supervising semantic Gaussians.Source: §A.3 · pp. 19–20

Compression-chain diagnostics

  • Full-stage macro accuracy rises from 44.36% to 45.74% across ten categories; table improves from 27.3% to 55.2% and picture from 63.8% to 76.7% versus OnlineLangSplat.Gains are selective: sofa falls to 0.0%, while lamp and ceiling remain at zero in both methods.Source: §A.3 · p. 19, Table A5 · p. 20

Semantic supervision gating

  • Feature-diversity and median-depth gates control OLAE updates and semantic-Gaussian supervision; repeated diversity rejections trigger a forced OLAE update without bypassing the semantic-map gate.The gates address observations dominated by few feature modes or extreme close-ups with insufficient semantic context.Source: §A.3 · p. 20

Purity by Gaussian creation time

  • ActiveLang exceeds OnlineLangSplat in purity in Groups 5–10; the newest group reaches 55.6% versus 19.5%.Groups are evaluated at the final checkpoint and formed independently by method; they may contain different scene content, so the curves are not longitudinal accuracy trajectories.Source: §A.3 · p. 20, Figure A5 · p. 21

A.4.1 EXPERIMENTAL RESULT REPRODUCIBILITY

The appendix points readers to system, protocol, and implementation details, and states that the code will be released upon acceptance.

A.4.1 EXPERIMENTAL RESULT REPRODUCIBILITY

  • The system is described in Section 3, with experimental protocol details in Section 4.Per-scene results and scene-specific text queries for both datasets are reported in Section A.1.Source: §A.4.1 · p. 21
  • ActiveLang is built on open-source software listed in Section A.4.3.Additional implementation details are in Section A.5.Source: §A.4.1 · p. 21
  • The authors state that their code will be released upon acceptance.Source: §A.4.1 · p. 21

A.4.2 COMPUTE RESOURCES FOR EXPERIMENTS

The appendix documents compute allocation, assets, simulation and data, semantic supervision, keyframes, exploration, baselines, and evaluation protocols. It specifies the implementation and measurement choices used for the experiments.

A.4.2 COMPUTE RESOURCES FOR EXPERIMENTS

  • The mapping GPU grows from about 11 GB at initialization to about 25 GB after Replica exploration, peaking near 35 GB by scene; the language pipeline stays near 6 GB.The two-GPU partition places mapping and planner rendering on one GPU, and language supervision on the other.Source: §A.4.2 · p. 21

A.4.3 LICENSES FOR EXISTING ASSETS

A.5.1 SIMULATION AND DATA

  • The experiments use Habitat-Sim-rendered RGB-D observations, SplaTAM mapping, and dense OpenCLIP features; eight scenes each are used from Replica and ScanNet++.Replica evaluation includes a 180-view novel-view orbit and 200 trajectory views; ScanNet++ uses a 180-view orbit and 50 real capture-path views.Source: §2 · p. 22, §A.5.1 · pp. 22–23

A.5.2 SEMANTIC SUPERVISION GATES

  • Semantic updates require sufficient feature diversity, and extreme close-ups with median valid depth below 0.5 m are rejected; RGB and depth reconstruction remain unaffected.After ten consecutive diversity-based rejections, one OLAE update is forced, but semantic-Gaussian supervision remains gated.Source: §A.5.2 · p. 23

A.5.3 KEYFRAME SELECTION

  • Global RGB-D and semantic keyframes are added every 100 frames from the lowest-scoring 30% of previously unseen views under color PSNR and semantic-feature similarity, respectively.The two criteria contribute to a shared global keyframe pool.Source: §A.5.3 · p. 23

A.5.4 EXPLORATION PLANNER

  • The planner samples candidate views in two stages, first with a coarse occupancy grid and then with a densified grid; candidates are scored with semantic uncertainty once mean view coverage reaches 0.5.Candidates leave the pool below 0.5% novel pixels; ScanNet++ runs took 452–1,458 steps, averaging 889, versus 1,000 passive-baseline poses.Source: §A.5.4 · pp. 23–24

A.5.5 BASELINES

  • Baselines share the RGB-D Gaussian backbone and evaluation code; OnlineLangSplat uses passive views, replay, and neither semantic gates nor global keyframes, while LangSplat follows a passive trajectory.LangSplat uses a scene-specific autoencoder trained before mapping and lacks a separate semantic branch or uncertainty.Source: §A.5.5 · p. 24

A.5.6 EVALUATION PROTOCOL

  • Evaluation measures rendered 2D IoU and localization hits, 3D mIoU and mAcc, geometry accuracy and completion, completion ratio, and RGB/depth rendering quality.The 3D semantic protocol uses opacity-filtered, 1 cm voxel-sampled Gaussians and reports mIoU and mAcc against top-10 ground-truth classes.Source: §A.5.6 · p. 24
Loading 2610.09518v1…