PaperlayerView exact revision on arXiv

Source-linked AI summary

HIGS: Hierarchical Implicit Grids for Joint Geometric and Semantic Scene Understanding

Hanwen Cao, Wenqiang Wu, Kuang-Ting Tu, Mathias Otnes, Jeffrey Delmerico, Rui Wang, Yulun Tian, Nikolay Atanasov

arXiv:2609.38620v1cs.CVcs.RO

TL;DR

Neural scene representations commonly emphasize geometry, while semantic information and efficient large-scale back-end optimization remain challenges. HIGS uses hierarchical grids, feature queries, and submaps to jointly represent geometric and semantic information. The paper reports improved computation and memory efficiency and strong performance in SDF construction and open-vocabulary grounding on large-scale real-world benchmarks.

  • Problem

    Most existing neural scene representations focus on geometry, while semantic information for high-level understanding and computational efficiency in large-scale back-end optimization remain challenges.

    Source: §1 · p. 1

  • Method

    HIGS uses hierarchical grids, query and decoding functions, and submap-based hierarchical optimization for joint geometric and semantic feature embedding.

    Source: §1 · p. 2

  • Results

    HIGS reports up to 40× speedup for SDF construction and over 20× memory compression with 2%−3% accuracy loss for object grounding, whose accuracy increases by up to 15 percentage points using SDF features.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HIGS demonstrates joint geometric and semantic scene modeling in SDF construction and open-vocabulary object grounding on large-scale real-world benchmarks.

    Source: §1 · p. 2

    The stated linear least-squares special case assumes a linear decoder and a quadratic cost function.

    Source: §4.2 · p. 6

Abstract

from arXiv · show

Neural implicit representations have had a significant impact on scene reconstruction by enabling robots to build continuous, differentiable, and high-fidelity 3D maps. Most existing works focus on geometric reconstruction and lack semantic information for high-level spatial understanding and task planning. Also, as the scale and complexity of the environment increase, neural representations face the challenge of maintaining computational efficiency in back-end optimization. To resolve these two challenges, we introduce a hierarchical neural field that leverages multiresolution submaps to achieve an efficient and scalable implicit representation, and a unified query and decoding mechanism to support both geometric and semantic features. More specifically, the learnable map features can be converted to the output with the query and decoding process for both training and inference. For large-scale representation, we decompose a scene into overlapping submaps and do hierarchical optimization within each local submap, thus enabling scalable computation. To further improve efficiency, we design feature encoders that predict initial hierarchical grid features to substantially reduce the time needed to optimize the submap features from scratch. To correct estimation drift among submaps, we align and fuse them entirely within the implicit feature space, leading to substantial acceleration by avoiding the need to decode the final output. Building upon this efficient hierarchical representation, we embed both geometric features and vision-language latent features into the map, and demonstrate it on both Signed Distance Field (SDF) construction and open-vocabulary object grounding. Our approach significantly improves computation and memory efficiency, maintains high estimation accuracy, and endows the robot with spatial awareness on large-scale real-world benchmarks.

1 Introduction

HIGS addresses the lack of semantic information in neural scene representations and the computational bottlenecks of large-scale back-end optimization. It unifies hierarchical geometric and semantic feature mapping, and reports gains in reconstruction speed, memory use, and object-grounding accuracy.

1 Introduction

  • Neural scene representations offer continuous, differentiable modeling, but most focus on geometry and large-scale back-end optimization can become a computational bottleneck.The introduction contrasts low-level geometric tasks with semantic understanding needed for tasks such as object search and physical interaction.Source: §1 · p. 1
  • Multiresolution grids and hierarchical optimization support geometric and semantic scene models, with implicit-space optimization used for local submaps and global fusion.The approach avoids decoding final outputs during implicit feature-space alignment and fusion.Source: §1 · pp. 1–2
  • HIGS is a unified hierarchical 3D representation and optimization approach for jointly embedding geometric and semantic features.The method is applied to SDF construction and open-vocabulary object grounding.Source: §1 · p. 2
  • HIGS uses pretrained encoders for hierarchical grid-feature initialization, submap decomposition, and hierarchical optimization in latent feature space for submap alignment.These techniques are designed to improve mapping efficiency.Source: §1 · p. 2
  • Geometric features improve object-grounding accuracy, while semantic features can improve submap alignment.The authors describe these as complementary benefits of combining feature types.Source: §1 · p. 2
  • SDF construction gains up to 40× speedup, while object grounding compresses memory over 20× with 2%−3% accuracy loss and gains up to 15 percentage points using SDF features.These results are reported on large-scale real-world benchmarks.Source: §1 · p. 2

2 Related work

Prior work develops neural implicit representations, language-aware scene understanding, and submap-based mapping, but the paper identifies remaining challenges in large-scale reconstruction, continuous spatial querying, and efficient alignment. HIGS is positioned as a joint geometric-semantic feature representation with feature-space submap alignment.

2.1 3D neural implicit representations

  • Neural implicit methods progress from coordinate-and-MLP representations to grid, factorized, point-based, and hierarchical features, yet high-fidelity, efficient, globally consistent large-scene reconstruction remains challenging.HIGS extends hierarchical implicit grids to geometry, semantic, and language features and explores their synergies in reconstruction and object grounding.Source: §2.1 · p. 3

2.2 Scene understanding

  • Language-aware methods include dense feature retrieval, scene graphs, point-cloud grounding, language-conditioned masks, and object-map classification, with different levels of spatial and relational representation.HIGS focuses on efficient joint geometric-semantic representation and uses implicit features for spatial reasoning; geometric features improve grounding results.Source: §2.2 · p. 3

2.3 Submap-based neural SLAM

  • Submap methods align local maps through pose refinement, rendering losses, ICP, or other strategies, while existing approaches may require geometry reconstruction for alignment.HIGS instead aligns submaps through hierarchical feature-space optimization and reports that semantic features can complement geometry in alignment.Source: §2.3 · p. 3

3 Overview

The overview frames spatial scene understanding as modeling both geometry and semantics through two tasks: SDF estimation and open-vocabulary object grounding. HIGS builds an implicit map from RGB-D images, with grid-feature initialization and task-specific use of the learned field.

3 Overview

  • SDF estimation predicts signed distance at 3D query points from an RGB-D-built implicit map, and marching cubes can use those predictions for scene construction.Construction quality is evaluated by Chamfer distance between point clouds sampled from the constructed and ground-truth meshes.Source: §3 · p. 4
  • Open-vocabulary grounding localizes a text-described object in a 3D bounding box, using an RGB-D-built map of vision-language features.Descriptions may combine attributes or spatial relationships; evaluation uses accuracy at specified IoU thresholds against ground-truth boxes.Source: §3 · p. 4
  • HIGS maps features through a query-and-decoding process, initializes grid features with an encoder, and supports both off-the-shelf grounding by feature queries and a model using SDF features to guide box prediction.The overview also describes optimizing sensor poses and aligning submaps with latent hierarchical grid features for SLAM.Source: §3 · p. 4

4 Methodology

HIGS represents scenes with multiresolution feature grids and a shared query-and-decoding process for geometric and vision-language outputs. It optimizes grid features with a fixed pretrained decoder and uses learned hierarchical initialization to reduce optimization from scratch.

4.1 Hierarchical 3D grids

  • Multiresolution grids combine per-level features through trilinear interpolation and a decoder to produce continuous outputs such as SDF or vision-language features.Dense grids store features throughout space for SDF, while sparse grids store near-surface features for vision-language fields.Source: §4.1.1 · p. 4

4.1.1 Overview

  • Local mapping lifts image features into 3D and optimizes grid features for observed points while keeping the offline-trained decoder fixed.The mapping objective sums observation-specific costs over transformed point-cloud observations.Source: §4.1.1 · p. 5

4.1.2 Cost functions

  • SDF costs supervise near-surface values and bound free-space values, while vision-language costs use on-surface feature similarity.The lower-bound violation receives an exponential penalty and the upper-bound violation a linear penalty; Eikonal regularization is omitted because it slowed optimization with limited accuracy impact.Source: §4.1.2 · p. 5

4.2 Hierarchical feature initialization

  • Learned encoders initialize each level's grid features from residuals of prior levels, then the initialized features are refined by minimizing the mapping objective.The encoders are applied recursively; pooled inputs are processed by a 3D CNN for dense SDF grids or a Fast Point Transformer for sparse vision-language grids, followed by an MLP.Source: §4.2 · pp. 6–7

4.3 Open-vocabulary object grounding

HIGS uses geometric and vision-language grid features to ground text queries in 3D bounding boxes. SDF-guided attention supplements semantic features with surface structure, while queried and decoded map features can also support existing grounding models without retraining.

4.3 Open-vocabulary object grounding

  • The grounding model takes grid features and query text to predict a queried object's 3D bounding box, including descriptions with attributes or spatial relations.Grid neighborhoods permit O(1) queries, unlike point-cloud approaches that may require KNN search.Source: §4.3.1 · p. 7, §4.3 · p. 7

4.3.1 SDF-guided object grounding

  • SDF features approximate grid-to-surface projections and guide positional attention, addressing the limited fine-grained spatial information in uniformly distributed grid coordinates.The projected coordinates are estimated from SDF features with an MLP and incorporated into positional embeddings.Source: §4.3.1 · p. 7
  • Queried and decoded HIGS features can feed existing grounding models in training-free inference, with reported substantial memory compression and small accuracy loss.The passage gives Locate-3D as an example of an existing model using pointwise CLIP and DINO features.Source: §4.3.2 · p. 8

4.4 Extension to SLAM

The SLAM extension jointly optimizes camera poses and hierarchical submaps, then aligns and fuses local maps in implicit feature space. Incremental pose and map updates and multilevel alignment culminate in a global representation decoded from averaged submap features.

4.4.1 Camera pose optimization

  • At each time step, HIGS adds observation costs and alternates tracking, which optimizes the current pose with features fixed, and mapping, which optimizes features with poses fixed.In the incremental setting, encoder initialization is disabled and recent and randomly sampled historical frames update the submap features.Source: §4.4.1 · p. 8

4.4.2 Global submap alignment and fusion

  • Overlapping submaps are aligned with correspondence-free feature costs, using coarser-level solutions to initialize progressively finer alignment.The method jointly optimizes submap poses for overlapping pairs.Source: §4.4.2 · p. 9
  • An optional final stage refines submap poses against predicted SDF or vision-language outputs, typically with only a few iterations.The paper reports that this preserves computational efficiency relative to methods aligning directly with SDF values.Source: §4.4.2 · p. 9
  • After alignment, HIGS averages features from contributing submaps and decodes the average into a global scene representation; this fusion scheme applies to different feature types.Each submap contributes when the query coordinate lies inside its bounding box.Source: §4.4.2 · pp. 9–10

5 Evaluation

The evaluation tests HIGS on scene construction, open-vocabulary grounding, pose estimation, outdoor mapping, and design ablations. Results show efficient reconstruction and compressed semantic maps, with performance depending on feature type and alignment stage.

5.1 Scene construction

  • On ScanNet, HIGS achieves either the best or second-best reconstruction results on all scenes; evaluation uses ground-truth poses with pose optimization disabled and reports Chamfer-L1 and F-score.HIGS is optimized for 20 epochs; iSDF and Neural Points are each optimized for 100 epochs.Source: §5.1 · p. 10, Table 1 · p. 11
  • At 0.5 m resolution, the level 1 encoder captures rough geometry and free-space SDF, while level 2 adds details such as the sofa and table.The qualitative comparison shows the initialization already captures much of the scene geometry, with remaining errors and missing details improved after 20 optimization epochs.Source: §5.1 · p. 10, Figure 7 · p. 10, Figure 8 · p. 10

5.2 Scene understanding

  • HIGS grids provide over 20× memory compression with 1%–3% overall accuracy loss, while adding SDF features improves grounding accuracy by over 15% at IoU 0.5.The evaluation uses ScanRefer, Sr3D, and Nr3D, reporting Top-1 accuracy at IoU thresholds 0.25 and 0.5.Source: Table 2 · p. 11, §5.2 · p. 12

5.3 Pose estimation

  • For local pose optimization, SDF features substantially improve over mapping only, whereas CLIP features have little effect and can increase rotation errors.The experiment uses colored ICP odometry and optimizes poses over sliding windows of 10 frames.Source: §5.3 · p. 12, Table 3 · p. 13
  • HIGS submap alignment is more robust to severe misalignment than MIPS, VFPP, and ICP, and is significantly faster while achieving best or second-best accuracy.With small initial errors, all methods align accurately; CLIP features can help submap alignment, especially rotation, when submap features are frozen.Source: §5.3 · p. 12, Table 4 · p. 13, Figure 10 · p. 13

5.5 Ablation studies

  • Pretrained encoder and decoder initialization speeds training-loss optimization, especially when both are used; after 100 epochs, all variants converge in training loss.The encoders are pretrained only on synthetic Replica scenes, which the authors attribute as a possible cause of Full's slight reconstruction F-score degradation versus No-E.Source: §5.5 · pp. 13–14, Table 6 · p. 14

5.4 Evaluation on large-scale outdoor scenes

  • On Newer College, HIGS is comparable to other methods on Quad-Easy and slightly worse than PIN-SLAM but on par with KISS-ICP on Maths Institute; it improves over point-to-point ICP.Comparing Odometry with Full also validates the submap optimization approach for global consistency.Source: §5.4 · p. 13, Table 5 · p. 14

5.5 Ablation studies

  • The full coarse-to-fine alignment with final SDF alignment achieves the smallest rotation and translation errors, while coarse-only alignment yields the largest errors.The final SDF stage requires decoding and increases computation time; adding fine and final decoded CLIP features brings no significant improvement over coarse-only alignment.Source: §5.5 · p. 14, Table 7 · p. 14

6 Conclusion and future work

HIGS unifies hierarchical 3D representation and optimization for joint geometric and semantic embedding. Experiments across indoor and outdoor datasets demonstrate scene construction and open-vocabulary grounding, while task planning remains a proposed direction for future work.

6 Conclusion and future work

  • HIGS combines hierarchical 3D grids, query and decoding functions, pretrained encoder initialization, submap decomposition, and hierarchical latent-feature alignment for efficient mapping.The work reports geometric features aiding semantic understanding and semantic features helping submap alignment.Source: §6 · p. 14

A Proofs

The proof expresses queried features as linear functions of grid features, then shows that a linear decoder yields a least-squares problem with a normal-equation solution.

A Proofs

  • For a linear decoder, each level's queried feature is represented by a matrix applied to that level's grid features, making the output a weighted sum across levels.Concatenating observations gives a least-squares problem through the model Jacobian with respect to level-l features, solved using normal equations.Source: A Proofs · p. 17

B.1 Map decoders

The appendix describes separate SDF and vision-language decoders, trained offline with shared decoder parameters and scene-specific grid features.

B.1 Map decoders

  • The SDF decoder is a single-layer MLP with hidden dimension 64, trained jointly with scene grids on six Replica scenes for 1200 epochs using a coarse-to-fine schedule.The fine-level features are activated after 200 epochs, and the scene grids are discarded after training.Source: §B.1 · p. 17
  • The vision-language decoder is a five-layer MLP trained on 1114 Sr3D, Nr3D, and ScanRefer scenes in chunks of 100 for 40 epochs per chunk.Each scene has dedicated grid features; the resulting features are used to train the object-grounding network.Source: §B.1 · p. 17

B.2 Map initialization encoders

The subsection describes separate initialization encoders for SDF and vision-language grid features. They predict hierarchical grid features using modality-specific networks and training procedures.

B.2 Map initialization encoders

  • The SDF encoder processes voxelized features with a two-layer 3D CNN and predicts each target feature grid using a shared two-layer MLP with hidden dimension 16.CNN feature channels double after each layer; all networks use ReLU.Source: §B.2 · p. 17
  • SDF encoders are trained sequentially from coarse to fine, using previously trained levels to compute features needed for the training loss.Training simulates pose noise with 1 deg rotation and 1 cm translation standard deviations; each encoder uses Adam at 10^-3 for 1000 epochs.Source: §B.2 · p. 18
  • The vision-language encoder embeds point features, distributes them to neighboring grid vertices with trilinear weights, and uses max pooling, an FPT attention block, and an MLP to predict latent grid features.It interpolates features from both levels, concatenates them into a 128-dimensional vector, and uses a pretrained decoder to predict raw point features.Source: §B.2 · p. 18
  • Vision-language encoders for both levels are trained jointly with a frozen decoder and cosine similarity loss for 3000 epochs using Adam at 10^-3.The loss weights are 0.25 for coarse-level prediction and 1 for prediction from both coarse and fine features.Source: §B.2 · p. 18

B.3 Object-grounding model

The object-grounding model uses distilled 3D-JEPA features to train encoders, then post-trains them jointly with the Locate-3D decoder. It is trained on annotations from three datasets.

B.3 Object-grounding model

  • The encoders distill interpolated 3D-JEPA features onto 3D grids, skipping Locate-3D’s self-supervised pretraining stage [Arnaud et al. 2025].The encoder uses the FPT architecture, following Locate-3D’s number of attention blocks and latent feature dimensions; its features are then jointly post-trained with the Locate-3D decoder.Source: §B.3 · p. 18
  • Without post-training, grounding accuracy is Acc@25 = 0.19 and Acc@50 = 0.08.The authors state that joint post-training is needed because accuracy without it is very low.Source: §B.3 · p. 18
  • The grounding network is post-trained for 20 epochs on annotations from Sr3D, Nr3D, and ScanRefer [Achlioptas et al. 2020] [Chen et al. 2020].Training uses a learning rate of 3 × 10^-5 and batch size 48, with stochastic weight averaging during the last 10 epochs [Izmailov et al. 2018].Source: §B.3 · p. 18
Loading 2609.38620v1…