PaperlayerView exact revision on arXiv

Source-linked AI summary

OREN-X: Octree Residual Network for Real-Time Multi-Modal Mapping

Zhirui Dai, Qihao Qian, Dinh Minh Nguyen, Quan-Dung Pham, Kiana Bronder, Carlos Nieto-Granda, Yiyu Chen, Quan Nguyen, Nikolay Atanasov

arXiv:2609.29157v1cs.RO

TL;DR

Separate modality-specific representations multiply memory and computation and cannot exploit synergies among modalities needed for long-horizon autonomy. OREN-X stores and jointly optimizes signed distance, occupancy, radiance, and vision-language fields in one shared octree, with flexible full or compressed vertex parameters. OREN-X runs all four modalities in real time within about 1 GiB of GPU memory, and its open-vocabulary responses are stronger and denser than the baselines’ qualitative outputs.

  • Problem

    Separate modality-specific representations multiply memory and computation and cannot exploit synergies among modalities needed for long-horizon autonomy.

    Source: §I · p. 1

  • Method

    OREN-X stores and jointly optimizes signed distance, occupancy, radiance, and vision-language fields in one shared octree, with flexible full or compressed vertex parameters.

    Source: §I · p. 1, A. Unified Multi-Modal Mapping · p. 3

  • Results

    OREN-X runs all four modalities in real time within about 1 GiB of GPU memory, and its open-vocabulary responses are stronger and denser than the baselines’ qualitative outputs.

    Source: F. Ablation Studies · p. 6, B. Qualitative Results · p. 6

  • Takeaways & Limitations

    The shared octree makes vision-language features queryable in 3D by interpolation and in 2D by rendering with radiance.

    Source: §I · p. 1

Abstract

from arXiv · show

To achieve general-purpose autonomy over long horizons, a robot needs to maintain spatial environment information that supports a variety of tasks: geometry for planning and control, radiance for rendering and relocalization, and vision-language features for open-vocabulary grounding. Existing methods represent and estimate each modality separately, multiplying memory and compute cost while forgoing potential synergy among the representations. We develop OREN-X, an online mapping method that uses an octree in 3D space as a shared data structure for indexing and storing a multi-modal field, capturing geometric, radiance, and vision-language information. OREN-X provides efficient unified storage and retrieval of these data in explicit/implicit and full/compressed form. Our unified representation yields cross-modality synergy: SDF estimates are sharpened by occupancy and radiance, while GPU-based ray-octree traversal and octree query enable real-time rendering. We also use online dictionary learning to compress the vision-language features, shrinking them 3.7x below full per-vertex storage while raising the query accuracy. On Replica, OREN-X maps in real time (80+ fps for SDF and 30+ fps for all four modalities), improves near-surface SDF accuracy by 33% over single-modality baselines, and improves mean open-vocabulary 3D mIoU by 71% and mean accuracy by 61% over the best prior method.

I. INTRODUCTION

Robots need geometric, radiance, and vision-language information, but separate maps multiply memory and computation and cannot exploit cross-modal synergies. OREN-X uses one octree to store and jointly train four modalities, with shared indexing for real-time queries and compressed vision-language features.

I. INTRODUCTION

  • Separate modality-specific representations multiply memory and computation, while independent modalities forgo potential synergies.The paper motivates unification as important for real-time, long-horizon autonomy.Source: §I · p. 1
  • OREN-X stores signed distance, occupancy, radiance, and vision-language information in one octree, with shared lookup and joint online optimization.Parameters can be explicit values, latent features decoded by a small network, or both, in full or compressed form.Source: §I · pp. 1–2
  • Radiance supervision improves SDF accuracy, SDF-occupancy consistency improves robustness to depth noise, and shared indexing supports 2D and 3D open-vocabulary queries.The shared octree supports VL-feature interpolation in 3D and rendering with radiance in 2D.Source: §I · pp. 1–2
  • One octree traversal serves all modalities in real time, while online dictionary learning compresses vision-language features, improves query accuracy, and scores text queries in code space.The method targets the vision-language features because they require the most memory.Source: §I · p. 2

II. RELATED WORK

Prior work maps geometry, appearance, and language in separate or layered representations, while OREN-X maintains them jointly in one octree. The problem statement formulates online estimation from posed RGB-D streams and vision-language features, with losses that can couple modalities.

II. RELATED WORK

  • Geometric mapping spans discrete grids and continuous fields, while radiance methods trade query flexibility against rendering speed; OREN-X supports ray-based and tile-based rendering on one octree.The related-work discussion describes octrees as suited to sparse 3D data through fast lookup, multi-resolution detail, and incremental growth.Source: §II · p. 2
  • OREN-X maintains vision-language features jointly with geometry and appearance in one octree, unlike methods that layer language onto appearance maps or keep separate semantic maps.Earlier approaches attach language features to NeRFs, Gaussians, points, or graphs, often using compression for memory.Source: §II · p. 2

III. PROBLEM STATEMENT

  • The target field has four components—signed distance, occupancy, radiance, and vision-language—with radiance also depending on viewing direction.The other three modalities have spatial domains in R^3; radiance is defined over R^3 × S^2.Source: §III · p. 2
  • Each posed RGB-D frame supplies colors, observed surface points, and pixel-aligned vision-language features for constructing modality-specific training data.SDF samples use distance to the nearest observed surface, negated behind it; occupancy labels are 0 in free space, 0.75 at the surface, and 1 behind it.Source: §III · p. 2
  • OREN-X estimates the field online from the stream through time t, using loss terms that may evaluate one or multiple modalities at positions or along rays.A loss depending on more than one modality is a cross-modality loss.Source: §III · p. 2

IV. TECHNICAL APPROACH

OREN-X maps four modalities in one octree, sharing storage, indexing, and online optimization while allowing each modality its own spatial resolution and parameterization. Its active-vertex updates keep per-frame work focused on observed regions.

A. Unified Multi-Modal Mapping

  • OREN-X represents all four modalities in one octree, with modality-specific parameters at vertices and joint online optimization.Parameters can be explicit values, implicit features, or both, and stored in full or compressed form.Source: §IV · p. 2, A. Unified Multi-Modal Mapping · p. 3
  • Each modality uses its own vertex pool and level range, sharing the octree while supporting different spatial resolutions and extents.Multi-pool indexing gathers the surrounding voxel corners at each selected level.Source: A. Unified Multi-Modal Mapping · p. 3
  • Per-vertex representations combine explicit values and implicit features as appropriate, with either component optionally compressed into dictionary codes.SDF uses an explicit coarse prior plus an implicit residual, radiance stores explicit colors, and VL features use explicit values that can be compressed.Source: A. Unified Multi-Modal Mapping · p. 3
  • Per-vertex bias correction and sparse updates touch only active vertices, keeping frame cost proportional to the observed region rather than the whole map.The per-vertex step count advances only when that vertex is active.Source: A. Unified Multi-Modal Mapping · p. 3

B. Cross-Modality Synergy

OREN-X couples modalities through shared supervision and computation: radiance and occupancy signals improve geometry, while shared octree operations serve multiple modalities. Its rendering and query pathways reuse the same spatial structure.

B. Cross-Modality Synergy

  • An occupancy-to-SDF sign-consistency loss improves robustness under strong depth noise by teaching SDF signs from detached, confident occupancy logits.Detaching occupancy prevents the SDF from pulling occupancy back.Source: B. Cross-Modality Synergy · p. 4
  • Radiance supervision uses SDF-derived density, coupling photometric loss to geometry and allowing radiance and SDF to improve one another.The SDF-indicated surface location also supports more accurate surface sampling.Source: B. Cross-Modality Synergy · p. 4
  • GPU ray-octree traversal trains radiance more efficiently and converges faster, while tile-based rasterization renders from the same octree.Shared octree traversal, ray traversal, or in-frustum filtering can serve several modalities at once.Source: B. Cross-Modality Synergy · p. 4
  • OREN-X stores vision-language features in a near-surface octree band alongside radiance, enabling color and feature rendering in one 2D pass and 3D feature queries; sparse shared-dictionary codes address prohibitive full-dimensional storage.Source: B. Cross-Modality Synergy · p. 4

C. Memory Compression for Vision-Language Features

OREN-X learns an append-only dictionary online to compress streamed vision-language features. Because query accuracy saturates before reconstruction fidelity, compact codes can preserve query performance without retaining every feature direction.

C. Memory Compression for Vision-Language Features

  • The append-only dictionary learns from streaming features, adding atoms when existing ones leave enough new-observation energy unexplained.Each vertex stores atom weights rather than a raw feature, and its code is updated as the dictionary grows.Source: C. Memory Compression for… · p. 4
  • Open-vocabulary query accuracy saturates well before reconstruction fidelity as dictionary size grows.Query scoring depends on concept-separating directions, while low-variance directions can affect reconstruction without changing score rankings.Source: Figure 3 · p. 5, C. Memory Compression for… · p. 5
  • Per-voxel PCA query accuracy saturates by k = 18 components, and aggregating features within a voxel raises accuracy above the per-point level.Feature spread within a voxel grows with voxel size, while accuracy peaks at intermediate voxel sizes.Source: C. Memory Compression for… · p. 5

V. EVALUATION

The evaluation examines whether joint octree training improves geometry and radiance, compares compressed vision-language features in 2D and 3D queries, and ablates design choices. The supplied feature-geometry analysis shows substantial class-cone overlap and query-accuracy saturation with compact per-voxel PCA.

V. EVALUATION

  • The evaluation tests joint-training effects on geometry and radiance, compares compressed VL features in 2D and 3D queries, and ablates design choices.Source: §V · p. 5
  • In Replica room0, VL features lie within a 35–80° cone around their mean, and 76% of class pairs have overlapping cones.The overlap indicates that a single feature is often ambiguous between classes.Source: Figure 4 · p. 5
  • Per-voxel PCA query accuracy saturates by k = 18 components and peaks at intermediate voxel sizes.Feature angular spread within a voxel increases with voxel size.Source: Figure 4 · p. 5

A. Experiment Setup

Experiments evaluate geometry, appearance, and open-vocabulary understanding on Replica and TUM, alongside runtime and memory. The setup specifies datasets, baselines, metrics, and implementation conditions.

Datasets

  • Evaluation uses Replica’s 8 scenes and TUM’s 3 sequences, with pseudo-ground-truth labels generated for TUM open-vocabulary evaluation.Replica provides semantic ground truth; TUM labels use LatentAM’s 10 classes and SAM 3.Source: A. Experiment Setup · p. 5

Baselines

  • Geometry, appearance, and vision-language results are compared against modality-specific baselines, with OREN-X matched to each baseline’s backbone.All methods use ground-truth camera poses; the setup also describes how baseline features are queried in 3D.Source: A. Experiment Setup · p. 5

Metrics

  • The evaluation reports geometry, photometric, and vision-language metrics, plus FPS and peak GPU memory measured on an NVIDIA RTX 5090.Geometry includes near- and far-surface SDF and gradient MAE; appearance includes PSNR, SSIM, LPIPS, and depth L1.Source: A. Experiment Setup · p. 5

Geometry table

  • Table I reports geometry on the 8-scene Replica mean, with mesh distances measured at a 5 cm threshold under OREN’s protocol.Best and second-best values are indicated by green and yellow shading, respectively.Source: Table I · p. 6

B. Qualitative Results

Qualitative comparisons show clean OREN-X renderings and stronger, denser open-vocabulary responses than the compared baselines on the displayed scenes and queries.

Radiance

  • OREN-X renders clean images, slightly smoother than LangSplatV2 and OnlineLangSplat, while LatentAM shows speckle artifacts.The rendering example is Replica room0.Source: B. Qualitative Results · p. 6

Vision-language

  • For the blanket, table, and monitor queries, OREN-X responds strongly, densely, and smoothly, unlike baselines’ noisy, diffuse, or missing responses.The queries are shown for Replica room1, Replica room2, and TUM 3, respectively.Source: B. Qualitative Results · p. 6

Comparison scope

  • The visual comparison covers OREN-X against LangSplatV2, OnlineLangSplat, and LatentAM for both RGB rendering and open-vocabulary queries.The RGB example is from Replica room0.Source: B. Qualitative Results · p. 6

C. Geometry

OREN-X combines improved near-surface geometry with balanced appearance results and strong open-vocabulary performance across the reported settings.

C. Geometry

  • OREN-X reduces near-surface SDF error by 33% and gradient error by 17% versus OREN, with on-par far-field and mesh metrics while maintaining three more modalities.The passage attributes the near-surface gains to occupancy and radiance supervising the SDF; nvblox has the best mesh but largest SDF and gradient errors.Source: C. Geometry · p. 6

D. Appearance

  • LangSplatV2 has the best images on both datasets but worst depth, while OREN-X balances the two with second-best Replica depth and better images than LatentAM.OREN-X has clearly the best depth on the real TUM sequences.Source: D. Appearance · p. 6

E. Open-Vocabulary Understanding

  • One OREN-X backbone is best in 46 of 48 Replica scene-metric entries, and TIPSv2-L remains best on all three-dimensional query metrics on TUM.LatentAM and OnlineLangSplat lead in 2D, where they are specifically optimized; OREN-X with CLIP is often lowest in fidelity but best in most mIoU and mAcc entries.Source: E. Open-Vocabulary Understanding · p. 6
  • Open-vocabulary feature fidelity does not determine query accuracy: OREN-X with CLIP is often lowest in fidelity yet best in most mIoU and mAcc entries.OnlineLangSplat reconstructs 2D features best in every Replica scene but trails in text queries.Source: E. Open-Vocabulary Understanding · p. 6

F. Ablation Studies

Ablations assess real-time efficiency, cross-modality effects, and design choices for radiance, rasterization, and vision-language compression. Results show trade-offs across speed, memory, and quality.

Ablation setup

Efficiency

  • OREN-X runs all four modalities in real time within about 1 GiB of GPU memory; SDF-only operation is faster than nvblox and uses less memory.Adding modalities lowers throughput moderately, and vision-language costs the most; LatentAM has similar throughput but about eight times the memory, while OnlineLangSplat and LangSplatV2 are one to two orders slower.Source: F. Ablation Studies · p. 6, Table IV · p. 7

Per-vertex bias correction

  • Per-vertex bias correction reduces near-surface SDF, gradient, and depth errors versus vanilla Adam, leaves SSIM unaffected, and lowers peak memory by about a third.The correction matters for rarely observed near-surface vertices; updating moments only for active vertices also reduces loss jitter.Source: F. Ablation Studies · p. 6, Figure 6 · p. 7

Cross-modality synergy and depth noise

  • Radiance reduces SDF error most on clean depth, occupancy helps under strong noise, and all modalities perform best on clean depth and the strongest noise.Replacing segment midpoints with SDF zero crossings leaves PSNR and SSIM unchanged but reduces LPIPS by 7%.Source: F. Ablation Studies · p. 6, Table V · p. 7

Radiance representation

  • Spherical-harmonic radiance improves quality only marginally on largely Lambertian Replica scenes while using 2.5× memory, so OREN-X retains RGB.Source: F. Ablation Studies · p. 6

Train/eval rasterizer

  • Ray-based training gives better quality with either evaluation backend and faster steps at higher memory, while tile-based rasterization renders two orders of magnitude faster at a small quality cost.The same octree supports both quality- and speed-oriented rendering.Source: F. Ablation Studies · p. 6

Online dictionary

  • The online dictionary achieves the best mAP at about a quarter of full-feature memory; sparse top-K codes retain most accuracy with less memory.An implicit MLP uses less memory but loses substantial accuracy and requires decoding before queries; the online dictionary scores text queries directly in code space.Source: F. Ablation Studies · p. 6

VL backbone

  • OREN-X reconstructs TIPSv2-L features best and DINOv3.txt features worst, but reconstruction fidelity does not determine query accuracy.CLIP has the best mAcc and Talk2DINOv3-ViT-L the best mIoU and mAP; the same dictionary hyperparameters work well across all four backbones without tuning.Source: F. Ablation Studies · pp. 6–7

VI. CONCLUSION

The conclusion presents OREN-X as an online framework maintaining SDF, occupancy, and radiance, and claims the best 3D open-vocabulary segmentation. The supplied comparison captions identify additional evaluation settings but do not state their outcomes.

VI. CONCLUSION

  • OREN-X is presented as an online multi-modal mapping framework maintaining SDF, occupancy, and radiance.Source: §VI · p. 7
  • The captions compare optimizer memory and stability, depth-noise robustness, and vision-language dictionary storage configurations on Replica room0.They identify the evaluation conditions but do not report comparative outcomes.Source: Figure 7 · p. 7, Figure 8 · p. 8, Table IX · p. 8
  • Other Replica room0 comparisons examine SDF zero-crossing versus segment-midpoint rendering, rasterizer choices, and RGB versus spherical-harmonic color.The captions describe the comparisons, not their outcomes.Source: Table VI · p. 7, Table VIII · p. 8, Table VII · p. 8
  • The conclusion claims OREN-X achieves the best 3D open-vocabulary segmentation.The supplied passage gives no comparison target or metric for this claim.Source: §VI · p. 8

Related papers

Loading 2609.29157v1…