Source-linked AI summary

LegoOcc++: Towards Consistent Open-Vocabulary 3D Occupancy Prediction

Changqing Zhou, Yueru Luo, Han Zhang, Zeyu Jiang, Changhao Chen

arXiv:2610.34093v1cs.CV

TL;DR

Open-vocabulary occupancy matters for embodied agents, but indoor scenes pose dense-occlusion, geometric, and long-tailed semantic challenges. LegoOcc++ combines Gaussian-based geometry and language-aligned semantics with object-level consistency and covisibility-aware view selection, and evaluates indoor and outdoor settings. It reports strong results across both domains, including improved Occ-ScanNet performance and competitive single-frame outdoor mIoU.

  • Problem

    Indoor open-vocabulary occupancy is less explored than driving scenes despite dense occlusion, complex geometry, and fine-grained long-tailed categories that challenge fixed taxonomies.

    Source: §I · p. 1
  • Method

    LegoOcc++ builds on Language-Embedded Gaussians and adds object-level semantic consistency and covisibility-aware support-view selection for open-vocabulary occupancy prediction.

    Source: §I · p. 2
  • Results

    LegoOcc++ achieves 61.73 IoU and 24.22 mIoU on Occ-ScanNet, and its single-frame 15.14 mIoU on Occ3D-nuScenes nearly matches eight-frame PGOcc's 15.15 mIoU.

    Source: §I · p. 3, C. Main Results · p. 9
  • Takeaways & Limitations

    Experiments support the framework's generality across indoor geometry-only supervision and outdoor annotation-free training settings.

    Source: §VI · p. 12

    Indoor geometry learning still relies on binary occupancy supervision, while broad categories such as furniture remain difficult to calibrate consistently.

    Source: §V · p. 12

Abstract

from arXiv · show

Open-vocabulary 3D occupancy prediction is important for embodied agents operating in complex environments whose semantic categories extend beyond fixed taxonomies. While recent studies have explored open-vocabulary occupancy in outdoor driving scenarios, indoor environments remain challenging due to dense occlusions, complex geometry, and fine-grained long-tailed semantics. To reduce the need for expensive voxel-level semantic annotations, we study a geometry-only supervision setting requiring only binary occupancy labels. We build upon Language-Embedded Gaussians as a unified representation coupling 3D geometry with language-aligned semantics. For geometry learning, we introduce an opacity-aware Poisson-based Gaussian-to-Occupancy formulation that enables stable volumetric aggregation under binary occupancy supervision. For semantic learning, we render Gaussian features into images and align them with a frozen open-vocabulary model, together with Progressive Temperature Decay to reduce feature mixing during Gaussian splatting. Building on this framework, we further introduce Instance-Level Semantic Consistency to alleviate fragmented object predictions, and Covisibility-Aware Support-View Selection to provide informative and less redundant multi-view semantic supervision. We also extend our framework to outdoor driving scenarios to evaluate its generality across substantially different environments. Extensive experiments on indoor and outdoor benchmarks demonstrate that our method achieves strong open-vocabulary occupancy performance, validating the effectiveness and generality of our framework. Code will be available at https://github.com/JuIvyy/LegoOcc.

I. INTRODUCTION

Open-vocabulary occupancy supports querying categories beyond fixed training taxonomies, but indoor scenes combine dense occlusion and complex geometry with fine-grained, long-tailed semantics. LegoOcc++ extends geometry-supervised indoor occupancy to outdoor driving scenes with object-level semantic consistency and geometry-aware multi-view supervision.

I. INTRODUCTION

  • Open-vocabulary occupancy enables arbitrary category queries, addressing the limits of fixed taxonomies for embodied agents encountering previously unseen objects.Indoor scenes add dense occlusions, complex geometry, and fine-grained long-tailed categories; Occ-ScanNet defines 11 semantic classes.Source: §I · pp. 1–2
  • LegoOcc uses Language-Embedded Gaussians, opacity-aware Poisson-based occupancy aggregation, and rendered-feature alignment with a frozen open-vocabulary model.Progressive Temperature Decay sharpens Gaussian opacities during training to reduce feature mixing while preserving differentiable supervision.Source: §I · p. 2
  • Instance-Level Semantic Consistency uses category-agnostic SAM masks and teacher-feature prototypes to regularize same-instance rendered features against intra-object semantic fragmentation.It complements pixel-wise alignment and Progressive Temperature Decay's ray-level feature disambiguation.Source: §I · pp. 2–3
  • Covisibility-Aware Support-View Selection filters views with insufficient overlap or near-duplicate content, then selects pose-diverse views for cross-view semantic alignment.It selects by shared visible 3D geometry rather than a fixed temporal interval.Source: §I · pp. 2–3
  • On Occ-ScanNet, LegoOcc++ reaches 61.73 IoU and 24.22 mIoU; on Occ3D-nuScenes, its single-frame 15.14 mIoU nearly matches eight-frame PGOcc's 15.15 mIoU.The Occ-ScanNet result improves on LegoOcc's 59.50 IoU and 21.05 mIoU.Source: §I · p. 3
  • The framework spans indoor geometry-only supervision and outdoor annotation-free training, testing generality across different scene domains and supervision settings.The outdoor setting also differs in scene scale and semantic distribution.Source: §I · pp. 2–3

II. RELATED WORKS

Occupancy prediction spans grid, transformer, and Gaussian-based approaches, with indoor methods adapting ideas from camera-based scene completion. Open-vocabulary occupancy has advanced further in driving scenes than indoors, where scalable approaches remain underexplored.

A. Occupancy Prediction

  • Indoor occupancy methods range from depth-guided feature lifting and volumetric decoding to transformer-augmented models and sparse Gaussian or point-based representations.Indoor scenes' layout complexity, viewpoint variation, severe occlusion, and long-tailed categories can make driving-specific designs transfer suboptimally.Source: A. Occupancy Prediction · p. 3

B. Open-Vocabulary Occupancy Prediction

  • Open-vocabulary occupancy methods transfer language-aligned cues to 3D through pixel, point, voxel, or rendered-field alignment, with geometry learned from pseudo-labels or multi-view consistency.These approaches are established mainly in outdoor driving scenarios.Source: B. Open-Vocabulary Occupancy Prediction · p. 3
  • Scalable open-vocabulary occupancy for large indoor scenes remains underexplored despite early indoor efforts such as OVO, OpenOcc, and OG.The cited approaches use pixel-wise alignment, RGB-D neural radiance fields, or instance-discriminating affinity fields, respectively.Source: B. Open-Vocabulary Occupancy Prediction · p. 3

III. METHOD

The method uses Language-Embedded Gaussians to couple 3D geometry with language-aligned semantics, then learns occupancy and semantics through complementary supervision paths. Its geometry branch maps Gaussian contributions to occupancy using an opacity-aware Poisson formulation.

A. Problem Setting

  • Language-Embedded Gaussians combine geometric parameters with language-aligned embeddings in a shared representation for occupancy prediction and semantic reasoning.The representation includes position, covariance, opacity, and a language-aligned feature.Source: A. Problem Setting · p. 4

B. Framework Overview

  • A feed-forward predictor produces Gaussians reused for occupancy from geometry and voxel semantics from embeddings, with binary occupancy and rendered-feature alignment providing supervision.The feature-alignment loss requires no 2D human annotations.Source: B. Framework Overview · p. 4

C. Poisson-based Gaussian-to-Occupancy

  • Opacity-agnostic Gaussian aggregation mismatches opacity-aware semantic rendering, while Bernoulli aggregation can shrink learned opacities and widen the feature gap.The Poisson formulation treats Gaussian contributions as event intensities and derives occupancy from their cumulative mean measure.Source: C. Poisson-based Gaussian-to-Occupancy · pp. 4–5
  • A Poisson process models occupancy as the probability of at least one event, using the cumulative mean measure of Gaussian contributions.Each Gaussian contributes a nonnegative intensity, and the total mean measure yields a closed-form occupancy probability.Source: C. Poisson-based Gaussian-to-Occupancy · p. 5

D. Progressive Temperature Decay

Rendered pixel features mix multiple Gaussian embeddings along a ray, so alignment may supervise a mixture rather than individual Gaussians. Progressive Temperature Decay continuously sharpens opacities during training to reduce this ambiguity while retaining gradient flow.

D. Progressive Temperature Decay

  • Ray-wise mixtures make rendered-feature supervision target combined embeddings rather than individual Gaussians, a mismatch that hampers later Gaussian-to-occupancy splatting.Hard Top-k registration avoids the mixture but produces sparse gradients through only selected primitives.Source: D. Progressive Temperature Decay · p. 5
  • Exponential temperature decay approaches Tmin rapidly, allocating more iterations to the low-temperature regime than linear decay.The schedule starts with smooth mixtures and gradually sharpens opacities for discriminative alignment.Source: Figure 3 · p. 5, D. Progressive Temperature Decay · p. 6
  • A tempered sigmoid sharpens predicted opacities as temperature falls, reducing feature mixing while maintaining gradient flow to all contributors.Unlike discrete Top-k selection, the continuous approach retains end-to-end differentiability.Source: D. Progressive Temperature Decay · p. 5

E. Instance-Level Semantic Consistency

Instance-Level Semantic Consistency supplements pixel-wise alignment with an object-level constraint that reduces fragmented semantics within objects. It uses reliable SAM masks to consolidate teacher cues into shared prototypes and align rendered LE-Gaussian features.

E. Instance-Level Semantic Consistency

  • The method aggregates teacher features within reliable SAM masks into instance-level prototypes, then aligns rendered LE-Gaussian features inside each instance to the shared prototype.This complements pixel-level supervision with an object-level constraint to reduce semantic fragmentation and improve feature consistency within instances.Source: E. Instance-Level Semantic Consistency · p. 6, Figure 4 · p. 6
  • Only pixels with rendered opacity above αinst are retained, and masks with fewer than τinst valid pixels are discarded to avoid unreliable prototypes.The opacity filter excludes pixels weakly explained by rendered Gaussians; the minimum-size rule addresses small or fragmented masks.Source: E. Instance-Level Semantic Consistency · p. 6
  • The shared prototype consolidates potentially fragmented local semantic cues while preserving the original pixel-wise loss’s fine-grained local supervision.The instance loss is complementary to, rather than a replacement for, pixel-wise feature alignment.Source: E. Instance-Level Semantic Consistency · p. 6

F. Covisibility-Aware Support-View Selection

Support views are selected by geometric covisibility to balance shared scene geometry with complementary observations. The method excludes views with too little overlap or near-duplicate content, then selects pose-diverse candidates.

F. Covisibility-Aware Support-View Selection

  • Geometric covisibility filters out candidate views with insufficient overlap or near-duplicate content, then greedy pose-diverse selection provides complementary cross-view supervision.The score measures shared visible geometry; the default bounds are cmin=0.2 and cmax=0.6.Source: F. Covisibility-Aware Support-View… · pp. 6–7
  • Fixed temporal intervals trade off redundant observations at small gaps against insufficient scene overlap at larger gaps, and are difficult to generalize across different video sampling rates.The selected views preserve shared geometry while providing more diverse viewpoints than fixed-interval selection.Source: F. Covisibility-Aware Support-View… · pp. 6–7
  • The figure contrasts redundant fixed-temporal support views with the method’s filtered, pose-diverse views around each reference view.The selected views retain shared scene geometry while providing complementary observations.Source: Figure 5 · p. 7, F. Covisibility-Aware Support-View… · p. 7

G. Losses

The training objective combines geometry and language-aligned semantic supervision, with different geometry supervision for indoor and outdoor settings. Both settings use rendered feature alignment and SAM-guided instance consistency.

G. Losses

  • Indoor geometry uses ground-truth depth and binary voxel occupancy labels with focal, Lovász–Softmax, and scene-class affinity losses.The indoor objective also includes rendered-feature alignment with the frozen open-vocabulary teacher and SAM-guided instance consistency.Source: G. Losses · p. 7
  • Outdoor geometry uses multi-view rendering and Huber depth loss on Metric3D pseudo-depth, without occupancy-supervision terms or occupancy annotations.The outdoor objective retains semantic feature alignment with the frozen teacher and SAM-guided instance consistency.Source: G. Losses · p. 7

A. Datasets and Metrics

The experiments use Occ-ScanNet for indoor monocular occupancy prediction and Occ3D-nuScenes for single-frame outdoor prediction. The datasets differ in voxel-grid coverage, supervision setup, and reported metrics.

A. Datasets and Metrics

  • Occ-ScanNet has 45,755 training and 19,764 validation samples, each represented by a 60×60×36 grid over a 4.8m×4.8m×2.88m volume.It includes 11 semantic categories plus empty; evaluation is restricted to the current camera frustum and reports occupancy IoU, per-class IoU, and mIoU.Source: A. Datasets and Metrics · p. 7
  • The indoor task uses one RGB image as input and binary occupancy labels for geometry supervision.The standard Occ-ScanNet split is used.Source: A. Datasets and Metrics · p. 7
  • Occ3D-nuScenes contains 1,000 driving scenes and uses a 200×200×16 grid of 0.4 m voxels over the stated [−40, 40] m x/y and [−1, 5.4] m z region.Evaluation reports per-class IoU and mIoU over 15 semantic categories.Source: A. Datasets and Metrics · p. 7
  • Outdoor inference uses six surround-view images from one timestamp, while neighboring frames provide training supervision without occupancy annotations.Geometry is learned through Gaussian rendering across neighboring frames, and semantic supervision comes from open-vocabulary segmentation.Source: A. Datasets and Metrics · p. 7

B. Implementation Details

The evaluation uses indoor Occ-ScanNet and outdoor Occ3D-nuScenes, with geometry-only open-vocabulary supervision indoors and no occupancy annotations during outdoor training. LegoOcc++ improves on LegoOcc in both settings and approaches an eight-frame outdoor baseline using one frame.

B. Implementation Details

  • Indoor models use Depth-Anything V2 for depth, surface-point expansion for Gaussian primitives, Trident for per-pixel language-aligned features, and CLIP text embeddings.Indoor training runs for 10 epochs with AdamW and global batch size 8.Source: B. Implementation Details · p. 7

C. Main Results

  • The Occ-ScanNet evaluation reports overall occupancy IoU, per-class IoU, and mIoU; Occ3D-nuScenes reports mIoU and per-class IoU.Indoor baseline reimplementations use binary occupancy supervision and the same input resolution and training recipe as LegoOcc++ for comparison.Source: C. Main Results · p. 8, Table I · p. 8
  • 61.73 IoU and 24.22 mIoU on Occ-ScanNet improve over LegoOcc by +2.23 IoU and +3.17 mIoU under the same geometry-only setting.LegoOcc++ also exceeds all closed-vocabulary methods in occupancy IoU.Source: C. Main Results · pp. 8–9, Table I · p. 8
  • 15.14 mIoU on Occ3D-nuScenes brings single-frame LegoOcc++ close to eight-frame PGOcc at 15.15 mIoU.LegoOcc++ improves over single-frame LegoOcc at 14.19 mIoU; its gains span foreground objects and background surfaces.Source: Table II · p. 8, C. Main Results · p. 9
  • The outdoor evaluation supports generalization from indoor monocular scenes to large-scale autonomous-driving environments with different scene layouts and semantic distributions.Outdoor methods are evaluated without occupancy annotations during training.Source: C. Main Results · pp. 8–9

D. Qualitative Visualization

Qualitative results show open-vocabulary indoor queries and more complete outdoor occupancy. Ablations attribute improvements to Poisson-based aggregation, temperature scheduling, and the two LegoOcc++ extensions, with no added inference overhead over LegoOcc.

D. Qualitative Visualization

  • Open-vocabulary indoor predictions query arbitrary categories, while closed-vocabulary comparisons evaluate occupancy against Occ-ScanNet labels.Figure 7 juxtaposes input images, 2D segmentation for queried nouns, and LegoOcc/LegoOcc++ 3D occupancies colored by those categories.Source: D. Qualitative Visualization · p. 9, Figure 6 · p. 10, Figure 7 · p. 11

E. Ablation Studies

  • 59.50 IoU and 21.05 mIoU with Poisson G2O surpass the Bernoulli variant by +12.85 IoU and +3.80 mIoU in the open-vocabulary setting.Poisson G2O also improves closed-vocabulary results over the GaussianFormer2 operator: 56.96→59.24 IoU and 51.88→52.80 mIoU.Source: E. Ablation Studies · p. 9, Table III · p. 9
  • The exponential temperature schedule achieves the best reported result, 21.05 mIoU/59.50 IoU, while constant low temperature collapses optimization.The authors report that the schedule reduces feature mixing; lowering Tmin to 10^-4 slightly hurts performance.Source: Table IV · p. 9, E. Ablation Studies · p. 10
  • LegoOcc++ improves over the LegoOcc baseline by +3.17 mIoU and +2.23 IoU, while retaining LegoOcc’s inference speed and requiring no additional inference overhead.Adding SAM-based instance-level semantic consistency further improves the result to 24.22/61.73.Source: E. Ablation Studies · pp. 10–11

D. Qualitative Visualization

  • On Occ3D-nuScenes, LegoOcc++ better preserves road layout and surrounding structure than GaussTR and recovers more complete long-range geometry.The visualization compares surround-view inputs and predictions from both methods with ground truth.Source: D. Qualitative Visualization · p. 9, Figure 8 · p. 11

E. Ablation Studies

  • 6.22 FPS makes LegoOcc++ approximately 15× faster than GaussTR and 26× faster than the eight-frame PG-Occ pipeline.Its complete inference pipeline has 71.9% fewer parameters than GaussTR and 67.8% fewer than PG-Occ.Source: E. Ablation Studies · p. 11, Table VII · p. 12

V. LIMITATIONS

LegoOcc++ performs across indoor and outdoor benchmarks, but broad categories remain difficult and indoor geometry still relies on binary occupancy supervision. The conclusion presents the framework as applicable to embodied indoor and autonomous-driving settings.

V. LIMITATIONS

  • Broad categories such as furniture and objects remain challenging because their diverse instances make language-aligned representations difficult to calibrate consistently.The authors identify this as a remaining limitation despite strong performance across indoor and outdoor benchmarks.Source: §V · p. 12
  • Indoor geometry still relies on binary occupancy supervision for reliable 3D geometry.The authors identify stronger multi-view geometric priors or self-supervised reconstruction as future directions for removing explicit geometry supervision in cluttered indoor environments.Source: §V · p. 12

VI. CONCLUSION

  • The framework is presented for indoor embodied environments and outdoor autonomous-driving scenarios, with experiments spanning geometry-only indoor supervision and annotation-free outdoor training.The conclusion describes this as a general open-vocabulary 3D occupancy framework.Source: §VI · p. 12
Loading 2610.34093v1…