PaperlayerView exact revision on arXiv

Source-linked AI summary

Progressive Gaussian Transformer with Anisotropy-aware Sampling for Open Vocabulary Occupancy Prediction

Chi Yan, Dan Xu

arXiv:2510.04759v2cs.CVcs.AI

TL;DR

Occupancy benchmarks restrict semantic labels to predefined categories, and text-aligned dense scene modeling carries memory and computation overhead. PG-Occ progressively densifies Gaussian representations and uses anisotropy-aware sampling. It reports state-of-the-art Occ3D-nuScenes performance with a 14.3% relative mIoU improvement over previous best results.

  • Problem

    Occupancy benchmarks constrain semantic information to predefined categories, while dense text-feature scene modeling incurs substantial memory and computational overhead.

    Source: §1 · pp. 1–2

  • Method

    PG-Occ uses coarse base Gaussians, feed-forward progressive densification guided by current predictions, and anisotropy-aware sampling for spatio-temporal feature fusion.

    Source: §1 · p. 2

  • Results

    14.3% relative mIoU improvement over previous best results establishes PG-Occ as state of the art on Occ3D-nuScenes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PG-Occ combines sparse Gaussian efficiency with refinement of scene regions that have insufficient perception.

    Source: §1 · p. 2

    As Gaussians increase, memory and computation costs may grow and affect real-time performance.

    Source: §5 · p. 9

Abstract

from arXiv · show

The 3D occupancy prediction task has witnessed remarkable progress in recent years, playing a crucial role in vision-based autonomous driving systems. While traditional methods are limited to fixed semantic categories, recent approaches have moved towards predicting text-aligned features to enable open-vocabulary text queries in real-world scenes. However, there exists a trade-off in text-aligned scene modeling: sparse Gaussian representation struggles to capture small objects in the scene, while dense representation incurs significant computational overhead. To address these limitations, we present PG-Occ, an innovative Progressive Gaussian Transformer Framework that enables open-vocabulary 3D occupancy prediction. Our framework employs progressive online densification, a feed-forward strategy that gradually enhances the 3D Gaussian representation to capture fine-grained scene details. By iteratively enhancing the representation, the framework achieves increasingly precise and detailed scene understanding. Another key contribution is the introduction of an anisotropy-aware sampling strategy with spatio-temporal fusion, which adaptively assigns receptive fields to Gaussians at different scales and stages, enabling more effective feature aggregation and richer scene information capture. Through extensive evaluations, we demonstrate that PG-Occ achieves state-of-the-art performance with a relative 14.3% mIoU improvement over the previous best performing method. Code and pretrained models will be released upon publication on our project page: https://yanchi-3dv.github.io/PG-Occ

1 INTRODUCTION

Existing occupancy benchmarks restrict semantics to predefined categories, while text-aligned modeling brings substantial memory and computational overhead. PG-Occ combines progressive Gaussian densification with anisotropy-aware sampling and reports a 14.3% relative mIoU improvement over previous best results.

1 INTRODUCTION

  • Existing 3D occupancy benchmarks provide semantic supervision but constrain semantic information to predefined categories.This limitation hinders perception of general objects outside those categories.Source: §1 · p. 1
  • Text-aligned features support occupancy queries beyond predefined categories, but dense scene modeling incurs substantial memory and computational overhead.Sparse Gaussian approaches can instead struggle to capture fine-grained scene details.Source: §1 · pp. 1–2
  • PG-Occ begins with coarse base Gaussians, then feed-forward densification refines regions with insufficient perception and anisotropy-aware sampling fuses spatio-temporal features.Sampling selects points according to each Gaussian’s anisotropy and projects them onto feature planes with varying receptive fields.Source: §1 · p. 2
  • 14.3% relative mIoU improvement over previous best results makes PG-Occ state of the art on Occ3D-nuScenes.The reported result is for the challenging Occ3D-nuScenes dataset.Source: §1 · p. 2

2 RELATED WORK

Related work addresses occupancy efficiency and label demands, open-vocabulary semantics, and feed-forward Gaussian scene modeling. The approaches differ in their representations, supervision, and modeling strategies.

Close-set 3D Occupancy Perception

  • Close-set occupancy work commonly supervises networks with strong 3D labels, while other approaches reduce computation or annotation demands.Examples include spatial cross-attention, three-view representations, lightweight BEV convolutions, and sparse Gaussians.Source: §2 · pp. 2–3

Open-vocabulary 3D Occupancy Perception

Generalizable 3D Gaussian Splatting with Feed-forward Networks

  • Feed-forward Gaussian reconstruction predicts scene representations from one or two views, and driving-scene methods extend the paradigm to multiple cameras or dynamic scenes.The paper distinguishes its own coarse-to-fine progressive Gaussian modeling from these approaches.Source: §2 · p. 3

3 METHODOLOGY

PG-Occ represents scenes with text-aligned feature Gaussians and progressively refines them using spatio-temporal image features. It trains through 2D rasterization supervision and converts final Gaussians to dense occupancy at inference.

3 METHODOLOGY

  • PG-Occ progressively transforms spatio-temporal image features into Gaussian representations, then converts the final representation into a dense 3D occupancy field.Source: §3 · p. 3, Figure 2 · p. 4

3.1 3D FEATURE GAUSSIAN SPLATTING

  • Each feature Gaussian encodes a 3D position, scale, rotation, opacity, and 512-dimensional text-aligned feature for open-vocabulary occupancy prediction.Source: §3.1 · pp. 3–4

3.2 PROGRESSIVE 3D GAUSSIAN MODELING

  • A base Gaussian layer models coarse geometry, followed by B progressive layers that adaptively expand and refine the Gaussian queries.Source: §3.2 · p. 4

3.2.1 PROGRESSIVE ONLINE DENSIFICATION (POD)

  • POD compares rendered expected depth with reference depth to identify under-represented regions and add Gaussians through a feed-forward process.This avoids gradient computation during densification.Source: §3.2.1 · pp. 4–5

3.2.2 ASYMMETRIC SELF-ATTENTION (ASA)

  • ASA prevents newly added Gaussians from influencing inherited Gaussians while allowing new ones to use existing features, stabilizing progressive modeling.Source: §3.2.2 · pp. 5–6

3.2.3 ANISOTROPY-AWARE FEATURE SAMPLING (AFS)

  • AFS scales and rotates learned offsets within each Gaussian ellipsoid, projects sample points onto feature planes, and aggregates features across views and timestamps.The aggregated feature is decoded into geometric properties and text-aligned features.Source: §3.2.3 · p. 6

3.3 EFFICIENT TRAINING VIA RASTERIZATION

  • Training uses only 2D text-aligned features and pseudo-depth maps, supervising rasterized Gaussians with depth and feature-rendering losses.The feature-rendering loss combines mean squared error and cosine similarity.Source: §3.3 · p. 6

3.4 TEST-TIME INFERENCE VIA VOXELIZATION

  • At inference, text prompts are matched to final-layer Gaussian features, and labeled Gaussians are voxelized into dense occupancy.The conversion is applied only at inference because training does not require dense occupancy labels.Source: §3.4 · p. 6, Figure 4 · p. 7

4 EXPERIMENTS

PG-Occ is evaluated on occupancy prediction, retrieval, depth estimation, and efficiency, with ablations examining its progressive densification and sampling components. It reports a 14.3% relative mIoU improvement over the previous best on Occ3D-nuScenes.

4.1 EXPERIMENTAL SETUP

  • Evaluation uses Occ3D-nuScenes (1,000 scenes) and nuScenes retrieval (105 samples); training uses seven prior frames, ResNet-50, and 8 × A800 GPUs.Open-vocabulary occupancy prediction is evaluated with IoU, mIoU, and ray-IoU; retrieval uses mAP and mAP(v).Source: §4.1 · p. 7

4.2 MAIN EXPERIMENT RESULTS

  • 15.15 mIoU on Occ3D-nuScenes is state of the art, a 14.3% relative gain over the previous best; PG-Occ also beats VEON without LiDAR training.The study groups methods by training modality; the comparison uses MaskCLIP text supervision for LangOcc, GaussTR, and PG-Occ.Source: §4.2 · p. 7, Table 1 · p. 8
  • Depth estimation reaches an error of 0.139, an +18.2% boost over Metric3D V2 labels, attributed to multi-view depth consistency and feature coherence.Depth results are reported on the nuScenes validation set, with Abs Rel as the primary metric.Source: Table 2 · p. 8, §4.2 · p. 8
  • 21.2 mAP(v) on nuScenes retrieval exceeds LangOcc’s 18.2; GaussTR reports no retrieval result and cannot be evaluated on LiDAR points.The benchmark contains 105 samples, each pairing a LiDAR point cloud with a text query.Source: §4.2 · pp. 7–8, §4.1 · p. 7, Figure 5 · p. 8

4.3 MAIN ABLATION STUDY

  • POD adaptively expands Gaussian queries from prior-layer outputs to refine under-reconstructed regions; removing it degrades all key metrics.The qualitative comparison also reports thicker, more realistic surfaces and finer structural details for PG-Occ than GaussTR.Source: §4.3 · p. 8, Table 3 · p. 8, Figure 6 · p. 9

4.2 MAIN EXPERIMENT RESULTS

  • On Occ3D-nuScenes, PG-Occ reports +14.3% mIoU, +41.1% FPS, and 25% less training time.The comparison covers mIoU, training time, and inference speed.Source: §4.2 · p. 8, Table 4 · p. 9

4.3 MAIN ABLATION STUDY

  • With base queries fixed at 4,000, mIoU rises from 14.84 to 15.15 as extended queries increase, but slightly drops at 2,000; mAP(v) continues improving.The authors attribute limited voxel-metric gains to evaluation at 0.4 m voxel size.Source: Table 5 · p. 9, §4.3 · p. 9
  • Removing anisotropy drops mIoU 0.12%; ASA raises mIoU from 14.85 to 15.15 by preventing extended queries from interfering with optimized queries.The anisotropy strategy supports multi-view feature capture and improves occupancy and retrieval metrics.Source: Table 3 · p. 8, §4.3 · p. 9

5 CONCLUSION AND LIMITATIONS

PG-Occ is a feed-forward progressive Gaussian framework for open-vocabulary occupancy prediction, and the authors report state-of-the-art results with high efficiency. They note that sparse driving viewpoints and growing Gaussian counts pose limitations, and provide video demonstrations on Occ3D-nuScenes validation and test sets.

5 CONCLUSION AND LIMITATIONS

  • PG-Occ models driving scenes as extendable feature Gaussian blobs in a feed-forward manner; anisotropy-aware sampling further improves detail capture.The conclusion reports state-of-the-art results with high efficiency.Source: §5 · p. 9
  • Sparse driving viewpoints make depth-scale constraints difficult and can cause popping artifacts; growing Gaussian counts may affect real-time performance.The authors identify 4D Gaussians and multi-view constraints as future directions.Source: §5 · p. 9

A VIDEO DEMONSTRATION

  • Video demonstrations show open-vocabulary occupancy inference on the Occ3D-nuScenes validation and test sets.The paper points readers to its project page for visualization quality.Source: A VIDEO DEMONSTRATION · p. 13

B.1 ADDITIONAL ABLATION STUDY

The additional ablations examine accuracy–compute trade-offs across Gaussian queries, densification, sampling, and losses, as well as pose-noise robustness and layer-wise inference time.

B.1 ADDITIONAL ABLATION STUDY

  • Increasing base-layer queries from 1,000 to 4,000 raises mIoU from 13.17 to 15.15, while 8,000 queries slightly reduce all three metrics.The authors attribute the decline to excessive queries overwhelming self-attention and weakening spatial interaction capture.Source: §B.1 · p. 13, Table 6 · p. 13
  • A densification threshold of 0.2 yields 15.15 mIoU, versus 15.11 at 0.0 and 15.13 at 0.4, while balancing point selection and computational cost.Lower thresholds select more points and raise subsequent FPS time from 30 ms to 50 ms.Source: §B.1 · pp. 13–14, Table 7 · p. 13
  • Thirty-two sampling offsets raise mIoU from 15.05 with 8 offsets to 15.46, while training time grows from 8 to 11.2 hours on 8×A800 GPUs.The authors identify 16 offsets as a balance of strong performance and moderate training time.Source: §B.1 · p. 14, Table 8 · p. 14
  • Using all five losses achieves 15.15% mIoU, 13.92% RayIoU, and 21.20% mAP; omitting any one slightly lowers these metrics.The evaluated losses are LL1, LSILog, Ltemp, Lmse, and Lcos.Source: §B.1 · p. 14, Table 10 · p. 14
  • Because outdoor scenes and limited view supervision make photometric cues unreliable for geometry and semantics, the ablation tests adding a 3D-color prediction head.The supplied passage describes the experiment but does not report its outcome.Source: §B.1 · p. 14, Table 9 · p. 14
  • PG-Occ remains robust to pose errors: slight pose noise marginally improves mIoU, while larger noise causes a small decrease followed by stabilization.The authors cite imperfect nuScenes poses and failed camera-plane feature sampling under significant pose errors as explanations.Source: §B.1 · p. 14, Table 11 · p. 14

B.2 LAYER-WISE TIME CONSUMPTION

  • Inference takes 27.4 ms in the base layer, 58.3 ms in the first progressive layer, and 60.6 ms in the second as Gaussian counts increase.The passage links the longer times to processing denser representations.Source: §B.2 · p. 15, Table 12 · p. 15

C ADDITIONAL VISUALIZATION RESULTS

The visualizations show PG-Occ’s open-vocabulary capabilities and its reconstructed BEV scene layouts. They illustrate semantic rendering and occupancy detail across objects and scenes.

C.1 PG-OCC CAPABILITIES

  • With single-frame multi-view inputs and feed-forward inference, PG-Occ renders open-vocabulary features and supports zero-shot semantic occupancy queries, retrieval, and localization.The examples include scene-depth estimates and semantics beyond fixed categories.Source: §C.1 · p. 15

C.2 BEV VISUALIZATION

  • BEV visualizations show large and small elements, including vehicles, pedestrians, and barriers, with sharp, consistent occupancy boundaries across diverse scenes.The BEV perspective makes spatial relationships among objects observable.Source: §C.2 · p. 15

C.3 EGO-CENTRIC PERSPECTIVE OCCUPANCY VISUALIZATION WITH PREVIOUS SOTA METHOD

The ego-centric visualization compares PG-Occ with GaussTR, while third-person views illustrate zero-shot semantic occupancy and note the challenge of occlusion.

C.3 EGO-CENTRIC PERSPECTIVE OCCUPANCY VISUALIZATION WITH PREVIOUS SOTA METHOD

  • Compared with GaussTR, PG-Occ detects small cars, bicycles, buses, trucks, and barriers more effectively, and identifies some pedestrians and bicycles absent from the ground truth annotations.The unannotated detections appear in the second visualization.Source: §C.3 · p. 15

C.4 THIRD PERSPECTIVE OCCUPANCY VISUALIZATION

  • Third-person views illustrate zero-shot semantic occupancy, but occlusion can hinder accurate predictions where visual observations are absent.The method may still produce reasonable inferences in some such areas.Source: §C.4 · p. 15

D ADDITIONAL IMPLEMENTATION DETAILS

The implementation converts text-aligned Gaussian features into occupancy fields and adapts voxelization for sparse-LiDAR retrieval evaluation. It also specifies prompt handling for occupancy and retrieval.

D.1 VOXELIZATION

  • For occupancy, CLIP encodes arbitrary text prompts, their embeddings are compared with Gaussian text-aligned features, and a softmax yields each Gaussian’s text probability.Voxelization then accumulates each Gaussian’s influence across an ROI grid based on anisotropy, opacity, and text probability.Source: §D.1 · p. 16
  • For nuScenes retrieval, each sparse LiDAR annotation point serves as a voxel center to obtain its occupancy probability and text feature.This adapts the occupancy calculation to the dataset’s sparse text annotations.Source: §D.1 · p. 16

D.2 TEXT PROMPT

  • For Occ3D-nuScenes occupancy prompts, PG-Occ omits “others” and “other flat” because their semantics are ambiguous; retrieval uses the dataset’s prompt directly.The authors note that finetuning the ambiguous occupancy prompts could improve performance.Source: §D.2 · p. 16

D.3 ADDITIONAL MODEL AND TRAINING DETAILS

This appendix specifies PG-Occ’s supervision and optimization settings and documents prompts used for zero-shot semantic occupancy estimation on Occ3D-nuScenes.

D.3.1 SUPERVISION STRATEGY

  • Metric3D V2 supplies depth supervision and MaskCLIP supplies feature supervision.The loss weights are λSILog = 0.15, λtemp = 10, and λmse = 10.Source: §D.3.1 · p. 16
  • Training uses AdamW with a 2e-4 initial learning rate and 0.01 weight decay.Experiments use the stated loss-weight settings for SILog, temporal, and MSE losses.Source: §D.3.1 · p. 16
  • Table 13 lists the text prompts used for zero-shot semantic occupancy estimation on Occ3D-nuScenes; a dash means no prompt was made for that class.Source: Table 13 · p. 17

D.3.2 MODEL ARCHITECTURE AND TRAINING SETUP

PG-Occ uses a multi-layer Gaussian setup and, from single-frame multi-view inputs with feed-forward passes, supports depth, feature rendering, occupancy, semantic occupancy, and text queries. Qualitative figures show its predictions across views and perspectives, including comparisons with GaussTR.

D.3.2 MODEL ARCHITECTURE AND TRAINING SETUP

  • PG-Occ starts with 4,000 Gaussian queries and adds 1,000 per progressive layer, using one base and two progressive layers.The setup uses ResNet-50, the previous seven frames, a 256-dimensional embedding, and 180 × 320 resolution for depth and feature rasterization and Gaussian initialization; training uses 8 A800 GPUs for 8 epochs, with inference on one A800 GPU.Source: §D.3.2 · p. 17
  • With single-frame multi-view inputs and feed-forward passes, PG-Occ estimates depth, renders open-vocabulary features, predicts zero-shot occupancy and semantic occupancy, and supports text queries.Source: Figure 7 · p. 18
  • The bird’s-eye-view visualization presents predicted occupancy and semantic structures to illustrate scene layout and spatial relationships.Source: Figure 8 · p. 18
  • In qualitative ego-centric multi-camera comparisons, PG-Occ better detects and reconstructs small or distant objects, preserves spatial relationships, and produces more accurate shapes than GaussTR.The figure shows input views, GaussTR predictions, ground truth, and PG-Occ predictions; colors encode semantic categories.Source: Figure 9 · p. 19
  • Qualitative zero-shot semantic occupancy results show PG-Occ predictions alongside ground truth for two views of a third perspective.The figure caption reports that the predictions capture semantic occupancy patterns across perspectives.Source: Figure 10 · p. 20

Related papers

Loading 2510.04759v2…