PaperlayerView exact revision on arXiv

Source-linked AI summary

GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction

Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, Jiwen Lu

arXiv:2405.17429v1cs.CVcs.AI

TL;DR

Vision-based 3D semantic occupancy prediction needs a scene representation that captures fine-grained geometry and semantics without wasting resources on sparse, unevenly scaled environments. GaussianFormer uses sparse semantic Gaussians learned from images and locally splats them into voxel predictions, achieving comparable state-of-the-art performance with much lower memory consumption.

  • Problem

    Vision-based occupancy prediction must describe fine-grained 3D geometry and semantics despite dense-grid methods overlooking occupancy sparsity and object-scale diversity.

  • Method

    GaussianFormer learns sparse 3D semantic Gaussians from images through attention and iterative refinement, then locally aggregates neighboring Gaussians into voxel occupancy predictions.

  • Results

    GaussianFormer achieves comparable state-of-the-art performance on nuScenes and KITTI-360 while using only 17.8% - 24.8% of competing methods' memory.

  • Takeaways & Limitations

    Object-centric Gaussians provide a sparse, adaptive scene representation that preserves fine-grained structure while reducing memory consumption.

  • Takeaways & Limitations

    Performance remains inferior to state-of-the-art methods, and satisfactory results require a large number of Gaussians.

Abstract

from arXiv · show

3D semantic occupancy prediction aims to obtain 3D fine-grained geometry and semantics of the surrounding scene and is an important task for the robustness of vision-centric autonomous driving. Most existing methods employ dense grids such as voxels as scene representations, which ignore the sparsity of occupancy and the diversity of object scales and thus lead to unbalanced allocation of resources. To address this, we propose an object-centric representation to describe 3D scenes with sparse 3D semantic Gaussians where each Gaussian represents a flexible region of interest and its semantic features. We aggregate information from images through the attention mechanism and iteratively refine the properties of 3D Gaussians including position, covariance, and semantics. We then propose an efficient Gaussian-to-voxel splatting method to generate 3D occupancy predictions, which only aggregates the neighboring Gaussians for a certain position. We conduct extensive experiments on the widely adopted nuScenes and KITTI-360 datasets. Experimental results demonstrate that GaussianFormer achieves comparable performance with state-of-the-art methods with only 17.8% - 24.8% of their memory consumption. Code is available at: https://github.com/huang-yh/GaussianFormer.

1 Introduction

Vision-centric 3D semantic occupancy prediction addresses limitations of camera-only perception by predicting fine-grained occupancy and semantics throughout the surrounding scene. GaussianFormer replaces dense or planar scene representations with sparse, object-centric 3D Gaussians and reports comparable state-of-the-art performance with substantially lower memory consumption.

  • Motivation: 3D semantic occupancy prediction improves vision-centric autonomous driving by representing occupancy and semantics throughout the surrounding 3D space.The task also supports applications including end-to-end autonomous driving, 4D occupancy forecasting, and self-supervised 3D scene understanding.
  • Motivation: Dense voxel and planar representations struggle to allocate resources effectively across sparse scenes and objects with diverse scales.Voxel methods process dense grids, while planar methods reduce redundancy but may lose height-related information.
  • Contribution: GaussianFormer introduces sparse 3D semantic Gaussians, each representing a flexible region with position, covariance, and semantic properties.The model learns these properties from multiscale image features using cross-attention and iterative refinement.
  • Results: 17.8% - 24.8% of existing methods' memory consumption accompanies comparable performance on nuScenes and KITTI-360.The method uses neighboring-Gaussian aggregation to convert the learned representation into semantic occupancy predictions.

2 Related Work

Prior occupancy methods primarily use voxels, BEV, or TPV representations, each trading off detail, efficiency, or height information. Gaussian splatting offers a compact alternative, but GaussianFormer differs from rendering-oriented 3D-GS by learning Gaussians online for semantic occupancy prediction.

  • Voxel representations: Voxel-based methods represent intricate 3D structures effectively by assigning feature vectors to regular voxels.Their dense representation can be costly for occupancy prediction because the scene space is largely unoccupied.
  • Planar representations: BEV methods project image information into bird's-eye-view features using lift-splat-shoot or query-based transformations.These approaches have succeeded in 3D object detection but are less suitable for occupancy when height compression loses information.
  • Planar representations: Existing occupancy methods remain grid-based, while BEV variants add backward projection or multiscale fusion to recover finer details.These designs improve planar features but retain the underlying grid representation.
  • Gaussian representations: 3D Gaussian splatting models intricate shapes with fewer parameters and renders views efficiently through splat-based rasterization.Its original use is radiance-field rendering rather than semantic occupancy prediction.
  • Gaussian representations: GaussianFormer learns Gaussians online and predicts 3D semantic occupancy, unlike 3D-GS's offline optimization and 2D RGB rendering.This distinction creates a different use of the Gaussian representation for vision-based occupancy prediction.

3 Proposed Approach

The proposed approach represents scenes with adaptive semantic Gaussians, learns them from multiview images through iterative attention and refinement, and splats only local Gaussian neighborhoods into voxel occupancy predictions. Gaussian properties encode flexible regions while preserving fine-grained structure and reducing representation redundancy.

  • Image to Gaussians: GaussianFormer learns an object-centric scene representation from multiview images using self-encoding, image cross-attention, and property refinement.The method initializes Gaussian properties and queries, then updates them through repeated model blocks.
  • Object-centric 3D Scene Representation: Each semantic Gaussian describes a flexible region using mean, scale, rotation, covariance, and semantic logits rather than a fixed grid cell.Occupancy at a location is obtained by evaluating and summing the semantic Gaussian distributions.
  • Object-centric 3D Scene Representation: The covariance is constructed from scale and rotation, allowing Gaussian support to adapt to object shapes and region complexity.This adaptive parameterization addresses the fixed resource allocation of dense voxels while retaining scene detail.
  • Gaussian-to-Voxel Splatting: Gaussian-to-voxel splatting assigns each Gaussian to nearby voxels, sorts Gaussian-voxel pairs by voxel index, and locally aggregates contributing Gaussians.The resulting occupancy prediction uses only neighboring Gaussians for each voxel rather than globally aggregating the full Gaussian set.
  • Image to Gaussians: Sparse convolution enables interactions among Gaussians by treating them as points at their means and processing their voxelized locations.Because the Gaussian count is much smaller than the target voxel volume, the operation exploits Gaussian sparsity.
  • Image to Gaussians: Image cross-attention samples multiscale image features at covariance-informed 3D reference points projected into image coordinates.Deformable attention aggregates visual information across camera views for each Gaussian query.

4 Experiments

Experiments on nuScenes and KITTI-360 evaluate GaussianFormer for surrounding-camera and monocular 3D semantic occupancy prediction. Results show competitive accuracy, substantially reduced memory use, and tunable efficiency through the number and refinement of 3D Gaussians.

  • Benchmark results: GaussianFormer achieves comparable performance with state-of-the-art methods on nuScenes and KITTI-360 occupancy prediction.The evaluations cover surrounding-camera prediction on nuScenes and monocular prediction on KITTI-360.
  • Surrounding-Camera 3D semantic occupancy prediction: GaussianFormer improves over planar representations on nuScenes and performs on par with dense-grid methods.The comparison includes BEVFormer and TPVFormer among planar methods, and OccFormer and SurroundOcc among dense-grid methods.
  • Monocular 3D semantic occupancy prediction: GaussianFormer performs competitively on KITTI-360, with advantages reported for motorcycle and other-vehicle categories.Its adaptive Gaussian positions and covariance can match small-object boundaries and accommodate objects with distinct shapes and appearances.
  • Efficiency comparisons with existing methods: GaussianFormer uses significantly less memory than existing representations while remaining competitive in latency.It is slightly slower than planar methods at approximately 70 ms but has the lowest latency among dense-grid representations.
  • Analysis of components of GaussianFormer: Residual refinement, sparse convolution, and deep supervision each contribute to GaussianFormer’s performance.Directly substituting Gaussian properties collapses because positions are sensitive to noise, while sparse convolution enables Gaussian interactions and deep supervision benefits intermediate refinement steps.
  • Effect of the number of Gaussians: Increasing the number of Gaussians beyond 38400 raises mIoU linearly, while latency and memory consumption also increase linearly.This trade-off provides flexibility for deployment by allowing the representation size to be adjusted according to efficiency requirements.

5 Conclusion and Discussions

GaussianFormer uses sparse, object-centric 3D semantic Gaussians and Gaussian-to-voxel splatting to represent scenes efficiently. It achieves comparable performance while reducing memory consumption, but remains inferior to state-of-the-art methods and requires many Gaussians.

  • Conclusion and Discussions: GaussianFormer represents scenes with sparse 3D Gaussians characterized by position, covariance, and semantics, then refines them from images through attention.Its Gaussian-to-voxel splatting module aggregates neighboring Gaussians for each voxel.
  • Conclusion and Discussions: GaussianFormer achieves comparable performance with state-of-the-art methods on nuScenes and KITTI-360 while reducing memory consumption by more than 75%.The method is evaluated across both datasets.
  • Conclusion and Discussions: Visualizations show that GaussianFormer captures object-shape details and allocates computation and storage resources reasonably.The Gaussians are visualized as ellipsoids whose centers and semi-axes derive from their means and covariance matrices.
  • Conclusion and Discussions: GaussianFormer remains inferior to state-of-the-art methods despite its lower memory consumption.The authors attribute this possibly to representation inaccuracies or limited hyperparameter tuning.
  • Conclusion and Discussions: The method requires a large number of Gaussians for satisfactory performance, partly because empty space is still modeled as a Gaussian category.The authors suggest modeling only solid objects as a possible improvement.

A Video Demonstration

The video demonstration shows GaussianFormer producing realistic and holistic semantic occupancy from surrounding images on nuScenes. The similarity between its Gaussians and occupancy predictions supports the representation’s expressiveness.

  • A Video Demonstration: GaussianFormer predicts realistic and holistic semantic occupancy using only surrounding images on the nuScenes validation set.The demonstration uses a sampled video frame.
  • A Video Demonstration: The similarity between the predicted 3D Gaussians and occupancy suggests that the representation is expressive.

B Additional Visualizations

Additional visualizations on KITTI-360 show GaussianFormer capturing overall scene structure and intricate details in monocular driving scenes. Gaussian density increases around vehicles, reflecting its object-centric allocation.

  • Additional Visualizations: Higher Gaussian density around vehicles demonstrates the object-centric representation and supports resource allocation.
  • Additional Visualizations: The visualization compares input images, predicted 3D Gaussians, predicted occupancy, and ground-truth occupancy.
  • Additional Visualizations: GaussianFormer captures overall structures and intricate details in KITTI-360 driving scenes from monocular input.

C Additional Ablation Study

Ablations examine Gaussian count, photometric supervision, splitting and pruning, initialization, and refinement visualizations. Performance improves with more Gaussians and splitting and pruning, while initialization—especially depth information—is important.

  • Additional Ablation Study: Performance improves consistently as the number of Gaussians increases.
  • Additional Ablation Study: Photometric supervision does not produce significant improvement on nuScenes, where surrounding cameras have little overlapping view.
  • Additional Ablation Study: Splitting and pruning improves performance compared with the baseline.
  • Additional Ablation Study: Initialization is important to performance, with depth information especially crucial across the tested initialization strategies.
  • Additional Ablation Study: The ablation study includes visualizations of refinement-layer outputs and corresponding mIoU, plus qualitative comparison with SurroundOcc.
Loading 2405.17429v1…