PaperlayerView exact revision on arXiv

Source-linked AI summary

OPUS-V2: Bridging the Gap between Sparse Points and Dense Voxels

Jiabao Wang, Qiang Meng, Liujiang Yan, Ke Wang, Qibin Hou, Ming-Ming Cheng

arXiv:2608.29187v1cs.CV

TL;DR

Point-based occupancy prediction offers an accuracy–efficiency trade-off but requires heuristic operations to align sparse predictions with dense voxel ground truth. OPUS-V2 introduces a learnable point-voxel transformation that directly generates dense occupancy, achieving state-of-the-art rayIoU on Occ3D while improving efficiency and scalability.

  • Problem

    Point-based predictions cannot directly match dense voxel ground truth, requiring heuristic training and post-processing operations that can be unstable and inaccurate.

  • Method

    OPUS-V2 integrates OPUS with a lightweight learnable point-voxel transformation module that converts sparse decoder points and features into voxel-based occupancy.

  • Results

    44.0 rayIoU on Occ3D establishes a new state of the art, while experiments on Occ3D and OpenOccupancy report improved accuracy and efficiency over prior methods.

  • Takeaways & Limitations

    Direct end-to-end dense-output generation removes manual post-processing, while decoupling feature extraction from occupancy generation supports adaptable output resolutions.

  • Takeaways & Limitations

    Sparse convolutions in the PVT module make deployment on edge devices more difficult.

Abstract

from arXiv · show

The point-based occupancy prediction paradigm has achieved an attractive trade-off between accuracy and efficiency by modeling 3D space sparsely. However, its predictions inherently mismatch the dense voxel-based occupancy required by self-driving systems, necessitating hand-crafted heuristics during training and inference that limit final performance. To overcome these limitations, we propose OPUS-V2, a novel framework built upon the pioneering OPUS (occupancy prediction using a sparse set) point-based approach. OPUS-V2 incorporates a lightweight point-voxel transformation (PVT) module behind the decoder to adaptively map sparse predictions into the dense voxel space, eliminating the need for suboptimal operations and improving model accuracy. Furthermore, our architecture decouples feature and occupancy generation processes, allowing OPUS-V2 to adapt to arbitrary occupancy resolutions. OPUS-V2 achieves a state-of-the-art rayIoU of 44.0 on the Occ3D dataset. On the more challenging OpenOccupancy dataset, it attains a competitive 16.4 mIoU while running in real time at 20.6 FPS.

1 Introduction

OPUS-V2 addresses the mismatch between sparse point predictions and dense voxel ground truth by introducing a learnable PVT module that removes heuristic alignment operations and supports spatial scaling. Experiments report improved accuracy and real-time performance on Occ3D and OpenOccupancy.

  • Motivation: Point-based occupancy prediction improves efficiency through sparse representations but must ultimately produce dense voxel outputs for autonomous-driving planning.Point modeling preserves precise occupancy boundaries while maintaining high sparsity.
  • Problem: Sparse predictions cannot directly match dense ground truth, so OPUS relies on nearest-neighbor label assignment and approximate voxel post-processing.These heuristics obstruct direct prediction–annotation interaction during training and inference.
  • Method: OPUS-V2 adds a lightweight point-voxel transformation module that learnably converts sparse points into voxel space, eliminating manual operations.The module enables direct interaction between predictions and voxel-based ground truth.
  • Method: OPUS-V2 decouples decoder feature extraction from occupancy generation, allowing occupancy resolution scaling within the PVT module.This design supports adaptation to broader perceptual ranges and finer voxel granularity.
  • Results: 44.0 rayIoU was achieved by OPUS-V2 on Occ3D, establishing a new state-of-the-art result alongside an mIoU of 38.6.The largest variant achieved these reported Occ3D results.
  • Results: 20.6 FPS and 16.4 mIoU were achieved on OpenOccupancy despite its broader coverage and finer voxel granularity.The result demonstrates real-time occupancy generation under more demanding spatial scaling.

2 Related Work

Related work develops occupancy representations that trade off efficiency, flexibility, and modeling capacity. OPUS uses sparse point sets and transformer-based set prediction, but this reformulation creates a gap between sparse predictions and dense ground truth.

  • Voxel representations: Dense voxel-based occupancy methods provide intuitive 3D geometry and semantics but waste computation in unoccupied regions.This limitation motivates sparse occupancy representations.
  • Gaussian representations: Gaussian-based representations improve flexibility over voxels, but ellipsoidal shape assumptions limit modeling of diverse structures.GaussianFormer and GaussianFormer-2 reduce or aggregate Gaussian kernels for efficiency.
  • Point representations: Point-based representations model arbitrary occupancy shapes sparsely, supporting efficiency across outdoor, dynamic, four-dimensional, and indoor settings.Examples include OPUS, ODG, SparseWorld, and DiScene.
  • Point representations: OPUS inherits transformer-based unordered set prediction, avoiding complex model design and improving efficiency through sparse point outputs.Its framework follows the set-prediction paradigm introduced by DETR and SparseBEV.

3.1 Preliminaries

OPUS reformulates occupancy prediction as sparse point-set prediction, then uses heuristic operations to align sparse outputs with dense voxel ground truth during training and inference. These heuristics introduce instability and post-processing errors.

  • Point-based occupancy prediction paradigm: OPUS reformulates occupancy prediction as generating a sparse set of points that mimics occupied voxel annotations.Occupied voxels are collected as point-set annotations, with coordinates representing voxel centers and classes representing semantic labels.
  • Point-based occupancy prediction paradigm: OPUS decouples prediction into separate regression and classification objectives, using Chamfer distance for point-position regression.Nearest-neighbor matching constructs a proxy ground-truth point set for supervising classification.
  • Point-based occupancy prediction paradigm: During testing, OPUS filters low-quality points, voxelizes the remainder, and applies 3D dilation and erosion through max-pooling.The resulting dense occupancy prediction is compared with voxel ground truth for evaluation.
  • The problem of heuristic operations: Sparse point predictions cannot directly align with dense voxel annotations, requiring nearest-neighbor proxy construction during training and post-processing during inference.These hand-crafted operations bridge the prediction-format mismatch so training and testing can proceed.
  • The problem of heuristic operations: The nearest-neighbor proxy ignores surrounding information and is sensitive to prediction variations, making training unstable.Its local optimum property contributes to this sensitivity.
  • The problem of heuristic operations: More than 2K voxels are mislabeled during post-processing in most Occ3D scenes, with some scenes exceeding 10K mislabeled voxels.These voxels were correctly represented by predicted points but incorrectly classified in the final occupancy output.

3.2 Design of OPUS-V2

OPUS-V2 combines transformer-based sparse point prediction with a lightweight PVT module that learnably produces dense voxel occupancy. It separately supervises sparse geometry and dense semantic occupancy, avoiding handcrafted representation-conversion operations.

  • Decoder structure: Decoder queries iteratively aggregate image geometry and semantic information while producing updated point positions and latent point features.Each decoder updates queries and positions, then generates latent features for its predicted points.
  • Architecture: OPUS-V2 uses image features, decoder-generated 3D sparse points with latent features, and a lightweight PVT module for dense occupancy prediction.The PVT adaptively transforms sparse features into dense voxel space.
  • Decoder structure: Learnable prototype sampling points are re-weighted by positional dispersion, preventing gradient vanishing in the sampling-offset layer.Sampled points are projected into image space to interpolate related features before query updating.
  • The point-voxel transformation module: PVT augments point features with relative query-center and voxel-center positions, aggregates points within each voxel, and generates sparse voxel predictions with two convolution layers.Dynamic VFE layers use FCN refinement and max-pooling; SparseConv3D propagates information to empty voxels, followed by SubMConv3D on non-empty voxels.
  • PVT and other sparse voxel-based models: PVT bridges sparse decoder predictions and dense occupancy, removing handcrafted training proxies and post-processing while retaining a lightweight sparse voxel-based design.The approach differs from prior sparse voxel heads by explicitly learning scene geometry through transformer-generated points.
  • Training objectives: OPUS-V2 separately supervises sparse point geometry with re-weighted Chamfer distance and dense voxel occupancy with class-balanced focal loss at every stage.An additional Chamfer-distance term optimizes the initial point-position distribution.

4.1 Experimental Setup

OPUS-V2 is evaluated on Occ3D and OpenOccupancy, which differ in voxel resolution, semantic classes, and perceptual range. Evaluation uses mIoU on both datasets and additionally reports distance-thresholded rayIoU on Occ3D.

  • Datasets: Occ3D uses 0.4 m voxels, ±40 m range, and 18 classes, whereas OpenOccupancy uses 0.2 m voxels, ±51.4 m range, and 17 classes.Both benchmarks are large-scale driving occupancy datasets.
  • Metrics: mIoU is reported on both datasets, while Occ3D additionally uses rayIoU at 1 m, 2 m, and 4 m thresholds.The final Occ3D rayIoU is the average of the three threshold-specific values.
  • Implementation: The experiments resize images to 704 × 256 and extract features with a ResNet50 backbone.Training also uses resizing, flipping, and random occupancy-flipping augmentation.
  • Implementation: OPUS-V2 is evaluated in T, S, M, and L configurations with 0.6K, 1.2K, 2.4K, and 4.8K queries, respectively.The number of predicted points increases across decoder stages, reaching 76.8K in the final stage.
  • Implementation: Chamfer-loss reweighting increases weights for uncovered ground-truth voxels and rare categories, each by 5.The reweighting targets coverage and low-frequency categories such as others, bicycle, construction vehicle, and traffic cone.

4.2 Main Results

OPUS-V2 improves occupancy prediction accuracy and efficiency over OPUS and prior methods on Occ3D, while scaling effectively to the more demanding OpenOccupancy setting.

  • 44.0 rayIoU was achieved by OPUS-V2-L on Occ3D, exceeding OPUS-L by 2.8 rayIoU and 2.4 mIoU.
  • 43.6 rayIoU was achieved after 50 epochs, only 0.4 below the 100-epoch counterpart, indicating faster convergence.The authors attribute the improved training efficiency to removing unstable nearest-neighbor matching.
  • OPUS-V2 achieved superior rayIoU at real-time speed and competitive mIoU against prior state-of-the-art methods.OPUS-V2-T surpassed FlashOcc in accuracy by up to 3.5 rayIoU; OPUS-V2-L surpassed ALOcc in both accuracy and speed.
  • 16.4 mIoU was attained by OPUS-V2-T on OpenOccupancy, while the heaviest model reached 18.1 mIoU.The 18.1 mIoU result established a new maximum for camera-based methods with a 4 mIoU advantage.
  • 20.6 FPS was maintained by OPUS-V2-T on OpenOccupancy after occupancy volume increased by more than 8 times.Models transferred from Occ3D to OpenOccupancy by enlarging only the PVT module; OPUS-V2-L reached 8.0 FPS.

4.3 Ablation and Related Studies

Ablation studies show that PVT, richer point features, decoder depth, and loss settings affect accuracy, speed, and convergence in distinct ways.

  • 15.1 mIoU was achieved by OPUS-V2-T under the shared 30-epoch single-frame setting, outperforming SparseOcc.Increasing input resolution improved performance to 15.7 mIoU, while 1900 × 600 inputs retained 6.9 FPS.
  • Replacing hand-crafted training and testing operations with PVT produced much higher accuracy than the original OPUS-like strategy.Models using PVT directly generated voxel occupancy without hand-crafted post-processing during testing.
  • 41.0 rayIoU was obtained with semantic and positional features, improving over 40.2 rayIoU from semantic features alone.Adding relative query-center and voxel-center positions provided a further 0.1 rayIoU gain.
  • Increasing decoder layers from 3 to 7 consistently improved accuracy but reduced inference speed.A 5-decoder configuration surpassed all 6-decoder original OPUS variants and was selected as the speed-accuracy balance.
  • Both focal loss and chamfer distance loss decreased smoothly during training across the tested chamfer-loss weights.The study examined chamfer distance loss on sparse predictions and focal loss on dense predictions.

4.4 Visualization

Visualizations compare sparse point predictions with dense voxel outputs and show that PVT refines geometric quality across datasets.

  • Sparse predictions roughly outlined environmental geometry and focused on occupied regions, supporting accuracy and efficiency.Point flexibility also produced uneven density and edge artifacts.
  • PVT suppressed noisy sparse points when producing dense voxel-based occupancy.
  • OPUS-V2 reduced surface holes and edge noise compared with OPUS within a ±40 m range.The visualization attributes this improvement to the effectiveness of PVT.
  • OpenOccupancy-trained OPUS-V2 produced sharper and more precise occupancy predictions than the Occ3D-trained model.The comparison indicates that higher-resolution ground truth supports learning finer geometric details.

5 Conclusion

OPUS-V2 uses PVT to bridge sparse decoder predictions and dense voxel occupancy while decoupling feature extraction from occupancy generation. Experiments report strong accuracy, efficiency, and scalability, with edge deployment remaining a challenge.

  • OPUS-V2 transforms sparse decoder predictions into dense voxel occupancy through a learnable PVT module.This enables direct end-to-end generation of dense outputs and eliminates manual post-processing.
  • Decoupling decoder feature extraction from PVT occupancy generation lets OPUS-V2 adapt its output resolution.
  • OPUS-V2 outperformed prior state-of-the-art methods in accuracy and efficiency on Occ3D and OpenOccupancy while demonstrating superior scalability.
  • Sparse convolutions in PVT make edge-device deployment more difficult.The authors plan to replace them with 2D convolutions to facilitate deployment on edge devices.
Loading 2608.29187v1…