Source-linked AI summary
RenderOcc: Vision-Centric 3D Occupancy Prediction with 2D Rendering Supervision
Mingjie Pan, Jiaming Liu, Renrui Zhang, Peixiang Huang, Xiaoqi Li, Bing Wang, Hongwei Xie, Li Liu, Shanghang Zhang
TL;DR
3D occupancy models typically require costly and ambiguous 3D labels, limiting scalable training. RenderOcc replaces them with 2D semantic and depth supervision through volume rendering, using Auxiliary-Rays to address sparse viewpoints. Experiments show competitive performance against 3D-label-supervised baselines, supporting 2D supervision as a viable training direction.
Problem
3D occupancy training depends on costly, ambiguous complete 3D labels, including about 4000 human hours to purify 30000 frames and 10–15% annotation variation.
Method
RenderOcc extracts a NeRF-style 3D volume from multi-view images, renders 2D semantic and depth outputs, and adds Auxiliary-Rays from adjacent frames for sparse viewpoints.
Results
RenderOcc achieves competitive performance using only 2D labels compared with baselines supervised by 3D labels on NuScenes and SemanticKiTTI.
Takeaways & Limitations
The experiments support 2D image supervision as a feasible alternative for training 3D occupancy models without complete 3D occupancy annotations.
Abstract
from arXiv · showhide
3D occupancy prediction holds significant promise in the fields of robot perception and autonomous driving, which quantifies 3D scenes into grid cells with semantic labels. Recent works mainly utilize complete occupancy labels in 3D voxel space for supervision. However, the expensive annotation process and sometimes ambiguous labels have severely constrained the usability and scalability of 3D occupancy models. To address this, we present RenderOcc, a novel paradigm for training 3D occupancy models only using 2D labels. Specifically, we extract a NeRF-style 3D volume representation from multi-view images, and employ volume rendering techniques to establish 2D renderings, thus enabling direct 3D supervision from 2D semantics and depth labels. Additionally, we introduce an Auxiliary Ray method to tackle the issue of sparse viewpoints in autonomous driving scenarios, which leverages sequential frames to construct comprehensive 2D rendering for each object. To our best knowledge, RenderOcc is the first attempt to train multi-view 3D occupancy models only using 2D labels, reducing the dependence on costly 3D occupancy annotations. Extensive experiments demonstrate that RenderOcc achieves comparable performance to models fully supervised with 3D labels, underscoring the significance of this approach in real-world applications.
I. INTRODUCTION
RenderOcc addresses the cost and ambiguity of 3D occupancy annotation by training multi-view 3D occupancy networks from 2D labels through rendering supervision. Auxiliary-Rays extend supervision across sparse viewpoints, while experiments report competitive performance against 3D-label-supervised baselines.
- Motivation: 3D occupancy annotation remains expensive and ambiguous, requiring about 4000 human hours to purify 30000 frames and differing by 10–15% across identical raw data.These costs and inconsistencies restrict practical use of 3D occupancy models.
- RenderOcc: RenderOcc constructs a NeRF-style 3D volume from multi-view images and renders 2D outputs for direct supervision of 3D occupancy learning.This rendering process supports multi-view consistency through intersecting rays from different cameras.
- RenderOcc: RenderOcc trains multi-view 3D occupancy networks solely with 2D labels, avoiding costly and challenging 3D occupancy annotation.The framework uses 2D rendering supervision to provide fine-grained pixel-level semantic and depth supervision.
- Auxiliary-Rays: Auxiliary-Rays use adjacent-frame rays to address sparse viewpoints in autonomous-driving scenarios and improve the learned 3D voxel representation.A dynamic sampling strategy balances and purifies these auxiliary rays.
- Results: RenderOcc achieves competitive performance with only 2D labels compared with baselines supervised by 3D labels, while also outperforming prior state-of-the-art methods when using 2D and corresponding 3D labels.Experiments are conducted on NuScenes and SemanticKiTTI.
II. RELATED WORK
Related work spans 3D occupancy prediction and rendering-oriented 3D representation methods. Occupancy prediction targets dense semantic scene understanding, whereas rendering methods prioritize visual quality and provide ideas for occupancy learning.
- 3D Occupancy Prediction: 3D occupancy prediction generates dense voxelized semantic scene representations and has become an important capability for autonomous vehicles.Recent benchmarks extend evaluation from front-view predictions to multi-view surrounding scenes.
- 3D Occupancy Prediction: Existing occupancy methods rely on extra 3D labels because manually annotating dense 3D occupancy is difficult.This annotation dependence motivates alternative supervision strategies such as RenderOcc.
- Rendering-Based Methods: Rendering-oriented methods emphasize rendering quality rather than semantic understanding and lack generalization, but their training ideas can inform 3D occupancy.Their focus differs from occupancy prediction’s dense semantic representation objective.
A. Problem Setup
RenderOcc replaces voxel-wise 3D-label supervision with a Semantic-Density-Field representation that supports 2D rendering supervision. The optimized field can then be converted into 3D occupancy using density-based occupancy filtering and semantic classification.
- A. Problem Setup: Multi-view images are transformed into 3D volume features, which an occupancy head maps to a dense semantic volume over voxel categories.The volume dimensions are H×W×D, and L includes the empty category.
- C. Semantic Density Field: The Semantic-Density-Field enables semantic rendering from arbitrary viewpoints, allowing the network to be optimized with 2D labels instead of complete voxel labels.The framework extracts multi-view features, predicts density and semantics, and performs volume rendering for supervision.
- C. Semantic Density Field: RenderOcc predicts voxel density and semantic logits with separate mappings to construct its Semantic-Density-Field.Density uses softplus activation so predicted density values remain nonnegative.
- C. Semantic Density Field: After optimization, occupied voxels are selected using a density threshold, while their semantic categories are determined from the semantic representation.Voxels meeting the occupancy condition receive the category given by the maximum semantic score; others receive the empty label.
D. Rendering Supervision with 2D Labels
RenderOcc bridges the Semantic-Density-Field and pixel-level supervision by rendering semantic and depth predictions along camera rays. Auxiliary rays from adjacent frames address limited viewpoint overlap, while semantic, depth, and regularization losses train the field.
- D. Rendering Supervision with 2D Labels: Each image pixel is associated with a camera ray carrying semantic and depth labels, and the Semantic-Density-Field is volume-rendered to produce corresponding predictions.This creates a direct training bridge between 3D field representations and 2D labels.
- D. Rendering Supervision with 2D Labels: Sampled points along each ray are combined using accumulated transmittance and termination probabilities to render semantic and depth values.The sampled points lie within a predefined range on the ray, and adjacent-point distance contributes to the rendering weights.
- D. Rendering Supervision with 2D Labels: Auxiliary rays from adjacent frames supplement sparse single-frame viewpoints and provide additional multi-view consistency constraints.The method targets the limited overlap and small perspective differences among surrounding cameras.
- D. Rendering Supervision with 2D Labels: Cross-entropy and SILog losses supervise rendered semantics and depth, while distortion and total-variation losses regularize the Semantic-Density-Field.The losses jointly train the rendering predictions and the field representation.
E. Auxiliary Rays: Boosting Multi-view Consistency
Auxiliary rays from adjacent frames address sparse viewpoint coverage, while weighted sampling prioritizes informative and temporally reliable rays during training.
- Auxiliary Rays: Auxiliary rays from adjacent frames complement sparse single-frame viewpoints and strengthen multi-view consistency constraints.The method transforms rays from nearby frames into the current frame to increase viewpoint coverage.
- Weighted Ray Sampling: Weighted Ray Sampling addresses increased computational cost and temporal mismatches introduced by auxiliary rays.It samples a fixed number of rays using probability weights rather than retaining every ray.
- Weighted Ray Sampling: Category Density Balance increases sampling emphasis for scarce, informative categories such as pedestrians and bicycles.Background rays from roads and buildings are abundant but lower-information-density, whereas pedestrian and bicycle rays are scarce.
- Weighted Ray Sampling: Temporal Misalignment Purification reduces the sampling probability of auxiliary rays affected by moving objects.The strategy masks dynamic objects and preserves current-frame rays to reduce incorrect voxel supervision.
- Weighted Ray Sampling: The final ray weight combines category-balance and temporal-misalignment weights as W = Wb ·Wt.These weights determine random sampling probabilities, reducing memory and computation while supporting performance.
IV. EXPERIMENTS
RenderOcc is evaluated against other baselines on NuScenes and SemanticKiTTI, with additional ablation experiments conducted on NuScenes.
- Experiments: RenderOcc is compared with other baselines on the NuScenes and SemanticKiTTI benchmarks.The evaluation covers both benchmark comparison and deeper analysis of the proposed method.
- Experiments: Extensive ablation experiments are conducted on NuScenes to analyze RenderOcc more deeply.
A. Dataset
The experiments use NuScenes for surrounding-view perception and SemanticKiTTI for front-view perception, covering outdoor driving scenes with different camera and occupancy configurations.
- Dataset: NuScenes contains 1000 outdoor driving scenes captured with six surrounding-view cameras.Its occupancy ground truth spans a specified 3D range, uses 0.4m voxel size, and includes 17 classes.
- Dataset: SemanticKiTTI contains 22 outdoor driving scenes and focuses on areas ahead of the car.It is used for the front-view setting.
B. Architecture and Implementation Details
RenderOcc uses BEVStereo as the underlying network and replaces its classification head with semantic-density-field prediction heads to support 2D rendering supervision.
- Architecture: BEVStereo serves as the network G, while its classification head is replaced with semantic density field heads.The remaining majority of the original network structure is retained.
- Implementation Details: Training uses a Swin Transformer backbone, 512x1408 resized images, Adam optimization, batch size 16, and approximately 10K iterations.The learning rate is set to 1e-4.
C. Main Results
RenderOcc achieves competitive occupancy prediction using 2D supervision, matching closely supervised baselines on NuScenes and remaining competitive on SemanticKITTI. Its strengths include small-object and boundary perception, while single-view settings create optimization limitations.
- 23.93 mIoU is achieved on NuScenes using only 2D labels, just 0.58 mIoU below BEVStereo and above MonoScene, OccFormer, and BEVFormer.With combined 2D and 3D supervision, performance reaches 26.11 mIoU.
- RenderOcc performs especially well on static background categories and small foreground objects but shows artifacts on dynamic objects.The Auxiliary Rays strategy alleviates the dynamic-object issue to some extent.
- 8.24 mIoU is achieved on SemanticKITTI using only 2D labels, remaining competitive with methods trained on 3D occupancy ground truth.Adding existing 3D labels raises performance to 12.87 mIoU.
- RenderOcc captures object boundaries and small objects more accurately than the 3D-supervised baseline in qualitative NuScenes results.It perceives a crane arm and distant traffic cones that the baseline misses.
- SemanticKITTI’s single front-facing camera limits multi-view consistency, trapping 2D rendering supervision in local optima.Auxiliary rays partially mitigate this limitation, while 3D labels overcome the optimization problem.
D. Ablation Study & Analysis
Ablations show that depth supervision, auxiliary rays, and weighted ray sampling each improve RenderOcc, while weighted sampling controls the computational cost of auxiliary rays. Additional auxiliary frames improve mIoU but increase memory use.
- Component ablations: Adding depth supervision increases mIoU by 2.34 over semantic-only 2D supervision, while auxiliary rays add a further 3.13 mIoU.RenSup-S achieves 16.94 mIoU before these additions.
- Component ablations: Weighted Ray Sampling yields a total mIoU improvement of 1.52 and addresses the training overhead introduced by auxiliary rays.It focuses sampling on rays with higher information density.
- Auxiliary-ray analysis: As auxiliary frames increase from 0 to 6, mIoU improves from 19.28 to 23.01.The improvement is accompanied by significantly higher GPU memory use when ray numbers double.
- Auxiliary-ray analysis: Weighted Ray Sampling keeps ray numbers fixed at 38400 while achieving higher mIoU without additional training overhead.It selects valuable rays with high information density.
- Depth supervision: Sparse image-derived depth labels still produce 21.11 mIoU, compared with 23.44 mIoU using raw LiDAR depth supervision.Without depth supervision, performance is 16.9 mIoU.
V. CONCLUSION
RenderOcc demonstrates that 2D image labels can train 3D occupancy networks while avoiding costly and ambiguous 3D occupancy annotations. Its experiments validate this framework as a new perspective for vision-centric occupancy training.
- RenderOcc uses 2D image labels to train 3D occupancy networks, circumventing costly and ambiguous 3D occupancy-label production.
- Extensive experiments validate the effectiveness of this framework and offer a new perspective for the community.