Source-linked AI summary
SelfOcc: Self-Supervised Vision-Based 3D Occupancy Prediction
Yuanhui Huang, Wenzhao Zheng, Borui Zhang, Jie Zhou, Jiwen Lu
TL;DR
3D occupancy prediction needs dense 3D supervision that is laborious to obtain, motivating learning from video alone. SelfOcc lifts images into 3D, regularizes the representation as an SDF, and uses temporal rendering with MVS-based depth optimization. It outperforms prior self-supervised occupancy methods and extends reasonable occupancy prediction to surround cameras while achieving strong depth results.
Problem
3D occupancy prediction usually requires laborious 3D occupancy or semantic supervision, limiting learning from large collections of unlabeled driving videos.
Method
SelfOcc lifts image features into 3D, represents them as an SDF field, and uses temporal supervision with an MVS-embedded strategy and multiple depth proposals.
Results
SelfOcc outperforms SceneRF by 58.7% in IoU on SemanticKITTI, produces reasonable surround-camera occupancy on Occ3D-nuScenes, and achieves strong depth results across several benchmarks.
Takeaways & Limitations
SelfOcc provides self-supervised vision-based occupancy prediction for both monocular and surround-camera settings using video sequences only.
Takeaways & Limitations
SelfOcc cannot synthesize high-quality novel views because of blurring and lacks specific designs for motion.
Abstract
from arXiv · showhide
3D occupancy prediction is an important task for the robustness of vision-centric autonomous driving, which aims to predict whether each point is occupied in the surrounding 3D space. Existing methods usually require 3D occupancy labels to produce meaningful results. However, it is very laborious to annotate the occupancy status of each voxel. In this paper, we propose SelfOcc to explore a self-supervised way to learn 3D occupancy using only video sequences. We first transform the images into the 3D space (e.g., bird's eye view) to obtain 3D representation of the scene. We directly impose constraints on the 3D representations by treating them as signed distance fields. We can then render 2D images of previous and future frames as self-supervision signals to learn the 3D representations. We propose an MVS-embedded strategy to directly optimize the SDF-induced weights with multiple depth proposals. Our SelfOcc outperforms the previous best method SceneRF by 58.7% using a single frame as input on SemanticKITTI and is the first self-supervised work that produces reasonable 3D occupancy for surround cameras on nuScenes. SelfOcc produces high-quality depth and achieves state-of-the-art results on novel depth synthesis, monocular depth estimation, and surround-view depth estimation on the SemanticKITTI, KITTI-2015, and nuScenes, respectively. Code: https://github.com/huang-yh/SelfOcc.
1. Introduction
Vision-centric autonomous driving depends on 3D scene representations, but obtaining them commonly requires laborious 3D annotations. SelfOcc explores self-supervised 3D occupancy learning from video sequences by lifting image features into 3D and optimizing an SDF field.
- Motivation: 3D representations such as BEV or TPV support vision-centric autonomous-driving tasks from 2D images.These representations are used for semantic mapping, 3D detection, and semantic occupancy prediction.
- Motivation: Existing methods generally rely on 3D annotations, whose laborious collection limits scalable training.
- Approach: SelfOcc lifts 2D image features into 3D space to enable feature interactions and reduce ambiguities across cameras.
- Approach: It represents the 3D scene as a signed distance function field, enabling regularization and direct determination of occupancy boundaries.
- Approach: An MVS-embedded strategy optimizes SDF-induced weights with multiple depth proposals using temporal consistency from video sequences.
2. Related Work
Prior 3D occupancy methods commonly use explicit 3D inputs or ground-truth supervision, while self-supervised NeRF-based methods address monocular settings. SelfOcc is positioned as a method that supports both monocular and surround-view occupancy learning.
- 3D Occupancy Prediction: Pioneering 3D occupancy methods use depth, occupancy grids, point clouds, or TSDFs as 3D inputs.
- 3D Occupancy Prediction: Supervised occupancy approaches require laborious 3D ground-truth annotation despite their promising performance.
- Self-supervised Occupancy: BTS and SceneRF learn occupancy self-supervisedly in monocular scenarios, but adapting them to surround views is nontrivial.
- Neural Radiance Fields: Neural radiance fields reconstruct scenes by mapping spatial locations to radiance and density and rendering novel views through volume integration.
- Self-supervised Depth Prediction: BTS and SceneRF incorporate reprojection losses from self-supervised depth estimation to enable explicit depth optimization in generalizable NeRFs.
3. Proposed Approach
SelfOcc learns vision-based 3D occupancy from video by lifting image features into a 3D representation, modeling it as an SDF field, and rendering views for self-supervision. Its pipeline combines multi-view-consistency supervision, MVS-based depth optimization, and SDF regularization to produce occupancy predictions.
- From Image to Occupancy: SelfOcc lifts 2D image features into BEV or TPV representations using deformable attention, enabling 3D feature interactions and reducing multi-camera ambiguity.Learnable 3D tokens query local image features using camera projection matrices, with interleaved attention blocks refining the representation.
- From Occupancy to Image: SelfOcc projects predicted occupancy back into 2D and uses differentiable volume rendering to synthesize color and depth views supervised by temporal multi-view consistency.The framework renders color, depth, and semantic views from 3D features, then applies 0-thresholding to obtain the occupancy volume.
- From Image to Occupancy: The decoder transforms the 3D representation into an SDF field, whose signed values determine whether continuous 3D points are occupied.The SDF represents distance to the nearest object surface; bicubic interpolation provides values at continuous coordinates, and occupancy follows from their sign.
- Occupancy-Oriented Supervision: The MVS-embedded loss evaluates multiple uniformly spaced depth proposals along each ray, extending optimization across the epipolar line rather than relying on local bilinear interpolation.Each proposal is weighted by its volume-rendering weight, providing a broader depth prior and addressing misleading initialization or low-texture regions.
- Occupancy-Oriented Supervision: SDF regularization combines Hessian smoothness, occupancy sparsity, and the Eikonal constraint to restrict sparse-view solutions and preserve the field’s physical meaning.The method also optionally uses off-the-shelf 2D segmentation maps to supervise semantic prediction.
4. Experiments
SelfOcc is evaluated on dense occupancy, novel depth synthesis, and depth estimation across nuScenes, SemanticKITTI, and KITTI-2015, using image sequences without 3D supervision. Experiments also analyze architecture, temporal supervision, and loss regularization, with strong results across the evaluated tasks.
- Task Descriptions: The experiments evaluate dense occupancy and semantic states on Occ3D-nuScenes, geometry-only occupancy on SemanticKITTI, novel depth synthesis, and depth estimation on nuScenes and KITTI-2015.IoU and mIoU measure geometric and semantic occupancy, while depth tasks use standard error and threshold metrics.
- 3D Occupancy Prediction: SelfOcc achieves comparable IoU and reasonable mIoU to supervised approaches on Occ3D-nuScenes without 3D supervision, while reaching 45.01% IoU.It outperforms MonoScene in mIoU and LiDAR-supervised TPVFormer in IoU, but predicts background classes better than foreground classes.
- 3D Occupancy Prediction: 58.7%: SelfOcc outperforms SceneRF in IoU for vision-based depth-supervised and self-supervised 3D occupancy prediction on SemanticKITTI.The improvement comes with much higher Precision and slightly lower Recall.
- Novel Depth Synthesis: SelfOcc outperforms SceneRF on all novel depth synthesis metrics on SemanticKITTI and also achieves better results after adaptation to nuScenes surround views.The novel-depth evaluation averages metrics over frames no farther than 10 meters from the input frame.
- Depth Estimation: SelfOcc achieves state-of-the-art self-supervised surround-view depth estimation on nuScenes and performs on par with state-of-the-art self-supervised monocular depth estimation on KITTI-2015.The nuScenes comparison also evaluates SurroundDepth with ground-truth poses to isolate pose effects.
- Architecture: MVS-embedded depth optimization consistently improves all three tasks, while SDF fields outperform density fields for occupancy and depth estimation but underperform for novel depth synthesis.The authors attribute the novel-depth result to the SDF field’s inherent smoothness prior, which is more difficult to optimize for that task.
5. Conclusion
SelfOcc is a self-supervised vision-based method that predicts meaningful geometry and semantics for 3D occupancy from video sequences only. It sets a new state of the art on SemanticKITTI and produces reasonable surround-camera occupancy on Occ3D for the first time.
- SelfOcc predicts meaningful geometry and semantics for autonomous-driving 3D occupancy using video sequences only.
- Conditioning NeRFs on 3D representations facilitates feature extraction in the 3D space.
- An MVS-embedded strategy boosts depth optimization for NeRFs.
- SelfOcc sets a new state of the art for self-supervised vision-based occupancy prediction on SemanticKITTI.
- SelfOcc produces reasonable 3D occupancy for surrounding cameras on Occ3D for the first time.
- Figure 6 visualizes semantic occupancy from six surrounding input images in global and bird’s-eye views on the nuScenes validation set.
A. Dataset Details
The evaluation uses nuScenes, KITTI-2015, and SemanticKITTI datasets with different camera configurations, sequence splits, and occupancy or LiDAR annotations. These datasets cover surround-camera, stereo, and monocular driving settings.
- nuScenes contains 1000 driving sequences split into 700 training, 150 validation, and 150 test sequences.Each sequence lasts 20 seconds and includes LiDAR and RGB images from six surround cameras, with keyframes annotated at 2Hz.
- Occ3D-nuScenes provides 3D semantic occupancy annotations at 200x200x16 resolution for 18 classes.
- KITTI-2015 provides stereo images from two forward-facing cameras and LiDAR point clouds.Using the Eigen split and removing static training images yields 39,810 monocular training triplets and 4,424 validation triplets.
- SemanticKITTI provides voxelized LiDAR scans for 22 sequences at 256x256x32 resolution.Voxels have 0.2m sides and 21 labels; the official split uses 10 training, 1 validation, and 11 test sequences.
B. Additional Implementation Details
The implementation combines a 2D semantic segmentor with dataset-specific geometric and image-resolution settings. Training uses fixed loss weights on eight RTX-3090 GPUs, with runtimes varying by dataset.
- nuScenes semantic supervision uses the tiny OpenSeeD segmentor trained on COCO2017 and Objects365v1.All classes remain included when calculating mIoU despite some classes being ignored by OpenSeeD.
- The represented cuboid covers [80, 80, 6.4] meters around the nuScenes ego car and [51.2, 51.2, 6.4] meters ahead for SemanticKITTI and KITTI-2015.Grid-cell resolution is 0.4m for nuScenes and 0.2m for SemanticKITTI and KITTI-2015.
- Input images have resolutions of 384x800 for nuScenes, 370x1220 for SemanticKITTI, and 320x1024 for KITTI-2015.
- Models are trained on eight RTX-3090 GPUs with 24GB memory.SemanticKITTI and KITTI-2015 experiments take less than one day, while nuScenes experiments finish within two days.
C. Mathematical Derivation
The mathematical derivation explains why MVS-embedded depth optimization improves on traditional reprojection loss. Moving the summation outside the dissimilarity metric and using multiple depth candidates enlarges the optimization receptive field.
- The section derives the advantages of MVS-embedded depth optimization over traditional reprojection loss.
- Traditional reprojection loss is expanded through bilinear interpolation to expose its optimization structure.
- Bilinear interpolation fixes the indexed corner pixels after perspective transformation, limiting the receptive field to four adjacent pixels.This restriction adversely affects the efficiency and stability of depth learning.
- MVS-embedded optimization moves the summation outside the dissimilarity metric and incorporates multiple depth candidates along each ray.These design choices effectively enlarge the receptive field.
D.1. 3D Occupancy Prediction
SelfOcc produces geometric and semantic occupancy predictions on nuScenes and SemanticKITTI, including occluded regions and accurate vehicle shapes. Its temporal supervision also supports novel depth synthesis and sharp depth estimation across challenging objects and scenes.
- SelfOcc successfully infers semantic and geometric occupancy in occluded areas on the nuScenes validation set.
- SelfOcc predicts accurate car shapes and sizes on SemanticKITTI without occupancy shadows.
- Temporal supervision enables SelfOcc to predict 3D structures beyond visible surfaces for high-quality novel depth views.The reported view transformations include translations along the x- and y-axes and ±10° yaw changes.
- SelfOcc predicts sharp, accurate depth for vehicles, thin poles, moving pedestrians, and cyclists on nuScenes and KITTI-2015.
E. Limitations and Future Work
SelfOcc’s main limitation is blurred novel-view synthesis despite color supervision. The authors identify motion awareness and higher-quality novel views as future-work directions.
- SelfOcc cannot synthesize high-quality novel views and suffers from blurring despite using color supervision.
- The authors identify high-quality novel-view synthesis and motion awareness as potential focuses for future work.