Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction

Yuanhui Huang; Wenzhao Zheng; Yunpeng Zhang; Jie Zhou; Jiwen Lu

Source-linked AI summary

Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction

Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, Jiwen Lu

arXiv:2302.07817v2Feb 15, 2023cs.CVcs.AIcs.LG

TL;DR

BEV is efficient for vision-based 3D perception but struggles to represent fine-grained 3D structure, motivating a richer scene representation. The paper introduces TPV and TPVFormer to lift image features into three perpendicular planes, and reports sparse-supervision occupancy prediction with camera-only LiDAR-segmentation performance comparable to LiDAR-based methods.

Problem
BEV efficiently represents scenes but has difficulty describing fine-grained 3D structures with a single plane.
Method
TPV augments BEV with two perpendicular planes, sums projected features for each 3D point, and uses TPVFormer attention to lift image features into TPV space.
Results
Using only camera inputs, TPVFormer predicts semantic occupancy for all voxels with sparse point supervision and achieves comparable performance with LiDAR-based methods on nuScenes LiDAR segmentation.
Takeaways & Limitations
TPV provides an efficient representation for fine-grained 3D structures and supports vision-based semantic occupancy prediction from sparse supervision.
Takeaways & Limitations
BEV reduces storage and computation to O(HW) but completely omits the z-axis, adversely affecting expressiveness.

Abstract

from arXiv · show

Modern methods for vision-centric autonomous driving perception widely adopt the bird's-eye-view (BEV) representation to describe a 3D scene. Despite its better efficiency than voxel representation, it has difficulty describing the fine-grained 3D structure of a scene with a single plane. To address this, we propose a tri-perspective view (TPV) representation which accompanies BEV with two additional perpendicular planes. We model each point in the 3D space by summing its projected features on the three planes. To lift image features to the 3D TPV space, we further propose a transformer-based TPV encoder (TPVFormer) to obtain the TPV features effectively. We employ the attention mechanism to aggregate the image features corresponding to each query in each TPV plane. Experiments show that our model trained with sparse supervision effectively predicts the semantic occupancy for all voxels. We demonstrate for the first time that using only camera inputs can achieve comparable performance with LiDAR-based methods on the LiDAR segmentation task on nuScenes. Code: https://github.com/wzzheng/TPVFormer.

1. Introduction

Vision-based 3D perception needs representations that retain fine-grained structure without the computational burden of full voxel grids. The paper proposes TPV and TPVFormer, then evaluates sparse-supervision occupancy prediction through proxy tasks.

Motivation: Vision-based perception offers a promising alternative to LiDAR-based perception but lacks direct depth sensing.It has nevertheless shown promising performance in depth estimation, semantic map reconstruction, and 3D object detection.
Motivation: Voxel representations describe fine-grained 3D structure accurately, but their large computational cost motivates more compact alternatives.Voxel features have storage and computation complexity proportional to O(HWD), whereas BEV reduces this burden by collapsing height.
Problem: BEV methods efficiently encode scenes for coarse 3D detection, but a flattened vector struggles to represent objects with varied 3D structures.This limitation motivates a more comprehensive and fine-grained understanding of 3D surroundings for occupancy prediction.
Method: TPV augments BEV with two perpendicular planes and represents each 3D point by summing bilinearly interpolated features projected onto all three planes.The representation supports arbitrary resolution and produces different features for different 3D points.
Method: TPVFormer lifts 2D image information into TPV space using image cross-attention and cross-view hybrid attention over TPV queries.The proposed task uses sparse LiDAR semantic labels for training while requiring semantic occupancy predictions for all voxels at test time.

2. Related Work

Prior work uses voxel, BEV, and implicit scene representations to balance 3D structure, efficiency, and resolution. TPV is motivated by the difficulty of applying compact BEV representations to fine-grained occupancy prediction.

Voxel-based Scene Representation: Voxel-based representations assign vectors to discretized 3D locations and support fine-grained structure modeling for segmentation and scene completion.Their inefficiency makes vision-centric multi-view applications challenging.
BEV-based Scene Representation: BEV representations improve efficiency by implicitly encoding height information in each ground-plane grid.Research has focused on transforming image features into BEV using depth prediction or BEV-query cross-attention.
BEV-based Scene Representation: Although BEV performs strongly for vision-centric 3D detection, it is difficult to apply to semantic occupancy prediction requiring fine-grained 3D descriptions.This exposes a task-specific limitation of collapsing the scene into a single plane.
Implicit Scene Representation: Implicit representations model a scene with a continuous function mapping 3D coordinates to point representations.They offer arbitrary-resolution modeling and computation-efficient architectures compared with explicit voxel and BEV representations.

3. Proposed Approach

The proposed TPV representation extends BEV with three orthogonal planes, preserving full-scale 3D structure without voxel-level cubic complexity. TPVFormer lifts multi-camera image features into these planes through deformable attention and cross-view interactions, then combines the plane features for occupancy prediction.

Scene Representation: Voxel features preserve fine-grained 3D structure but require O(HWD) storage and computation, challenging real-time onboard deployment.BEV reduces this burden but suppresses the height axis.
Scene Representation: BEV represents each point using a ground-plane projection, reducing complexity to O(HW) but omitting z-axis information and limiting expressiveness.A BEV feature corresponds to a pillar spanning the full real-world z-axis.
TPV Representation: TPV models the 3D space with three axis-aligned orthogonal planes representing top, side, and front views without suppressing any axis.Each point is projected onto the three planes, whose sampled features are aggregated to form its 3D feature.
TPV Representation: TPV expands into a full-scale 3D feature space with O(HW + DH + WD) storage and computation, an order of magnitude lower than voxel complexity.Its orthogonal planes diversify features along directions that a single BEV plane ignores.
TPVFormer: TPV queries encode view-specific pillar information, interact across planes for context, and aggregate projected plane features for 3D semantic occupancy prediction.The framework uses a lightweight prediction head on the summed projected features.
TPVFormer: TPVFormer lifts multi-scale, multi-camera image features to TPV planes with deformable image cross-attention and cross-view hybrid attention.HCAB blocks combine visual querying and cross-view attention, while HAB blocks specialize in contextual encoding.

4. Experiments

The experiments evaluate TPVFormer on vision-based semantic occupancy prediction, nuScenes LiDAR segmentation, and SemanticKITTI semantic scene completion using RGB inputs. Results show dense occupancy predictions, comparable LiDAR-segmentation performance, scene-completion gains, and design sensitivities.

Task descriptions: TPVFormer is evaluated on 3D semantic occupancy prediction, LiDAR segmentation, and semantic scene completion using only RGB images.The first two tasks use Panoptic nuScenes, while semantic scene completion uses SemanticKITTI.
Task descriptions: Sparse LiDAR semantic labels supervise training, while 3D semantic occupancy prediction requires semantic labels for all voxels at test time.Because no benchmark exists for this setting, the paper reports qualitative analysis.
3D semantic occupancy prediction: TPVFormer produces denser and more realistic occupancy predictions than LiDAR-segmentation ground truth and correctly predicts objects missed by Cylinder3D in an example scene.Querying LiDAR points yields predictions close to ground truth in the visualization comparison.
3D semantic occupancy prediction: TPVFormer adjusts TPV-plane resolution at test time without retraining; increasing resolution from 50x50x4 to eight times larger captures finer object details such as shape.The lightweight segmentation head enables interpolation-based resolution adjustment.
LiDAR segmentation: TPVFormer-Base achieves comparable mIoU (∼70%) with most LiDAR-based methods on nuScenes LiDAR segmentation despite using only 2D image inputs.This is presented as the first vision-based method for the LiDAR segmentation task.
Semantic scene completion: On SemanticKITTI semantic scene completion, TPVFormer outperforms all compared methods in IoU and mIoU, using 6.0M parameters and 128G FLOPS per image versus MonoScene’s 15.7M and 500G.The comparison uses RGB-inferred versions of the baselines.
Ablation study: Using both voxel and point predictions in the loss keeps their mIoUs high and close, whereas single-type supervision favors the corresponding prediction type.The authors associate this with learning both voxel-space discretization and point-feature interpolation.
Ablation study: Increasing TPV-plane resolution improves fine-grained structure modeling more directly than increasing feature dimension, and TPVFormer outperforms BEVFormer across configurations.The ablation attributes the advantage to three complementary cross-views.

5. Conclusion

The paper presents TPV and TPVFormer for efficient fine-grained 3D scene description from images. Visualization and nuScenes results support consistent occupancy prediction with sparse supervision and comparable performance to LiDAR-based methods.

5. Conclusion: TPV represents a 3D scene with three mutually perpendicular planes to describe fine-grained structure efficiently.TPVFormer lifts image features into these planes using attention.
5. Conclusion: TPVFormer produces consistent semantic voxel occupancy predictions when trained with only sparse point supervision.The conclusion bases this claim on visualization results.
5. Conclusion: Using vision alone, TPVFormer achieves comparable performance with LiDAR-based methods on the nuScenes LiDAR segmentation task.The paper presents this as the first such demonstration.

A. Dataset Details

The experiments use Panoptic nuScenes and SemanticKITTI, which provide multi-camera imagery alongside LiDAR-derived annotations for autonomous-driving scene understanding.

Panoptic nuScenes: Panoptic nuScenes contains 1,000 twenty-second driving scenes with six RGB cameras, a 360° horizontal field of view, and 32-beam LiDAR data.The official split is 700 training, 150 validation, and 150 test scenes; keyframes are annotated at 2Hz.
SemanticKITTI: SemanticKITTI voxelizes automotive LiDAR scans into 256 × 256 × 32 grids with 0.2m voxels and 21 labels.RGB images from cam2 are also used, with 22 sequences split into 10/1/11 for training, validation, and test.

B. Implementation Details

Implementation choices vary by task and model size while retaining TPV-based processing. Training uses AdamW-style optimization and task-specific image backbones, resolutions, and losses.

3D occupancy and LiDAR segmentation: TPVFormer-Base uses ResNet101-DCN initialized from FCOS3D, while TPVFormer-Small uses ImageNet-pretrained ResNet-50.Their TPV resolutions are 200x200x16 and 100x100x8, respectively, with 128-dimensional features.
Optimization: Training uses AdamW with learning rate 2e-4, weight decay 0.01, cosine scheduling, 500-iteration linear warmup, and 24 epochs on eight A100 GPUs.The batch size is 8 and image augmentation follows BEVFormer.
Semantic scene completion: For semantic scene completion, TPVFormer uses a pretrained EfficientNetB7-based 2D UNet and 128x128x16 TPV planes to match MonoScene’s voxel tensor size.The model uses cam2 RGB images cropped to 1220x370 and a feature dimension of 96.

C. 3D Semantic Occupancy Prediction Results

TPVFormer produces detailed 3D semantic occupancy from surround-camera images with sparse LiDAR supervision, including small and rare objects that are barely visible in RGB images.

TPVFormer successfully predicts bicycles, motorcycles, and pedestrians despite their limited visibility in the RGB images.The visualizations highlight these small and rare objects and connect them to corresponding ground truths.

D. LiDAR segmentation Results

On nuScenes LiDAR segmentation, TPVFormer-Base reaches parity with LiDAR-based methods and substantially outperforms a comparable BEVFormer-Base model.

12.7% higher mIoU than BEVFormer-Base demonstrates TPV’s effectiveness for modeling fine-grained 3D scene structures.The comparison uses self-attention and matched feature dimensions to make the model sizes comparable.
TPVFormer-Base achieves mIoU on par with LiDAR-based methods despite using a different sensing modality.

E. Semantic Scene Completion Results

On SemanticKITTI semantic scene completion, TPVFormer does not achieve the highest IoU but attains a clear-margin advantage in mIoU over other methods.

TPVFormer outperforms other methods in mIoU with a clear margin for semantic scene completion.Its IoU is not the highest, so the reported advantage is specifically in mean intersection over union.