PaperlayerView exact revision on arXiv

Source-linked AI summary

Unifying Voxel-based Representation with Transformer for 3D Object Detection

Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, Jiaya Jia

arXiv:2206.00630v2cs.CV

TL;DR

UVTR addresses the challenge of representing camera and LiDAR inputs together for robust 3D detection. It unifies both modalities in explicit, uncompressed voxel spaces, adds knowledge transfer and fusion, and uses a transformer decoder for object-level prediction. The framework reports leading nuScenes detection and tracking performance across single- and multi-modality settings.

  • Problem

    Camera and LiDAR offer complementary information, but images lack accurate depth for natural voxel representation and prior alignment methods can damage spatial structure or create semantic ambiguity.

  • Method

    UVTR represents image and point-cloud features in modality-specific uncompressed voxel spaces, enables spatial and cross-modality interaction, and decodes object predictions with learnable-position transformer sampling.

  • Results

    UVTR attains leading nuScenes test-set performance, reaching 69.7%, 55.1%, and 71.1% NDS for point-cloud, image, and multi-modality detection, respectively, with leading tracking results as well.

  • Takeaways & Limitations

    A unified voxel representation supports single- and multi-modality detection while facilitating cross-modality knowledge transfer and feature fusion.

  • Takeaways & Limitations

    Constructing voxel space for multi-view images requires processing all images in a shared backbone, creating computational cost that is especially pronounced for multi-frame settings.

Abstract

from arXiv · show

In this work, we present a unified framework for multi-modality 3D object detection, named UVTR. The proposed method aims to unify multi-modality representations in the voxel space for accurate and robust single- or cross-modality 3D detection. To this end, the modality-specific space is first designed to represent different inputs in the voxel feature space. Different from previous work, our approach preserves the voxel space without height compression to alleviate semantic ambiguity and enable spatial connections. To make full use of the inputs from different sensors, the cross-modality interaction is then proposed, including knowledge transfer and modality fusion. In this way, geometry-aware expressions in point clouds and context-rich features in images are well utilized for better performance and robustness. The transformer decoder is applied to efficiently sample features from the unified space with learnable positions, which facilitates object-level interactions. In general, UVTR presents an early attempt to represent different modalities in a unified framework. It surpasses previous work in single- or multi-modality entries. The proposed method achieves leading performance in the nuScenes test set for both object detection and the following object tracking task. Code is made publicly available at https://github.com/dvlab-research/UVTR.

1 Introduction

UVTR addresses the difficulty of unifying camera and LiDAR inputs by preserving explicit 3D voxel structure and enabling cross-modality interaction. It reports leading detection performance across point-cloud, image, and multi-modality settings.

  • Motivation: Camera and LiDAR provide complementary information, but images cannot naturally occupy voxel space because cameras lack accurate depth.Point clouds provide accurate positions, while images provide rich context.
  • Motivation: Existing input-level alignment damages spatial structure through inaccurate depth or collapsed 3D geometry, while frustum-to-BEV methods introduce semantic ambiguity through height compression.These limitations motivate preserving the voxel space without height compression.
  • Approach: UVTR represents image and point-cloud features in an explicit voxel space, uses voxel encoding for spatial interaction, and supports cross-modality learning.Its design includes image-plane sampling for images and natural voxel association for point clouds.
  • Approach: The unified representation supports 3D spatial interaction, cross-modality knowledge transfer and fusion, and synchronized augmentation without complex alignment.These are presented as key advances over previous and concurrent studies.
  • Results: 69.7%, 55.1%, and 71.1% NDS are achieved on the nuScenes test set with point clouds, images, and multi-modality inputs, respectively.UVTR also reports 67.0%, 51.9%, and 70.1% AMOTA in the corresponding tracking settings.

2 Related Work

Related work organizes unified 3D detection around LiDAR, camera, and cross-modality representations. Prior approaches align modalities through transformed inputs or compressed feature spaces, whereas UVTR retains fine-grained voxel structure.

  • LiDAR-based 3D Detection: LiDAR-based detectors process point clouds with point, voxel, or range-view representations before generating predictions, commonly in bird-eye-view space.Voxel methods use regular grids with sparse or 2D convolutions.
  • LiDAR-based 3D Detection: UVTR follows the voxel-based pipeline but preserves fine-grained voxel space without height compression.This design is shown as distinct from approaches that collapse the vertical dimension.
  • Camera-based 3D Detection: Camera-based methods use direct image features, intermediate representations, or constructed 3D geometry volumes for single- and multi-view detection.Recent methods project multi-view features into frustum space with predicted depth before collapsing them to BEV.
  • Cross-modality Interaction: Cross-modality interaction includes modality fusion and knowledge transfer, with fusion performed at point or instance level.These approaches seek to combine information from different sensors during feature learning or prediction.

3 UVTR Framework

UVTR constructs modality-specific voxel spaces, interacts with them spatially, transfers or fuses cross-modality features, and predicts through a transformer decoder. Images are mapped into voxels using predicted depth and calibration, while point clouds retain explicit 3D structure.

  • Framework Overview: UVTR constructs modality-specific spaces, designs cross-space feature learning, and uses a transformer decoder for object-level interaction and prediction.The framework supports single- and multi-modality inputs through modality selection.
  • Modality-specific Space: Image voxel features are formed from multi-view or multi-frame backbone features by sampling into predefined voxels using predicted depth distributions and geometric constraints.The depth distribution is generated from image features, while calibration maps voxel sampling points to image-plane coordinates.
  • Modality-specific Space: For multi-frame images, calibrated voxel spaces are aligned to the initial ego frame, enriched with relative time offsets, concatenated, and fused.The shared network processes all frames before space-level fusion.
  • Modality-specific Space: Point clouds are split into regular voxels and processed with sparse convolutions and multi-scale heads, while the 3D voxel space remains uncompressed.Preserving the vertical dimension supports cross-modality and fine-grained object interaction.
  • Voxel Encoder: Voxel encoders aggregate adjacent features across coplanar and vertical dimensions to establish local spatial connections.The encoder uses three basic convolutional blocks in each modality-specific space.
  • Cross-modality Interaction: Knowledge transfer uses point-cloud geometry to guide image features during training, while modality fusion adds processed modality features and applies convolution in the unified voxel space.Knowledge transfer targets single-modality inference; fusion uses both modalities during training and inference.
  • Transformer Decoder: The transformer decoder initializes object queries and reference points, samples unified voxel features with deformable attention, and iteratively refines predicted boxes.The decoder uses three blocks for LiDAR-based settings and six for camera-based settings.
  • Training: Hungarian matching trains one-to-one query assignments with classification and box-regression losses, optionally adding knowledge-transfer loss weighted by 0.01.The full network is optimized end to end.

4 Experiments

Experiments on nuScenes evaluate UVTR across component ablations, modalities, conditions, and downstream tracking. The results show benefits from preserving voxel height, spatial interaction, multi-frame input, cross-modality transfer and fusion, with leading detection and tracking performance.

  • Experimental Setup: UVTR is evaluated on nuScenes using synced LiDAR and six-camera data across single- and multi-modality settings.The benchmark contains 700, 150, and 150 train, validation, and test scenes, respectively.
  • Component-wise Analysis: Camera-based NDS improves by 3.1% and 4.2% when voxel height increases to 5 and 11, while LiDAR gains are smaller.The height axis is especially important for camera-based voxel representations; LiDAR benefits less because positions are accurate.
  • Component-wise Analysis: 3D spatial interaction improves NDS by 2.6% for camera-based and 0.6% for LiDAR-based methods over 2D convolution.Without spatial interaction, the camera-based network cannot converge, whereas LiDAR remains satisfactory without the voxel encoder.
  • Component-wise Analysis: Using 5 and 10 sweeps yields performance gaps of 5% for camera-based and 18.1% for LiDAR-based methods.Both modalities achieve consistent gains as more multi-frame inputs are represented in voxel space.
  • Component-wise Analysis: Knowledge transfer provides up to 1.9% NDS gain for camera-based students and reaches 2.6% NDS and 1.8% mAP with context features.LiDAR-based student gains saturate at 0.6% NDS.

5 Discussion and Conclusion

UVTR unifies voxel-based representations across modalities and supports knowledge transfer, modality fusion, and object-level transformer interactions for 3D detection. It achieves leading performance, while remaining limited by image-processing cost, voxel resolution, and unresolved autonomous-driving corner cases.

  • Discussion and Conclusion: UVTR uniformly encodes sensor inputs in modality-specific voxel spaces, reducing semantic ambiguity and enabling spatial interaction.Its unified space supports cross-modality knowledge transfer and modality fusion, while a transformer decoder facilitates object-level interactions.
  • Discussion and Conclusion: UVTR achieves consistent improvements over benchmarks and leading performance on nuScenes across evaluated detection settings.
  • Limitations: Image voxel construction processes all multi-view images through a shared backbone, increasing computational cost, especially for multi-frame settings.
  • Limitations: The current voxel space uses 128 × 128 resolution, while higher resolution and more image frames remain unexplored possibilities for better results.
  • Societal Impacts: The current technique cannot solve all corner cases and extreme situations, creating potential risks in real-world autonomous-system decision processes.

A Experimental Details

The experimental details describe unified sampling and voxel-space augmentation for aligning LiDAR and camera training, alongside modality-specific schedules, inference settings, and head comparisons. The section also documents the evaluation tables and computational setup used to assess UVTR.

  • Unified Sampler: A unified sampler generates point clouds and image crops, attaching sampled objects to scenes and reorganizing crops by actual depth for multi-modality training.The sampler is disabled during the last two epochs to fit the normal distribution.
  • Unified Augmentation: Global scaling, rotation, and flipping are applied to the image voxel space after construction so image and point-cloud modalities share voxel-space augmentations.
  • Training Schedule: Multi-modality training uses 20 fine-tuning epochs, while pretrained modality branches and CBGS reduce the total schedule to 10 epochs under the described procedure.
  • Experimental Comparisons: Tables 11 and 12 compare multi-modality methods and L2 versus partial L2 distance, with L2C denoting LiDAR-to-camera knowledge transfer.
  • Training Setting: Batches are constructed on 8 devices with 32, 8, and 16 inputs for LiDAR-, camera-, and multi-modality settings, respectively.Most models use NVIDIA V100 GPUs; memory-intensive multi-frame or multi-modality models use NVIDIA A100 GPUs.
  • Model Inference: Table 13 reports model inference runtime on the nuScenes validation set using a single NVIDIA Tesla V100 GPU.
  • Model Inference: Inference retains 300 top-scoring predictions within the specified spatial ranges, with runtime costs distributed across modality-specific backbones, view transformation, and the decoder.
  • Decoder Head: The transformer decoder provides object-level interaction and efficient voxel-space feature capture, and is compared with the convolution-based CenterPoint head.For CenterPoint compatibility, the unified voxel space is compressed along Z after voxel encoding.

B Qualitative Analysis

Qualitative visualizations show that UVTR detects most objects across LiDAR, camera, and multi-modality views, while each modality exhibits complementary strengths. Multi-modality predictions are generally accurate but can miss far or small objects.

  • Qualitative Analysis: UVTR visualizations on LiDAR BEV and camera views show that most objects are detected in the nuScenes scenes.
  • Multi-modality Results: Multi-modality predictions accurately estimate object location, category, speed, and orientation compared with ground truth.
  • Multi-modality Results: Multi-modality detection still misses some objects, especially those that are far away or small.
  • Visualization Setup: Figures 6 and 7 visualize UVTR predictions with different modalities on the nuScenes validation set, marking areas needing attention with dotted red boxes.
  • LiDAR-based Results: LiDAR-based detection can misclassify objects without image context; a tree is detected as a vehicle in the last row of Figure 6.
  • Camera-based Results: Camera-based predictions have less accurate locations because images lack precise positions, but image context improves recognition.
  • Camera-based Results: Barriers are well detected by camera-based and multi-modality methods, unlike the LiDAR-based method in the second row of Figure 7.
Loading 2206.00630v2…