PaperlayerView exact revision on arXiv

Source-linked AI summary

BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection

Junjie Huang, Guan Huang

arXiv:2203.17054v3cs.CV

TL;DR

Single-frame vision-based multi-camera 3D detection is limited, especially for velocity prediction. BEVDet4D extends BEVDet with aligned temporal BEV-feature fusion and simplified velocity targets, achieving major velocity-error reductions and a 54.5% NDS result on nuScenes val. Its design retains a scalable framework with negligible added inference cost, while its alignment uses interpolation that can reduce precision.

  • Problem

    Single-frame vision-based multi-camera 3D detection has limited information and performs relatively poorly on time-relevant targets such as velocity.

  • Method

    BEVDet4D extends BEVDet by aligning and concatenating previous- and current-frame BEV features while simplifying velocity learning by removing ego-motion and time factors.

  • Results

    BEVDet4D-Tiny reduces velocity error by -62.9% from 0.909 mAVE to 0.337 mAVE, while BEVDet4D-Base reaches 54.5% NDS on nuScenes val.

  • Takeaways & Limitations

    BEVDet4D makes vision-based velocity precision comparable with LiDAR- or radar-based methods while adding negligible inference cost.

  • Takeaways & Limitations

    The interpolation used for spatial alignment is suboptimal and causes precision degeneration, with the magnitude negatively correlated with BEV-feature resolution.

Abstract

from arXiv · show

Single frame data contains finite information which limits the performance of the existing vision-based multi-camera 3D object detection paradigms. For fundamentally pushing the performance boundary in this area, a novel paradigm dubbed BEVDet4D is proposed to lift the scalable BEVDet paradigm from the spatial-only 3D space to the spatial-temporal 4D space. We upgrade the naive BEVDet framework with a few modifications just for fusing the feature from the previous frame with the corresponding one in the current frame. In this way, with negligible additional computing budget, we enable BEVDet4D to access the temporal cues by querying and comparing the two candidate features. Beyond this, we simplify the task of velocity prediction by removing the factors of ego-motion and time in the learning target. As a result, BEVDet4D with robust generalization performance reduces the velocity error by up to -62.9%. This makes the vision-based methods, for the first time, become comparable with those relied on LiDAR or radar in this aspect. On challenge benchmark nuScenes, we report a new record of 54.5% NDS with the high-performance configuration dubbed BEVDet4D-Base, which surpasses the previous leading method BEVDet-Base by +7.3% NDS. The source code is publicly available for further research at https://github.com/HuangJunJie2017/BEVDet .

1. Introduction

BEVDet4D extends BEVDet into the spatial-temporal 4D space to address weak vision-based prediction of time-relevant targets. It fuses aligned BEV features from adjacent frames with negligible extra inference cost and achieves substantial nuScenes improvements.

  • Motivation: Single-frame vision-based paradigms perform poorly on time-relevant targets such as velocity compared with LiDAR- and radar-based methods.BEVDet’s velocity error is reported as three times CenterPoint’s and twice CenterFusion’s.
  • Method: BEVDet4D retains previous-frame intermediate BEV features, spatially aligns them, and concatenates them with current-frame features.The remaining BEVDet framework details are largely unchanged.
  • Method: BEVDet4D accesses temporal cues by querying and comparing candidate features while adding negligible computational budget during inference.
  • Results: 62.9%: BEVDet4D-Tiny reduces velocity error from 0.909 mAVE to 0.337 mAVE on nuScenes val.It also improves mAP by +2.6%, mAOE by -12.0%, and mAAE by -25.1%.
  • Results: 54.5% NDS: BEVDet4D-Base surpasses the previous leading vision-based results on nuScenes val.The configuration also scores 42.1% mAP.

2. Related Works

Vision-based 3D detection has expanded from image-view methods toward multi-camera BEV representations, but temporal cues remain underused. BEVDet4D addresses this gap through end-to-end temporal feature fusion rather than postprocessing or multi-frame attention-heavy designs.

  • Vision-based 3D object detection: Image-view methods such as FCOS3D perform relatively well on appearance-related attributes but are weaker for translation, velocity, and orientation.
  • Vision-based 3D object detection: BEVDet encodes features in BEV space and is scalable to multi-task learning, multi-sensor fusion, and temporal fusion.
  • Temporal modeling: Few vision-based 3D detection methods exploit temporal cues, leaving velocity prediction weaker than in LiDAR- or radar-based methods.
  • Temporal modeling: BEVDet4D uses temporal cues in an end-to-end learning framework, unlike approaches that apply a 3D Kalman filter during postprocessing.
  • Temporal modeling: BEVFormer achieves comparable velocity precision by fusing features from four adjacent frames with an attention mechanism.

3. Methodology

BEVDet4D extends BEVDet with temporal fusion of aligned BEV features from adjacent frames, while simplifying velocity learning by removing time and ego-motion factors from the target.

  • Temporal feature fusion: BEVDet4D retains previous-frame BEV features and merges them with current-frame features after alignment, using an additional BEV encoder before temporal fusion.The baseline image-view encoder, view transformer, BEV encoder, and task-specific head remain unchanged; two residual units adjust sparse candidate features.
  • Simplify the velocity learning task: Predicting adjacent-frame target translation removes the time factor and makes positional shifting measurable from differences between BEV features.The method also targets shifting that is irrelevant to ego-motion, simplifying the learning task.
  • Feature alignment: Figure 3 contrasts unaligned features, which require modules to model motion tied to ego-motion, with aligned features that simplify the learning targets.The figure uses a static target example to illustrate how ego-motion changes the apparent motion distribution.
  • Simplify the velocity learning task: Feature alignment compensates for ego-motion before fusion so the learning target is object motion in the current frame’s ego coordinate system.Without alignment, static objects appear to move in ego coordinates and target distributions become more complicated.
  • Feature alignment: The alignment operation is implemented by transforming the previous-frame feature into the current frame’s coordinate system before interpolation.Bilinear interpolation may query invalid positions in sparse previous-frame features and can cause precision degeneration; the effect is estimated in ablation studies.

4.1. Experimental Settings

Experiments use the large-scale nuScenes benchmark and its official detection metrics, with standard BEVDet training and data-processing settings and reported test-set comparisons.

  • Dataset: nuScenes contains 1000 scenes with six surrounding-view cameras, five radars, one LiDAR, and up to 1.4M annotated boxes across ten classes.The official split is 700/150/150 scenes for training, validation, and testing.
  • Evaluation metrics: The evaluation reports mAP, ATE, ASE, AOE, AVE, AAE, and NDS, with NDS combining the other indicators into a composite detection score.mAP measures precision and recall using ground-plane center-distance matching rather than IoU.
  • Training parameters: Models are trained with AdamW, gradient clipping, a 2e-4 learning rate, batch size 64 across eight RTX 3090 GPUs, and a cyclic learning-rate policy.The learning rate rises to 1e-3 during the first 40% of training and then decreases to zero.
  • Data processing: Training data processing follows BEVDet with random flipping, scaling, rotation, and cropping of 1600×900 source images to the selected input size.The default random scaling and rotation ranges are specified in the experimental protocol.
  • Comparison protocol: Table 2 compares nuScenes test-set state-of-the-art methods, with a dagger marking pre-training on DDAD.The comparison includes methods and sensor paradigms described in the accompanying experimental discussion.
  • Inference speed: Inference speed is averaged over 6019 validation samples, with monocular methods’ speeds divided by six and BEVDet acceleration enabled by default.The division accounts for the six images in each nuScenes sample.

4.2. Benchmark Results

On nuScenes, BEVDet4D improves vision-based 3D detection over BEVDet across composite accuracy and time-relevant prediction, while retaining competitive speed and requiring only negligible extra inference latency.

  • nuScenes val set: 47.6% NDS: BEVDet4D-Tiny surpasses BEVDet-Tiny’s 39.2% NDS by +8.4% on the nuScenes validation set.The gain mainly derives from reduced orientation, velocity, and attribute errors.
  • nuScenes val set: -62.9% velocity error: BEVDet4D-Tiny reduces mAVE from 0.909 mAVE to 0.337 mAVE versus BEVDet-Tiny.Its velocity precision exceeds CenterFusion’s 0.540 mAVE and is comparable to PointPillar’s 0.323 mAVE at similar inference speed.
  • nuScenes val set: -12.0% mAOE and -25.1% mAAE: BEVDet4D-Tiny reduces orientation error from 0.523 mAOE to 0.460 mAOE and attribute error from 0.247 mAAE to 0.185 mAAE.The paper attributes the orientation improvement partly to strong coupling between orientation and velocity.
  • nuScenes val set: 54.5% NDS: BEVDet4D-Base improves over BEVDet-Base’s 47.2% NDS by +7.3% on nuScenes validation.It surpasses BEVFormer’s 51.7% NDS by +2.8% while running at 1.9 FPS versus 1.7 FPS, and reaches 55.2% NDS with test-time augmentation.
  • nuScenes test set: 56.9% NDS: BEVDet4D ranks first on the nuScenes vision-based 3D object detection test leaderboard, exceeding BEVDet by +8.7% NDS and BEVFormer by +3.4% NDS.The reported test result uses BEVDet4D-Base trained on the train and validation sets with test-time augmentation.

4.3. Ablation Studies

The ablations progressively refine temporal feature alignment, learning targets, fusion placement, and test-time interval to improve BEVDet4D’s velocity and overall detection performance.

  • Temporal alignment and targets: Direct feature concatenation reduces NDS from 39.2% to 37.6% by -1.6%, particularly degrading translation and velocity performance.Ego-motion creates positional shifts between corresponding features, complicating object localization and velocity prediction.
  • Temporal alignment and targets: Translation-only alignment reduces translation error by -5.4% to 0.672 and velocity error by -23.2% from 1.544 mAVE to 1.186 mAVE.The velocity target is simplified by removing the ego-motion component, but inconsistent frame intervals still limit the result.
  • Temporal alignment and targets: Removing the time factor further reduces velocity error by -59.6% to 0.479 mAVE, while an extra BEV encoder raises NDS by +0.9% from 44.0% to 44.9%.The module predicts positional shift directly, simplifying the learning target and improving validation robustness.
  • Temporal alignment and targets: Considering ego-pose rotation reduces velocity error by 13.6% from 0.435 to 0.376, indicating that precise alignment improves velocity prediction.The comparison is between configurations Tab. 3 (E) and (F).
  • Test-time interval: The optimal test interval is around 15T, where T ≈0.083s denotes the interval between adjacent 12Hz camera sweeps.The interval is selected by comparing three test-time choices in each training configuration.
  • Interpolation precision: At high BEV resolution of 0.4m×0.4m, alignment methods differ negligibly, with 45.2 NDS versus 45.3 NDS, while higher resolution reduces interpolation degeneration.Aligning within the view transformer is deprecated for inference acceleration despite its precision advantage at lower resolution.
  • Temporal fusion position: Temporal fusion after the extra BEV encoder achieves the lowest velocity error of 0.429 mAVE; moving fusion earlier raises it by +11.9% to 0.480 mAVE.Moving fusion later reduces overall performance to 39.4% NDS, close to the 39.2% NDS BEVDet baseline.

5. Conclusion

BEVDet4D extends BEVDet from spatial-only 3D into spatial-temporal 4D for multi-camera 3D object detection, especially velocity prediction. It preserves BEVDet’s scalable design while improving performance.

  • Conclusion: BEVDet4D lifts BEVDet from a spatial-only 3D working space into a spatial-temporal 4D working space for vision-based autonomous driving.The paper presents this as a first attempt to exploit temporal information in this setting.
  • Conclusion: The paradigm retains BEVDet’s scalability while substantially improving multi-camera 3D object detection, particularly velocity prediction.Future work is directed toward frameworks and paradigms that actively mine temporal cues.
Loading 2203.17054v3…