PaperlayerView exact revision on arXiv

Source-linked AI summary

Sparse4D v3: Advancing End-to-End 3D Detection and Tracking

Xuewu Lin, Zixiang Pei, Tianwei Lin, Lichao Huang, Zhizhong Su

arXiv:2311.11722v1cs.CVcs.AIcs.RO

TL;DR

Sparse 3D perception faces convergence challenges, while detection and tracking are central autonomous-driving tasks. Sparse4Dv3 adds auxiliary training tasks and decoupled attention, and extends Sparse4D into an end-to-end tracker. On nuScenes, the paper reports state-of-the-art performance in detection and tracking.

  • Problem

    Sparse-based 3D detectors face convergence challenges from unstable one-to-one matching and sparse positive samples, limiting decoder training efficiency.

    Source: §1 · p. 2

  • Method

    Sparse4Dv3 adds temporal instance denoising, quality estimation, and decoupled attention, and extends Sparse4D into an end-to-end tracking model.

    Source: §1 · p. 3

  • Results

    On nuScenes, Sparse4Dv3 achieves state-of-the-art performance in both detection and tracking.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    The tracker outputs motion trajectories without detector-training changes or ground-truth tracking IDs.

    Source: §1 · p. 2

    The tracking attempts are preliminary, and tracking performance has substantial room for improvement.

    Source: §5 · p. 9

Abstract

from arXiv · show

In autonomous driving perception systems, 3D detection and tracking are the two fundamental tasks. This paper delves deeper into this field, building upon the Sparse4D framework. We introduce two auxiliary training tasks (Temporal Instance Denoising and Quality Estimation) and propose decoupled attention to make structural improvements, leading to significant enhancements in detection performance. Additionally, we extend the detector into a tracker using a straightforward approach that assigns instance ID during inference, further highlighting the advantages of query-based algorithms. Extensive experiments conducted on the nuScenes benchmark validate the effectiveness of the proposed improvements. With ResNet50 as the backbone, we witnessed enhancements of 3.0\%, 2.2\%, and 7.6\% in mAP, NDS, and AMOTA, achieving 46.9\%, 56.1\%, and 49.0\%, respectively. Our best model achieved 71.9\% NDS and 67.7\% AMOTA on the nuScenes test set. Code will be released at \url{https://github.com/linxuewu/Sparse4D}.

1 Introduction

Sparse4Dv3 targets convergence and feature-interference challenges in sparse 3D perception with auxiliary training tasks and decoupled attention, then extends Sparse4D to end-to-end tracking. Experiments on nuScenes demonstrate state-of-the-art performance in detection and tracking.

1 Introduction

  • Sparse-based methods avoid image-to-3D vector-space transforms and keep detection-head computation independent of perception distance and image resolution.The paper identifies these properties as advantages over dense-BEV-based algorithms.Source: §1 · p. 1
  • Temporal instance denoising stabilizes positive matching and increases positive samples, while quality estimation improves confidence-based detection ranking.Source: §1 · pp. 2–3
  • Decoupled attention replaces addition with concatenation to reduce incorrect attention weights and feature interference.The paper attributes outlier attention weights to adding anchor embeddings and instance features before attention calculation.Source: §1 · pp. 2–3
  • Sparse4D is extended into an end-to-end tracking model that outputs object motion trajectories without modifying detector training or losses.Source: §1 · pp. 2–3
  • Sparse4Dv3 demonstrates state-of-the-art performance in both detection and tracking on nuScenes.Source: §1 · p. 3

2 Related Works

Related work covers convergence and detection improvements for end-to-end detectors, alongside query-based approaches to multi-object tracking. Sparse4D-v3 is presented against these lines of work as a sparse perception framework and a tracker that avoids changes to detector training and inference strategies.

2.1 Improvements for End-to-End Detection

  • DETR improvements address convergence, attention, query initialization, and training matching; Sparse4D-series methods add instance-feature decoupling, multipoint sampling, and temporal fusion.Source: §2.1 · p. 3

2.2 Multi-Object Track

  • End-to-end tracking methods use track queries and inter-frame transfer, while Sparse4D-v3 requires no detector-training or inference-strategy changes and no ground-truth tracking IDs.Source: §2.2 · p. 3

3 Methodology

The methodology adds temporal instance denoising and quality estimation to Sparse4D, modifies attention through concatenated feature encoding, and extends the detector to tracking through inference-time ID assignment.

3.1 Temporal Instance Denoising

  • Temporal denoising adds GT-noised anchors to learnable anchors, uses per-group bipartite matching for positive and negative labels, and propagates selected noisy groups across frames.Ego-pose and velocity compensation move anchors forward; instance features initialize the next frame, while group independence prevents cross-group feature interaction and limits each GT to at most one positive sample. At test time, only learnable instances are retained.Source: §3.1 · p. 4, Figure 4 · p. 4

3.2 Quality Estimation

  • Quality estimation adds centerness and yawness predictions alongside classification confidence to represent positive-sample quality and support convergence and prediction ranking.The stated losses are cross-entropy and focal loss, respectively.Source: §3.2 · p. 5

3.3 Decoupled Attention

  • Decoupled attention independently encodes anchor components and concatenates them, modifying query attention while retaining Sparse4D’s deformable aggregation for query-image cross-attention.The architecture caption reports lower computational and parameter overhead than original Sparse4D; E and F denote anchor embedding and instance feature.Source: §3.3 · p. 5, Figure 5 · p. 5

3.4 Extend to Tracking

  • Tracking redefines detections as trajectories and assigns an ID when confidence exceeds threshold T; the ID persists during temporal propagation, with lifecycle management handled by the existing top-k strategy.The trained temporal model shows tracking characteristics without fine-tuning with tracking constraints.Source: §3.4 · pp. 5–6

4 Experiment

The experiments evaluate Sparse4Dv3 on nuScenes detection and tracking, including controlled ablations and larger cloud-based configurations. Results report improvements over Sparse4Dv2 and competing methods across both tasks.

4.1 Benckmark

  • The 1000-scene nuScenes benchmark is split into 700 training, 150 validation, and 150 test scenes, with six camera views per scene.Detection is evaluated with mAP, mATE, mASE, mAOE, mAVE, mAAE, and NDS; tracking uses AMOTA, AMOTP, recall, and IDS.Source: §4.1 · p. 6

4.2 Implementation Details

  • Training uses a 6-layer decoder with 900 instances and 600 temporal instances; models train for 100 epochs with AdamW and no tracking fine-tuning.Training also uses dense depth regression as an auxiliary task; three of five denoising groups are randomly selected as temporal instances.Source: §4.2 · p. 6

4.3 Main Results

  • Validation gains are 3.0% mAP and 2.2% NDS with ResNet50; test-set gains are 1.3% mAP and 1.8% NDS.The test configuration uses VoVNet-99 with weights pretrained from DD3D.Source: Table 2 · p. 7, §4.3 · p. 7, Table 1 · p. 7
  • Validation AMOTA is 0.490 versus DORT's 0.424 and 0.567 versus DQTrack's 0.407; ID switches fall 44.5% versus DQTrack.On the test set, Sparse4Dv3 achieves state-of-the-art results across IDS, recall, MOTAR, MOTA, and MOTP.Source: §4.3 · p. 7, Table 3 · p. 8, Table 4 · p. 8

4.4 Ablation Studies

  • Single-frame and temporal denoising increase mAP by 0.8% and 0.4%; decoupled attention improves mAP by 1.1% and mAVE by 1.9%.Centerness reduces distance error by 1.8% but harms orientation estimation, partly mitigated by yawness; together, the quality estimations improve mAP by 0.8%, mATE by 2.8%, and mAVE by 1.9%. Sparse4Dv3 also converges faster and better than Sparse4Dv2, while centerness improves confidence ranking and mitigates errors at low recall or high thresholds.Source: §4.4 · p. 8, Table 5 · p. 8, Figure 6 · p. 9

4.5 Cloud-Based Performance Boost

  • 0.682 mAP, 0.719 NDS, and 0.677 AMOTA are achieved on the nuScenes test set using EVA02 and future frames.Adding the next 8 frames reduces mAVE by 5.67% and increases NDS by 3.23%; EVA02-Large raises mAP by 5.98% over ResNet101.Source: §4.5 · p. 9, Table 6 · p. 9

5 Conclusion and Outlook

Sparse4Dv3 combines three detection improvements with an end-to-end tracking extension, and nuScenes experiments place it at the forefront of the field. The outlook identifies preliminary tracking performance and several directions for extending the framework.

5 Conclusion and Outlook

  • Sparse4Dv3 combines Temporal Instance Denoising, Quality Estimation, and Decoupled Attention, with nuScenes experiments placing it at the forefront of the field.The paper also describes extending Sparse4D into an end-to-end tracking model.Source: §5 · p. 9
  • The authors describe their tracking attempts as preliminary and say tracking performance has substantial room for improvement.Source: §5 · p. 9
  • Future directions include lidar-only or multimodal models, downstream prediction and planning, online mapping, and 2D sign and traffic-light detection.Source: §5 · p. 9

Related papers

Loading 2311.11722v1…