PaperlayerView exact revision on arXiv

Source-linked AI summary

SRCN3D: Sparse R-CNN 3D for Compact Convolutional Multi-View 3D Object Detection and Tracking

Yining Shi, Jingyan Shen, Yifan Sun, Yunlong Wang, Jiaxin Li, Shiqi Sun, Kun Jiang, Diange Yang

arXiv:2206.14451v3cs.CV

TL;DR

Camera-based 3D detection faces reconstruction and cross-view fusion challenges, while prior approaches may require dense queries or interactions. SRCN3D uses sparse box-wise sampling and twin-track refinement, and the authors report effective detection and tracking with a faster-than-transformer-based pipeline.

  • Problem

    Camera-based 3D detection faces ill-conditioned reconstruction without geometric or depth cues and cross-view fusion when adjacent cameras capture only parts of an object; prior approaches require dense queries or interactions.

    Source: §1 · p. 1

  • Method

    SRCN3D iteratively updates paired 3D query boxes and features using projected local RoI features, while its tracker uses multi-hypothesis association with RoI and query features.

    Source: §1 · pp. 1–2

  • Results

    Experiments on nuScenes demonstrate effectiveness for 3D detection and tracking, and the authors describe the pipeline as faster than transformer-based detectors.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Final refined query boxes directly provide detections, avoiding complicated post-processing or regression from latent features.

    Source: §1 · p. 1

    RoI features are less effective than grid sampling for 3D localization, leading to larger errors in object center and scale estimation.

    Source: §4.7 · p. 8

Abstract

from arXiv · show

Detection and tracking of moving objects is an essential component in environmental perception for autonomous driving. In the flourishing field of multi-view 3D camera-based detectors, different transformer-based pipelines are designed to learn queries in 3D space from 2D feature maps of perspective views, but the dominant dense BEV query mechanism is computationally inefficient. This paper proposes Sparse R-CNN 3D (SRCN3D), a novel two-stage fully-sparse detector that incorporates sparse queries, sparse attention with box-wise sampling, and sparse prediction. SRCN3D adopts a cascade structure with the twin-track update of both a fixed number of query boxes and latent query features. Our novel sparse feature sampling module only utilizes local 2D region of interest (RoI) features calculated by the projection of 3D query boxes for further box refinement, leading to a fully-convolutional and deployment-friendly pipeline. For multi-object tracking, motion features, query features and RoI features are comprehensively utilized in multi-hypotheses data association. Extensive experiments on nuScenes dataset demonstrate that SRCN3D achieves competitive performance in both 3D object detection and multi-object tracking tasks, while also exhibiting superior efficiency compared to transformer-based methods. Code and models are available at https://github.com/synsin0/SRCN3D.

1. Introduction

SRCN3D addresses challenges in camera-based 3D detection and tracking with a sparse, two-stage detection pipeline and multi-hypothesis tracking. The authors report effective detection and tracking on nuScenes and describe the detector as faster than transformer-based alternatives.

1. Introduction

  • Camera-based 3D perception faces ill-conditioned reconstruction without geometric or depth cues and cross-view fusion when adjacent cameras capture only parts of an object.Source: §1 · p. 1
  • The detector iteratively updates 3D query boxes and features using projected local RoI features, with final boxes serving directly as detection outputs.The approach avoids complicated post-processing or regression from latent features.Source: §1 · p. 1
  • The tracker uses multi-hypothesis probabilistic matching with RoI and query features to address unstable detections and reduce failed tracking.Source: §1 · pp. 1–2
  • SRCN3D is presented as a transformerless two-stage MV3D approach with box-wise sampling, characterized as lightweight and faster than transformer-based detectors.The authors describe it as the first such approach to their knowledge.Source: §1 · p. 2
  • Sparse cross-attention replaces dense attention with local interactions to refine 3D queries from 2D feature maps at lower computation cost.Source: §1 · p. 2
  • Experiments on nuScenes demonstrate SRCN3D's effectiveness for 3D object detection and tracking.Source: §1 · p. 2

2. Related Work

Prior multi-camera 3D detection approaches use perspective-view, geometry-based, or network-based transformations, while transformer-based systems vary in their query and attention designs. Tracking work includes deterministic matching and multi-hypothesis association; SRCN3D explores hybrid-feature probabilistic association.

2.1. Multi-camera 3D Object Detection

  • Perspective-view methods face cross-view fusion challenges, geometry-based methods are sensitive to depth errors, and SRCN3D adopts network-based cascade refinement without depth estimation.Its two-stage pipeline differs from previous single-stage approaches.Source: §2.1 · p. 2

2.2. Transformer-based Object Detection

  • Transformer-based 3D detectors use differing query and attention schemes, while SRCN3D explores sparse feature sampling without global attention.DETR3D projects reference points, BEVFormer uses BEV grids and deformable attention, and PETR/PETRv2 project image features into 3D space.Source: §2.2 · p. 2

2.3. 3D Multi-Object Tracking

  • Tracking approaches include deterministic one-to-one matching and online multi-hypothesis RFS; SRCN3D explores probabilistic association using hybrid feature embeddings.The framework figure depicts its backbone, twin-track detection module, and sparse feature sampling for local RoI features.Source: §2.3 · p. 3, Figure 2 · p. 3

3. Methodology

SRCN3D uses a two-stage, fully convolutional architecture that iteratively updates paired 3D query boxes and query features. Its sparse pipeline samples local image features for box refinement and extends multi-hypothesis association to tracking.

3.1. Overview

  • SRCN3D estimates 3D boxes directly in world space without depth supervision or NMS, using sparse local features in a fully convolutional pipeline.A fixed set of paired query boxes and features is updated in a cascade.Source: §3.1 · p. 3

3.2. SRCN3D Head

  • Each cascade round projects box corners into views, aggregates RoIAlign features, refines query features, predicts classes and offsets, then adjusts box parameters.The head refines location, dimensions, rotation, and velocity.Source: §3.2 · pp. 3–4

3.3. Sparse Feature Sampling Module

  • Projected boxes yield local RoIAlign features; invisible projections produce empty or partial RoIs, avoiding masking and positional encoding, while cross-view aggregation keeps feature expression fixed.Boxes are projected into visible camera regions before feature extraction.Source: §3.3 · p. 4

3.4. Box Adjustment

  • Location and dimensions are adjusted from predicted cues; dimension parameters use logarithms for non-negative constraints, while rotation and velocity use predicted deltas directly.Source: §3.4 · p. 4

3.5. Loss Design

  • Training combines Focal Loss classification with L1 3D-box regression and uses DETR [3] set-prediction loss.Source: §3.5 · pp. 4–5

3.6. Multi-object Tracking

  • MBM represents uncertain associations as global and single-target hypotheses; matching combines 3D-box Mahalanobis distance with RoI/query cosine similarities, with α=β=0.5 balancing attributes and appearance.Each single-target hypothesis represents a detected object or a missed detection.Source: §3.6 · p. 5

4. Experiments

On nuScenes, SRCN3D reports competitive detection and tracking results, while its inference speed exceeds the cited transformer-based detectors. Ablations examine the detector and tracking components, and the discussion identifies localization and duplicate-box limitations.

4.3. Training and Inference

  • SRCN3D infers at 3.2 FPS on one RTX3090, faster than DETR3D at 2.7 FPS, BEVFormer at 2.1 FPS, and PETR.Inference uses ResNet101 and 1600×900 images, without NMS or test-time augmentation.Source: §4.3 · p. 5

4.4. Comparison with State-of-the-art

  • SRCN3D achieves 42.8% NDS and 33.8% mAP on nuScenes validation, and exceeds DETR3D by 0.3% NDS under the same backbone and image-size settings.It achieves the best mAOE and mAVE, while translation and scale predictions remain limitations.Source: §4.4 · pp. 5–6, Table 1 · p. 6

4.5. Ablation Study

  • Replacing query-box modules with DETR3D components shows query features alone are insufficient for box refinement; random box initialization outperforms fixed initialization.Removing box adjustment slightly reduces accuracy, while sparse interaction performs equally effectively without self-attention.Source: §4.5 · p. 6, Table 4 · p. 7

4.4. Comparison with State-of-the-art

  • On nuScenes test, SRCN3D reaches 0.398 AMOTA, more than 12 points above recent camera-only trackers, with moderate AMOTP error and the highest recall rate.On validation, it performs best among camera-based methods on all reported metrics.Source: §4.4 · p. 6, Table 3 · p. 7

4.5. Ablation Study

  • Reducing query count affects accuracy by less than 3%, whereas reducing candidate objects causes greater accuracy loss; cascade-stage gains diminish after the first three stages.The ablation attributes lightweight region-of-interest capture to sparse feature sampling using local features.Source: §4.5 · pp. 7–8, Table 6 · p. 8
  • For camera-only tracking, probabilistic association outperforms deterministic matching, and RoI and query features facilitate matching and improve tracking precision.The features provide affinity cues for appearance and classification, respectively.Source: Table 5 · p. 7, §4.5 · p. 8

4.6. Visualization

  • On validation visualizations, predicted boxes are close to ground truth in BEV, cross-view fusion captures a bus across overlapping views, and small pedestrians are detected precisely.Tracking examples visualize unique objects across five key frames in a crowded intersection.Source: Figure 4 · p. 7, §4.6 · p. 8, Figure 5 · p. 8

4.7. Discussion

  • RoI features suit classification, orientation, and velocity better than 3D localization, where they lead to larger object-center and scale errors.Visualizations also expose overlapping predicted boxes and remaining duplicates in dense-feature regions.Source: §4.7 · p. 8, §4.6 · p. 8

5. Conclusion

SRCN3D combines sparse queries, sparse attention, and sparse prediction in a cascade that refines boxes and features using local RoI attention. The authors propose it as a foundation for fully sparse surround-view 3D detection, with segmentation and temporal information as future directions.

  • SRCN3D combines sparse queries, sparse attention, and sparse prediction with twin-track box-feature queries and cascade refinement using local RoI attention.The authors identify this combination as their insight for 3D detection and cross-view fusion.Source: §5 · p. 8
  • The authors hope SRCN3D can serve as a foundation for fully sparse surround-view 3D object detection.Source: §5 · p. 8
  • Future work will investigate combining segmentation and temporal information to enhance SRCN3D’s accuracy and robustness.Source: §5 · p. 8

Related papers

Loading 2206.14451v3…