Source-linked AI summary
Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion
Xuewu Lin, Tianwei Lin, Zixiang Pei, Lichao Huang, Zhizhong Su
TL;DR
Multi-view 3D detection must handle missing depth cues, while sparse methods face performance limitations despite avoiding dense view transformation. Sparse4D addresses this with deformable 4D sampling, hierarchical fusion, iterative anchor refinement, and depth reweighting, and reports leading nuScenes detection performance among sparse methods and most BEV methods.
Problem
Sparse 3D detection avoids dense perspective transformation but has limited capacity, inefficient or imprecise feature sampling, insufficient temporal context, and a performance gap relative to BEV methods.
Method
Sparse4D samples multi-keypoint features across timestamps, views, and scales, hierarchically fuses them, iteratively refines anchors, and reweights instance features using depth confidence.
Results
Sparse4D outperforms all existing sparse algorithms and most BEV-based algorithms on nuScenes 3D detection, while also performing well on tracking.
Takeaways & Limitations
Sparse4D provides a sparse spatial-temporal fusion framework for precise multi-view 3D detection without dense view transformation or global attention.
Takeaways & Limitations
BEV-based methods still present an important comparison boundary because their dense transformations achieve strong performance despite deployment, range, and height-information trade-offs.
Abstract
from arXiv · showhide
Bird-eye-view (BEV) based methods have made great progress recently in multi-view 3D detection task. Comparing with BEV based methods, sparse based methods lag behind in performance, but still have lots of non-negligible merits. To push sparse 3D detection further, in this work, we introduce a novel method, named Sparse4D, which does the iterative refinement of anchor boxes via sparsely sampling and fusing spatial-temporal features. (1) Sparse 4D Sampling: for each 3D anchor, we assign multiple 4D keypoints, which are then projected to multi-view/scale/timestamp image features to sample corresponding features; (2) Hierarchy Feature Fusion: we hierarchically fuse sampled features of different view/scale, different timestamp and different keypoints to generate high-quality instance feature. In this way, Sparse4D can efficiently and effectively achieve 3D detection without relying on dense view transformation nor global attention, and is more friendly to edge devices deployment. Furthermore, we introduce an instance-level depth reweight module to alleviate the ill-posed issue in 3D-to-2D projection. In experiment, our method outperforms all sparse based methods and most BEV based methods on detection task in the nuScenes dataset.
1. Introduction
The paper motivates Sparse4D as a sparse alternative to BEV-based 3D detection, targeting efficient spatial-temporal feature fusion while improving sparse detection performance.
- Sparse4D: The method hierarchically fuses sampled features across views, scales, timestamps, and keypoints to generate instance features for precise 3D detection.This contribution is presented as a deformable 4D aggregation module for sparse multidimensional feature sampling and fusion.
- BEV-based methods: BEV methods achieve strong performance but require dense image-to-BEV transformation, limit the perception-range trade-off, and compress height information.These properties complicate low-cost edge deployment and can lose texture cues needed for tasks such as sign detection.
- Sparse-based methods: Existing sparse methods avoid dense perspective transformation but have limited model capacity, inefficient or imprecise multi-view sampling, and insufficient temporal context.The paper identifies a performance gap between sparse methods and state-of-the-art BEV methods.
- Sparse4D: Sparse4D assigns multiple keypoints within each 3D anchor to sample multi-timestamp, multi-view, and multi-scale features for anchor refinement.The approach is designed to extract richer context inside each anchor and align temporal information.
- Sparse4D: Sparse4D adds an instance-level depth reweight module that uses sampled depth confidence to address ambiguity in camera-based 3D perception.The module is trained sparsely without additional LiDAR point-cloud supervision.
- Results: On nuScenes, Sparse4D outperforms existing sparse methods and most BEV-based methods on 3D detection and also performs well on tracking.The reported contributions describe this as the method’s benchmark-level outcome.
2. Related Work
Related work spans dense and sparse detection paradigms, while Sparse4D uses an encoder-decoder architecture with iterative anchor refinement over multi-view temporal features.
- Sparse detection paradigms: DETR performs global cross attention between object queries and image context, creating high computation cost and making it unsuitable as a pure sparse method.Deformable DETR replaces global attention with local reference-point attention to reduce complexity.
- Monocular 3D detection: Monocular 3D detection is ill-posed because a single image lacks depth information, making it more challenging than 2D detection.Existing approaches estimate depth directly or construct pseudo point-cloud signals from images.
- Dense methods: Dense multi-view 3D detection methods use dense feature vectors for view transformation, feature fusion, or box prediction.BEVFormer, BEVDet, BEVDepth, BEVStereo, SOLOFusion, and PETR represent prominent dense approaches.
- Sparse methods: Sparse methods sample and fuse features from sparse reference points rather than using dense view transformation.DETR3D is identified as representative, while Graph DETR3D improves spatial fusion in multi-view overlapping regions.
- Sparse4D architecture: Sparse4D extracts multi-view, multi-scale feature maps and recent-frame features, then iteratively refines 3D anchors through decoder modules.Each refinement module consumes the image feature queue, 3D anchor boxes, and instance features.
3. Methodology
Sparse4D iteratively refines 3D anchors by extracting and fusing sparse spatial-temporal image features. Its deformable 4D aggregation samples multi-keypoint features across views, scales, and timestamps, while depth reweighting addresses ambiguity in 3D-to-2D projection.
- Overall Framework: Sparse4D uses an encoder-decoder with temporal image features, iteratively refined 3D anchors, updated instance features, and final classification confidences.Each refinement module applies self-attention, deformable 4D aggregation, depth reweighting, and regression.
- 4D Keypoints Generation: Each anchor receives fixed and learnable 4D keypoints, allowing instance-dependent sampling within the anchor region.Fixed points include the stereo center and six face centers; learnable points vary with instance features.
- 4D Keypoints Generation: The method shifts keypoints across past timestamps using a constant-velocity model and ego-motion transformations.This constructs 4D keypoints in the coordinate systems of historical frames.
- Deformable 4D Aggregation: Projected keypoints are bilinearly sampled from multi-view, multi-scale, multi-timestamp feature maps, producing f_m ∈ R^(K×T×N×S×C).Indices represent anchor, keypoint, timestamp, camera, and feature-scale dimensions.
- Deformable 4D Aggregation: Hierarchical fusion first weights view-scale features, then fuses timestamps and finally aggregates multi-keypoint features into an instance feature.Group-wise channel weighting is predicted through a linear layer before view-scale aggregation.
- Depth Reweight Module: Depth reweighting estimates a discrete depth distribution and uses anchor-center depth confidence to suppress features from depth-misaligned instances.The module targets ambiguity where different 3D points project to the same 2D coordinates.
4. Experiment
Experiments on nuScenes evaluate Sparse4D’s detection performance, ablations, refinement and temporal settings, and computational cost against sparse and BEV-based methods.
- Datasets and Metrics: The nuScenes benchmark uses 1,000 scenes split into 700 training, 150 validation, and 150 testing scenes, with six camera views per frame.Each scene is a 20-second video recorded at 2 frames per second and includes 3D box and object-ID annotations.
- Datasets and Metrics: Detection evaluation uses mAP, mATE, mASE, mAOE, mAVE, mAAE, and NDS, while tracking uses AMOTA, AMOTP, and Recall.NDS is a weighted average of the other detection metrics.
- Ablation Studies and Analysis: Adding DRM and LKP together increases mAP by 0.38% and NDS by 0.79% on the nuScenes validation set.Individually, DRM and LKP improve NDS by 0.33% and 0.35%, respectively.
- Ablation Studies and Analysis: Motion compensation makes temporal fusion substantially more effective, increasing mAP by 4.2%, mAVE by 28.4%, and NDS by 6.4%.Without motion compensation, adding temporal information increases mAVE by 20.8% and NDS by 2.3%; modeling object motion further reduces speed-estimation error by 6.9%.
- Ablation Studies and Analysis: Refinement improves overall metrics with diminishing returns, while ten refinement modules produce the highest NDS of 38.1%.The sixth module differs little from the fifth, whereas the second module gives a substantial gain over the first.
- Ablation Studies and Analysis: Adding three history frames increases FLOPs by 9.3% and parameters by 1.4%, while improving mAP by 5.4% and NDS by 9.0%.With one frame, Sparse4D uses 1019.2G FLOPs and 58.1M parameters; compared with DETR3D, mAP and NDS increase by 3.6% and 2.6%.
- Main Results: On the nuScenes validation set, Sparse4D leads DETR3D by 3.3% mAP and 1.7% NDS, and leads BEVFormer by 0.7% mAP and 0.3% NDS.With T = 4, it exceeds BEVDepth by 2.4% mAP and 0.6% NDS; increasing T from 4 to 9 adds 0.9% mAP and 0.6% NDS.
- Main Results: On the nuScenes test set, Sparse4D achieves 51.1% mAP and 59.5% NDS, outperforming non-BEV methods and exceeding BEVDistill by 1.5% mAP.Against DETR3D, mAP and NDS increase by 9.9% and 11.6%, respectively.
5. Conclusion
Sparse4D fuses multi-timestamp and multi-view features through deformable 4D aggregation and iterative 3D box refinement. It achieves strong nuScenes performance, surpassing existing sparse methods and most BEV-based methods, while remaining open to further improvement.
- Sparse4D performs feature-level fusion across multiple timestamps and views using deformable 4D aggregation, followed by iterative 3D box regression.
- Sparse4D outperforms all existing sparse algorithms and most BEV-based algorithms on the nuScenes leaderboard.
- The authors identify substantial room for improvement, including more accurate depth from multi-view stereo and better 3D generalization using camera parameters.