PaperlayerView exact revision on arXiv

Source-linked AI summary

3D-MAN: 3D Multi-frame Attention Network for Object Detection

Zetong Yang, Yin Zhou, Zhifeng Chen, Jiquan Ngiam

arXiv:2103.16054v1cs.CV

TL;DR

Single-frame 3D detection can underuse temporal information, despite partial LiDAR views leaving object boxes ambiguous. 3D-MAN stores proposals and features from multiple frames and uses attention to align and aggregate them. The paper reports state-of-the-art Waymo Open Dataset performance compared with published single-frame and multi-frame methods.

  • Problem

    Single-frame 3D detectors do not fully use information from multiple frames, while partial LiDAR views can leave object boxes ambiguous.

    Source: §1 · p. 1

  • Method

    3D-MAN stores fast single-frame detector proposals and features in a memory bank, then uses multi-view attention to align and aggregate features across frames.

    Source: §1 · p. 2

  • Results

    3D-MAN achieves state-of-the-art performance on Waymo Open Dataset and outperforms published single-frame and multi-frame methods.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The method is reported to remain effective across long time horizons on a challenging large-scale dataset.

    Source: §5 · p. 8

Abstract

from arXiv · show

3D object detection is an important module in autonomous driving and robotics. However, many existing methods focus on using single frames to perform 3D detection, and do not fully utilize information from multiple frames. In this paper, we present 3D-MAN: a 3D multi-frame attention network that effectively aggregates features from multiple perspectives and achieves state-of-the-art performance on Waymo Open Dataset. 3D-MAN first uses a novel fast single-frame detector to produce box proposals. The box proposals and their corresponding feature maps are then stored in a memory bank. We design a multi-view alignment and aggregation module, using attention networks, to extract and aggregate the temporal features stored in the memory bank. This effectively combines the features coming from different perspectives of the scene. We demonstrate the effectiveness of our approach on the large-scale complex Waymo Open Dataset, achieving state-of-the-art results compared to published single-frame and multi-frame methods.

1. Introduction

Single-frame LiDAR can leave partially observed objects with multiple plausible box predictions, while multiple frames provide different perspectives. 3D-MAN stores detector proposals and features in a memory bank, then uses multi-view attention to aggregate them; the paper reports state-of-the-art results on Waymo Open Dataset.

1. Introduction

  • Partial LiDAR views can make 3D detection ill-posed, because the observed points may support multiple reasonable box predictions.The paper notes that autonomous-driving platforms use precise 3D detection for world representation and downstream driving decisions.Source: §1 · p. 1
  • Multiple frames provide different perspectives of an object, and recent sequence datasets make well-calibrated multi-frame data available.Point concatenation may be less effective for fast-moving objects or longer time horizons when points are no longer aligned; feature-map concatenation faces a similar misalignment challenge, while recurrent approaches can be computationally expensive.Source: §1 · pp. 1–2
  • 3D-MAN combines a fast single-frame detector, a memory bank, and a multi-view alignment and aggregation module.The bank stores proposals and feature maps from previous frames; MVAA aligns relevant features per stored frame, then aggregates across frames for each target proposal.Source: §1 · p. 2
  • The fast single-frame detector uses max-pooling NMS and a Hungarian-matching loss to produce proposals at real-time speeds.Its proposals and last feature map are stored to maintain different perspectives of each instance across frames.Source: §1 · p. 2
  • 3D-MAN achieves state-of-the-art performance on Waymo Open Dataset and outperforms published single-frame and multi-frame methods.The paper also reports thorough ablation studies.Source: §1 · p. 2
  • MVAA extracts and aggregates relevant features from multiple frames to produce multi-perspective features for classification and box regression.The alignment and aggregation stages factorize attention across proposals and frames.Source: §1 · p. 2

2. Related Work

Related work spans voxel-based, point-based, and combined single-frame 3D detectors, alongside 2D multi-frame methods that aggregate proposal features. Existing 3D fusion approaches include point or feature concatenation and recurrent layers; 3D-MAN instead factorizes attention across frames and proposals.

2. Related Work

  • Single-frame 3D detectors use voxel-based, point-based, or combined representations, with combined methods using voxel proposals and point-based refinement.Voxel methods discretize points into fixed-size 2D feature maps, whereas point-based methods extract features directly from point clouds.Source: §2 · p. 2
  • Many 2D multi-frame detectors aggregate temporal proposal features with self-attention, while 3D-MAN first aligns independently across frames and then aggregates across proposals.The cited 2D methods use a target frame and reference frames with proposals generated per frame.Source: §2 · pp. 2–3
  • Point concatenation can improve 3D detection, but its gains diminish over longer horizons as LiDAR points become less aligned.Feature-map concatenation can also misalign for fast-moving objects, while recurrent fusion approaches can face a single-memory-state bottleneck and high computational cost.Source: §2 · p. 3

3. 3D-MAN Framework

3D-MAN combines a fast proposal detector and a memory bank with attention-based alignment and aggregation across frames. Its losses supervise single-frame detection, multi-view prediction, and cross-view features.

3. 3D-MAN Framework

  • The framework uses a fast single-frame detector to produce proposals, a memory bank for features from different frames, and MVAA to combine them into final predictions.MVAA aligns and aggregates information across frames for the target proposals.Source: §3 · p. 3, §3.3 · p. 5

3.1. Fast Single-frame Detector

  • MaxPoolNMS finds local peaks in parallel, and Hungarian matching trains the detector to assign one positive prediction to each ground-truth box.Matching is applied after MaxPoolNMS to reduce the number of predictions; unmatched predictions are negatives.Source: §3.1 · p. 4

3.2. Memory Bank

  • The memory bank retains proposals and feature maps from the last n frames, while features are generated for proposals from both stored and target frames.Using all stored proposals can improve recall when an object is missed in one frame because of occlusion or partial observation.Source: §3.2 · p. 4, Figure 3 · p. 4

3.3. Multi-view Alignment and Aggregation

  • MVAA aligns target proposals independently with each stored frame using cross-attention informed by frame indices and pairwise box residuals, then aggregates the resulting features across time.The aggregation can focus on the new frame when an object has newly appeared.Source: §3.3 · p. 5, Figure 4 · p. 5

3.4. Losses

  • Training minimizes equally weighted FSD, MVAA prediction, and cross-view losses, using objectness and box-regression losses with Hungarian-matched boxes for regression supervision.Orientation uses 12 bins for FSD and 1 bin for MVAA and cross-view losses.Source: §3.4 · p. 6

4. Experiments

Experiments on Waymo Open Dataset evaluate 3D-MAN against published detectors and ablate its frame inputs, proposal assignment, fusion approaches, and auxiliary loss. The results report state-of-the-art comparisons and gains from multi-frame inputs and the proposed components.

4.2. Main Results

  • With 16 stored frames, overall 3D AP (LEVEL 1) improves by 5.50% on vehicles labeled LEVEL 1 difficulty.The comparison is between the model with and without multi-frame inputs.Source: §4.2 · p. 6, Table 3 · p. 7
  • 3D-MAN exceeds PV-RCNN by 3.56% (30-50m range) and 9.49% (>50m range) AP (LEVEL 1) on vehicles, and outperforms Conv-LSTM by 10.93% on vehicle detection.It also achieves the best reported pedestrian performance on the validation set.Source: §4.2 · p. 7, Table 4 · p. 7
  • 78.71% AP and 78.28 APH; 3D-MAN outperforms RCD by 6.74% and 6.69%, respectively, among single-model vehicle results on the Waymo testing set.The reported comparison is for LEVEL 1 difficulty vehicles.Source: Table 5 · p. 7, §4.2 · p. 7

4.3. Ablation Studies

  • The mini-validation ablations use vehicle LEVEL 1 results from a uniformly sampled 10% of validation data; the best model scores 74.3% there versus 74.5% on full validation.The authors describe this performance gap as negligible.Source: §4.3 · p. 7
  • Hungarian matching gives FSD its highest AP among the tested assignment strategies, outperforming centeredness and mask strategies.The authors note that mask assignments can mismatch classification score and localization accuracy, while central pillars may not provide the best localization prediction.Source: §4.3 · pp. 7–8
  • MVAA outperforms Concat and Relation by 2.0% and 2.4%, respectively; Concat improves on the baseline by 2.3% AP, while Relation performs between them.The comparison uses four input frames and predicts boxes for the last frame.Source: §4.3 · p. 8, Table 7 · p. 8
  • The cross-view auxiliary loss outperforms no auxiliary loss by 1.8% and correspondence loss by 1.1%.Correspondence loss encourages alignment-stage attention between a query proposal and its matching stored proposal.Source: §4.3 · p. 8, Table 8 · p. 8
  • Performance steadily improves with more input frames, and increasing from 4 to 16 frames improves results for fast-moving vehicles.For 7- through 16-frame comparisons, computation is fixed using overlapping windows of four concatenated frames.Source: §4.3 · p. 8, Table 9 · p. 8, Table 10 · p. 8

5. Conclusion

3D-MAN uses attention networks to extract and aggregate features across frames for 3D object detection. Its fast detector, memory bank, and multi-view alignment and aggregation module support state-of-the-art results on a challenging large-scale dataset and effectiveness across long time horizons.

5. Conclusion

  • 3D-MAN uses attention networks to extract and aggregate features across multiple frames for 3D object detection.Its pipeline combines a fast single-frame detector, memory bank, and multi-view alignment and aggregation module.Source: §5 · p. 8
  • The fast single-frame detector uses Hungarian matching to align objectness scores with the best-localized box.Source: §5 · p. 8
  • The method is effective across long time horizons and achieves state-of-the-art performance on a challenging large-scale dataset.Source: §5 · p. 8

Related papers

Loading 2103.16054v1…