PaperlayerView exact revision on arXiv

Source-linked AI summary

Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds

Chaoda Zheng, Xu Yan, Jiantao Gao, Weibing Zhao, Wei Zhang, Zhen Li, Shuguang Cui

arXiv:2108.04728v2cs.CV

TL;DR

Incomplete LiDAR scans make accurate feature comparisons difficult because point-cloud features do not encode object size or part structure. BAT uses first-frame box information through BoxCloud and box-aware feature fusion, outperforming prior state of the art on KITTI and NuScenes, especially for sparse objects.

  • Problem

    Incomplete point clouds leave feature comparisons without object-size and part-structure cues, which can make similar objects ambiguous.

    Source: §1 · pp. 1–2

  • Method

    BAT represents points through their distances to the target box’s corners and center, then uses box-aware comparison and fusion for tracking.

    Source: §1 · p. 2, §3.3 · p. 4

  • Results

    BAT outperforms state-of-the-art methods on KITTI and NuScenes, with especially strong gains for sparse objects.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    BoxCloud provides a flexible representation for future work seeking to improve tracking performance.

    Source: §5 · p. 8

    The tracker assumes the target’s 3D bounding-box size remains unchanged across frames and does not re-predict it.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Current 3D single object tracking approaches track the target based on a feature comparison between the target template and the search area. However, due to the common occlusion in LiDAR scans, it is non-trivial to conduct accurate feature comparisons on severe sparse and incomplete shapes. In this work, we exploit the ground truth bounding box given in the first frame as a strong cue to enhance the feature description of the target object, enabling a more accurate feature comparison in a simple yet effective way. In particular, we first propose the BoxCloud, an informative and robust representation, to depict an object using the point-to-box relation. We further design an efficient box-aware feature fusion module, which leverages the aforementioned BoxCloud for reliable feature matching and embedding. Integrating the proposed general components into an existing model P2B, we construct a superior box-aware tracker (BAT). Experiments confirm that our proposed BAT outperforms the previous state-of-the-art by a large margin on both KITTI and NuScenes benchmarks, achieving a 15.2% improvement in terms of precision while running ~20% faster.

1. Introduction

Incomplete LiDAR scans make feature matching difficult because point-cloud features do not encode object size or explicit part structure. BAT uses first-frame bounding-box information through BoxCloud and box-aware feature fusion, reporting gains over state of the art on KITTI and NuScenes, especially for sparse data.

1. Introduction

  • Pure LiDAR point-cloud features omit object size and explicit part structure, making comparisons ambiguous when scans are incomplete.Source: §1 · pp. 1–2
  • BoxCloud represents each object point by its distances to the target box’s corners and center, encoding size and part information.Source: §1 · p. 2
  • BAFF compares template and search-area BoxClouds, then aggregates the top-k similar template points into each search point’s features.Source: §1 · p. 2
  • BAT combines BoxCloud and box-aware feature fusion with P2B, improving on state-of-the-art methods on KITTI and NuScenes, especially for extremely sparse data.Source: §1 · p. 2

2. Related Work

Prior 3D single-object trackers use point-cloud features and proposal-generation methods, but do not exploit the first-frame box as an input cue. BAT’s pipeline pairs box-aware matching with a 3D proposal network.

2. Related Work

  • P2B [27] adapts SiamRPN to point clouds, fusing target information into the search area before detection with VoteNet [24].Source: §2 · p. 2
  • Unlike SOT, 3D MOT first detects objects in every frame and then associates their boxes into trajectories using motion estimates.Source: §2 · p. 2
  • BAT’s pipeline uses a 3D RPN for final proposals; its BAFF module finds template neighbors by BoxCloud distance and aggregates their features with Mini-PointNet.Source: Figure 2 · p. 3
  • Earlier 3D SOT methods use the first-frame box to crop a template, while part-supervised detection [30] captures parts but ignores box size and does not use boxes as input.Source: §2 · p. 3

3. Method

The method uses the first-frame bounding box’s shape priors to represent points with BoxCloud, then uses box-aware comparison and feature aggregation to form target-specific search features for P2B’s proposal stage.

3.1. Problem Statement

  • Prior trackers use the first-frame BBox mainly to crop and center the template, leaving incomplete templates without explicit size and part information.The method uses the amodal BBox to exploit those shape priors, including when points are occluded or truncated.Source: §3.1 · p. 3, §3.3 · p. 4

3.2. Box-Aware Tracker (BAT)

  • BAT builds on P2B, replacing its initial feature-comparison stage with BoxCloud comparison and aggregation before using P2B’s target-proposal process.The shared PointNet++ extracts template and search features; the template BoxCloud is calculated from the template and its BBox, while the search BoxCloud is predicted.Source: §3.2 · pp. 3–4

3.3. BoxCloud Representation

  • BoxCloud represents each object point by nine ordered distances to the BBox corners and center, encoding full-object size and intra-object part location despite incomplete scans.The full size is inferable from BoxCloud regardless of shape incompleteness; each vector also indicates a point’s relative location within the object.Source: §3.3 · p. 4

3.4. Box-Aware Feature Fusion

  • BAFF augments search-area features with template information to create an enhanced target-specific search area for the 3D RPN.The module consists of BoxCloud comparison and feature aggregation.Source: §3.4 · p. 4

3.4.1 BoxCloud Comparison

  • Search BoxClouds are predicted under ground-truth-BBox supervision, then pairwise L2 distances select top-k template matches, helping discard false matches and avoid noise interference.The distance map compares each template BoxCloud point with each search-area BoxCloud point.Source: §3.4.1 · pp. 4–5

3.4.2 Feature Aggregation

  • For each search point, a mini-PointNet max-pools MLP embeddings of its top-k template points and the search feature into enhanced target-specific features.The template-point inputs include spatial coordinates, extracted features, and BoxCloud coordinates.Source: §3.4.2 · p. 5

3.5. Implementation

  • Training uses consecutive frames, merging first-frame and previous-frame target points for the template; testing updates that template with the previous prediction and searches within a 2-meter-enlarged box.The search area is formed by enlarging the shifted training BBox or previous predicted testing BBox by 2 meters in each direction.Source: §3.5 · p. 5

4. Experiments

Experiments evaluate BAT on KITTI and NuScenes, compare it with prior trackers, and examine robustness and design choices. BAT improves on prior methods, including in sparse scenes, while the ablations support BoxCloud comparison and BAFF.

4.1. Experimental Setting

  • The experiments use KITTI and NuScenes, evaluate Success and Precision with OPE, and compare BAT with SC3D, its follow-up, and P2B.For NuScenes, the authors evaluate SC3D and P2B using their official open-source code; KITTI results are taken from the corresponding papers.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.2. Comparison with State-of-the-arts

  • BAT outperforms P2B by over 10% on average on KITTI and is consistently superior to the other competitors on NuScenes.Table 1 compares KITTI and NuScenes results; the prose reports these overall comparisons.Source: Table 1 · p. 6, §4.2 · p. 6
  • For targets with fewer than 10 points, BAT achieves over 55% success, 10% higher than P2B and 30% higher than SC3D.For sequences with more than 50 points, performance drops as those sequences are longer on average (86.3 vs 47.7 frames), with accumulated tracking errors offered as the explanation.Source: Figure 4 · p. 6, §4.2 · p. 7
  • BoxCloud distances provide more reliable feature comparisons than PointNet++ features in the illustrated case, where target search points have relatively low feature similarity.Figure 6 compares pairwise distances between search-area points and a template point.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3. Ablation Study

  • Ablations show that adding BoxCloud to P2B improves tracking, while BoxCloud comparison and kNN grouping further exploit its encoded cues.Table 2 presents the ablation results, including the baseline, BAT-Vanilla, and the best results.Source: §4.3 · p. 7, Table 2 · p. 8

4.3.1 Analysis on Box-Aware Feature Fusion (BAFF)

  • With default k = 4, BAT achieves its best performance and a satisfactory high speedup, running about 20% faster than P2B; speedup is measured during model forwarding with batch size 96.The speedup calculation excludes pre- and post-processing time.Source: §4.3.1 · p. 8
  • Replacing BoxCloud comparison with PointNet++ features significantly reduces BAT performance, while adding further features can also cause drastic drops.The authors suggest that redundant features may confuse feature fusion; Table 2 reports ablation results for feature aggregation and BoxCloud comparison.Source: §4.3.1 · p. 8, Table 2 · p. 8

5. Conclusion

BAT extends P2B with BoxCloud and box-aware feature fusion, using free bounding-box information to improve LiDAR point-cloud tracking, especially for sparse objects.

5. Conclusion

  • BAT combines BoxCloud with box-aware feature fusion to enhance object representation and generate box-aware target-specific search areas.The paper presents these as its two main components for LiDAR point-cloud tracking.Source: §5 · p. 8
  • Each component independently improves tracking performance, and they benefit from one another when applied jointly.Source: §5 · p. 8
  • With small modifications to P2B, BAT outperforms current state-of-the-art methods by a large margin, especially on sparser object tracking.Source: §5 · p. 8
  • The authors describe BoxCloud as a flexible tool for future work to enhance performance.Source: §5 · p. 8

A. Overview

The supplementary material presents additional analysis experiments and then describes the BAT-Vanilla architecture.

A. Overview

  • Section B provides additional analysis experiments.Source: A. Overview · p. 11
  • Section C describes the architecture of BAT-Vanilla.Source: A. Overview · p. 11
  • The supplementary material moves from further experiments in Section B to the architecture description in Section C.Source: A. Overview · p. 11

B. More Analysis Experiments

The supplementary analyses compare tracking setups and template strategies, examine MOT methods, and evaluate alternative BoxCloud designs and prediction accuracy.

Other BoxCloud Design

  • The 27-dimensional point-offset BoxCloud alternative declines in performance despite being three times larger and containing almost the same information as the proposed design.The authors attribute the decline to the extra dimension burdening network training, and describe the proposed Euclidean-distance design as more compact and effective.Source: B. More Analysis Experiments · p. 11, Table 1 · p. 11

Performance on Long/Short Term Tracking

  • BAT outperforms P2B and SC3D in both long- and short-term car tracking setups, with a more notable advantage in the long-term setup.Long-term tracking uses model-predicted prior locations, whereas short-term tracking uses the previous frame’s ground-truth location; the authors consider long-term evaluation more representative of realistic tracking.Source: B. More Analysis Experiments · p. 11, Table 2 · p. 11

Template Generation Strategy

  • BAT retains notable advantages across template-generation strategies and beats P2B by about 7% under the “previous result” strategy.The tested strategies use the first ground truth, the previous result, all previous results, or the default first-ground-truth-plus-previous-result template.Source: B. More Analysis Experiments · p. 11, Table 3 · p. 11

Comparison with MOT Approaches

  • BAT significantly outperforms the compared 3D MOT methods AB3DMOT and PC3T on KITTI, while both MOT methods are slower and cannot achieve real-time speed.The evaluation converts MOT objects into SOT tasks by nearest-neighbor matching; tracking stops after an identity change.Source: B. More Analysis Experiments · pp. 11–12, Table 4 · p. 11

Visualization of BoxCloud Learning

C. Architecture of BAT-Vanilla

BAT-Vanilla adds a BoxCloud prediction branch and concatenates template BoxClouds with coordinates and features; its RPN votes for target centers and forms final proposals.

C. Architecture of BAT-Vanilla

  • After shared feature extraction, a shared MLP predicts BoxClouds from search-area features, while template BoxClouds are concatenated with template coordinates and extracted features.This feature augmentation adds BoxCloud priors to an approach otherwise almost the same as P2B’s.Source: C. Architecture of BAT-Vanilla · p. 12, Figure 4 · p. 13
  • The RPN applies a point-wise MLP to sampled seed features to predict object-center offsets and feature offsets, with only coordinate offsets supervised.A separate MLP predicts a targetness score for each search-area seed.Source: C. Architecture of BAT-Vanilla · pp. 12–13, Figure 3 · p. 13
  • Predicted vote centers and targetness scores are clustered using furthest-point sampling and ball query, then a mini-PointNet generates each final target proposal.Source: C. Architecture of BAT-Vanilla · p. 13, Figure 3 · p. 13

Related papers

Loading 2108.04728v2…