PaperlayerView exact revision on arXiv

Source-linked AI summary

P2B: Point-to-Box Network for 3D Object Tracking in Point Clouds

Haozhe Qi, Chen Feng, Zhiguo Cao, Feng Zhao, Yang Xiao

arXiv:2005.13888v1cs.CV

TL;DR

Point-cloud 3D tracking is difficult under sparse, disordered observations and cannot always rely on RGB-D cues. P2B embeds template information into search-area seeds for end-to-end point-driven proposal and verification, achieving about a 10% gain on KITTI while exposing limitations from sparse initial templates and data demands.

  • Problem

    Point-cloud 3D tracking must handle sparse, disordered data without depending on RGB information that may be degraded or inaccessible.

    Source: §1 · p. 1

  • Method

    P2B uses permutation-invariant target-specific feature augmentation to embed template clues into search-area seeds, which vote for target centers and support joint proposal and verification.

    Source: §1 · p. 1, §3.2 · p. 4, §5 · p. 8

  • Results

    ∼10% improvement on both Success and Precision over the state-of-the-art method was achieved on the KITTI tracking dataset.

    Source: §1 · p. 2

  • Takeaways & Limitations

    P2B reduces the search space by operating on sampled seeds instead of 3D boxes while providing an end-to-end point-driven tracking formulation.

    Source: §5 · p. 8

    Most failures arose when the initial template was too sparse, and the experiments indicate that P2B needs more data for satisfying results.

    Source: §4.4.2 · p. 8, §5 · p. 8

Abstract

from arXiv · show

Towards 3D object tracking in point clouds, a novel point-to-box network termed P2B is proposed in an end-to-end learning manner. Our main idea is to first localize potential target centers in 3D search area embedded with target information. Then point-driven 3D target proposal and verification are executed jointly. In this way, the time-consuming 3D exhaustive search can be avoided. Specifically, we first sample seeds from the point clouds in template and search area respectively. Then, we execute permutation-invariant feature augmentation to embed target clues from template into search area seeds and represent them with target-specific features. Consequently, the augmented search area seeds regress the potential target centers via Hough voting. The centers are further strengthened with seed-wise targetness scores. Finally, each center clusters its neighbors to leverage the ensemble power for joint 3D target proposal and verification. We apply PointNet++ as our backbone and experiments on KITTI tracking dataset demonstrate P2B's superiority (~10%'s improvement over state-of-the-art). Note that P2B can run with 40FPS on a single NVIDIA 1080Ti GPU. Our code and model are available at https://github.com/HaozheQi/P2B.

1. Introduction

P2B addresses point-cloud 3D tracking without RGB by embedding template information into search-area points, then jointly proposing and verifying targets in an end-to-end network. Experiments on KITTI show substantial gains over the prior state of the art.

Approach

  • P2B first localizes potential target centers, then performs point-driven 3D target proposal and verification jointly.This replaces intuitive 3D box search with a point-to-box formulation that can be trained end to end.Source: §1 · pp. 1–2
  • The P2B pipeline has two parts: target-specific feature augmentation and 3D target proposal and verification using a modified PointNet++ backbone.Augmented search seeds regress target centers, while seed-wise targetness scores support joint proposal and verification.Source: Figure 2 · p. 2, §1 · p. 2
  • Target-specific feature augmentation combines search-point coordinates, point-wise template similarity, and encoded global template features.The resulting features retain spatial geometry while incorporating local tracking clues and global target information.Source: §1 · p. 2

Results

  • ∼10% improvement on both Success and Precision over the state-of-the-art method was achieved on the KITTI tracking dataset.P2B also runs at about 40FPS on a single NVIDIA 1080Ti GPU.Source: §1 · p. 2

2. Related Works

Related work spans point-cloud tracking, 2D Siamese tracking, point-set learning, target proposal, and Hough voting. P2B adapts these ideas to point-driven 3D tracking while emphasizing end-to-end joint proposal and verification.

3D object tracking

  • Earlier 3D tracking methods generally use RGB-D information, whereas point-cloud-only tracking was only recently studied.RGB dependence can limit use when visual information is degraded or inaccessible.Source: §2 · p. 2

2D Siamese tracking

  • 2D Siamese trackers compare template and search-area branches with shared weights, but their 2D CNN foundations do not directly address point-cloud data.These methods avoid time-consuming multiscale search and online fine-tuning through a region-proposal-style paradigm.Source: §2 · p. 2

Deep learning on point set

  • Point-set learning addresses disorder, sparsity, and rotation variance across recognition, detection, pose estimation, and tracking tasks.The prior 3D tracking network could not execute end-to-end target proposal and verification jointly, which is P2B’s focus.Source: §2 · pp. 2–3

Target proposal

  • P2B treats each point as a regressor toward a potential target center, unlike 2D target-proposal methods that operate on area-based pixels.The point-driven formulation directly connects target proposal to 3D point-cloud tracking.Source: §2 · p. 3

Hough voting

  • Hough voting provides a learned representation that aggregates information from different observations and has been embedded in end-to-end 3D point-cloud networks.P2B applies this voting idea to 3D tracking after target-specific feature augmentation.Source: §2 · p. 3, Figure 3 · p. 3

3. P2B: A Novel Network on Point Set for 3D Object Tracking

P2B embeds template target information into search-area seeds, localizes potential target centers, and jointly generates and verifies 3D target proposals in an end-to-end pipeline.

3D target proposal and verification

  • Each augmented search seed predicts a potential target center through Hough voting and receives a seed-wise targetness score.Source: §3.1 · p. 3, §3.3 · p. 4, §3.4 · p. 4
  • Potential centers are clustered with ball queries, and each cluster produces a 3D target proposal with a proposal-wise targetness score.A subset of K centers is sampled for efficiency, producing K proposals.Source: §3.1 · p. 3, §3.3 · p. 4
  • P2B has two main stages: target-specific feature augmentation and 3D target proposal and verification.Source: §3.1 · p. 3

Target-specific feature augmentation

  • Template and search-area points are encoded into seeds, with each seed represented by its 3D position and learned feature.Source: §3.2 · p. 3
  • Permutation-invariant augmentation combines each search seed’s similarities to template seeds with template coordinates and features through an MLP-Maxpool-MLP network.Source: §3.1 · p. 3, §3.2 · p. 4

Ablation

  • Removing template features or similarities from augmentation performs worse than the full feature design in the reported ablation study.Source: §3.2 · p. 4

Final target verification

  • The highest-scoring proposal is selected as the final tracking result, while targetness losses regularize feature learning and proposal generation.Source: §3.4 · pp. 4–5, §3.5 · p. 5

4. Experiments

Experiments use the lidar-based KITTI tracking dataset, following prior work’s data split, tracklet generation, and evaluation metric for fair comparison.

  • The experiments benchmark P2B on KITTI tracking data collected with lidar point clouds.Source: §4 · p. 5
  • The study mainly evaluates car tracking and also reports experiments for pedestrians, vans, and cyclists.Source: §4 · p. 5

4.1. Experimental setting

The experiments use KITTI tracking data, with point-cloud sparsity motivating the setup and One Pass Evaluation measuring tracking quality. P2B samples normalized template and search-area points, uses PointNet++ seeds, and evaluates several search-area and proposal configurations.

Dataset

  • KITTI tracking data were split into training, validation, and testing scenes, with tracklets generated for target instances.The tailored dataset contained 21 outdoor scenes and 8 target types.Source: §4.1.1 · p. 5
  • 34% of KITTI cars contain fewer than 50 points, highlighting severe target sparsity for point-cloud tracking.The paper notes that sparsity may be worse for smaller pedestrians and cyclists.Source: §4.1.1 · p. 5, Figure 5 · p. 5

Evaluation metric

  • Success is the predicted-to-ground-truth box IoU, while Precision is the AUC of center-distance errors from 0 to 2m.Both metrics are measured with One Pass Evaluation.Source: §4.1.2 · p. 5

Implementation details

  • Templates and search areas are normalized to 512 and 1024 points, respectively, before feature extraction.Points may be randomly abandoned or duplicated during normalization.Source: §4.1.3 · p. 5
  • PointNet++ produces 64 template seeds and 128 search-area seeds from three set-abstraction layers with receptive radii of 0.3, 0.5, and 0.7 meters.The backbone output feature dimension is 256.Source: §4.1.3 · p. 5
  • Testing uses the first ground truth and previous result to form the template, while the previous result enlarged by 2 meters defines the search area.The model infers boxes frame by frame within tracklets.Source: §4.1.3 · p. 6

4.2. Comprehensive comparisons

P2B is compared with SC3D under multiple search-area and target-type settings. It achieves roughly 10% average gains and benefits from transferable information when training data are limited.

Comprehensive comparison

  • P2B’s advantage remains large when search areas are centered on previous results, previous ground truth, or current ground truth.Later experiments use the previous-result setting as the realistic configuration.Source: §4.2 · p. 6

Extensive comparisons

  • P2B outperformed SC3D by about 10% on average across Car, Pedestrian, Van, and Cyclist tracking.The comparison reports results across different target types and their mean.Source: §4.2 · p. 6
  • P2B degrades with less training data, especially for Van and Cyclist, whereas SC3D requires relatively less data for similarity measurement.The authors conjecture that center regression benefits from more training data.Source: §4.2 · p. 6

Feature augmentation

  • P2B’s default feature augmentation combines template features with similarity information, while the tested search-area feature variants do not improve performance.The alternatives are illustrated as two ways of incorporating search-area features.Source: Figure 6 · p. 6

Extensive comparisons

  • Training on Car and testing on Van improves P2B’s Success/Precision from 40.8/48.4 to 49.9/59.9.The corresponding SC3D result declines from 40.4/47.0 to 37.2/45.9.Source: §4.2 · p. 6

4.3. Ablation study

Ablations show that target-specific feature components and seed-wise targetness contribute to performance, while P2B remains robust to fewer proposals and varied template-generation strategies.

Target-specific feature augmentation

  • Removing template features lowers performance by about 1%, while removing the similarity map lowers it by about 3%.These results support both components of the default target-specific feature augmentation.Source: §4.3.1 · p. 6

Effectiveness of seed-wise targetness

  • Removing the seed-wise targetness branch reduces performance by about 3%, while omitting its concatenation reduces performance by about 1%.The branch provides supervision for target proposal and verification.Source: §4.3.2 · p. 7

Robustness with different number of proposals

  • P2B remains satisfactory with only 20 proposals, whereas SC3D degrades dramatically below 40 proposals.The comparison was performed without retraining.Source: §4.3.3 · p. 7, Figure 8 · p. 7

Template generation

  • P2B consistently outperforms SC3D across four template-generation settings, including first ground truth, previous result, their fusion, and all previous results.Performance degrades in the all-previous-shapes setting, which the authors attribute to missing shape completion and mismatched training.Source: §4.3.4 · p. 7, Table 6 · p. 7

4.4. Qualitative analysis

Qualitative analyses show that P2B identifies target-relevant seeds and maintains boxes near targets in sparse scenes. Its main failures occur when the initial template contains too little target information.

Targetness and center localization

  • P2B learns to distinguish target seeds from background and gathers informative potential centers near the ground-truth target center.The visualization uses seed-wise targetness and seed-to-center projections.Source: §4.4.1 · p. 7

Comparison with SC3D

  • In sparse scenes where SC3D tracks off course or fails, P2B’s predicted box remains close to the target center.The comparison is presented for dense and sparse sequences.Source: §4.4.1 · p. 7, Figure 9 · p. 8

Failure cases

  • Most P2B failures arise when the initial template is too sparse to provide sufficient target information.In cluttered backgrounds, erroneous predictions may prevent the tracker from recovering an informative template.Source: §4.4.2 · p. 8, Figure 10 · p. 8
  • More points containing rich target information can produce higher-quality proposals and better results.The authors report that this intuition is validated by Fig. 11.Source: §4.4.2 · p. 8

4.5. Running speed

P2B achieves high real-time tracking speed on a single NVIDIA 1080Ti GPU, with most time spent in network forward propagation.

  • 45.5 FPS was achieved on average across all test frames for car tracking on a single NVIDIA 1080Ti GPU.Processing point clouds took 7.0 ms, network forward propagation 14.3 ms, and post-processing 0.9 ms.Source: §4.5 · p. 8

5. Conclusions

P2B embeds target information into search-space seeds and performs point-driven target proposal and verification jointly, reducing the reliance on exhaustive 3D box search. Experiments support its superiority, while also indicating that more data is needed for satisfying results.

  • P2B embeds template target information into search space and jointly performs point-driven target proposal and verification.The method operates on sampled seeds rather than 3D boxes to reduce the search space.Source: §5 · p. 8
  • P2B reduces the search space by using sampled seeds instead of 3D boxes.Source: §5 · p. 8
  • The experiments support P2B's superiority.Source: §5 · p. 8
  • P2B needs more data to obtain satisfying results, and future work includes improving feature augmentation and testing more challenging scenarios.Source: §5 · p. 8

Related papers

Loading 2005.13888v1…