Source-linked AI summary
P2B: Point-to-Box Network for 3D Object Tracking in Point Clouds
Haozhe Qi, Chen Feng, Zhiguo Cao, Feng Zhao, Yang Xiao
TL;DR
Point-cloud 3D tracking is difficult under sparse, disordered observations and cannot always rely on RGB-D cues. P2B embeds template information into search-area seeds for end-to-end point-driven proposal and verification, achieving about a 10% gain on KITTI while exposing limitations from sparse initial templates and data demands.
Problem
Point-cloud 3D tracking must handle sparse, disordered data without depending on RGB information that may be degraded or inaccessible.
Method
P2B uses permutation-invariant target-specific feature augmentation to embed template clues into search-area seeds, which vote for target centers and support joint proposal and verification.
Results
∼10% improvement on both Success and Precision over the state-of-the-art method was achieved on the KITTI tracking dataset.
Takeaways & Limitations
P2B reduces the search space by operating on sampled seeds instead of 3D boxes while providing an end-to-end point-driven tracking formulation.
Takeaways & Limitations
Most failures arose when the initial template was too sparse, and the experiments indicate that P2B needs more data for satisfying results.
Abstract
from arXiv · showhide
Towards 3D object tracking in point clouds, a novel point-to-box network termed P2B is proposed in an end-to-end learning manner. Our main idea is to first localize potential target centers in 3D search area embedded with target information. Then point-driven 3D target proposal and verification are executed jointly. In this way, the time-consuming 3D exhaustive search can be avoided. Specifically, we first sample seeds from the point clouds in template and search area respectively. Then, we execute permutation-invariant feature augmentation to embed target clues from template into search area seeds and represent them with target-specific features. Consequently, the augmented search area seeds regress the potential target centers via Hough voting. The centers are further strengthened with seed-wise targetness scores. Finally, each center clusters its neighbors to leverage the ensemble power for joint 3D target proposal and verification. We apply PointNet++ as our backbone and experiments on KITTI tracking dataset demonstrate P2B's superiority (~10%'s improvement over state-of-the-art). Note that P2B can run with 40FPS on a single NVIDIA 1080Ti GPU. Our code and model are available at https://github.com/HaozheQi/P2B.
1. Introduction
P2B addresses point-cloud 3D tracking without RGB by embedding template information into search-area points, then jointly proposing and verifying targets in an end-to-end network. Experiments on KITTI show substantial gains over the prior state of the art.
- Approach: P2B first localizes potential target centers, then performs point-driven 3D target proposal and verification jointly.This replaces intuitive 3D box search with a point-to-box formulation that can be trained end to end.
- Approach: The P2B pipeline has two parts: target-specific feature augmentation and 3D target proposal and verification using a modified PointNet++ backbone.Augmented search seeds regress target centers, while seed-wise targetness scores support joint proposal and verification.
- Approach: Target-specific feature augmentation combines search-point coordinates, point-wise template similarity, and encoded global template features.The resulting features retain spatial geometry while incorporating local tracking clues and global target information.
- Results: ∼10% improvement on both Success and Precision over the state-of-the-art method was achieved on the KITTI tracking dataset.P2B also runs at about 40FPS on a single NVIDIA 1080Ti GPU.
2. Related Works
Related work spans point-cloud tracking, 2D Siamese tracking, point-set learning, target proposal, and Hough voting. P2B adapts these ideas to point-driven 3D tracking while emphasizing end-to-end joint proposal and verification.
- 3D object tracking: Earlier 3D tracking methods generally use RGB-D information, whereas point-cloud-only tracking was only recently studied.RGB dependence can limit use when visual information is degraded or inaccessible.
- 2D Siamese tracking: 2D Siamese trackers compare template and search-area branches with shared weights, but their 2D CNN foundations do not directly address point-cloud data.These methods avoid time-consuming multiscale search and online fine-tuning through a region-proposal-style paradigm.
- Deep learning on point set: Point-set learning addresses disorder, sparsity, and rotation variance across recognition, detection, pose estimation, and tracking tasks.The prior 3D tracking network could not execute end-to-end target proposal and verification jointly, which is P2B’s focus.
- Target proposal: P2B treats each point as a regressor toward a potential target center, unlike 2D target-proposal methods that operate on area-based pixels.The point-driven formulation directly connects target proposal to 3D point-cloud tracking.
- Hough voting: Hough voting provides a learned representation that aggregates information from different observations and has been embedded in end-to-end 3D point-cloud networks.P2B applies this voting idea to 3D tracking after target-specific feature augmentation.
3. P2B: A Novel Network on Point Set for 3D Object Tracking
P2B embeds template target information into search-area seeds, localizes potential target centers, and jointly generates and verifies 3D target proposals in an end-to-end pipeline.
- 3D target proposal and verification: Each augmented search seed predicts a potential target center through Hough voting and receives a seed-wise targetness score.
- 3D target proposal and verification: Potential centers are clustered with ball queries, and each cluster produces a 3D target proposal with a proposal-wise targetness score.A subset of K centers is sampled for efficiency, producing K proposals.
- P2B has two main stages: target-specific feature augmentation and 3D target proposal and verification.
- Target-specific feature augmentation: Template and search-area points are encoded into seeds, with each seed represented by its 3D position and learned feature.
- Target-specific feature augmentation: Permutation-invariant augmentation combines each search seed’s similarities to template seeds with template coordinates and features through an MLP-Maxpool-MLP network.
- Ablation: Removing template features or similarities from augmentation performs worse than the full feature design in the reported ablation study.
- Final target verification: The highest-scoring proposal is selected as the final tracking result, while targetness losses regularize feature learning and proposal generation.
4. Experiments
Experiments use the lidar-based KITTI tracking dataset, following prior work’s data split, tracklet generation, and evaluation metric for fair comparison.
- The experiments benchmark P2B on KITTI tracking data collected with lidar point clouds.
- The study mainly evaluates car tracking and also reports experiments for pedestrians, vans, and cyclists.
4.1. Experimental setting
The experiments use KITTI tracking data, with point-cloud sparsity motivating the setup and One Pass Evaluation measuring tracking quality. P2B samples normalized template and search-area points, uses PointNet++ seeds, and evaluates several search-area and proposal configurations.
- Dataset: KITTI tracking data were split into training, validation, and testing scenes, with tracklets generated for target instances.The tailored dataset contained 21 outdoor scenes and 8 target types.
- Dataset: 34% of KITTI cars contain fewer than 50 points, highlighting severe target sparsity for point-cloud tracking.The paper notes that sparsity may be worse for smaller pedestrians and cyclists.
- Evaluation metric: Success is the predicted-to-ground-truth box IoU, while Precision is the AUC of center-distance errors from 0 to 2m.Both metrics are measured with One Pass Evaluation.
- Implementation details: Templates and search areas are normalized to 512 and 1024 points, respectively, before feature extraction.Points may be randomly abandoned or duplicated during normalization.
- Implementation details: PointNet++ produces 64 template seeds and 128 search-area seeds from three set-abstraction layers with receptive radii of 0.3, 0.5, and 0.7 meters.The backbone output feature dimension is 256.
- Implementation details: Testing uses the first ground truth and previous result to form the template, while the previous result enlarged by 2 meters defines the search area.The model infers boxes frame by frame within tracklets.
4.2. Comprehensive comparisons
P2B is compared with SC3D under multiple search-area and target-type settings. It achieves roughly 10% average gains and benefits from transferable information when training data are limited.
- Comprehensive comparison: P2B’s advantage remains large when search areas are centered on previous results, previous ground truth, or current ground truth.Later experiments use the previous-result setting as the realistic configuration.
- Extensive comparisons: P2B outperformed SC3D by about 10% on average across Car, Pedestrian, Van, and Cyclist tracking.The comparison reports results across different target types and their mean.
- Extensive comparisons: P2B degrades with less training data, especially for Van and Cyclist, whereas SC3D requires relatively less data for similarity measurement.The authors conjecture that center regression benefits from more training data.
- Feature augmentation: P2B’s default feature augmentation combines template features with similarity information, while the tested search-area feature variants do not improve performance.The alternatives are illustrated as two ways of incorporating search-area features.
- Extensive comparisons: Training on Car and testing on Van improves P2B’s Success/Precision from 40.8/48.4 to 49.9/59.9.The corresponding SC3D result declines from 40.4/47.0 to 37.2/45.9.
4.3. Ablation study
Ablations show that target-specific feature components and seed-wise targetness contribute to performance, while P2B remains robust to fewer proposals and varied template-generation strategies.
- Target-specific feature augmentation: Removing template features lowers performance by about 1%, while removing the similarity map lowers it by about 3%.These results support both components of the default target-specific feature augmentation.
- Effectiveness of seed-wise targetness: Removing the seed-wise targetness branch reduces performance by about 3%, while omitting its concatenation reduces performance by about 1%.The branch provides supervision for target proposal and verification.
- Robustness with different number of proposals: P2B remains satisfactory with only 20 proposals, whereas SC3D degrades dramatically below 40 proposals.The comparison was performed without retraining.
- Template generation: P2B consistently outperforms SC3D across four template-generation settings, including first ground truth, previous result, their fusion, and all previous results.Performance degrades in the all-previous-shapes setting, which the authors attribute to missing shape completion and mismatched training.
4.4. Qualitative analysis
Qualitative analyses show that P2B identifies target-relevant seeds and maintains boxes near targets in sparse scenes. Its main failures occur when the initial template contains too little target information.
- Targetness and center localization: P2B learns to distinguish target seeds from background and gathers informative potential centers near the ground-truth target center.The visualization uses seed-wise targetness and seed-to-center projections.
- Comparison with SC3D: In sparse scenes where SC3D tracks off course or fails, P2B’s predicted box remains close to the target center.The comparison is presented for dense and sparse sequences.
- Failure cases: Most P2B failures arise when the initial template is too sparse to provide sufficient target information.In cluttered backgrounds, erroneous predictions may prevent the tracker from recovering an informative template.
- Failure cases: More points containing rich target information can produce higher-quality proposals and better results.The authors report that this intuition is validated by Fig. 11.
4.5. Running speed
P2B achieves high real-time tracking speed on a single NVIDIA 1080Ti GPU, with most time spent in network forward propagation.
- 45.5 FPS was achieved on average across all test frames for car tracking on a single NVIDIA 1080Ti GPU.Processing point clouds took 7.0 ms, network forward propagation 14.3 ms, and post-processing 0.9 ms.
5. Conclusions
P2B embeds target information into search-space seeds and performs point-driven target proposal and verification jointly, reducing the reliance on exhaustive 3D box search. Experiments support its superiority, while also indicating that more data is needed for satisfying results.
- P2B embeds template target information into search space and jointly performs point-driven target proposal and verification.The method operates on sampled seeds rather than 3D boxes to reduce the search space.
- P2B reduces the search space by using sampled seeds instead of 3D boxes.
- The experiments support P2B's superiority.
- P2B needs more data to obtain satisfying results, and future work includes improving feature augmentation and testing more challenging scenarios.