PaperlayerView exact revision on arXiv

Source-linked AI summary

PTT: Point-Track-Transformer Module for 3D Single Object Tracking in Point Clouds

Jiayao Shan, Sifan Zhou, Zheng Fang, Yubo Cui

arXiv:2108.06455v3cs.CV

TL;DR

Existing point-cloud 3D trackers struggle in sparse scenarios and do not weight features by points’ differing importance, making attention to important clues key to improving performance. PTT applies transformer self-attention and position encoding to weight point-cloud features, and is embedded in P2B to create PTT-Net. On KITTI car tracking, PTT-Net achieved state-of-the-art performance across evaluation metrics and improved ∼11% over baseline algorithm P2B.

  • Problem

    Existing point-cloud 3D trackers struggle in sparse scenarios and do not weight features by points’ differing importance, making attention to important clues key to improving performance.

    Source: §I · p. 1

  • Method

    PTT applies transformer self-attention and position encoding to weight point-cloud features, and is embedded in P2B to create PTT-Net.

    Source: §I · pp. 1–2

  • Results

    On KITTI car tracking, PTT-Net achieved state-of-the-art performance across evaluation metrics and improved ∼11% over baseline algorithm P2B.

    Source: 1) Comparisons on car category · p. 5

  • Takeaways & Limitations

    PTT-Net applies transformer-based feature weighting to point-cloud 3D tracking and is reported to outperform previous state-of-the-art methods.

    Source: §V · p. 6

Abstract

from arXiv · show

3D single object tracking is a key issue for robotics. In this paper, we propose a transformer module called Point-Track-Transformer (PTT) for point cloud-based 3D single object tracking. PTT module contains three blocks for feature embedding, position encoding, and self-attention feature computation. Feature embedding aims to place features closer in the embedding space if they have similar semantic information. Position encoding is used to encode coordinates of point clouds into high dimension distinguishable features. Self-attention generates refined attention features by computing attention weights. Besides, we embed the PTT module into the open-source state-of-the-art method P2B to construct PTT-Net. Experiments on the KITTI dataset reveal that our PTT-Net surpasses the state-of-the-art by a noticeable margin (~10%). Additionally, PTT-Net could achieve real-time performance (~40FPS) on NVIDIA 1080Ti GPU. Our code is open-sourced for the robotics community at https://github.com/shanjiayao/PTT.

I. INTRODUCTION

The introduction motivates point-cloud 3D tracking by its geometric and illumination advantages alongside challenges from sparse, disordered data and feature scarcity. It presents PTT, embeds it in P2B as PTT-Net, and reports improved KITTI tracking results.

I. INTRODUCTION

  • Point-cloud tracking offers geometric information and lower sensitivity to illumination changes, but faces sparse, disordered inputs, higher-dimensional estimates, and unstable features for non-rigid objects.The introduction identifies attention to important clues as key because sparse template and search clouds contain less potential object information.Source: §I · p. 1
  • PTT uses transformer self-attention and position encoding to weight point-cloud features, and is embedded in P2B to form PTT-Net.The introduction describes the module as focusing on deeper-level object clues during tracking.Source: §I · pp. 1–2
  • On KITTI, PTT-Net reports ∼10% improvement on both Success and Precision and outperforms prior state-of-the-art methods.The paper also makes its method open source.Source: §I · p. 2
  • The paper presents PTT as a point-cloud-only tracking module that weights features to focus on deeper-level object clues.Source: §I · p. 2
  • PTT-Net is end-to-end trainable, and the authors describe it as the first application of transformers to point-cloud-based 3D object tracking..Source: §I · p. 2

II. RELATED WORK

The related work reviews point-cloud 3D trackers and transformer methods, identifying limitations in prior tracking approaches and motivating transformer-based feature weighting for 3D tracking.

II. RELATED WORK

  • Prior methods span shape completion, BEV proposals, Siamese tracking, and Hough voting; reported limits include speed, geometry detail, target sizing, and point weighting, [19],,.P2B also loses raw-cloud location distribution through random sampling.Source: A. 3D SOT Using Point Cloud · p. 2
  • The paper proposes weighting point features and using farthest point sampling instead of random sampling to retain more raw point-cloud information.Source: A. 3D SOT Using Point Cloud · p. 2

B. Transformer and Self-attention

III. METHODOLOGY

The methodology applies transformer self-attention to point features for 3D single object tracking, then introduces PTT to process point-cloud shape and geometry information. PTT embeds features, encodes positions, and computes attention features with a residual output.

III. METHODOLOGY

  • The method starts with M XYZ points, extracts N interest-point seeds with 3D coordinates and local-geometry descriptors, and refines their features through attention weighting.The refined features retain the input feature dimension.Source: §III · p. 2

A. Transformer

  • Self-attention uses feature transformations to form Q, K, and V, relates Q and K with position encoding, and normalizes the result to produce attention features.The relation function, normalization function, and Hadamard product are identified in the general formula.Source: A. Transformer · p. 2

B. PTT Module

  • PTT processes point coordinates and descriptors through feature embedding, position encoding, and self-attention, then adds attention features to input features as a residual output.Its position encoding uses k-nearest neighbors to obtain local position information learned by an MLP.Source: Figure 2 · p. 3, B. PTT Module · p. 3

1) Feature Embedding:

Feature embedding maps point features into a higher-dimensional space, while position encoding uses relative coordinates to represent spatial relationships and local geometric shape.

1) Feature Embedding:

  • A linear layer maps point features from dimension D to M, bringing semantically similar features closer in the embedding space.The passage also says the mapping gives the network stronger fitting ability.Source: 1) Feature Embedding · p. 3

2) Position Encoding:

  • Position encoding feeds relative point coordinates into an MLP to capture spatial correlation and local geometric shape information.The relative coordinates are formed between each point and its K-nearest neighbors.Source: 2) Position Encoding · p. 3

3) Self-Attention:

The self-attention module derives pointwise attention from local-neighborhood features and position encodings, then uses the normalized weights to combine value features.

3) Self-Attention:

  • K and V aggregate features from each point’s k-neighborhood to encode local context, while Q, K, and V are computed through linear layers.Q is in R^M, while K and V are in R^(K×M).Source: 3) Self-Attention · p. 3
  • PTT-Net places one transformer block in seed voting to refine augmented features and another in proposal generation to refine cluster features.The second block is intended to help produce better proposals.Source: Figure 3 · p. 4
  • Vector attention uses Q − K for pointwise weights, adds position features to the attention vector and transformed features, and sums weighted V vectors into residual features.The paper motivates vector attention as supporting adaptive modulation of individual feature channels.Source: 3) Self-Attention · pp. 3–4

C. PTT-Net

PTT-Net inserts PTT modules into P2B’s seed-voting and proposal-generation stages to reweight point and target-context features. The paper reports that it outperforms P2B with remarkable margins.

C. PTT-Net

  • In seed voting, PTT reweights augmented features to focus on foreground points with geometric information and suppress background noise.The authors note that P2B does not differentiate search-area point features or locations when generating votes.Source: C. PTT-Net · p. 4
  • In proposal generation, PTT reweights target-wise context features to add deeper target clues that help distinguish similar objects.P2B’s proposal generation uses local context features and can fail to distinguish similar objects such as two pedestrians.Source: C. PTT-Net · p. 4
  • PTT-Net outperforms P2B with remarkable margins.The passage does not quantify the margins.Source: C. PTT-Net · p. 4

1) Loss Function:

The paper follows P2B's loss design and evaluates PTT-Net on KITTI against dominant trackers across rigid and non-rigid object categories.

1) Loss Function:

  • The overall loss follows P2B and combines voting and proposal classification losses with voting and proposal box regression losses.The coefficients λ1, λ2, and λ3 weight the component losses.Source: 1) Loss Function · p. 4

IV. EXPERIMENTS

  • PTT-Net is compared with dominant methods on KITTI for rigid and non-rigid objects, with the authors reporting substantial margins at 40 fps.The evaluation includes Car, Van, Pedestrian, and Cyclist categories.Source: §IV · p. 4

1) Dataset:

  • The KITTI training set contains more than 20,000 manually labeled 3D objects captured with a Velodyne HDL-64E lidar.The 20 sequences are split into training (00–16), validation (17–18), and test (19–20) sets.Source: 1) Dataset · p. 4

2) Evaluation Metric:

The experiments use Success and Precision metrics on KITTI, with quantitative comparisons spanning car and other object categories.

2) Evaluation Metric:

  • Success and Precision are reported using One Pass Evaluation, representing overlap and error AUC, respectively.The evaluation follows prior work.Source: 2) Evaluation Metric · p. 4

3) Implementation Details:

  • Training uses farthest point sampling, Adam with an initial learning rate of 0.001, batch size 48, and 60 epochs.The learning rate is reduced fivefold after 12 epochs; template augmentation also extends the offset to include z.Source: 3) Implementation Details · pp. 4–5

B. Quantitative Experiments

1) Comparisons on car category:

On KITTI, PTT-Net improves over P2B on car and other-category tracking, and performs strongly across four template-generation settings.

1) Comparisons on car category:

  • PTT-Net achieves state-of-the-art car-tracking performance and improves by approximately 11% over P2B.Against RGB+LiDAR methods, its Success/Precision results outperform by 4.7%/12.1% and by 30.7%/31.2%, respectively.Source: 1) Comparisons on car category · p. 5

2) Comparisons on other categories:

  • PTT-Net's average performance on Pedestrian, Van, and Cyclist exceeds P2B by approximately 13%.For Pedestrian tracking, Success/Precision improve by 9.7%/15.8%.Source: 2) Comparisons on other categories · p. 5

1) Template Area Generation:

  • PTT-Net achieves state-of-the-art performance across four template-generation settings: first ground truth, previous result, their fusion, and all previous results.The table distinguishes GT and “First & Previous” as ground truth and the first ground truth plus previous result.Source: 1) Template Area Generation · p. 5, Table III · p. 5

2) Embedding location of PTT module:

Embedding PTT in both P2B stages gives the best improvement among tested placements, while qualitative examples show foreground focus and robust tracking in difficult scenes.

2) Embedding location of PTT module:

1) Advantageous cases:

D. Qualitative Experiment

1) Advantageous cases:

  • When scenes contain fewer than 50 points, PTT-Net tracks tightly while SC3D and P2B eventually fail.The comparison is shown for sparse scenarios.Source: 1) Advantageous cases · p. 6

2) Failure cases:

PTT-Net outperforms previous state-of-the-art methods, but tracking can fail when the initial frame contains too few points. It runs at 40 FPS on an NVIDIA 1080Ti GPU.

2) Failure cases:

  • Tracking failures occurred mostly when the first-frame car had fewer than 20 points; four tracklets initialized with zero points also failed.In these cases, PTT-Net could not learn effective object characteristics.Source: 2) Failure cases · p. 6, Figure 6 · p. 6

E. Timing Breakdown

  • 40 FPS on a single NVIDIA 1080Ti GPU, compared with 1.8 FPS for SC3D, 45.5 FPS for P2B, and 20.8 FPS for 3D-SiamRPN.The PTT-Net total includes 8.3 ms for point-cloud preparation, 16.2 ms for model forward propagation, and 0.5 ms for post-processing.Source: E. Timing Breakdown · p. 6

V. CONCLUSIONS

  • PTT-Net outperforms previous state-of-the-art methods with remarkable margins.The conclusion does not quantify the margins.Source: §V · p. 6

Related papers

Loading 2108.06455v3…