PaperlayerView exact revision on arXiv

Source-linked AI summary

DEFT: Detection Embeddings for Tracking

Mohamed Chaabane, Peter Zhang, J. Ross Beveridge, Stephen O'Hara

arXiv:2102.02267v2cs.CV

TL;DR

Tracking-by-detection can achieve strong benchmark scores with simple associations, while systems combining appearance and motion features can be complex and slow. DEFT jointly learns detector-based appearance matching and motion constraints; it is comparable to leading methods on KITTI and MOT and substantially outperforms the prior top method on nuScenes.

  • Problem

    Tracking-by-detection separates object detection from association, while jointly learned approaches combining appearance and motion can be complex and slow.

    Source: §1 · p. 1

  • Method

    DEFT jointly trains an object-matching network that uses detector-backbone embeddings for appearance association, with an LSTM providing geometric motion constraints.

    Source: §1 · p. 1, §3 · p. 2

  • Results

    3.8x on AMOTA and 2.1x on MOTAR versus CenterTrack on nuScenes; DEFT is comparable in accuracy and speed to leading methods on KITTI and MOT.

    Source: §5 · p. 9

  • Takeaways & Limitations

    DEFT performs better when occlusions or inter-frame displacements are challenging, conditions that represent the majority of nuScenes samples.

    Source: §5 · p. 9

    The stated efficiency and accuracy advantage is framed relative to methods that treat detection as a black-box input to association.

    Source: §1 · p. 1

Abstract

from arXiv · show

Most modern multiple object tracking (MOT) systems follow the tracking-by-detection paradigm, consisting of a detector followed by a method for associating detections into tracks. There is a long history in tracking of combining motion and appearance features to provide robustness to occlusions and other challenges, but typically this comes with the trade-off of a more complex and slower implementation. Recent successes on popular 2D tracking benchmarks indicate that top-scores can be achieved using a state-of-the-art detector and relatively simple associations relying on single-frame spatial offsets -- notably outperforming contemporary methods that leverage learned appearance features to help re-identify lost tracks. In this paper, we propose an efficient joint detection and tracking model named DEFT, or "Detection Embeddings for Tracking." Our approach relies on an appearance-based object matching network jointly-learned with an underlying object detection network. An LSTM is also added to capture motion constraints. DEFT has comparable accuracy and speed to the top methods on 2D online tracking leaderboards while having significant advantages in robustness when applied to more challenging tracking data. DEFT raises the bar on the nuScenes monocular 3D tracking challenge, more than doubling the performance of the previous top method. Code is publicly available.

1 Introduction

Tracking-by-detection has enabled strong performance with simple associations, but learned detection-and-tracking systems can be complex and slow. DEFT jointly learns appearance matching with detection and is presented as efficient and more robust under challenging tracking conditions.

1 Introduction

  • Tracking-by-detection separates frame-level detection from cross-frame association, while jointly learned approaches can improve benchmark performance but may combine appearance and motion cues at the cost of complexity and speed.Recent work has also shown that simple tracking mechanisms paired with state-of-the-art detectors can outperform more complex trackers using older detection architectures.Source: §1 · p. 1
  • DEFT extracts object embeddings from a detector’s multi-scale backbone and uses them as appearance features in an object-to-track association sub-network.The approach jointly learns detection and tracking association.Source: §1 · p. 1
  • DEFT matches leading methods in speed on popular benchmarks and is more robust to occlusions and large inter-frame displacements, dramatically outperforming competitors on nuScenes.The paper attributes this robustness to retaining appearance embeddings over time.Source: §1 · p. 2

2 Related Work

  • Earlier appearance-based association methods often used separate, time-consuming feature extractors; DEFT reuses the detector backbone, adding only small latency beyond detection.The paper contrasts this with treating detection as a black-box input to association.Source: §2 · p. 2
  • DEFT is similar to JDE and FairMOT, but its longer track memory and LSTM-based motion model address challenging cases that CenterTrack’s consecutive-frame association may miss.The paper identifies longer occlusions and large inter-frame displacements as such cases.Source: §2 · p. 2

3 DEFT Network

DEFT jointly trains an object detector and matching network, using detector-backbone embeddings for appearance association and an LSTM for geometric constraints. Its online pipeline combines remembered track embeddings, motion filtering, and bipartite matching.

3 DEFT Network

  • DEFT’s CenterNet implementation achieves state-of-the-art performance on several tracking benchmarks and adds only a few milliseconds of latency beyond detection.Its association module is a small addition within the detection network.Source: §3 · pp. 2–3

3.1 Object Embeddings

  • Embeddings are extracted at detected object centers from multiple detector feature maps, with convolution layers adjusting selected map dimensions before concatenation.Multiple receptive fields provide additional robustness over single-receptive-field embeddings.Source: §3.1 · p. 3

3.2 Matching Head

  • The matching head builds pairwise affinity scores in both directions, adds a non-match option, and averages forward and backward probabilities for each detection pair.It processes padded embedding-pair tensors with 1 × 1 convolutions.Source: §3.2 · pp. 3–4

3.3 Online Data Association

  • Tracks retain embeddings from recent observations, including through missed detections; Hungarian matching assigns detections to tracks, while unmatched detections start new tracks.Tracks without observations beyond the maximum age are deleted.Source: §3.3 · p. 4

3.4 Motion Forecasting

  • An LSTM predicts future track locations and rules out associations to detections too distant from those predictions.The module uses past track information to forecast locations over future frames.Source: §3.4 · p. 4

3.5 Training

  • Training uses frame pairs separated by a random gap, forward and backward matching targets, and a jointly optimized detection-and-matching loss with learned task-balancing weights.Random frame gaps encourage robustness to temporary occlusions or missed detections.Source: §3.5 · pp. 4–5

4 Experiments

DEFT performs similarly to CenterTrack on 2D benchmarks but gains substantially on nuScenes, where it is more robust to occlusions and large inter-frame displacements. Experiments evaluate multiple detector backbones, benchmark settings, and tracking-component ablations.

4.1 Datasets and Metrics

  • MOT Challenge, KITTI, and nuScenes evaluate 2D tracking, vehicle tracking, and monocular 3D tracking, respectively; nuScenes uses 2 FPS key frames, making inter-frame motion challenging.KITTI evaluation uses the Car class; nuScenes evaluation covers seven annotated classes.Source: §4.1 · p. 5

4.2 Implementation Details

  • DEFT achieves its strongest KITTI validation performance with CenterNet, scoring 88.1 MOTA, and the authors use it for the remaining experiments.The compared backbones also include YOLO v3, FPN, and Faster R-CNN; CenterNet yields fewer ID switches than Faster R-CNN despite similar detection performance.Source: §4.2 · p. 6

4.3 Comparative Evaluation

  • 13.1 percentage points in AMOTA, 25.3 in MOTAR, and 11.3 in MOTA are DEFT’s gains over CenterTrack on nuScenes.The comparison is on the monocular 3D tracking leaderboard.Source: Table 4 · p. 6, §4.3 · p. 7

4.4 Performance Analysis

  • CenterTrack’s performance drops 7.6 percentage points from easy to hard occlusion videos, compared with 2.8 for DEFT.On easier videos, their performance difference is marginal; DEFT is superior on moderate and hard videos.Source: §4.4 · p. 7, Figure 2 · p. 7

4.5 Ablation Studies

  • The CenterNet plus motion-and-IoU baseline performs reasonably on MOT17 and KITTI validation data but poorly on nuScenes; multi-scale embeddings yield modest improvements across benchmarks.The ablation results compare MOT17 and KITTI cross-validation with nuScenes validation results.Source: §4.5 · p. 7, Table 5 · p. 8
  • The motion forecasting module improves MOT17 and KITTI modestly and nuScenes more strongly; the LSTM outperforms a Kalman filter more markedly on nuScenes.Adding an IoU-based second-stage association can provide a small gain in some cases.Source: §4.5 · pp. 7–9

5 Conclusion

Local association has performed well on MOT and KITTI, but driving scenarios can involve large displacements and prolonged occlusions. DEFT is comparable to leading methods on those 2D benchmarks and reports substantially stronger results on challenging nuScenes tracking.

5 Conclusion

  • Unlike the local-regime assumption effective on MOT and KITTI, driving-camera tracking can involve large inter-frame displacements and occlusions lasting seconds.The conclusion also notes lower-frame-rate tracking use cases in bandwidth-constrained domains.Source: §5 · p. 9
  • DEFT reports 3.8x on AMOTA and 2.1x on MOTAR versus CenterTrack on nuScenes, with the moderate and hard samples making up most of the benchmark.The conclusion calls the improvement in these cases of considerable practical significance.Source: §5 · p. 9
  • On MOT and KITTI, DEFT is comparable in both accuracy and speed to leading methods.The conclusion also reports near parity with CenterTrack when occlusions and inter-frame displacements are low.Source: §5 · p. 9

1 Motion Forecasting Module

The motion forecasting module predicts future track locations from past bounding-box information. Its input features are represented differently for 2D and 3D tracking.

1 Motion Forecasting Module

  • The motion forecasting module predicts each track’s locations over the next ΔTpred frames from information in the previous ΔTpast frames.It uses features from past bounding-box coordinates.Source: §1 · p. 10
  • For 2D tracking, each detection is represented by an 8-dimensional vector containing center, size, velocity, and changes in width and height over time.The listed components are x and y center, width, height, x/y velocity, and width/height change divided by the interval between detections.Source: §1 · p. 10
  • For 3D tracking, each detection is represented by an 11-dimensional vector containing 3D center, size, orientation, velocity, and rotational velocity.The components include x, y, and z center and velocity, height, width, length, orientation about the z-axis, and rotational velocity.Source: §1 · p. 10

2 Parameter Settings

Parameter settings vary by benchmark. The supplied settings specify embedding dimensions and tracker parameters for MOT16/MOT17, KITTI, and nuScenes.

2 Parameter Settings

  • For MOT16/MOT17, the embedding dimension is 416 and the motion module uses ΔTpast=15 and ΔTpred=10.The tracker parameters Nmax, ngap, δ, ΔTIoU, γ1, γ2, Nage, c are 100, 60, 50, 5, 0.1, 0.4, 50, 10, respectively.Source: §2 · p. 10
  • For KITTI, the embedding dimension is 672 and the motion module uses ΔTpast=10 and ΔTpred=5.The tracker parameters Nmax, ngap, δ, ΔTIoU, γ1, γ2, Nage, c are 100, 30, 25, 3, 0.1, 0.6, 30, 10, respectively.Source: §2 · p. 10
  • For nuScenes monocular tracking, the embedding dimension is 704, frames are resized to 800×448, and the motion module uses ΔTpast=10 and ΔTpred=4.The tracker parameters Nmax, ngap, δ, ΔTIoU, γ1, γ2, Nage, c are 100, 6, 5, 1, 0.1, 0.2, 6, 10, respectively.Source: §2 · p. 10

3 Detectors Implementation Details

DEFT is implemented with four detector backbones for MOT and KITTI, while nuScenes uses CenterNet. The embedding dimension and backbone feature maps vary with the detector and benchmark.

3 Detectors Implementation Details

  • DEFT was trained and tested with CenterNet, YOLO v3, FPN, and Faster R-CNN; nuScenes used only CenterNet.The detectors were pre-trained on COCO, and embedding dimensions were 416 for MOT, 672 for KITTI, and 704 for nuScenes.Source: §3 · p. 10
  • CenterNet used a modified DLA-34 backbone, with embeddings extracted from all 13 feature maps.The backbone details are referred to [51].Source: §3 · p. 10
  • YOLO v3 used a Darknet-53 backbone comprising 53 convolutional layers and extracted feature embeddings from 12 feature maps output by selected backbone layers.Source: §3 · p. 10
  • For FPN and Faster R-CNN, the authors used a ResNet101 [18] backbone and extracted feature embeddings from 11 feature maps, using outputs from selected ResNet101 layers.Source: §3 · p. 10

4 Joint vs Separate Training

The section compares jointly and separately optimized DEFT training. Joint training improves tracking on KITTI and nuScenes without hurting detection performance.

4 Joint vs Separate Training

  • Joint training improves tracking performance on both KITTI and nuScenes without hurting detection performance.The comparison is between jointly and separately optimized DEFT.Source: Table 1 · p. 11, §4 · p. 11
  • Table 1 compares DEFT's joint and separate training strategies on KITTI and nuScenes.Source: Table 1 · p. 11
  • KITTI detection is evaluated with 2D bounding-box AP at easy, moderate, and hard difficulty levels; nuScenes uses mAP over seven classes.KITTI results use 3-fold cross-validation; nuScenes results use the validation set.Source: Table 1 · p. 11

5 MOT16 Results

The MOT16 results provide comparisons with additional methods; the text notes that MOT16 results are generally similar to MOT17.

5 MOT16 Results

  • MOT16 contains the same seven training and seven test sequences as MOT17, and its results are generally similar.Source: §5 · p. 11
  • Table 2 presents accumulated results over all MOT16 sequences.Source: §5 · p. 11
  • JDE is included for comparison using results from its paper because it is absent from the public leaderboard.Source: §5 · p. 11

6 Difficulty Splits on nuScenes

The nuScenes validation videos are partitioned by occlusion, displacement, and combined scores, with Figure 1 showing score distributions and split thresholds.

6 Difficulty Splits on nuScenes

  • Occlusion divides the videos into 76 easy and 74 hard cases, split at the median score.Source: §6 · p. 11
  • Figure 1 shows the distributions of occlusion, displacement, and combined scores, with red lines marking split thresholds.Source: Figure 1 · p. 11, §6 · p. 11
  • Displacement is split into 44 easy, 55 moderate, and 51 hard videos using thresholds half a standard deviation from the median; combined scores yield 43 easy, 60 moderate, and 47 hard.Source: §6 · p. 11

7 Additional 3D Monocular Tracking results

Per-class results on nuScenes validation front-camera imagery show a particular challenge for Trailer tracking with private visual detections.

7 Additional 3D Monocular Tracking results

  • Trailer tracking is particularly difficult with private vision-based detections, while lidar-based detections yield performance more consistent with other classes.Source: §7 · pp. 11–12
  • The authors note that some classes may lack enough training samples for a robust visual detector without the lidar signal.Source: §7 · pp. 11–12
  • The reported table covers nuScenes 3D monocular tracking on the validation set, comparing private and public detections.Source: Table 3 · p. 12
Loading 2102.02267v2…