PaperlayerView exact revision on arXiv

Source-linked AI summary

How To Train Your Deep Multi-Object Tracker

Yihong Xu, Aljosa Osep, Yutong Ban, Radu Horaud, Laura Leal-Taixe, Xavier Alameda-Pineda

arXiv:1906.06618v3cs.CV

TL;DR

Deep MOT systems often rely on component losses that do not directly align with MOTA and MOTP, while Hungarian matching makes these metrics non-differentiable. DeepMOT addresses this with differentiable CLEAR-MOT proxies and a Deep Hungarian Net for soft assignment, improving existing trackers and reaching state-of-the-art MOTChallenge results.

  • Problem

    Existing MOT components use losses only indirectly related to MOTA and MOTP, which cannot be directly optimized with gradient descent because standard matching is non-differentiable.

  • Method

    DeepMOT combines differentiable proxies of CLEAR-MOT measures with a Deep Hungarian Net that softly approximates prediction-to-ground-truth assignment for end-to-end training.

  • Results

    DeepMOT-Tracktor establishes new state-of-the-art results on MOT17 and MOT16, improving over Tracktor in MOTA, IDF1, and IDS.

  • Takeaways & Limitations

    The framework improves existing deep MOT methods and jointly optimizes tracking components using losses aligned with standard MOT measures.

  • Takeaways & Limitations

    The distance matrix excludes objects and tracks thresholded out as too far from any possible assignment, while left-out objects or tracks are accounted for separately.

Abstract

from arXiv · show

The recent trend in vision-based multi-object tracking (MOT) is heading towards leveraging the representational power of deep learning to jointly learn to detect and track objects. However, existing methods train only certain sub-modules using loss functions that often do not correlate with established tracking evaluation measures such as Multi-Object Tracking Accuracy (MOTA) and Precision (MOTP). As these measures are not differentiable, the choice of appropriate loss functions for end-to-end training of multi-object tracking methods is still an open research problem. In this paper, we bridge this gap by proposing a differentiable proxy of MOTA and MOTP, which we combine in a loss function suitable for end-to-end training of deep multi-object trackers. As a key ingredient, we propose a Deep Hungarian Net (DHN) module that approximates the Hungarian matching algorithm. DHN allows estimating the correspondence between object tracks and ground truth objects to compute differentiable proxies of MOTA and MOTP, which are in turn used to optimize deep trackers directly. We experimentally demonstrate that the proposed differentiable framework improves the performance of existing multi-object trackers, and we establish a new state of the art on the MOTChallenge benchmark. Our code is publicly available from https://github.com/yihongXU/deepMOT.

1. Introduction

DeepMOT addresses the mismatch between deep MOT training losses and standard tracking measures by making matching and CLEAR-MOT-inspired objectives differentiable. Its Deep Hungarian Net enables soft assignment and gradient-based end-to-end tracker optimization.

  • Vision-based MOT supports applications including mobile robotics and autonomous driving, while many pedestrian trackers associate detector responses over time.
  • Existing deep MOT methods often train individual components with proxy losses that are only indirectly related to standard tracking measures.Examples include triplet loss for identity embeddings.
  • The Hungarian algorithm is needed to compute optimal prediction-to-ground-truth matching, but its non-differentiable operations prevent direct gradient-based optimization of MOTA and MOTP.
  • DeepMOT introduces differentiable variants of CLEAR-MOT measures and combines them into a loss suitable for end-to-end MOT training.
  • The Deep Hungarian Net uses a bi-directional recurrent architecture to approximate Hungarian matching and provide soft prediction-to-ground-truth assignments for back-propagation.The assignment is computed from a prediction-to-ground-truth distance matrix.
  • Training Tracktor with the proposed framework improves over the baseline and establishes a new state-of-the-art result on MOTChallenge datasets.

2. Related Work

Related MOT work spans discrete association, probabilistic inference, reinforcement learning, and deep component-wise learning. DeepMOT is positioned as a differentiable alternative that trains tracking systems with losses aligned to tracking measures.

  • Tracking-by-detection methods detect objects per frame and associate detections over time using matching or offline network-flow optimization.
  • Alternative MOT formulations use MAP estimation with CRFs, MCMC, or variational expectation-maximization, typically relying on hand-crafted appearance and motion descriptors.
  • Some deep MOT methods learn tracking policies with reinforcement learning because standard evaluation measures are not differentiable.
  • Other methods learn appearance models or identity-preserving embeddings with losses such as contrastive, triplet, or quadruplet loss, then use them for data association.
  • Wang et al. learn linear association costs with structured SVM and a MOTA-like loss, but their loss is non-differentiable and limited to network-flow tracker parameters.
  • Tracktor is a tracking-by-regression method trained with smooth L1 bounding-box regression; sequence-level DeepMOT training further improves it on MOTChallenge.

3. Overview and Notation

The MOT evaluation pipeline matches predicted boxes to ground-truth objects frame by frame, then derives tracking counts and metrics from the assignment. DeepMOT replaces the non-differentiable matching step with a soft, differentiable approximation.

  • Each track Xi has an identity i and consists of Li image bounding boxes, while the tracker estimates boxes for all identities through time.
  • At frame t, Nt predicted boxes are compared with Mt ground-truth objects, requiring correspondence between the two sets.Overlapping ground-truth boxes can fit several track hypotheses, making correspondence non-trivial.
  • The standard procedure forms a prediction-to-ground-truth distance matrix D and uses the Hungarian algorithm to obtain an optimal binary assignment matrix A*.An IoU-based distance is commonly used for vision-based tracking.
  • The assignment constraints enforce mutually consistent associations by preventing multiple assignments between predicted tracks and ground-truth objects.
  • True positives count matched tracks, false positives count unmatched predictions, and false negatives count ground-truth objects without matches.
  • Identity switches are counted when a track's assignment to a ground-truth object changes relative to past frames.
  • The distance matrix excludes objects or tracks thresholded out as too far from any possible assignment, with accounting also for left-out objects or tracks.
  • Because MOTA and MOTP are not differentiable, they cannot be directly optimized with gradient descent in their standard form.

4. DeepMOT

DeepMOT replaces the non-differentiable track-to-object matching step with a differentiable soft assignment, enabling proxies of MOTA and MOTP to train trackers end to end. Its DHN uses globally informed recurrent processing, while the loss softly estimates false positives, false negatives, identity switches, and matched-track distances.

  • Differentiable assignment: The framework first performs soft matching between predicted tracks and ground-truth objects, then uses that assignment to approximate the standard MOT measures.DHN produces a differentiable proxy assignment matrix from the prediction-to-ground-truth distance matrix.
  • Deep Hungarian Net: DHN processes row-wise and column-wise flattenings with Bi-RNNs so each assignment decision can account for the entire distance matrix.The two recurrent stages encode intermediate and final assignments before fully connected layers and sigmoid activation produce the soft matrix.
  • Distance matrix: The distance matrix can use differentiable distances beyond IoU, including normalized Euclidean center distance and learned appearance cosine distance.The alternative distance formulation avoids zero gradients when boxes do not intersect.
  • Differentiable MOTA: The soft assignment matrix is converted into estimates of FP and FN by appending threshold-valued entries and applying row-wise and column-wise softmax operations.The added column and row use a threshold such as δ = 0.5; their unmatched outputs estimate false positives and false negatives.
  • DeepMOT loss: dMOTP measures the average distance of matched tracks, while the DeepMOT loss combines 1 − dMOTA and λ(1 − dMOTP).The loss penalizes FP, FN, and IDS through dMOTA while also incorporating matched-track distances through dMOTP.
  • Differentiable MOTA: Differentiable identity switches are estimated by comparing current and previous-frame assignment structures, with padding handling changing numbers of tracks and objects.The binary true-positive matrices are aligned across frames before the soft assignment is used to approximate IDS.

5. Experimental Evaluation

Experiments evaluate DeepMOT’s differentiable matching and loss framework across Tracktor and MOT-by-SOT trackers on pedestrian-tracking datasets. The framework improves tracking performance, including state-of-the-art results on MOT16 and MOT17.

  • Experimental setup: DHN is evaluated as a differentiable approximation to Hungarian matching before tracker experiments on pedestrian-tracking datasets.The DHN training set contains 114,483 matrices for training and 17,880 for testing; its reported weighted accuracy is 92.88%.
  • Tracktor evaluation: Jointly training Tracktor with DeepMOT improves performance over vanilla training, while its impact on identity switches is minimal without an identity-aware component.The experiments compare external and jointly trained ReID strategies and assess the proposed loss on Tracktor.
  • Tracktor evaluation: The jointly trained ReID head improves over the Base Tracktor and external ReID, achieving MOTA +0.04 and IDS ↓31 with approximately 131 K parameters versus 25 M.The ReID head enables the identity-switch term to affect jointly trained Tracktor optimization.
  • Loss ablation: The combined dMOTA+dMOTP loss consistently improves all tracking evaluation measures over Vanilla and Smooth L1 training, while removing the IDS term increases identity switches.The dMOTA term most affects MOTA and IDF1, whereas dMOTP has the strongest effect on MOTP.
  • MOT-by-SOT ablation: DeepMOT improves MOT-by-SOT trackers beyond Smooth L1, including GOTURN MOTA +1.81% and 211 fewer IDS and SiamRPN MOTA +0.65% and 6 fewer IDS.Smooth L1 itself improves MOTA over pretrained models for GOTURN and SiamRPN before DeepMOT provides further gains.
  • Benchmark results: DeepMOT-Tracktor establishes state-of-the-art results on MOT17 and MOT16, improving Tracktor’s MOTA by 0.2% and 0.4% and IDF1 by 1.5% and 0.9%.The corresponding reductions in identity switches are 125 and 37, respectively.

6. Conclusion

The paper proposes an end-to-end MOT training framework based on differentiable approximations of Hungarian assignment and CLEAR-MOT metrics, improving existing deep MOT methods and achieving new state-of-the-art scores on MOT16 and MOT17.

  • The framework differentiably approximates Hungarian assignment and CLEAR-MOT metrics for end-to-end MOT training.
  • The experiments show improved performance for existing deep MOT methods.
  • The method achieves new state-of-the-art scores on the MOT16 and MOT17 datasets.

A.1. DHN

The appendix describes separate DHN and tracker-training procedures, including generated distance-matrix data, optimizer settings, sequence sampling, and augmentations designed to mimic imperfect detections.

  • DHN is trained with RMSprop, while trackers use Adam; the DHN weights are frozen after convergence during tracker training.
  • DHN training uses distance matrices computed from MOTChallenge ground-truth labels and detections, with threshold-based augmentation to discourage selected assignments.
  • The tracker experiments use MOT17 training data, with 2,664 frames, 35,836 ground-truth boxes, and 306 identities in the training split.
  • Training samples use consecutive frames, compute distance matrices from predictions and ground truth, and back-propagate the resulting loss through DHN to the tracker.
  • Bounding-box rescaling and random offsets augment training data to mimic imperfect detections and reduce over-fitting.
  • Tracktor ReID training samples ten consecutive frames and averages past embeddings to compute cosine distances.
  • Test-time track management combines detector refinement, score-based filtering, and multi-frame criteria for track birth and death.

B. Additional DHN Ablation

Additional ablations compare DHN architectures, recurrent units, discretization rules, validity, and matrix-size effects. Sequential DHN with GRUs performs best and generalizes to training-time matrices, although accuracy decreases as matrices grow.

  • The ablation evaluates weighted accuracy, missing assignments, and several assignments while assessing whether discretized outputs preserve permutation-matrix validity.
  • Sequential, parallel, and 1D convolutional DHN architectures are compared, with LSTM and GRU recurrent units additionally tested.
  • 92.88% and 93.49% weighted accuracy are achieved by sequential GRU DHN under row-wise and column-wise maximum discretization, respectively.
  • Sequential GRU exceeds the 1D convolutional variant by 36.45 and 37.31 percentage points in weighted accuracy for row-wise and column-wise maximum, respectively.
  • Sequential GRU has the lowest missing-assignment rates, at 4.79% for row-wise and 6.41% for column-wise maximum.
  • On distance matrices collected during tracker training, sequential GRU reaches 92.71% and 92.36% weighted accuracy for row-wise and column-wise maximum, respectively.
  • DHN assignment accuracy decreases as matrix size grows, while sequential GRU consistently outperforms the alternatives.

C. Training Gradient Visualization

The gradient visualization examines how DeepMOT loss terms affect predicted bounding boxes in manually constructed false-positive, false-negative, and identity-switch cases.

  • Negative gradients are visualized for false-positive and false-negative terms, MOTP, and identity-switch terms across adjacent time steps.
  • The experiment manually constructs FP, FN, and IDS cases to inspect the direction and magnitude of loss gradients.
  • The visualization uses blue predicted boxes, green ground-truth boxes, and red arrows for gradient direction.

D. MOT15 Results

On MOT15, the proposed framework improves the MOT-by-SOT baselines, while DeepMOT-Tracktor remains comparable to vanilla Tracktor and does not improve MOTA.

  • DeepMOT-SiamRPN increases MOTA by 2.3%, MOTP by 0.7%, and IDF1 by 2.0%.
  • DeepMOT-SiamRPN suppresses 2,416 false positives and 143 identity switches, with similar performance gains observed for DeepMOT-GOTURN.
  • DeepMOT-Tracktor obtains results comparable to vanilla Tracktor on MOT15.
  • MOT15 shows no MOTA improvement for DeepMOT-Tracktor, which the authors attribute to noisy labels and vanilla Tracktor's already impressive performance; MOTP increases by 0.3% and false positives decrease by 392.
Loading 1906.06618v3…