PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Affinity Network for Multiple Object Tracking

ShiJie Sun, Naveed Akhtar, HuanSheng Song, Ajmal Mian, Mubarak Shah

arXiv:1810.11780v2cs.CV

TL;DR

MOT requires associating detected objects across frames, while data association has relied substantially on handcrafted constraints. DAN jointly learns object appearances and affinities end-to-end, and its online tracker performs competitively across three MOT challenges, with limitations for crowded scenes containing similar-looking objects.

  • Problem

    Data association in MOT has not fully benefited from deep learning, despite advances in object detection and the need to establish cross-frame object correspondences.

  • Method

    DAN jointly models hierarchical appearances and cross-frame affinities for nonconsecutive frame pairs, then associates current objects with multiple previous frames for online tracking.

  • Results

    The tracker runs at 6.3 frames per second and surpasses existing leading approaches on a majority of metrics across MOT15, MOT17, and UA-DETRAC.

  • Takeaways & Limitations

    End-to-end affinity learning supports online MOT with reliable trajectory generation across multiple previous frames and strong challenge performance.

  • Takeaways & Limitations

    DAN can underperform when crowded scenes contain multiple similar-looking objects at close locations, sometimes causing identity switches.

Abstract

from arXiv · show

Multiple Object Tracking (MOT) plays an important role in solving many fundamental problems in video analysis in computer vision. Most MOT methods employ two steps: Object Detection and Data Association. The first step detects objects of interest in every frame of a video, and the second establishes correspondence between the detected objects in different frames to obtain their tracks. Object detection has made tremendous progress in the last few years due to deep learning. However, data association for tracking still relies on hand crafted constraints such as appearance, motion, spatial proximity, grouping etc. to compute affinities between the objects in different frames. In this paper, we harness the power of deep learning for data association in tracking by jointly modelling object appearances and their affinities between different frames in an end-to-end fashion. The proposed Deep Affinity Network (DAN) learns compact; yet comprehensive features of pre-detected objects at several levels of abstraction, and performs exhaustive pairing permutations of those features in any two frames to infer object affinities. DAN also accounts for multiple objects appearing and disappearing between video frames. We exploit the resulting efficient affinity computations to associate objects in the current frame deep into the previous frames for reliable on-line tracking. Our technique is evaluated on popular multiple object tracking challenges MOT15, MOT17 and UA-DETRAC. Comprehensive benchmarking under twelve evaluation metrics demonstrates that our approach is among the best performing techniques on the leader board for these challenges. The open source implementation of our work is available at https://github.com/shijieS/SST.git.

1 INTRODUCTION

MOT separates object detection from data association, but association still relies heavily on handcrafted appearance, motion, and related constraints. DAN jointly learns object appearances and cross-frame affinities end-to-end, supporting nonconsecutive frames, entering or leaving objects, and online tracking.

  • Motivation: MOT combines per-frame object detection with cross-frame association, yet association has not fully benefited from deep learning.The challenge is especially pronounced when computing affinities between objects separated by multiple video frames.
  • Prior approaches: Existing trackers estimate affinities from separate appearance, motion, or composite representation models.Motion models may use linear or nonlinear dynamics, while composite methods seek a balance between motion and appearance.
  • Proposed approach: DAN jointly learns hierarchical object appearances and affinities for frame pairs through a shared two-stream convolutional network trained end-to-end.The network uses exhaustive permutations of compact features and separately handles objects entering or leaving between frames.
  • Tracking strategy: DAN does not require input frames to be consecutive, promoting robustness against object occlusions in the induced model.Its affinity computations are used to associate current-frame objects with objects in multiple previous frames for trajectory generation.
  • Evaluation: 6.3 frames per second; DAN surpasses existing leading approaches on a majority of evaluation metrics across MOT15, MOT17, and UA-DETRAC.The authors also report significant gains over nearest competitors for online multiple pedestrian tracking.

2 RELATED WORK

Related MOT research commonly follows tracking-by-detection, detecting objects before associating them across frames. Prior work spans local and global association, detector-robustness strategies, online classifiers, deep feature extraction, and dataset development; DAN targets end-to-end affinity learning within this landscape.

  • Tracking-by-detection: Tracking-by-detection detects objects first and then associates them across frames, with local and global methods forming broad categories.Local methods are computationally efficient because they use only two frames, whereas global methods associate over larger frame sets.
  • Local methods: Local association methods can be sensitive to camera motion and pose variation despite their computational efficiency.These factors are described as tracking-irrelevant influences that can affect local association performance.
  • Global methods: Global methods cast multi-frame association as network-flow or related optimization problems, while dense detections aim to mitigate occlusion and close-proximity issues.These approaches use larger temporal contexts or avoid suppressing potentially useful detections.
  • Online adaptation: Online trainable classifiers have had limited success in MOT, although they are more beneficial in single-object tracking.The related work describes their MOT adoption as restricted to limited scenarios.
  • Deep representations: Deep tracking methods commonly reuse classification-pretrained models to extract object features for association.Related examples include Siamese architectures for single-object and multiple-object tracking.
  • Datasets: MOT research also emphasizes dataset collection and annotation, with MOT and UA-DETRAC identified as particularly relevant datasets.The paper positions its contribution against the broader dataset and association literature.
  • Positioning DAN: DAN addresses the stated gap by jointly modeling object appearance and inter-frame affinity under end-to-end training.This links deep representation learning to the association component rather than using appearance features independently of affinity modeling.

3 PROPOSED APPROACH

The proposed approach uses DAN to learn object features and cross-frame affinities, then performs online association across multiple previous frames. Its pipeline prepares detections and training pairs, computes bidirectional affinity losses, and updates trajectories without future frames.

  • 3 PROPOSED APPROACH: DAN jointly models object appearances and cross-frame affinities for frame pairs that may be separated by multiple timestamps.Training uses frame pairs with n ∈ N sampled between 1 and NV, while deployment associates the current frame with multiple previous frames.
  • 3 PROPOSED APPROACH: The tracker represents each frame with detected-object centers and feature columns, while association matrices encode cross-frame correspondences and affinities.Ct stores object centers, Ft stores per-object feature vectors, Lt−n,t encodes correspondence, and At−n,t encodes bounding-box similarities.
  • 3 PROPOSED APPROACH: Training data are augmented with photometric distortions, frame expansion, sequential transformations, and horizontal flips to expose DAN to tracking-irrelevant variation.Each preprocessing step is applied to frame pairs with probability 0.3, and the frames are resized to fixed dimensions before possible horizontal flipping.
  • 3.2.3 Affinity estimator: DAN trains with four sub-losses covering forward association, backward association, directional consistency, and suppression of non-maximum affinity predictions.The final loss is the mean of the four sub-losses, and the forward/backward losses maximize probabilities for relevant affinity coefficients.
  • 3.4 Deep track association: Online track association applies the Hungarian algorithm to an accumulator matrix and bounds retained track history and disappearance waiting time with δb and δw.The accumulator integrates affinities over previous δb frames; trajectories exceeding δb are shortened, while disappearing objects remain for δw frames before removal.

4 EXPERIMENTS

The experiments evaluate DAN on established online MOT challenges and describe how its deployment computes affinities across recent frames to update trajectories.

  • DAN is evaluated on MOT17, MOT15, and UA-DETRAC, with challenge servers performing evaluation after submissions.
  • For frame t, detector-provided object centers produce F_t, which is paired with previous feature matrices to compute affinity matrices.
  • The current feature matrix is stored so future frames can compute affinities with earlier frames.
  • The trajectory set T_t is updated by associating the current frame with t previous frames using the computed affinities.

4.1 Implementation details

DAN is implemented in PyTorch and tuned using MOT17 validation data, including grid search for its temporal association parameters.

  • DAN is implemented with PyTorch and trained on an NVIDIA GeForce GTX Titan GPU.
  • MOT17 is used for hyper-parameter optimization because of its manageable size, with a specified validation set for training.
  • The learning rate starts at 0.01 and is reduced by a factor of ten at epochs 50, 80, and 100.
  • Grid search optimizing validation MOTA selects δ_w = 12 and δ_b = 15 from multiples of three between 3 and 30.

4.2 Multiple Object Tracking 17 (MOT17)

MOT17 contains varied pedestrian scenes and detector outputs, and DAN is benchmarked with twelve metrics, leaderboard comparisons, and qualitative occlusion examples.

  • MOT17 contains seven indoor and outdoor pedestrian scenes, each divided into training and testing clips.
  • The dataset provides detections from SDP, Faster-RCNN, and DPM, and accepts both online and offline tracking approaches.
  • The training scenes vary in resolution, frame rate, pedestrian density, track count, background, illumination, and camera viewpoint.
  • Testing data contains 17,757 frames, 2,355 tracks, and 564,228 boxes, while labels and boxes remain private for server evaluation.
  • DAN is benchmarked using twelve standard metrics, including CLEAR MOT and MT/ML metrics.
  • DAN outperforms existing online and offline methods on five metrics and is significantly better than online methods on MOTA and MOTAL.
  • Qualitative MOT17 examples show recovery from inter-frame occlusions, although one severe partial occlusion causes a temporary identity error.

4.3 Multiple Object Tracking 15 (MOT15)

MOT15 provides a broader benchmark of pedestrian trackers, and DAN significantly outperforms existing methods on three metrics, especially MOTA.

  • MOT15 contains 11 pedestrian video sequences with ground-truth tracks and detections from the detector proposed by Dollar et al.
  • Four MOT15 sequences share scenes with MOT17, enabling fair comparison using average results on those corresponding scenes.
  • DAN significantly outperforms existing MOT15 methods on three metrics, especially MOTA.
  • Against CDA-DDAL, DAN uses only learned appearance affinities, whereas CDA-DDAL combines l2-distance appearance affinities with shape and motion affinities.
  • CDA-DDAL significantly underperforms DAN despite using additional affinity types under a related tracking pipeline.

4.4 UA-DETRAC

UA-DETRAC evaluates vehicle tracking across diverse traffic scenes, using precision-recall-based metrics. DAN maintains identities through missed detections and its performance improves with more accurate detectors.

  • Dataset: UA-DETRAC contains about 140k frames, 8,250 vehicles, and 1,210k bounding boxes from 100 traffic videos across 24 locations.The dataset covers urban highways, crossings, T-junctions, vehicle types, and traffic conditions.
  • Evaluation: UA-DETRAC reports PR-prefixed metrics by integrating performance over precision-recall curves generated from varying detector thresholds.PR-MOTA, for example, integrates MOTA values across the resulting three-dimensional curve.
  • Results: 18.6 PR-MOTA and 35.8 PR-MOTAP are obtained by CompACT+DAN, while RCNN+DAN obtains 15.1 PR-MOTA and 37.1 PR-MOTAP.The reported values differ with detector choice, and overall performance improves with detector accuracy.
  • Results: DAN’s UA-DETRAC results support effective vehicle tracking on roads, with detector selection remaining an empirical component of each compared technique.The authors state that these results ascertain the overall effectiveness of the proposed approach.
  • Tracking examples: DAN correctly restores vehicle identities after detector failures caused by occlusion in the illustrated UA-DETRAC scenes.In one scene, identity-2 is missed across frames 130–134; in another, identity-1 is missed across frames 132–143 before correct reassignment.

5 DISCUSSION

The discussion links DAN’s performance to jointly learned appearance and affinity representations, tests robustness across difficult frame pairs, and examines architectural choices and runtime. It also identifies similar-looking nearby objects as a residual weakness.

  • Robustness: DAN associates objects across randomly selected frame gaps of 1–30 timestamps despite illumination changes, partial occlusions, and multiple similar-looking objects.The examples are described as representative of similar performance across tested cases.
  • Affinity evaluation: DAN’s predicted affinity error is evaluated against DCML and SSIM on MOT17 frame gaps of n = 1, 5, 10, 15, 20, 25, and 30.Mean absolute error is computed against ground-truth association matrices for all possible frame pairs at each gap.
  • Limitation: DAN sometimes switches identities when similar-looking objects are close together across multiple timestamps, although Deep Track Association enables recovery.The authors expect weaker performance in crowded scenes containing multiple similar-looking objects, a limitation shared by appearance-based trackers.
  • Ablation: DAN achieves better results in fewer epochs than the tested variants, while DAN-Mean is the closest alternative at 120 epochs.The ablation compares training losses and MOT17 training-set MOTA and IDF1 scores at 120 epochs.
  • Ablation: Removing feature dimension reduction, gradual compression, or the Extension sub-network degrades performance relative to the proposed architecture.The discussion attributes poor performance to DAN-Remove and a significant drop to DAN-Curtail, while favoring gradual feature-map compression.
  • Runtime: DAN processes a single training frame pair in 188.54 ms on an NVIDIA GeForce GTX Titan GPU, corresponding to 1508.33 ms for eight samples.Runtime measurements also report major tracking-component timings on the MOT17 validation set.

6 CONCLUSION

The paper concludes with an online tracker that associates current detections with multiple previous frames using DAN’s multilevel features and exhaustive affinity analysis. Across MOT15, MOT17, and UA-DETRAC, it reports strong benchmark performance and 6.3 frames per second.

  • Conclusion: DAN performs online tracking by associating detections in the current frame with objects from multiple previous frames.The tracker records cross-frame similarities and object features to trace trajectories.
  • Conclusion: DAN models pre-detected objects at multiple abstraction levels and infers affinities by analyzing exhaustive permutations of extracted features.The architecture is presented as the source of the tracker’s association capability.
  • Conclusion: 6.3 frames per second is achieved across the MOT17, MOT15, and UA-DETRAC online tracking challenges.The evaluation uses twelve different metrics, and the tracker leads many online-tracking leaderboard results.
  • Conclusion: DAN achieves the highest Multiple Object Tracking Accuracy on all three evaluated challenges.The conclusion also describes its overall performance as excellent.
Loading 1810.11780v2…