PaperlayerView exact revision on arXiv

Source-linked AI summary

FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking

Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, Wenyu Liu

arXiv:2004.01888v6cs.CV

TL;DR

MOT benefits from combining detection and re-ID in one efficient network, but prior one-shot systems can create task conflicts and degrade association quality. FairMOT addresses these issues with an anchor-free, CenterNet-based design that treats detection and re-ID equally. It achieves high detection and tracking accuracy and outperforms previous state-of-the-art methods across multiple benchmark datasets.

  • Problem

    One-shot MOT seeks efficient joint detection and re-ID, but prior systems can bias the network toward detection and substantially degrade tracking or re-ID performance.

  • Method

    FairMOT is an anchor-free, CenterNet-based single network that treats detection and re-ID equally and addresses anchor, feature-sharing, and feature-dimension issues.

  • Results

    FairMOT achieves high detection and tracking accuracy and outperforms previous state-of-the-art methods by a large margin on multiple benchmark datasets.

  • Takeaways & Limitations

    FairMOT provides a simple, efficient one-shot tracking design that offers a good trade-off between detection and re-ID performance.

  • Takeaways & Limitations

    Video object detection methods discussed in the paper can be extremely slow when videos contain many objects.

Abstract

from arXiv · show

Multi-object tracking (MOT) is an important problem in computer vision which has a wide range of applications. Formulating MOT as multi-task learning of object detection and re-ID in a single network is appealing since it allows joint optimization of the two tasks and enjoys high computation efficiency. However, we find that the two tasks tend to compete with each other which need to be carefully addressed. In particular, previous works usually treat re-ID as a secondary task whose accuracy is heavily affected by the primary detection task. As a result, the network is biased to the primary detection task which is not fair to the re-ID task. To solve the problem, we present a simple yet effective approach termed as FairMOT based on the anchor-free object detection architecture CenterNet. Note that it is not a naive combination of CenterNet and re-ID. Instead, we present a bunch of detailed designs which are critical to achieve good tracking results by thorough empirical studies. The resulting approach achieves high accuracy for both detection and tracking. The approach outperforms the state-of-the-art methods by a large margin on several public datasets. The source code and pre-trained models are released at https://github.com/ifzhang/FairMOT.

1 Introduction

FairMOT addresses the competition between detection and re-ID in one-shot MOT by treating both tasks equally in an anchor-free, CenterNet-based network. The approach is designed to improve tracking while retaining efficient inference.

  • Motivation: Two-step trackers face scalability problems because separate models do not share features and re-ID is applied independently to every bounding box.This limits real-time inference when many objects appear in the environment.
  • Motivation: One-shot MOT combines object detection and re-ID in one network, but prior systems can preserve detection accuracy while substantially degrading tracking performance.The degradation is associated with many more ID switches, making joint optimization non-trivial.
  • Failure Analysis: Anchor-based one-shot architectures can overlook effective re-ID learning, while shared features conflict because detection and re-ID require different feature characteristics.Detection features should be similar across instances, whereas re-ID features need low-level information that discriminates instances of the same class.
  • Failure Analysis: Large re-ID feature dimensions can harm both tasks, whereas experiments indicate that low-dimensional re-ID features improve tracking accuracy and efficiency.The paper identifies feature dimension as a third issue distinguishing joint MOT from standalone re-ID.
  • FairMOT: FairMOT uses an anchor-free CenterNet-based design with homogeneous detection and re-ID branches, treating the two tasks equally rather than making re-ID secondary.The network extracts high-resolution feature maps, predicts object centers and sizes, and produces re-ID features for object-centered locations.
  • Results: FairMOT ranks first on 2DMOT15, MOT16, MOT17, and MOT20, while running at 30 FPS on a single RTX 2080Ti GPU.The authors report additional gains on all datasets after single-image pre-training.

2 Related Work

Related MOT work spans separate detection-and-association pipelines, joint detection and re-ID or motion prediction, appearance and motion cues, and offline global optimization. Single-network methods improve efficiency but must manage task conflicts and can face limitations in challenging scenes or deployment settings.

  • Tracking-by-Detection: Tracking-by-detection methods first detect objects and then associate them over time, while separate models allow task-specific design and scale-normalized re-ID crops.Their main trade-off is that independent processing can be computationally expensive when object counts are high.
  • Association Cues: Location- and motion-based methods use predicted positions, overlap, or learned motion cues for association, but challenging crowded scenes and fast motion remain difficult.Some single-object tracking approaches improve localization but are extremely slow with many people.
  • Association Cues: Appearance-based methods crop detections for re-ID feature extraction, while other approaches enhance or fuse appearance, pose, motion, and location cues.These methods use feature similarity and assignment procedures to link detections with tracklets.
  • Offline Methods: Offline methods perform global optimization over an entire sequence and often achieve better results than online approaches.Their global formulation can represent detections across all frames and search for an optimal assignment.
  • Joint Models: Joint detection-and-re-ID methods use one network to reduce inference time, whereas joint detection-and-motion methods learn motion features alongside detection.Track-RCNN adds a re-ID head to Mask R-CNN, while other methods predict inter-frame displacements or exploit tracklets.
  • Joint Models: Joint models can struggle with association quality or occlusion: one-shot trackers may suffer degraded association, and adjacent-frame methods have difficulty re-initializing lost tracks.FairMOT investigates task conflicts in the first class, while motion-based methods may fail when tracks are occluded.
  • Multi-Task Learning: Multi-task learning methods offer strategies for balancing detection and re-ID losses, including uncertainty-based weighting and common-gradient directions.These approaches address optimization among competing task objectives in shared networks.
  • Video Object Detection: Video object detection uses temporal information to improve detection, but related methods can be extremely slow when videos contain many objects.The cited work includes tube-based detection and methods that leverage neighboring frames.

3 Unfairness Issues in One-shot Trackers

Existing one-shot trackers face three linked unfairness issues: anchor-based designs can bias training toward detection, mix identities in extracted features, and create ambiguous identity assignments. Shared high-dimensional features can also intensify competition between detection and re-ID.

  • Anchor Design: Anchor-based one-shot trackers often produce many ID switches despite good detection because anchors are unsuitable for learning re-ID features.The paper identifies three perspectives underlying this mismatch.
  • Anchor Design: Cascaded proposal-first training biases Track R-CNN toward accurate object proposals rather than high-quality re-ID features.Re-ID features depend heavily on proposal quality during training.
  • Anchor Design: ROI-Align sampling can include other instances or background, whereas extracting features at a single estimated object center better represents the target.The paper presents center-only extraction as a remedy for mixed-content anchors.
  • Anchor Design: Multiple adjacent anchors may predict one identity, while small perturbations can force one anchor to predict different identities, creating severe training ambiguity.These conflicts arise when anchors correspond to different patches or shift under augmentation.
  • Feature Sharing and Dimension: Detection and re-ID require different feature characteristics: detection favors deep features, while re-ID benefits from low-level appearance features and lower-dimensional embeddings for one-shot MOT.The paper states that high-dimensional re-ID features can harm detection and final tracking accuracy while increasing computation.

4 FairMOT

FairMOT combines anchor-free CenterNet-style detection with center-based re-ID features in a jointly trained network. Its design includes multi-layer feature aggregation, center heatmaps, precise box regression, identity classification, uncertainty-based task balancing, and online association.

  • Backbone: FairMOT uses a ResNet-34 backbone with enhanced DLA feature aggregation and deformable convolutions to fuse feature levels and adapt receptive fields.The modifications add skip connections, dynamically adjust receptive fields for object scales and poses, and help alleviate alignment issues.
  • Detection Branch: The anchor-free detection branch appends parallel heads to estimate object heatmaps, center offsets, and bounding-box sizes.It is built on CenterNet, although other anchor-free detectors can also be used.
  • Detection Branch: The heatmap head represents object centers with peak responses that decay exponentially with distance from each ground-truth center.This provides the center-based representation used for landmark-style object localization.
  • Detection Branch: Offset and size heads use continuous center offsets and box dimensions to reduce stride-induced quantization errors and recover target boxes.The final feature-map stride can introduce errors up to four pixels, which the offset branch mitigates.
  • Re-ID Branch: The re-ID branch extracts a 128-dimensional feature at each location and trains center features with identity classification across all training identities.Only embedding vectors located at object centers are used for training because centers are available from the objectness heatmap at test time.
  • Training and Inference: FairMOT jointly trains detection and re-ID losses using uncertainty-based weighting, and performs hierarchical online association with Kalman filtering, re-ID matching, and IoU matching.Unmatched detections become new tracks, while unmatched tracklets are retained for 30 frames.

5.1 Datasets and Metrics

FairMOT is trained across six datasets with varying annotation support and evaluated on four MOT benchmarks using detection AP and re-ID TPR at a specified false accept rate.

  • Datasets: Six training datasets provide either box annotations alone or both box and identity annotations, determining whether FairMOT trains its detection branch alone or both branches.ETH and CityPerson provide boxes only; CalTech, MOT17, CUHK-SYSU, and PRW also provide identities.
  • Metrics: The approach is evaluated on 2DMOT15, MOT16, MOT17, and MOT20 using Average Precision for detection and TPR@FAR=0.1 for re-ID.Re-ID evaluation retrieves similar candidates using features extracted from ground-truth boxes.

5.2 Implementation Details

The default FairMOT implementation uses a COCO-pretrained DLA-34 variant, Adam optimization for 30 epochs, and 1088 × 608 inputs with standard augmentation.

  • Data and Compute: Inputs are resized to 1088 × 608, producing 272 × 152 feature maps, with rotation, scaling, and color-jitter augmentation.Training takes about 30 hours on two RTX 2080 Ti GPUs.

5.3 Ablative Studies

The ablations examine feature sampling, loss balancing, backbone fusion, feature dimensions, and association choices. Together, they show that FairMOT benefits from designs that balance detection and re-ID rather than optimizing either task alone.

  • 5.3.1 Anchors: Center sampling obtains higher IDF1 and True Positive Rate than ROI-Align, POS-Anchor, and the two-stage approach, with fewer ID switches.The comparison keeps the remaining FairMOT components unchanged and evaluates alternative re-ID feature sampling strategies.
  • 5.3.2 Balancing Multi-task Losses: Fixed loss weighting favors MOTA and AP but worsens IDs and TPR, whereas MGDA-UB favors TPR at the expense of MOTA and AP.GradNorm achieves the best overall tracking accuracy, while the simpler Uncertainty method is slightly worse but used thereafter.
  • 5.3.3 Multi-layer Feature Fusion: ResNet-34-FPN raises TPR from 90.9% to 94.2% over ResNet-50 despite having fewer parameters, showing the value of multi-layer feature fusion.RegNetY-4.0GF-FPN adds +1.9 MOTA, +1.3 IDF1, -36.9% IDs, +2.2 AP, and +2.3 TPR over RegNetY-4.0GF.
  • 5.3.3 Multi-layer Feature Fusion: DLA-34 increases TPR from 90.9% to 94.4% and reduces ID switches from 435 to 299, while deformable convolution supports its detection performance.Replacing DLA-34 deformable convolutions with normal convolutions yields 65.0 MOTA and 78.1 AP.
  • 5.3.4 Feature Dimension: 64-dimensional re-ID features outperform 512-dimensional features for JDE, while FairMOT trades stronger IDF1 and TPR at 512 dimensions for higher MOTA at 64 dimensions.The paper attributes FairMOT’s MOTA change to competition between detection and re-ID tasks.
  • Association and Runtime: Using re-ID features alongside Kalman filtering improves IDF1 and reduces ID switches compared with using box IoU alone, especially in crowded scenes and fast camera motion.Runtime for joint detection and re-ID is minimally affected by object density, while re-ID matching time increases linearly with density.

5.4 Single Image Training

FairMOT is pretrained on CrowdHuman with single-image identity assignments and then fine-tuned on MOT17. This pretraining improves detection and association performance compared with direct MOT17 training.

  • 5.4 Single Image Training: FairMOT is pretrained on CrowdHuman by assigning each bounding box a unique identity label, then fine-tuned on the target MOT17 dataset.The procedure uses single-image training rather than simulated consecutive frames.
  • 5.4 Single Image Training: CrowdHuman pretraining improves human detection and domain generalization, while re-ID training further enhances the tracker’s association ability.The pretrained model can also be directly used as a tracker on MOT datasets such as MOT17.
  • 5.4 Single Image Training: Pretraining on CrowdHuman outperforms direct training on MOT17 by a large margin.

5.5 Results on MOTChallenge

FairMOT achieves strong tracking accuracy and near-video-rate inference across MOTChallenge evaluations, outperforming prior one-shot and two-step methods. Its performance remains high with limited or varied training data.

  • Evaluation setting: FairMOT is evaluated under the private-detector protocol on the testing sets of four MOTChallenge datasets.The reported results are obtained from the official MOT challenge evaluation server.
  • Comparison with SOTA MOT methods: FairMOT ranks first among online and offline trackers on 2DMOT15, MOT16, MOT17, and MOT20.It also achieves video-rate inference and ranks second under the ALTA metric.
  • Training data ablation: 69.8 MOTA is achieved using only MOT17 training data, while 72.9 MOTA is achieved with JDE’s training data.Single-image training on CrowdHuman further raises the MOTA score to 73.7.

5.6 Qualitative Results

Qualitative MOT17 results show that FairMOT can preserve identities when pedestrians cross, using high-quality re-ID features rather than relying only on bounding-box overlap.

  • Identity preservation: FairMOT assigns correct identities when pedestrians cross, whereas trackers using bounding-box IoUs usually cause identity switches.The qualitative examples are visualized on sampled MOT17 test-set frames in chronological order, with colors marking identities.

6 Summary and Future Work

The paper attributes prior one-shot MOT degradation to anchor ambiguity, feature unfairness, and feature-dimension mismatch. FairMOT addresses these issues with an anchor-free single-shot network that improves accuracy, speed, and training-data efficiency.

  • Summary and Future Work: Multiple nearby anchors can correspond to different object parts while estimating one identity, creating training ambiguities.The paper identifies anchors in object detection and identity embedding as a main reason previous one-shot methods degrade.
  • Summary and Future Work: Detection and re-ID require different features, while their shared representation creates feature conflict and reduces task performance.Re-ID benefits from low-level instance-discriminative information, whereas detection features should be similar across instances.
  • Summary and Future Work: FairMOT addresses anchor, feature-sharing, and feature-dimension issues in an anchor-free single-shot deep network.The approach treats detection and re-ID equally rather than making re-ID secondary to detection.
  • Summary and Future Work: FairMOT outperforms prior state-of-the-art methods on several benchmarks in tracking accuracy and inference speed.The paper also reports inherent training-data efficiency and single-image training using only bounding-box annotations.
Loading 2004.01888v6…