PaperlayerView exact revision on arXiv

Source-linked AI summary

Rethinking the competition between detection and ReID in Multi-Object Tracking

Chao Liang, Zhipeng Zhang, Xue Zhou, Bing Li, Shuyuan Zhu, Weiming Hu

arXiv:2010.12138v3cs.CV

TL;DR

One-shot MOT improves efficiency by jointly learning detection and ReID, but shared representations can suffer from task competition and semantic level misalignment. CSTrack introduces REN and SAAN to separate task-dependent learning while improving cross-task cooperation and multi-scale embedding alignment. It reports state-of-the-art performance across MOT16, MOT17, and MOT20, with a lightweight version reaching 34.6 FPS.

  • Problem

    One-shot MOT overlooks differences between detection and ReID, allowing task competition and semantic level misalignment to limit representation learning and association.

  • Method

    CSTrack combines a reciprocal network using self- and cross-relations with a scale-aware attention network for task-dependent representations and aligned multi-resolution ID embeddings.

  • Results

    CSTrack achieves state-of-the-art performance on MOT16, MOT17, and MOT20; its lightweight version runs at 34.6 FPS with a minor performance drop.

  • Takeaways & Limitations

    The framework improves one-shot MOT performance, especially ReID association, while retaining near-real-time efficiency.

  • Takeaways & Limitations

    CSTrack can fail under image blur when the detector misclassifies objects as background.

Abstract

from arXiv · show

Due to balanced accuracy and speed, one-shot models which jointly learn detection and identification embeddings, have drawn great attention in multi-object tracking (MOT). However, the inherent differences and relations between detection and re-identification (ReID) are unconsciously overlooked because of treating them as two isolated tasks in the one-shot tracking paradigm. This leads to inferior performance compared with existing two-stage methods. In this paper, we first dissect the reasoning process for these two tasks, which reveals that the competition between them inevitably would destroy task-dependent representations learning. To tackle this problem, we propose a novel reciprocal network (REN) with a self-relation and cross-relation design so that to impel each branch to better learn task-dependent representations. The proposed model aims to alleviate the deleterious tasks competition, meanwhile improve the cooperation between detection and ReID. Furthermore, we introduce a scale-aware attention network (SAAN) that prevents semantic level misalignment to improve the association capability of ID embeddings. By integrating the two delicately designed networks into a one-shot online MOT system, we construct a strong MOT tracker, namely CSTrack. Our tracker achieves the state-of-the-art performance on MOT16, MOT17 and MOT20 datasets, without other bells and whistles. Moreover, CSTrack is efficient and runs at 16.4 FPS on a single modern GPU, and its lightweight version even runs at 34.6 FPS. The complete code has been released at https://github.com/JudasDie/SOTS.

I. INTRODUCTION

One-shot MOT jointly performs detection and ReID efficiently, but their conflicting representation needs and resolution preferences create task competition and semantic misalignment. CSTrack addresses these issues with reciprocal task learning and scale-aware attention, improving one-shot association performance toward two-stage methods.

  • Motivation: One-shot MOT jointly predicts detection and ID embeddings, offering efficiency but risking lower accuracy than two-stage tracking-by-detection.Two-stage methods separately detect objects and associate them using ID embeddings, whereas one-shot methods share computation across both tasks.
  • Motivation: Detection seeks similar semantics across pedestrians, whereas ReID must distinguish specific pedestrians, making their shared representation requirements contradictory.The paper identifies this conflict as competition in object representation learning.
  • Proposed approach: REN learns task-dependent representations while preserving collaboration between detection and ReID through self-relation and cross-relation design.The reciprocal network separates task branches and uses cross-task semantic exchange.
  • Proposed approach: SAAN fuses spatially and channel-attended features across resolutions to prevent semantic level misalignment in ID embedding extraction.The resulting consistent representation improves resilience to objects with different scales during matching.
  • Results: Extensive experiments show improved one-shot MOT performance, especially in ReID association, with competitiveness against two-stage methods.The reported benefit centers on the association capability of re-identification features.

II. RELATED WORK

MOT methods evolved from computationally expensive two-stage detection-and-association pipelines toward efficient one-shot models that jointly produce detections and ID embeddings. Related work also uses alternative joint tracking mechanisms and graph-based association.

  • Two-stage methods: Two-stage trackers first detect bounding boxes, then associate them with existing tracklets using appearance embeddings.Additional networks may extract embeddings per bounding box, increasing computation.
  • Two-stage methods: Graph neural networks have been introduced to improve matching by modeling association information and replacing or supplementing traditional assignment procedures.MPN uses edge classification to associate targets.
  • One-shot methods: One-shot trackers simultaneously generate detection boxes and ID embeddings in a unified network to improve efficiency.This paradigm avoids forwarding an ID network separately for every detected bounding box.

C. Joint Detection and Tracking without ReID

CSTrack builds on JDE’s one-shot architecture while targeting its shared-feature competition and scale-dependent ReID misalignment. REN decouples and reconnects task features, while SAAN aggregates attended multi-resolution features for stronger embeddings.

  • A. Overview: JDE processes each frame with a backbone-and-neck feature extractor that produces multi-resolution features for joint prediction.These features feed parallel detection and ReID branches.
  • A. Overview: JDE predicts detection results and raw ID embedding maps, then uses NMS boxes and corresponding resolution-level embeddings for tracklet association.Each candidate box retrieves an embedding from the feature-resolution level assigned by detection.
  • A. Overview: Directly feeding shared features into independent branches can create ambiguous learning because detection and ReID have different task requirements.The ReID branch may also extract embeddings from an arbitrary detection-guided feature level, causing semantic misalignment.
  • A. Overview: REN decouples features into task branches and exchanges their semantic information through cross-relation processing.Its self-relation and cross-relation structure enhances task-dependent representations and collaboration.
  • A. Overview: SAAN combines spatial and channel attention across resolutions to suppress background noise and learn object-related representations.Its fused multi-resolution output supports consistent ID embedding extraction.

B. Reciprocal Network

The reciprocal network separates detection and ReID into task-driven branches while learning both task-specific relations and cross-task commonalities. It enhances each branch through fused relation weights and residual attention.

  • Self-relation learning: Self-relation weights model correlations between channels within each task to strengthen task-specific representations.The weights are computed from task-specific tensors using matrix multiplication and row softmax.
  • Cross-relation learning: Cross-relation weights model semantic commonalities between detection and ReID by comparing tensors from different tasks.The resulting maps are generated with matrix multiplication, transposition, and row softmax.
  • Relation fusion: A trainable parameter λ fuses self-relation and cross-relation weights for each task.The fused weight matrices are used to enhance task representations.
  • Reciprocal Network: REN separates detection and ReID feature maps into different task-driven branches for task-dependent representation learning.The branches receive enhanced representations derived from shared features.
  • Relation fusion: Enhanced representations are reshaped, fused with the original feature map through residual attention, and sent to separate task branches.Residual fusion is used to prevent information loss.

C. Scale-aware Attention Network

SAAN aggregates multi-resolution features and applies spatial and channel attention to align semantic levels for ID embeddings. It produces a 512-channel embedding map for subsequent ReID matching.

  • Attention modules: Spatial attention enhances target-related features and suppresses background noise using pooled channel maps, convolution, and sigmoid activation.Average- and max-pooling produce two 2D maps that are concatenated before a 7 × 7 convolution.
  • Embedding extraction: A 3×3 convolution maps the fused features to 512 channels, producing an ID embedding at each spatial location for extraction from detected boxes.Each embedding denotes identity information used in the subsequent ReID task.

D. Training details

Training jointly optimizes detection and ReID with weighted losses. Detection combines classification and box regression, while ReID uses cross-entropy classification for ID embeddings.

  • Joint optimization: The joint objective is a weighted linear sum of detection loss Ldet and ReID loss Lid.Detection loss consists of classification and box regression losses.
  • Detection loss: Detection loss combines classification loss with CIOU-based bounding-box regression loss.The detection loss is normalized by the number of positive samples across three resolutions and weighted by β = 0.05.
  • ReID loss: The ReID objective models identity learning as classification using a fully connected layer and cross-entropy loss.ID embeddings are mapped to a C-class distribution and compared with one-hot ground-truth labels.
  • Joint optimization: η is set to 0.02 to balance object detection and ReID tasks during joint optimization.The weight is selected experimentally.

E. Online Tracking

Online tracking links detected candidate boxes to existing tracklets using cosine similarity, Kalman-filter constraints, and template updates. Unmatched tracks and candidates are managed through inactivity and initialization rules.

  • Cascade matching: Candidate boxes are matched to existing tracklet templates using cosine similarity between their ID embeddings.A Kalman filter removes unreasonable matches by providing distance restrictions.
  • Track management: Unmatched candidates initialize new tracklets with their ID embeddings as templates, while unmatched tracklets become inactive.A tracklet is terminated after 30 frames without a matched box.
  • Track management: Inactive tracklets return to the active state when successfully matched before termination.Template embeddings are updated using the stated weighted update rule with ε = 0.9.

IV. EXPERIMENTS

The experiments evaluate CSTrack through component analysis, benchmark testing, and qualitative failure-case inspection. Evaluation uses MOT16, MOT17, and MOT20 with MOTA and IDF1 among the reported metrics.

  • The study examines component effectiveness, state-of-the-art comparisons, data association, qualitative tracking results, and failure cases.These analyses are organized across Sections IV-B through IV-D and the qualitative-results discussion.
  • CSTrack is trained with the same six datasets as JDE, using detection-only supervision where identity annotations are unavailable.ETH and CityPerson train only the detection branch, while MOT17, CalTech, CUDK-SYSU, and PRW support both tasks.
  • Evaluation covers MOT16, MOT17, and crowded-scene MOT20 using CLEAR metrics, particularly MOTA and IDF1.MOT16 and MOT17 share seven testing videos with different annotations, while MOT20 contains four extremely crowded-scene videos.
  • Experiments run on a single RTX 2080Ti GPU and Xeon Gold 5218 2.30GHz CPU.The implementation uses Python 3.7 and PyTorch 1.6.0.
  • The component-wise analysis is reported in Table I on the MOT16 testing set.

B. Ablation Studies 2

Ablation studies test REN, SAAN, training-data choices, and ID-loss weighting. The results support learned reciprocal relations and scale-aware feature aggregation while exposing trade-offs in task balancing and dataset composition.

  • Some ablations use validation sets because the MOT Challenge permits only four submissions per model on each benchmark.
  • REN Ablation: 66.0% MOTA and 70.7% IDF1 result when learned weights fuse REN self-relation and cross-relation features.The learned weights converge to λ1=0.12122 and λ2=0.31519, while ID Sw. decreases by about 25%.
  • SAAN Ablation: SAAN improves MOTA and IDF1 over the basic ReID head by aggregating information across resolutions with different attention arrangements.The ablation is conducted on the MOT17 validation set and evaluates serial arrangements of the attention modules.
  • ID Loss Weight: A large ID-loss weight degrades detection performance, while a small weight degrades matching performance.The ablation varies η to balance detection and ID-embedding learning.

C. Comparisons with State-of-the-art Trackers

CSTrack is compared with two-stage, one-shot, and other joint detection-and-tracking methods on MOT benchmarks, plus state-of-the-art trackers on HiEve. The reported results emphasize tracking accuracy, association quality, and speed.

  • The comparison includes two-stage methods, one-shot methods, and joint detection-and-tracking methods without ReID association.Evaluations on MOT16, MOT17, and MOT20 use the official online server.
  • HiEve: CSTrack is also evaluated against state-of-the-art trackers on the HiEve benchmark with private detection.
  • Two-stage Methods: CSTrack outperforms POI by +9.5% on MOTA and +8.2% on IDF1 while running at 16.4 FPS versus 0.5∼8.6 FPS on MOT16.CSTrack-S reaches 34.6 FPS with a small performance drop: MOTA -1.8∼2.4% and IDF1 -2.3∼3.6%.
  • One-shot Trackers: On MOT16, CSTrack achieves 75.6 MOTA and 73.3 IDF1, exceeding JDE by +11.2% MOTA and +17.5% IDF1.On MOT20, false positives decrease from 103440 to 25404 and ID switches from 5243 to 3196 versus FairMOTv2.
  • Other Joint Detection and Tracking Methods: Against other joint detection-and-tracking methods, CSTrack improves IDF1 by 11.1%∼18.4% on MOT16 and 7.6%∼17.2% on MOT17.These methods use non-ReID approaches for data association.

D. Further Analysis

Further analyses show CSTrack improves association and handles scale variation, while image blur and over-occlusion remain failure cases for its detector.

  • HiEve Challenge: CSTrack outperforms baseline JDE by +16.5% on MOTA and +15.4% on IDF1 on the HiEve benchmark.HiEve contains 19 training videos and 13 testing videos focused on human-centric complex events.
  • Upper-bound Analysis: Replacing detector outputs with ground-truth boxes evaluates CSTrack’s upper-bound data-association ability while controlling detector influence.All compared methods are trained on the same MOT17 training set.
  • Discriminative Ability of ID Embeddings: CSTrack produces more discriminative ID embeddings than JDE across current frames and previous-tracklet templates.The comparison uses cosine metric matrices for within-frame, within-template, and cross-frame-template embeddings.
  • Discriminative Ability of ID Embeddings: CSTrack’s association advantage over JDE is most pronounced under occlusion and scale changes, consistent with SAAN’s scale-aware design.SAAN is reported to improve resilience to objects with different scales.
  • Qualitative Results and Failure Cases: Qualitative MOT17 and MOT20 results show that CSTrack handles large-scale variations while maintaining correct identities.The reported examples attribute this behavior to SAAN’s use of multi-resolution information for discriminative embeddings.
  • Qualitative Results and Failure Cases: Image blur can cause detector misclassification as background, while over-occlusion causes missed targets and breaks tracklet temporal consistency.Both cases are linked to reliance on single-frame image-based detection; future work proposes using temporal information.

V. CONCLUSION

CSTrack is a one-shot online MOT model combining REN and SAAN to reduce detection–ReID competition and improve collaboration. It achieves strong benchmark performance while retaining near-real-time efficiency, although blur and over-occlusion remain documented failure cases.

  • V. CONCLUSION: CSTrack combines reciprocal and scale-aware attention networks to mitigate competition and improve collaboration between detection and ReID.REN and SAAN are integrated into a one-shot online MOT framework.
  • V. CONCLUSION: +11.2% on MOTA and +17.5% on IDF1 are reported for CSTrack over baseline JDE on public benchmarks.The paper describes this as state-of-the-art performance on the evaluated public benchmarks.
  • V. CONCLUSION: 34.6 FPS is achieved by CSTrack’s lightweight version with only a minor performance drop, supporting real-application use.The full model is described as nearly real-time, while the lightweight version prioritizes higher speed.
  • V. CONCLUSION: The qualitative evaluation includes MOT17 and MOT20 testing sequences, with different colors denoting identities and lines denoting target tracklets.Frame numbers appear in the upper-left of the qualitative-result figures.
  • V. CONCLUSION: Failure-case visualization identifies image blur and over-occlusion as conditions producing false negatives.The failure-case figure marks false negatives with red dotted boxes.
Loading 2010.12138v3…