PaperlayerView exact revision on arXiv

Source-linked AI summary

RelationTrack: Relation-aware Multiple Object Tracking with Decoupled Representation

En Yu, Zhuoling Li, Shoudong Han, Hongwei Wang

arXiv:2105.04322v1cs.CV

TL;DR

MOT systems must jointly support detection and ReID despite their different feature requirements, while prior methods often overlook global semantic relations. RelationTrack addresses these issues with GCD and GTE, and its experiments on MOT16, MOT17, and MOT20 report superior performance, including 70.5% IDF1 and 67.2% MOTA on MOT20.

  • Problem

    Detection and ReID require different features, creating an optimization contradiction, while prior MOT methods often use local information and neglect global semantic relations.

  • Method

    RelationTrack uses GCD to decouple representations into detection-specific and ReID-specific embeddings, and GTE to combine deformable attention with a Transformer encoder for global relations.

  • Results

    70.5% IDF1 and 67.2% MOTA are reported on MOT20, while RelationTrack surpasses preceding methods across experiments on MOT16, MOT17, and MOT20.

  • Takeaways & Limitations

    RelationTrack establishes new state-of-the-art performance while using decoupled task representations and efficient global-relation modeling.

Abstract

from arXiv · show

Existing online multiple object tracking (MOT) algorithms often consist of two subtasks, detection and re-identification (ReID). In order to enhance the inference speed and reduce the complexity, current methods commonly integrate these double subtasks into a unified framework. Nevertheless, detection and ReID demand diverse features. This issue would result in an optimization contradiction during the training procedure. With the target of alleviating this contradiction, we devise a module named Global Context Disentangling (GCD) that decouples the learned representation into detection-specific and ReID-specific embeddings. As such, this module provides an implicit manner to balance the different requirements of these two subtasks. Moreover, we observe that preceding MOT methods typically leverage local information to associate the detected targets and neglect to consider the global semantic relation. To resolve this restriction, we develop a module, referred to as Guided Transformer Encoder (GTE), by combining the powerful reasoning ability of Transformer encoder and deformable attention. Unlike previous works, GTE avoids analyzing all the pixels and only attends to capture the relation between query nodes and a few self-adaptively selected key samples. Therefore, it is computationally efficient. Extensive experiments have been conducted on the MOT16, MOT17 and MOT20 benchmarks to demonstrate the superiority of the proposed MOT framework, namely RelationTrack. The experimental results indicate that RelationTrack has surpassed preceding methods significantly and established a new state-of-the-art performance, e.g., IDF1 of 70.5% and MOTA of 67.2% on MOT20.

I. INTRODUCTION

RelationTrack addresses the conflicting feature requirements of detection and ReID with GCD, while GTE captures global semantic relations efficiently. Across MOT16, MOT17, and MOT20, the framework outperforms preceding methods.

  • Motivation: Jointly integrating detection and ReID reduces inference cost but creates an optimization contradiction because their feature objectives differ.Detection favors similarity within categories, whereas ReID favors distinction among targets of the same category.
  • Guided Transformer Encoder: GTE combines deformable attention with a Transformer encoder to model long-range, global pixel-to-pixel relations across the image.It selects a small set of key samples rather than analyzing every pixel, reducing complexity from O(n^2) to O(n).
  • Experiments: RelationTrack is evaluated on MOT16, MOT17, and MOT20, where it significantly outperforms preceding counterparts.On IDF1, it surpasses FairMOT by 3.0% on MOT16 and 2.4% on MOT17.
  • Global Context Disentangling: GCD decouples learned features into detection-specific and ReID-specific embeddings to alleviate this contradiction.The module provides separate representations for the two subtasks during training.
  • Experiments: 70.5% IDF1 and 67.2% MOTA are achieved by RelationTrack on MOT20, establishing new state-of-the-art performance.These results are reported across five groups of experiments on three benchmark datasets.

II. RELATED WORKS

Related MOT work includes tracking-by-detection, joint-detection-and-prediction, and joint-detection-and-embedding paradigms. Unified approaches improve over separate models but still demand further improvement in sophisticated scenarios.

  • Tracking-by-detection: Tracking-by-detection methods commonly estimate target locations and associate them with trajectories, often using kinematics or appearance features.Kalman filtering and Hungarian assignment represent kinematic approaches, while DeepSort uses ReID features.
  • Tracking-by-detection: Separate detection and association sub-models can suffer poor final precision when either component fails, while their unshared layers increase inference cost.These restrictions limit further development of the tracking-by-detection paradigm.
  • Joint-detection-and-prediction: Joint-detection-and-prediction methods combine localization and trajectory prediction in one network, with examples including Tracktor, CTracker, and MAT.These methods use bounding-box regression, adjacent-frame information, or kinematic cues to estimate target positions.
  • Joint-detection-and-prediction: Siamese and center-based methods explore feature similarity or bounding-box offsets for trajectory estimation, including DeepMOT and CenterTrack.DeepMOT decomposes MOT into single-object tracking tasks, while CenterTrack follows CenterNet-inspired offset prediction.
  • Joint-detection-and-prediction: Joint-detection-and-prediction models generally outperform tracking-by-detection mainly through their trajectory-prediction blocks, but sophisticated scenarios still require improvement.The cited comparison identifies stronger behavior without removing the need for further advances.

C. Joint-detection-and-embedding

Joint-detection-and-embedding models combine detection and identification in one-stage networks and associate targets using embeddings. RelationTrack targets their remaining optimization and global-relation limitations.

  • Joint-detection-and-embedding: Joint-detection-and-embedding models detect targets and identify them within a one-stage network, associating targets through extracted embeddings rather than motion offsets.JDE and FairMOT are cited examples.
  • Limitations: Their detection and identification branches can support competitive tracking precision, but conflicting objectives hinder optimization.Detection and identification require different representations for localization and identity discrimination.
  • Limitations: Previous frameworks primarily use local information and neglect global semantic relations among targets and background regions.The paper identifies this as a second obstacle in the joint-detection-and-embedding paradigm.
  • Problem Formulation: RelationTrack consists of a detector, feature extractor, and associator that produce boxes, embeddings, and trajectories from video frames.The associator links detected boxes across frames using the extracted embeddings.
  • Problem Formulation: The formulation requires accurate bounding boxes and identity-consistent embeddings, after which even a simple Hungarian-algorithm associator can produce promising tracking results.Same-identity embeddings should be more alike than embeddings from different identities.
  • Framework: RelationTrack processes video through feature extraction, feature decoupling, detection, representation extraction, and association.Its backbone is DLA-34, and the association stage uses the Hungarian algorithm.

C. Global Context Disentangling (GCD)

GCD decouples backbone features by first producing a global context vector and then transforming it into task-specific representations. The shared context enables efficient feature computation.

  • GCD Design: GCD has two phases: producing a global context vector and using it to decompose input feature maps.The resulting representations are detection-specific and ReID-specific.
  • Global Context Vector: The global context vector aggregates input feature-map information and is invariant to the selected spatial index during aggregation.The input feature maps contain Np = H′×W′ spatial positions.
  • Feature Decoupling: Two transformations convert the global context vector into task-specific vectors, which are broadcast-added to the input features to obtain detection and ReID embeddings.The transformations use learnable matrices, ReLU, and layer normalization.
  • Efficiency: GCD has O(C^2) calculation complexity because all detection and ReID elements reuse the same global context vector.The paper contrasts this with O(HWC^2) complexity for earlier global-attention methods.
  • Effectiveness: Experiments report that GCD successfully decouples learned features and addresses the feature contradiction problem.The efficiency claim and the experimental validation are presented together in the GCD analysis.

D. Guided Transformer Encoder (GTE)

GTE combines Transformer encoding with deformable attention to model long-range pixel relations while selecting only a small set of informative key samples. This design preserves global receptive-field reasoning while reducing attention complexity.

  • Guided Transformer Encoder (GTE): GTE combines a Transformer encoder and deformable attention to explore long-range pixel-to-pixel relations with a global receptive field.The Transformer contributes reasoning over dependencies, while deformable attention supplies self-adaptive global sampling.
  • Transformer encoder: Standard Transformer encoding forms relation maps through query-key dot products, normalizes them, aggregates key information, and applies an FFN.In GTE, this encoder-like structure produces embeddings for subsequent association operations.
  • Deformable attention: Deformable attention selects valuable key samples across the whole image for each query node instead of relating every query to all pixels.The selected samples interact with query nodes to produce discriminative representations.
  • Deformable attention: Deformable attention has complexity O(HWC), compared with O(H2W 2C) for global attention.The reduction avoids computing similarity between query nodes and all feature-map values.

E. Detection and Association

RelationTrack detects objects from decoupled detection-specific features and associates detections across frames using GTE embeddings, bounding boxes, and Hungarian matching.

  • Detection: The detection module localizes objects, regresses center offsets, and estimates bounding-box shapes from detection-specific representations.Combining these outputs determines the regions containing targets.
  • Association: The association stage uses GTE embeddings and estimated bounding boxes to match objects across frames with the Hungarian algorithm.Matching is based on cosine distances between embedding vectors; unmatched detections initialize new trajectories.

F. Optimization objectives

RelationTrack trains its detection and ReID components with separate task-specific objectives and combines them into an overall weighted optimization objective.

  • Detection branch: The detection branch optimizes target center-point estimation, bounding-box shape prediction, and center-offset prediction.Center heatmaps use Gaussian ground truth, while box shapes and offsets are optimized with an l1 measurement.
  • ReID branch: The ReID branch treats identity recognition as classification and transforms bounding-box features into a class-distribution vector.Targets with identical identities are assigned to the same category, with one-hot annotations used for supervision.
  • Overall optimization objective: The overall RelationTrack objective combines the component loss functions using learnable coefficients ω1 and ω2.This objective jointly trains the framework’s various parts.

IV. EXPERIMENT

The experiments use multiple MOT benchmarks and evaluate RelationTrack against existing methods and through component ablations. Additional experiments examine the effectiveness of GCD and GTE and the framework’s robustness.

  • IV. EXPERIMENT: MOT15, MOT16, and MOT17 are used to validate the models in the experiments.The section also reports comparisons with existing state-of-the-art methods, ablations, module-specific evaluations, and robustness analysis.
  • IV. EXPERIMENT: The experiments compare RelationTrack with existing state-of-the-art MOT methods and assess its components through ablation experiments.Separate subsections evaluate GCD, GTE, and the robustness of RelationTrack.

A. Datasets and evaluation metrics

RelationTrack is evaluated on MOT16, MOT17, and MOT20, with additional datasets used for training and CLEAR-MOT metrics used for verification.

  • Datasets: MOT16 contains 14 sequences spanning varied scenarios, viewpoints, camera poses, and weather conditions, with seven sequences for training and seven for validation.
  • Datasets: MOT17 reconstructs MOT16 with more reliable ground truth and additional detection bounding boxes from DPM, SDP, and Faster R-CNN detectors.
  • Datasets: MOT20 contains eight sequences from three crowded scenes, with some frames including more than 220 pedestrians across indoor, outdoor, day, and night settings.
  • Datasets: ETH, CityPerson, CalTech, CUHK-SYSU, PRW, and CrowdHuman are adopted for training, while performance verification and analysis focus mainly on MOT16, MOT17, and MOT20.
  • Evaluation metrics: RelationTrack is evaluated using CLEAR-MOT metrics including IDF1, HOTA, MOTA, MOTP, MT, ML, FP, FN, IDS, and inference speed.

C. Comparison with preceding SOTAs

RelationTrack outperforms preceding state-of-the-art trackers across MOT16, MOT17, and MOT20, while ablations attribute gains to GCD, GTE, and their combination.

  • Overall comparison: RelationTrack ranks among the top across metrics on MOT16, MOT17, and MOT20, with especially large margins on IDF1, HOTA, MOTA, and IDS.
  • MOT16/17: 75.8% IDF1 on MOT16 and 74.7% on MOT17 exceed FairMOTv2 by 3.0% and 2.4%, respectively.
  • MOT20: 70.5% IDF1, 56.5% HOTA, and 67.2% MOTA on MOT20 surpass FairMOTv2 by 3.2%, 1.9%, and 5.4%, respectively.
  • Ablation study: GCD improves the baseline by 1.6% on IDF1 and 0.9% on MOTA, supporting feature decoupling to alleviate the detection–ReID optimization contradiction.
  • Ablation study: GTE improves the pure baseline by 0.3% on IDF1 and 1.0% on MOTA, while combining GCD and GTE yields gains of 2.0% and 1.6%, respectively.

E. Visualization of GCD

Visualization shows that GCD separates detection-focused and pedestrian-focused information from the original representation. Additional experiments examine key-sample choices and RelationTrack’s behavior in difficult tracking conditions.

  • Visualization of GCD: Decoupled features highlight target centers for detection and pedestrian regions for ReID, unlike original maps that miss small targets and attend to irrelevant areas.These visual patterns demonstrate the intended separation of detection-specific and ReID-specific embeddings.
  • Key-sample analysis: 75.3% IDF1 is obtained with 9 key samples, while increasing samples to 12 or 15 lowers IDF1 to 75.0% and 74.9%.MOTA changes little beyond 9 samples, whereas additional samples increase computing-resource demand.
  • Robustness analysis: RelationTrack successfully identifies a partly occluded target that FairMOT and CSTrack misidentify or overlook in the illustrated frames.The example compares performance in Frame #514 and Frame #517.
  • Robustness analysis: Tracking examples span indoor and outdoor scenes, day and night, and huge and small targets, with the authors reporting robust and precise estimates under these conditions.The examples are presented on the MOT17 dataset.
Loading 2105.04322v1…