PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Object Tracking with Multiple Cues and Switcher-Aware Classification

Weitao Feng, Zhihao Hu, Wei Wu, Junjie Yan, Wanli Ouyang

arXiv:1901.06129v1cs.CV

TL;DR

MOT remains challenging when detections are inaccurate and targets are occluded or move abnormally. This paper combines SOT and ReID cues with switcher-aware classification for association. Its conclusion reports strong performance on MOT benchmarks, while a multi-task SOT/ReID network reduced both MOTA and IDF1 in the reported experiment.

  • Problem

    Tracking-by-detection is limited by inaccurate detections, while occlusion and abnormal motion also challenge MOT; prior methods did not fully combine short- and long-term cues or consistently use local interaction information.

    Source: §1 · pp. 1–2, §2.2 · p. 3

  • Method

    The framework combines SOT short-term cues and ReID long-term cues in data association, with a switcher-aware classifier that uses potential switcher information.

    Source: §1 · p. 2, §2.2 · p. 3

  • Results

    The authors report outstanding performance on MOT benchmarks and find that short- and long-term cues improve different metrics in ablation experiments.

    Source: §4.3 · p. 8, §5 · p. 9

  • Takeaways & Limitations

    Within the reported framework, long-term cues can help correct short-term cue mistakes such as SOT drift during occlusion.

    Source: §5 · p. 9

    In the reported experiment, multi-task training of SOT and ReID reduced MOTA by 0.6% and IDF1 by 2.5% when replacing Siamese-RPN in the MOT task.

    Source: §4.3 · p. 8

Abstract

from arXiv · show

In this paper, we propose a unified Multi-Object Tracking (MOT) framework learning to make full use of long term and short term cues for handling complex cases in MOT scenes. Besides, for better association, we propose switcher-aware classification (SAC), which takes the potential identity-switch causer (switcher) into consideration. Specifically, the proposed framework includes a Single Object Tracking (SOT) sub-net to capture short term cues, a re-identification (ReID) sub-net to extract long term cues and a switcher-aware classifier to make matching decisions using extracted features from the main target and the switcher. Short term cues help to find false negatives, while long term cues avoid critical mistakes when occlusion happens, and the SAC learns to combine multiple cues in an effective way and improves robustness. The method is evaluated on the challenging MOT benchmarks and achieves the state-of-the-art results.

1. Introduction

MOT tracking-by-detection is challenged by inaccurate detections, occlusion, and abnormal motion. The framework combines short- and long-term cues and uses switcher-aware classification to support data-driven association; experiments on MOT16 and MOT17 show its effectiveness.

1. Introduction

  • Short-term SOT cues can reduce false negatives, while stable tracklet appearance helps counter SOT drift during occlusion.The paper notes that occluded regions can make updated appearance unreliable for SOT.Source: Figure 1 · p. 1, §1 · p. 2
  • Prior approaches often omitted one cue, failed to combine SOT with association, or relied on rule-based combinations that may overfit specific cases.The authors report difficulty combining all cues in one network because SOT cannot distinguish similar objects and long-term features do not predict precise position.Source: §1 · p. 2
  • The switcher-aware classifier considers the potential identity-switch causer to improve tracking robustness.The paper identifies local interaction information as relevant to correct matching.Source: §1 · p. 2
  • The framework combines SOT short-term cues and ReID long-term cues with switcher-aware classification to make adaptive data-association decisions.The classifier uses target and potential switcher features to generate matching scores.Source: §1 · p. 2
  • Experiments on MOT16 and MOT17 benchmarks show the proposed framework's effectiveness.Source: §1 · p. 2

2. Related Work

Prior MOT work uses SOT trackers, feature-based association, and optimization methods, but the paper identifies gaps in cue integration and local interaction modeling. Its approach combines SOT and long-term cues and includes potential switchers in association.

2.1. MOT Using SOT Tracker

  • Prior SOT-for-MOT approaches often separated tracking from data association, whereas this work combines SOT information with long-term trajectory cues for learned association.The paper also notes that online-updating SOT can be slow and memory-intensive, while offline Siamese-RPN trackers can operate at more than 80 frames (targets) per second.Source: §2.1 · p. 2

2.2. Data Association

  • Earlier data-association methods used optimization formulations, feature-learning approaches, or heuristic cue ensembles, but often omitted SOT for false negatives and local switcher information.The authors argue that pairwise matching can lose local interaction information relevant to tracking robustness.Source: §2.2 · pp. 2–3
  • The proposed SAC uses the potential switcher in training and inference, while combining short- and long-term cues in a balanced, data-driven way.Source: §2.2 · p. 3, Figure 2 · p. 3

3. The Proposed Framework

The framework represents a tracked target as a trajectory across frames, with each frame’s state describing bounding-box position and size and target appearance.

3. The Proposed Framework

  • A target trajectory is represented as X = {X_t}, with X_t denoting its frame-specific state.Source: §3 · p. 3
  • The stated trajectory variables include the bounding box’s top-left coordinate, width, and height, alongside target appearance at frame t.The passage also identifies q_X as the target’s overall tracking quality.Source: §3 · p. 3

3.1. Overall Design

The online framework uses SOT to predict target locations and ReID to extract long-term features, then scores detections with SAC using target and switcher information. Matching updates tracks, while unmatched detections may start new tracks.

3.1. Overall Design

  • For each target, SOT predicts its next-frame location, while ReID extracts long-term features from the detection region and historical target appearances.Source: §3.1 · p. 3
  • The framework finds a potential switcher and uses its SOT and ReID features with target matching features to score each detection through SAC.Source: §3.1 · p. 3
  • A bipartite graph of targets and detections is matched using minimum-cost network flow; matched targets update from detections, while unmatched targets use SOT positions.Unreliable or lost targets are dropped, and qualifying isolated detections can be added as new targets.Source: §3.1 · p. 4

3.2. Using SOT Tracker for Short Term Cues

The framework uses Siamese-RPN to track targets between neighbouring frames and derive short-term matching cues. It also refines tracking quality using detection matches to identify unreliable tracks.

3.2. Using SOT Tracker for Short Term Cues

  • Siamese-RPN compares a current-frame exemplar with a next-frame search region, producing score maps and bounding-box predictions for short-term tracking.The exemplar is resized to 127 × 127 and the search region to 255 × 255; the network uses shared weights and separate scoring and box-regression branches.Source: §3.2 · p. 4
  • The SOT sub-net outputs a tracking score and predicted box, while detections provide boxes for matching.The short-term features are calculated from these quantities, though their formula is not included in the supplied passages.Source: §3.2 · p. 4
  • A refined tracking-quality score incorporates detection matches or applies decay and a score penalty otherwise, allowing tracks below threshold ζt to be dropped.The tracker is adapted to pedestrian scales and refined using matching results to address drift to background distractors.Source: §3.2 · p. 4

3.3. Using ReID Network for Long Term Cues

The ReID sub-net extracts long-term appearance features using a modified Inception-v4 backbone. A quality filter selects history images from successive time periods before their features are used for matching.

3.3. Using ReID Network for Long Term Cues

  • A modified GoogLeNet Inception-v4 backbone supplies the ReID feature from its final fully connected layer before classification.Tables 1 and 2 give architecture details, including one Inception-A block.Source: §3.3 · p. 4, Table 2 · p. 5
  • A quality-aware filter selects the best K images from the target's history, choosing one high-quality frame within each of K past time periods.A ResNet-18 network produces quality scores; the interval parameter δ defines the frame-selection ranges.Source: §3.3 · pp. 4–5
  • The selected history images and candidate detection are passed through ReID to obtain long-term features, with each tracklet image's features extracted only once and cached.Source: §3.3 · p. 5

3.4. Switcher-Aware Classifier

The switcher-aware classifier uses the target most likely to cause an identity switch as additional association context. It combines features for the main target and switcher to score matching edges.

3.4. Switcher-Aware Classifier

  • For each tracklet, the potential switcher is the other tracked target with the largest positional overlap, motivated by identity switches occurring when targets meet with large overlap.Source: §3.4 · p. 5
  • Classifier inputs concatenate features for the main target and the switcher, with each feature vector having dimension K + 1.Source: §3.4 · p. 5
  • A regularized Newton boosting decision tree classifies matches; scores above threshold ζm add graph edges with cost 1 − y.Source: §3.4 · p. 5

3.5. Training

Training separately fits the SOT and ReID sub-nets, then generates labeled classifier examples from baseline tracklets and detections. The losses include SOT classification and regression, ReID classification, and binary classification for the boosting tree and quality filter.

3.5.1 Training data generation

  • SOT and ReID sub-nets are trained independently, using ground-truth target pairs with IoU above 0.65 as SOT positives and below 0.45 as negatives.ReID treats each target as a class and predicts the class of each input target image region.Source: §3.5.1 · p. 5
  • Classifier examples come from baseline-generated tracklets associated to ground truth by maximum-sum IoU assignment, with detections and potential switchers labeled from ground truth.Valid tracklets allow at most one unassociated ground-truth match among the sampled frames; swapping the switcher and main target creates an additional negative example for a positive sample. The quality filter uses IoU above 0.6 as positive and the rest as negative.Source: §3.5.1 · pp. 5–6

3.5.2 Loss functions.

  • The SOT loss combines cross-entropy classification and regression losses, while ReID optimizes cross-entropy for target-class prediction.The boosting-tree classifier and quality filter use a binary cross-entropy loss.Source: §3.5.2 · p. 6

3.6. Pre-Processing and Offline Clustering

The method refines noisy detections and uses a batch clustering procedure to merge tracklet fragments and repair false negatives.

3.6. Pre-Processing and Offline Clustering

  • A stricter NMS method or a confidence-score refiner addresses noisy detections, false positives, and unreliable confidences.The quality filter used in long-term cue selection serves as the confidence refiner.Source: §3.6 · p. 6
  • Tracklet clustering links frames with similar ReID features, then forms slices as connected subgraphs.An edge is added when the cosine distance between two nodes' ReID features falls below a threshold.Source: §3.6 · p. 6
  • Slices are merged across targets using frame overlap, spatial distance, and mean ReID-feature distance, followed by interpolation to repair false negatives.Source: §3.6 · p. 6

4. Experiments

The experiments evaluate online and batch tracking on MOT16 and MOT17 and report strong benchmark performance. Ablations examine the contributions of short- and long-term cues, SAC, feature dimensionality, and classifier choice.

4.1. Implement Details

  • Implementation used ImageNet-pretrained CNNs, private pedestrian video for training the two subnetworks, and public data for the classifier and quality filter.The classifier used XGBoost; long-term cue selection sampled K = 3 frames with δ = 15.Source: §4.1 · p. 6

4.2. Evaluation on MOT Benchmarks

  • The benchmarks use MOT16 and MOT17 test videos with different detection inputs; MOT17 has more accurate ground truth.The evaluation follows CLEAR MOT metrics, with MOTA and IDF1 treated as most important.Source: §4.2 · pp. 6–7
  • The framework outperforms many state-of-the-art trackers on MOT16 and MOT17, with MOTA and IDF1 leading among online and batch algorithms; batch mode achieves the highest MOT17 MOTA.Source: §4.2 · p. 7, Table 3 · p. 7, Table 4 · p. 7

4.3. Ablation Study and Discussion

  • Ablations on MOT16 train data found short-term cues improved MOTA by 1.3% and IDF1 by 0.3%, while long-term cues added 1.4% IDF1.Combining both cues outperformed using either alone; adding SAC increased IDF1 by another 1.1% and had little effect on MOTA.Source: §4.3 · pp. 7–8, Figure 4 · p. 8
  • In MOT16 private, SAC reduced identity switches from 642 to 569 and increased IDF1 by 1.1%, helping distinguish targets during occlusion.The authors attribute matching errors to pairwise matching that lacks comparison with a local switcher.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Jointly training SOT and ReID in one network decreased MOTA by 0.6% and IDF1 by 2.5%; the authors identify conflicting feature-learning needs.They state that handling both cues in one network remains difficult and needs further research.Source: §4.3 · p. 8
  • Concatenating raw ReID features with short position features decreased IDF1 by 1.3% and MOTA by 0.1%, while increasing identity switches by 88.The authors use lower-dimensional appearance features to balance the input and emphasize position and motion information; shorter features also make association faster.Source: §4.3 · p. 8
  • Among NN, SVM, and BDT, the reported MOTA scores were 67.0%, 67.6%, and 67.8%, respectively.The authors report that neural networks perform poorly with such a small input feature.Source: §4.3 · p. 8

5. Conclusions

The framework learns to combine long- and short-term cues and adds switcher-aware classification; its reported MOT benchmark performance is outstanding.

5. Conclusions

  • The proposed framework learns to combine long-term and short-term cues for multi-object tracking.Source: §5 · p. 9
  • Long-term cues can correct short-term cue mistakes, including SOT-subnet drift during occlusion.Source: §5 · p. 9
  • Switcher-aware classification is proposed to improve tracking robustness, and the framework reports outstanding performance on MOT benchmarks.Source: §5 · p. 9
Loading 1901.06129v1…