PaperlayerView exact revision on arXiv

Source-linked AI summary

SiamMOT: Siamese Multi-Object Tracking

Bing Shuai, Andrew Berneshawi, Xinyu Li, Davide Modolo, Joseph Tighe

arXiv:2105.11595v1cs.CV

TL;DR

Online MOT requires accurate temporal association while avoiding the expensive computation and offline graph solving used by earlier approaches. SiamMOT combines region-based detection with Siamese motion modelling, and achieves top results across multiple tracking benchmarks, including outperforming previous methods on MOT17 and TAO-person.

  • Problem

    Online MOT needs to detect instances and associate them into trajectories without the expensive computation and offline graph solving of earlier approaches.

  • Method

    SiamMOT combines Faster-RCNN region-based detection with Siamese trackers that model instance-level motion using implicit or explicit motion models.

  • Results

    SiamMOT achieves top results across three multi-person tracking datasets, including 65.9 MOTA / 63.3 IDF1 on MOT17 and 41.1 TrackAP on TAO-person.

  • Takeaways & Limitations

    The results demonstrate that tracker capability and motion modelling are particularly important to SiamMOT's multi-object tracking performance.

  • Takeaways & Limitations

    For direct frame-to-frame comparisons, the evaluation immediately kills a trajectory when it is unseen for one frame.

Abstract

from arXiv · show

In this paper, we focus on improving online multi-object tracking (MOT). In particular, we introduce a region-based Siamese Multi-Object Tracking network, which we name SiamMOT. SiamMOT includes a motion model that estimates the instance's movement between two frames such that detected instances are associated. To explore how the motion modelling affects its tracking capability, we present two variants of Siamese tracker, one that implicitly models motion and one that models it explicitly. We carry out extensive quantitative experiments on three different MOT datasets: MOT17, TAO-person and Caltech Roadside Pedestrians, showing the importance of motion modelling for MOT and the ability of SiamMOT to substantially outperform the state-of-the-art. Finally, SiamMOT also outperforms the winners of ACM MM'20 HiEve Grand Challenge on HiEve dataset. Moreover, SiamMOT is efficient, and it runs at 17 FPS for 720P videos on a single modern GPU. Codes are available in \url{https://github.com/amazon-research/siam-mot}.

1. Introduction

Online MOT improves local association for real-time tracking, and SiamMOT investigates how motion modelling affects this capability. Its region-based Siamese design achieves strong results across multiple benchmarks, including outperforming prior methods and a challenge winner.

  • Motivation: Online MOT targets efficient local linking between consecutive frames instead of expensive offline graph optimization across long temporal gaps.Earlier tracking-by-detection methods often required costly visual and motion cues and large offline graphs.
  • Approach: SiamMOT combines region-based detection with implicit and explicit Siamese motion models to estimate instance movement for association.The explicit model uses template matching and region-based features, targeting challenging fast-motion scenarios.
  • Findings: Instance-level motion modelling is especially important when cameras move quickly or people undergo substantial pose deformation.Ablation results across three multi-person tracking datasets support this conclusion.
  • Results: 65.9 MOTA / 63.3 IDF1: SiamMOT with EMM and DLA-34 outperforms previous methods on MOT17 using public detection.This result is reported on the MOT17 Challenge.
  • Results: 41.1 TrackAP versus 36.7: SiamMOT substantially improves over Tracktor++ on the TAO-person dataset.The comparison is reported as a state-of-the-art improvement on the large-scale TAO-person benchmark.
  • Results: SiamMOT outperforms the winner of the ACM MM’20 HiEve Grand Challenge on the HiEve dataset.The comparison is reported as an additional benchmark result beyond MOT17 and TAO-person.

2. Related work

Related work frames SiamMOT as an online MOT system built from Siamese tracking ideas in single-object tracking. It extends tracking-by-detection and SORT-style local association by integrating motion modelling into a unified network.

  • Siamese tracking: Siamese single-object trackers match a target from one frame within a search region in another frame, with the matching function learned offline.This differs from general Siamese networks, which typically learn affinity between two detected instances.
  • SiamMOT: SiamMOT formulates Siamese tracking inside an end-to-end trainable multi-object tracking network rather than relying on an off-the-shelf single-object tracker.DeepMOT also trains Siamese trackers with other components but focuses on structured loss instead of a unified detector-tracker formulation.
  • Tracking-by-detection: Tracking-by-detection first detects objects in each frame and then associates them across time using visual coherence and spatiotemporal consistency.Some approaches add separate computationally expensive components such as optical flow and re-identification.
  • Online MOT: Online MOT performs association without future frames, emphasizing accurate local links rather than globally optimal links across long temporal gaps.SORT is a fast online association method used by later deep MOT models, and SiamMOT is based on SORT.
  • Motion modelling: SORT evolved from geometric track features toward models combining visual and geometric features to learn instance motion.SiamMOT builds on this progression by exploring region-based Siamese motion modelling.

3. SiamMOT: Siamese Multi-Object Tracking

SiamMOT combines Faster-RCNN detection with a region-based Siamese tracker that propagates and associates instances across frames. It supports implicit and explicit motion models, parallel per-instance inference, and spatial matching for online tracking.

  • 3.1. Motion modelling with Siamese tracker: SiamMOT combines Faster-RCNN detection with a region-based Siamese tracker to propagate each detected instance and associate it with detections in a later frame.The tracker processes pairs of frames and uses region features to search for each instance in a contextual window.
  • 3.1. Motion modelling with Siamese tracker: The tracker applies the same motion-and-association process to every detected instance, with operations running in parallel after computing backbone features once.This design supports efficient online tracking inference.
  • 3.2. Implicit motion model: SiamMOT provides implicit and explicit motion models to estimate instance motion between frames.The implicit model uses an MLP over paired region features, while the explicit model uses pixel-level matching supervision.
  • 3.3. Explicit motion model: The explicit motion model uses channel-wise cross-correlation to form a pixel-level response map, then predicts dense visibility and bounding-box offset maps.The predicted offsets recover candidate regions from each response-map location.
  • 3.3. Explicit motion model: Explicit motion modelling improves matching by learning instance correspondence directly and providing finer-grained supervision that reduces false matches to distractors.A penalty map additionally discourages dramatic movements during tracking.
  • Training and inference: Training combines RPN, detection, and motion losses, while inference applies independent NMS followed by IoU-based spatial matching and online trajectory management.Trajectories are continued, born, or killed according to visibility and detection-confidence thresholds.

4. Experimental settings

The experiments use MOT17, TAO-person, and Caltech Roadside Pedestrians (CRP), which present different tracking challenges including crowding, camera motion, pose variation, and video corruption. SiamMOT is trained with image or video frame pairs and evaluated with standard tracking metrics and fixed inference thresholds.

  • Datasets: MOT17 contains crowded indoor and outdoor scenes, while TAO-person is a large-scale multi-person benchmark with diverse videos and artifacts.MOT17 evaluation reports MOTA, IDF1, false positives, false negatives, and ID switches; TAO-person uses Track-AP to emphasize temporal trajectory consistency.
  • Datasets: CRP contains roadside videos captured from a moving car, producing relatively fast pedestrian motion because of large camera movement.CRP contains seven approximately 20-minute videos and uses the MOT17 metrics.
  • Datasets: The datasets differ in their dominant challenges: MOT17 emphasizes occlusion and crowding, CRP emphasizes rapid position changes, and TAO-person includes varied scenes and corruption artifacts.The paper analyzes short-term ground-truth motion between consecutive annotated frames to characterize these differences.
  • Implementation: SiamMOT uses a DLA-34 feature-pyramid backbone, with a search region set to twice the tracking-target size.The implementation uses a Faster-RCNN backbone and separate spatial configurations for the implicit and explicit motion models.
  • Training: Training uses image pairs generated with spatial and motion-blur transformations when video annotations are unavailable, or random frames at most one second apart when they are available.The tracker and detector are jointly trained with SGD and momentum using image-pair batches.

5. Ablation analysis

The ablations test motion models, training-triplet composition, joint detector-tracker training, and trajectory persistence. They show progressively stronger tracking with stronger motion models, benefits from combining triplet types and joint training, and improved temporal consistency when tracks survive short occlusions.

  • Instance-level motion modelling: Tracktor performs poorly on CRP because its motion model is too weak for rapidly moving people, whereas adding flow substantially improves CRP and TAO-person.The comparison evaluates SiamMOT against Tracktor and Tracktor + Flow on MOT17, CRP, and TAO-person.
  • Instance-level motion modelling: +35 MOTA and +25 IDF1 over Tracktor + Flow are reported for SiamMOT on CRP.The performance gap is especially large on the challenging CRP dataset.
  • Instance-level motion modelling: Tracking performance increases across the motion-model sequence Tracktor < Flow < IMM < EMM, with EMM outperforming IMM on MOT17 and TAO-person.IMM and EMM are SiamMOT’s implicit and explicit motion models, respectively.
  • Training-triplet sampling: Combining positive, hard-negative, and negative triplets achieves the best overall results, whereas positive-plus-hard-negative training can yield relatively low MOTA.Positive triplets contain the same person, negative triplets contain no person, and hard triplets exclude the target person.
  • Joint training: Jointly training the detection and Siamese branches is reported as very important for SiamMOT and leads to the best results.The jointly trained detection models achieve 73.3% and 73.4% AP@IOU=0.5, indicating no negative impact on detection.
  • Inference: Tracking performance, especially IDF1 and TrackAP, increases as trajectories remain active through more unseen frames and saturates around τ = 30.The authors attribute saturation to people likely moving outside the search region after that duration and identify longer occlusions as future work.

6. Comparison to State-of-the-art

SiamMOT achieves state-of-the-art results across MOT17, TAO-person, and HiEve, with explicit motion modelling delivering strong benchmark performance. Its gains are especially notable on challenging tracking datasets and against established competitors.

  • MOT17: SiamMOT outperforms all previous methods on MOT17, including Tracktor++ v2 by 9.4 MOTA and CenterTrack by 4.4 MOTA.
  • Design comparison: Region-based features and explicit template matching distinguish SiamMOT from CenterTrack's point-based, implicit motion modelling.
  • TAO-person: SiamMOT+ sets new state-of-the-art results on TAO-person, with motion modelling reducing the cases requiring re-identification linking.
  • HiEve: On HiEve, the lightweight DLA-34 model matches the challenge's top performance, while the heavier DLA-169 model surpasses all winning methods.

7. Conclusion

SiamMOT is a region-based MOT network that detects and associates instances simultaneously through a Siamese tracker modelling motion across frames. It achieves top results on three multi-person tracking datasets while remaining adaptable to multi-class MOT.

  • SiamMOT simultaneously detects and associates object instances using a region-based MOT network.
  • A Siamese tracker temporally links detected instances by modelling their motion across frames.
  • SiamMOT achieves top results on three multi-person tracking datasets.
  • The framework can be adapted to multi-class multi-object tracking, although this direction remains future work.

Appendix A. Implicit Motion Model

The Implicit Motion Model learns instance displacement from visual features in two frames. Its configuration uses spatial feature maps, but it requires substantially more Siamese-tracker parameters than the explicit model.

  • IMM learns relative location and scale changes from visual features in both frames.
  • The IMM feature map is empirically set to c × 15 × 15, with diminished performance gains at c × 30 × 30.
  • IMM has 400× more learnable Siamese-tracker parameters than EMM under the reported configuration.
  • During inference, λ is set to 0.4 by default and reduced to 0.1 for CRP's large person motion.

Appendix C. Caltech Roadside Pedestrians (CRP)

CRP provides a fast-moving, incompletely annotated tracking setting used for ablation analysis. Hard examples reduce identity switches, while forward tracking beyond two frames fails in CRP; public-detection evaluation follows the reported protocol.

  • CRP contains long videos with very fast-moving people and is evaluated only on frames with available ground truth.
  • Hard training examples are important for reducing identity switches in SiamMOT.
  • CRP's incomplete annotation means many apparent false positives come from tracking unannotated background people rather than actual false detections.
  • τ > 1 frame has negligible effect because SiamMOT fails to track people forward beyond two frames in CRP.
  • Test-set results use public detections, with the provided detections re-scored by SiamMOT's detector under the allowed protocol.

Appendix F. TAO-person

SiamMOT is evaluated across TAO-person subsets and MOT Challenge metrics, with competitive performance across varied tracking scenarios and significant gains over Tracktor++.

  • SiamMOT performs competitively on self-driving street scenes such as BDD and Argoverse, as well as on the movie dataset Charades.
  • The appendix reports detailed results on MOT17 test videos and dataset-wise performance on TAO-person.
  • SiamMOT significantly outperforms Tracktor++ on MOT Challenge metrics on the TAO-person validation set.

Appendix G. Sensitivity analysis of parameters

The sensitivity analysis examines inference thresholds α and β on MOT17, finding that threshold choices affect tracking performance and that a balanced setting avoids excessive false negatives.

  • α is the detection confidence threshold for starting a new trajectory, while β is the visibility confidence threshold for continuing one.
  • Large α and β values make the solver too cautious, leading to high FN.
  • A good balance is achieved with β = 0.4 and α = 0.6, the setting used throughout the paper.
Loading 2105.11595v1…