PaperlayerView exact revision on arXiv

Source-linked AI summary

MAT: Motion-Aware Multi-Object Tracking

Shoudong Han, Piao Huang, Hongwei Wang, En Yu, Donghaisheng Liu, Xiaofeng Pan, Jun Zhao

arXiv:2009.04794v2cs.CV

TL;DR

MOT struggles to maintain long-range, pure tracks when camera or object motion, occlusion, and blur degrade detections, while ReID remains costly and unreliable. MAT combines pedestrian and camera motion modeling with dynamic reconnection and temporal-spatial association filtering, and experiments report state-of-the-art accuracy and high efficiency on MOT16 and MOT17.

  • Problem

    MOT faces unreliable long-range association and false negatives under camera motion, fast motion, occlusion, blur, and difficult appearances, while ReID is time-consuming and unreliable.

  • Method

    MAT combines pedestrian and camera motion in IML, dynamically determines reconnection windows and fills fragments with DRC, and filters associations using 3DII.

  • Results

    MAT demonstrates state-of-the-art accuracy and high efficiency on challenging MOT16 and MOT17 benchmarks.

  • Takeaways & Limitations

    Motion-aware prediction, reconnection, and temporal-spatial filtering provide an enhanced tracking-by-detection paradigm that can extend existing trackers.

  • Takeaways & Limitations

    The camera-motion modeling assumes adjacent-frame objects have slow motion and constant shapes when establishing pixel alignment.

Abstract

from arXiv · show

Modern multi-object tracking (MOT) systems usually model the trajectories by associating per-frame detections. However, when camera motion, fast motion, and occlusion challenges occur, it is difficult to ensure long-range tracking or even the tracklet purity, especially for small objects. Although re-identification is often employed, due to noisy partial-detections, similar appearance, and lack of temporal-spatial constraints, it is not only unreliable and time-consuming, but still cannot address the false negatives for occluded and blurred objects. In this paper, we propose an enhanced MOT paradigm, namely Motion-Aware Tracker (MAT), focusing more on various motion patterns of different objects. The rigid camera motion and nonrigid pedestrian motion are blended compatibly to form the integrated motion localization module. Meanwhile, we introduce the dynamic reconnection context module, which aims to balance the robustness of long-range motion-based reconnection, and includes the cyclic pseudo-observation updating strategy to smoothly fill in the tracking fragments caused by occlusion or blur. Additionally, the 3D integral image module is presented to efficiently cut useless track-detection association connections with temporal-spatial constraints. Extensive experiments on MOT16 and MOT17 challenging benchmarks demonstrate that our MAT approach can achieve the superior performance by a large margin with high efficiency, in contrast to other state-of-the-art trackers.

1 Introduction

Tracking-by-detection is effective but becomes unreliable in complex scenes when detections are degraded and motion is large. MAT addresses these challenges through motion-aware prediction, reconnection, and temporally constrained association.

  • Motivation: Tracking-by-detection estimates object trajectories from per-frame detections, but its detection dependence creates limitations in complex scenes.The paradigm can operate online or offline, yet its visible detection requirements and non-temporal concerns constrain robustness.
  • Motivation: Camera motion, fast motion, and occlusion can cause poor detections and large object displacements, undermining adjacent-frame association.These conditions challenge assumptions of high frame rates and low pedestrian speeds.
  • Motivation: Re-identification is costly and unreliable under noisy detections, similar appearances, and missing observations, while occlusion and blur still produce unrecovered fragments.These issues can also lead to identity switches and extensive false negatives.
  • MAT: MAT replaces ReID-based association with motion-based prediction and association through Integrated Motion Localization, Dynamic Reconnection Context, and 3D Integral Image modules.The framework focuses on different object motion patterns rather than requiring clean, discriminative appearance features.
  • MAT: IML combines pedestrian and camera motion, DRC supports dynamic long-range reconnection and fragment filling, and 3DII filters unnecessary track-detection connections.Together, the modules target motion compatibility, smooth trajectory completion, and efficient temporal-spatial association.

2 Related Work

Related MOT methods use appearance, spatial, or motion cues for association and trajectory prediction. MAT instead combines motion-aware prediction, reconnection, and temporal-spatial filtering in an integrated architecture.

  • ReID-based data association: Tracking-by-detection research includes ReID-based association methods that extract appearance features for long-range matching or per-frame association.These methods first locate objects using detector outputs and then compare corresponding appearance features.
  • ReID-based data association: Existing trackers also use spatial center distances, joint spatial-embedding association, and detector-integrated regression for tracking.These approaches include Tracktor++, CenterTrack, and FairMOT.
  • Motion models for trajectory prediction: Trajectory prediction methods model nonrigid object motion with constant velocity, observation smoothing, Kalman filtering, or social-force models.These methods address temporal-spatial variation in pedestrian trajectories.
  • Motion models for trajectory prediction: Rigid camera motion is commonly handled through 3D information, structure-from-motion, or affine transformations.These approaches target changes in camera pose rather than pedestrian dynamics.
  • MAT: MAT focuses on motion-based prediction, association, and reconnection, using IML, DRC, and 3DII within a regression-based tracking architecture.The architecture predicts motion, fills deactivated trajectories, and efficiently matches tracks within corrected detection regions.

3 Proposed Method

MAT extends tracking-by-detection with motion-based prediction, reconnection, trajectory filling, and temporospatial association filtering. Its modules combine camera and pedestrian motion, adapt reconnection to object motion, smooth occlusion gaps, and reduce unnecessary associations.

  • Framework: MAT uses motion-based predictions and detections within a tracking-by-detection architecture instead of relying on ReID association.The framework is designed as an extension of tracking-by-detection trackers.
  • 3.1 Integrated Motion Localization: IML predicts pedestrian motion with a Kalman Filter and then aligns bounding-box coordinates using ECC-estimated camera motion.The method processes pedestrian motion before camera motion, applying ECC alignment only to predicted box coordinates.
  • 3.2 Dynamic Reconnection Context: DRC maintains full-lifecycle trajectory predictions and dynamically sets each object's reconnection window from camera-motion intensity and individual velocity.The adaptive window addresses differing motion patterns and avoids a unified reconnection duration for all deactivated trajectories.
  • 3.2 Dynamic Reconnection Context: Cyclic pseudo-observation filling initializes boxes between interruption and reconnection, then forward- and backward-smooths the fragment using IML predictions.The two-direction smoothing is intended to reduce trajectory offsets and discontinuities at reconnection.
  • 3.3 3D Integral Image: 3DII encodes detections into K binary spatial layers and uses integral-image computation to filter track-detection connections within partial-overlap regions.Dynamic programming accelerates the integral-image computation, while spatial encoding removes unnecessary fully connected associations.

4 Experiments

Experiments evaluate MAT on MOT16 and MOT17, using ablations to isolate its motion and reconnection designs and comparisons against state-of-the-art trackers. The results report improved tracking accuracy, long-range reconnection, and association efficiency.

  • Experimental setup: Experiments use MOT16 and MOT17 benchmarks with CLEAR MOT metrics including MOTA, IDF1, FP, FN, IDS, and tracker speed.MOT16 and MOT17 contain the same sequences but differ in public detections and labels.
  • Ablation study: The ablation study compares motion models and reconnection mechanisms while controlling detector-related interference on the MOT17 training set.The baseline uses a Kalman model and fixed 10-frame reconnection, while variants replace motion or reconnection modules.
  • Ablation study: IML improves the baseline across metrics, especially IDS, while dynamic-window or DRC reconnection improves MOTA, IDF1, FN, and IDS with little FP sacrifice.DRC’s cyclic pseudo-observation filling is more robust than inertia prediction, with lower FP and FN and higher MOTA and IDF1.
  • Ablation study: Dynamic-window DRC increasingly outperforms fixed-window reconnection as maximal length grows, with a peak up to 120 frames for long-range tracking.This comparison uses the same cyclic pseudo-observation trajectory filling strategy.
  • Ablation study: The 3DII module accelerates temporal-spatial association with nearly constant time cost, outperforming IoU-filtering especially as the number of pedestrians per frame increases.Changing the cell size enables temporal-spatial constraints at different extended scales, whereas IoU-filtering only tests intersection.
  • Benchmark evaluation: On MOT16 and MOT17, MAT significantly outperforms existing state-of-the-art trackers, particularly in MOTA and FN, while achieving near-best IDF1 and IDS and lower time cost than most compared trackers.MAT wins the primary MOTA metric despite the re-implemented detector not matching some private detectors in accuracy or speed.

5 Conclusion

MAT is an enhanced tracking-by-detection paradigm centered on motion-based prediction, reconnection, and association. Its three modules support compatible motion modeling, trajectory filling, and efficient temporal-spatial filtering, with experiments demonstrating state-of-the-art accuracy and high efficiency.

  • Conclusion: MAT combines IML, DRC, and 3DII for motion-based prediction, dynamic reconnection with trajectory filling, and fast temporal-spatial filtering.The architecture is described as simple, general, and embeddable in tracking-by-detection trackers or video detectors.
  • Conclusion: Experiments on MOT16 and MOT17 demonstrate state-of-the-art accuracy and high efficiency for MAT.The paper presents MAT as an extensible enhancement to tracking-by-detection architectures.

(Supplementary Material)

The supplementary material supports reproduction by adding structured pseudocode and detector implementation details, along with sensitivity analyses for selected parameters.

  • Supplementary material: The supplementary material adds structured pseudocode for MAT with public or private detectors and implementation details for the re-implemented detector.It also reports sensitivity analyses for alpha in formula (2) and backward tracklet length.

1 Detailed Implementation

The supplementary implementation integrates MAT with public or private detections and specifies its processing algorithm and detector configuration. It uses temporal-spatial filtering before association and a cascaded RCNN detector implementation.

  • Association: The 3D Integral Image module filters temporal-spatial track-detection connections before association.The implementation evaluates candidate similarity with IoU and can retain connections above an IoU threshold of 0.3 before the KM algorithm.
  • Algorithm: MAT accepts a video sequence with either public or private detections and outputs object trajectories.The algorithm is presented for both detector settings.
  • Detector: The re-implemented detector uses ResNet50, FPN, RPN, three cascaded RCNN heads, RoIAlign, classification, and bounding-box regression branches.The final third-head outputs undergo naive NMS to produce detections.
  • Training: Training uses cross-entropy classification loss, smooth L1 regression loss, and learning-rate reductions after 15,000 and 25,000 iterations.The first 2,000 iterations use warm-up, and later learning rates become 0.1 and 0.01 of the initial value.

2 More Experiments

Additional experiments examine parameter sensitivity, sequence-level results, qualitative tracking, and trajectory filling. They support MAT’s motion-aware behavior under occlusion, fast or irregular camera motion, and direction changes.

  • 2.1 Parameter Sensitivity Analysis: The maximum reconnection length can reach 120 frames, while MOTA is not very sensitive to α and uses α=0.95 in all experiments.Camera-motion intensity should play the main role in determining the dynamic reconnection-window length, but removing the individual-velocity term is not optimal.
  • 2.1 Parameter Sensitivity Analysis: MOTA is almost completely insensitive to backward tracklet length, so the experiments fix this parameter at 3.The backward tracklet trains a backward IML model for cyclic pseudo-observation trajectory filling.
  • 2.2 Detailed Experimental Results: Sequence-level MOT16 and MOT17 results support the main conclusions, although the re-implemented detector is not equally strong on every sequence.The authors attribute the detector limitation to their focus on motion-aware designs rather than detector optimization.
  • 2.2 Detailed Experimental Results: On sequences with extensive occlusion and fast or camera motion, MAT works better with the proposed IML and DRC modules.The supplementary sequence results are presented as additional evidence of MAT’s motion-aware ability.
  • 2.2 Detailed Experimental Results: In a typical occlusion comparison, MAT is evaluated against Tracktor++ and DeepSort using their stated detector settings.The figure focuses on tracking differences for an occluded target pedestrian.
  • 2.2 Detailed Experimental Results: MAT produces consistent pedestrian identities under irregular camera motion, crowded scenes, and different camera viewpoints.The qualitative visualizations use SDP detections as observations.
  • 2.2 Detailed Experimental Results: DRC produces smoother trajectory filling closer to ground truth than inertia IML when pedestrians change direction during occlusion.Inertia IML predictions progressively move farther from the ground-truth boxes in the illustrated case.
Loading 2009.04794v2…