Source-linked AI summary
Online Multi-Object Tracking with Dual Matching Attention Networks
Ji Zhu, Hua Yang, Nian Liu, Minyoung Kim, Wenjun Zhang, Ming-Hsuan Yang
TL;DR
The paper addresses online MOT under noisy detections, occlusion, and frequent target interactions. It combines single-object tracking with cost-sensitive learning and DMAN’s spatial-temporal attention for data association, achieving favorable identity-preserving performance against online and offline trackers.
Problem
Online MOT must maintain target identities despite noisy or missing detections, occlusion, misalignment, and frequent interactions between targets.
Method
The unified online framework uses a cost-sensitive tracking loss for single-object tracking and DMAN with spatial and temporal attention for data association.
Results
The method performs favorably against online and offline MOT trackers on benchmark datasets, especially on identity-preserving metrics.
Takeaways & Limitations
Spatial attention focuses matching on corresponding target regions, while temporal attention reduces the influence of unreliable tracklet observations.
Abstract
from arXiv · showhide
In this paper, we propose an online Multi-Object Tracking (MOT) approach which integrates the merits of single object tracking and data association methods in a unified framework to handle noisy detections and frequent interactions between targets. Specifically, for applying single object tracking in MOT, we introduce a cost-sensitive tracking loss based on the state-of-the-art visual tracker, which encourages the model to focus on hard negative distractors during online learning. For data association, we propose Dual Matching Attention Networks (DMAN) with both spatial and temporal attention mechanisms. The spatial attention module generates dual attention maps which enable the network to focus on the matching patterns of the input image pair, while the temporal attention module adaptively allocates different levels of attention to different samples in the tracklet to suppress noisy observations. Experimental results on the MOT benchmark datasets show that the proposed algorithm performs favorably against both online and offline trackers in terms of identity-preserving metrics.
1 Introduction
Online MOT must maintain identities despite unreliable detections, occlusion, target interactions, and distractors. The paper combines single-object tracking, cost-sensitive online learning, and attention-based data association to address these challenges.
- Offline MOT uses future frames, whereas online MOT uses only information available through the current frame for real-time tracking.
- Tracking-by-detection methods depend heavily on detection quality, so missing or inaccurate detections can cause targets to be lost.
- The unified framework tracks targets with a single-object tracker and invokes data association when tracking becomes unreliable.Data association is used when tracking scores fall below a threshold, indicating possible occlusion or major appearance changes.
- Single-object tracking faces severe positive-negative imbalance because only a few search locations are positives while most background locations are easy negatives.The paper therefore emphasizes hard examples during online updates to alleviate drifting.
- DMAN uses dual spatial attention maps to focus on matching regions and temporal attention to weight reliable trajectory observations.The proposed contributions target misaligned, incomplete, or occluded detections and noisy tracklet samples.
- A cost-sensitive tracking loss focuses online training on sparse hard samples, while experiments and ablations evaluate the proposed modules on MOT benchmarks.
2 Related Work
Related MOT work includes offline global optimization, online tracking-by-detection, attention-based methods, and approaches addressing data imbalance. The paper positions its online pipeline within these strands.
- Multi-Object Tracking: Offline MOT methods use past and future detections for batch processing and often formulate tracking as global optimization.Examples include network-flow and multi-cut formulations.
- Multi-Object Tracking: Online MOT methods avoid future detections but may struggle when targets are heavily occluded or mis-detected.
- Proposed Pipeline: The proposed pipeline combines detection, single-object tracking, and data association, switching targets between tracked and lost states based on reliability.Tracking generates tracklets for tracked targets, while association compares tracklets with candidate detections for lost targets.
- Attention Model: Attention mechanisms help models focus on relevant input regions, and this paper integrates spatial and temporal attention into online MOT.
- Data Imbalance: Data imbalance can be addressed through hard-negative mining or by re-weighting samples according to their observed loss.
3 Proposed Online MOT Algorithm
The proposed online MOT algorithm combines single-object tracking with data association, using confidence-based state management, motion-guided candidate selection, and DMAN appearance matching. Its tracking component adds a cost-sensitive loss, while DMAN uses spatial and temporal attention to handle noisy, occluded observations.
- Online MOT pipeline: The pipeline tracks each target with a single-object tracker and invokes data association when tracking becomes unreliable.A target is marked tracked when its tracking score and mean past observation score exceed thresholds; otherwise it enters a lost state.
- Single Object Tracking: ECO learns a continuous multi-channel convolution filter from image features and predicts target confidence over spatial locations.The objective combines sample-weighted data fitting with regularization, and the Fourier-domain formulation enables efficient optimization.
- Single Object Tracking: The cost-sensitive loss re-weights circularly shifted training samples according to their losses, emphasizing hard samples during online updates.The factor q(t) is precomputed from the previous model update and preserves the efficiency of the original ECO optimization.
- Data Association: When a target is lost, motion cues predict its location before appearance association selects nearby candidate detections.The prediction uses the previous box scale and a linear motion model; detections within distance threshold τd are considered, and affinity threshold τa controls relinking.
- Data Association: DMAN combines spatial and temporal attention to compare a candidate detection with tracklet observations under misalignment and occlusion.SAN extracts spatially matched features from paired images, while TAN aggregates tracklet information to infer whether the detection and tracklet belong to the same target.
4 Experiments
Experiments evaluate the method on MOT16 and MOT17, visualize its attention mechanisms, analyze module contributions, and compare benchmark performance against existing trackers.
- Datasets: The experiments use MOT16 and MOT17; MOT17 additionally provides DPM, Faster-RCNN, and SDP detections.MOT16 has 14 sequences split evenly between training and testing, while MOT17 uses the same sequences with three detection sets.
- Attention Visualization: Spatial attention locates matching target regions despite misalignment, scale change, and occlusion while suppressing distractors and backgrounds.The attention maps focus explicitly on target regions without part-level correspondence annotation.
- Attention Visualization: Temporal attention assigns low scores to occluded or unreliable observations by considering the overall tracklet sequence.This behavior is demonstrated for both same-target detections with noisy tracklets and different-target detections containing misleading observations.
- Ablation Studies: The full model reaches 41.2% MOTA on MOT16 training, while disabling DMAN causes a 7.1% MOTA drop.Disabling spatial or temporal attention also degrades performance, and the cost-sensitive loss provides a slight MOTA improvement.
- Benchmark Performance: On MOT16, the method improves IDF by 4.8%, IDP by 3.9%, IDR by 4%, and MT by 2.8% over the second-best published online tracker.It achieves the best IDF and IDP among online and offline methods.
- Benchmark Performance: On MOT17, the method performs favorably in MOTA against online trackers and achieves the best identity-preserving metrics among all methods.It also achieves the best AR score among online trackers on both MOT16 and MOT17.
5 Conclusions
The framework unifies single-object tracking and data association for online MOT, using cost-sensitive learning and spatial-temporal attention to address detection noise and occlusions.
- The proposed approach integrates single-object tracking and data association within a unified online MOT framework.
- A cost-sensitive loss mitigates data imbalance during single-object tracking.
- Spatial and temporal attention mechanisms handle noisy detections and occlusions during data association.
- Experiments on public MOT benchmark datasets demonstrate the effectiveness of the proposed approach.