Source-linked AI summary
Online Multi-Object Tracking Using CNN-based Single Object Tracker with Spatial-Temporal Attention Mechanism
Qi Chu, Wanli Ouyang, Hongsheng Li, Xiaogang Wang, Bin Liu, Nenghai Yu
TL;DR
Multi-object tracking remains challenging in crowded videos because occlusion and target interactions complicate consistent identity maintenance, while directly applying single-object trackers introduces additional problems. The paper proposes STAM, a dynamic CNN-based online MOT framework with shared computation and spatial-temporal attention, and demonstrates its effectiveness on MOT15 and MOT16 benchmarks.
Problem
Multi-object tracking must estimate multiple object locations while maintaining identities, but crowded scenes with occlusion and target interactions remain challenging, and direct single-object tracking introduces problems.
Method
STAM is a dynamic CNN-based online MOT framework that gives each object an individually learned tracker, shares computation, and uses online spatial-temporal attention to address occlusion-related drift and interactions.
Results
The proposed online MOT algorithm demonstrates effectiveness on the challenging MOT15 and MOT16 benchmarks.
Takeaways & Limitations
Shared CNN features and ROI-Pooling improve computational efficiency, while spatial-temporal attention helps address drift caused by frequent occlusions and target interactions.
Takeaways & Limitations
The method produces slightly more Frag than some offline methods, a common limitation of online MOT under long-term occlusions and severe camera-motion fluctuations.
Abstract
from arXiv · showhide
In this paper, we propose a CNN-based framework for online MOT. This framework utilizes the merits of single object trackers in adapting appearance models and searching for target in the next frame. Simply applying single object tracker for MOT will encounter the problem in computational efficiency and drifted results caused by occlusion. Our framework achieves computational efficiency by sharing features and using ROI-Pooling to obtain individual features for each target. Some online learned target-specific CNN layers are used for adapting the appearance model for each target. In the framework, we introduce spatial-temporal attention mechanism (STAM) to handle the drift caused by occlusion and interaction among targets. The visibility map of the target is learned and used for inferring the spatial attention map. The spatial attention map is then applied to weight the features. Besides, the occlusion status can be estimated from the visibility map, which controls the online updating process via weighted loss on training samples with different occlusion statuses in different frames. It can be considered as temporal attention mechanism. The proposed algorithm achieves 34.3% and 46.0% in MOTA on challenging MOT15 and MOT16 benchmark dataset respectively.
1. Introduction
The paper targets online MOT challenges caused by occlusion, target interaction, and the rising computational cost of independently applying single-object trackers. STAM addresses these issues with shared CNN computation and online spatial-temporal attention, and is evaluated on MOT15 and MOT16.
- MOT estimates multiple object locations while maintaining identities and trajectories, but remains difficult in crowded scenes with frequent occlusion and target interaction.
- Directly applying single-object tracking to MOT can corrupt appearance learning during occlusion, causing trackers to drift toward occluders and eventually fail.Occluded targets may retain high scores initially, so corrupted samples can still update the appearance model.
- Adding a new single-object tracker for each appearing target can make computational cost intolerable as the number of tracked objects increases.
- STAM uses an online-learned individual tracker for each object within a dynamic CNN-based online MOT framework.
- Shared computation among multiple objects improves efficiency, while learned spatial-temporal attention addresses drift from occlusion and target interactions.Visibility maps infer spatial attention for feature weighting and temporal attention for reducing updates from severely occluded samples.
- STAM is evaluated on challenging MOT15 and MOT16 benchmarks.
2. Related Work
Related work includes detector-and-association methods, single-object trackers integrated into MOT, and occlusion-handling approaches. STAM focuses on online single-object tracking with attention-based feature learning as a complement to detection methods.
- Multi-object Tracking by Data Association: Data-association MOT methods apply detectors frame by frame and associate detections, while offline approaches can use future frames and online approaches cannot.Association-based online tracking depends heavily on detector quality; single-object tracking can provide candidates and help with missing detections.
- Single Object Tracker in MOT: Prior work often integrates single-object trackers into only limited MOT subproblems because the approach faces occlusion and computational challenges.
- Occlusion handling in MOT: Most occlusion-handling methods improve detectors, whereas this work learns more robust target appearance models using spatial and temporal attention.The feature-learning approach is presented as complementary to detection-based occlusion handling.
- Proposed approach: STAM combines motion-based search, spatially weighted candidate features, and temporally weighted online updates.
3.1. Overview
STAM tracks each target by searching a motion-defined region, weighting candidate features spatially, estimating visibility, and updating target-specific models with temporally weighted losses.
- The motion model defines each target’s search area, within which candidate states are sampled.
- ROI-Pooling extracts candidate features, spatial attention weights them, and a binary classifier selects the candidate with the maximum score.
- The estimated target state is used to infer a visibility map and temporal attention from target and neighboring-target spatial configurations.
- Temporal attention weights current and historical training losses when updating each target-specific CNN branch, while the motion model is updated from the estimated state.
- Object management initializes new targets and terminates targets that are no longer tracked.
- The procedure repeats for successive frames until the video ends.
3.2. Dynamic CNN-based MOT Framework
The dynamic CNN framework combines shared frame-level feature extraction with separately learned target-specific branches. This shares computation while preserving individual appearance models for changing target populations.
- Shared CNN layers encode the whole frame once into a feature map, and ROI-Pooling extracts individual target representations.
- The shared layers are ImageNet-pretrained and remain fixed during tracking for computational efficiency.
- Target-specific CNN branches share an architecture but are trained separately to capture different target appearances.These branches function as a set of single-object trackers.
- The number of target-specific branches changes with the tracked population: branches are added for new targets and removed for disappeared targets.
3.3. Online Tracking with STAM
The tracker searches candidate states using motion- and detection-based proposals, then extracts visibility-aware features and classifies candidates for each target.
- 3.3.1 Candidate States: Candidate states combine samples drawn around the motion-model prediction with nearby object detections.
- 3.3.2 Feature Extraction with Spatial Attention: Shared feature maps are ROI-Pooled for each candidate, while spatial attention emphasizes unoccluded regions that remain reliable under occlusion.The spatial attention map is derived from a learned visibility map.
- 3.3.3 Target State Estimation Using Binary Classifier and Detection Results: The binary classifier outputs each candidate’s probability of belonging to target T_i.The classifier uses the refined attention-weighted feature representation.
- 3.3.3 Target State Estimation Using Binary Classifier and Detection Results: The primitive target state is selected as the candidate with maximum classification score.
- 3.3.3 Target State Estimation Using Binary Classifier and Detection Results: A primitive state with classification score below p0 is marked untracked; otherwise, detection results further refine the state.The refinement uses object detection states and their bounding-box IoU overlap with the primitive state.
3.4. Model Initialization and Online Updating
Each target-specific branch is initialized from synthetic visibility-map examples and jointly updated online using current and historical samples weighted by estimated occlusion.
- 3.4.1 Model Initialization: Each target-specific branch contains visibility-map, attention-map, and binary-classifier modules that are jointly learned after initialization.
- 3.4.1 Model Initialization: Initialization uses augmented samples from the initial target state, including feature replacement that labels replaced regions as occluded.The visibility-map training samples are optimized with cross-entropy loss.
- 3.4.2 Online Updating Appearance Model: Online updates use current-frame positives, historical positives, and current-frame negatives; when a target is untracked, only historical positives are used.Negative samples include other tracked targets and randomly sampled background regions.
- 3.4.2 Online Updating Appearance Model: Temporal attention dynamically balances current and historical visual cues according to the target’s occlusion status.The visibility map provides the occlusion-related cue for this weighting.
- 3.4.2 Online Updating Appearance Model: As occlusion becomes more severe, the weight assigned to current positive samples becomes smaller.States with sufficiently low temporal attention can be added to the target’s historical sample set.
- 3.4.3 Motion Model: A constant-velocity motion model with Gaussian noise predicts the next target state and expands the search area during successive untracked frames.
3.5. Object Management
Object management dynamically initializes targets from reliable unmatched detections and terminates them after sustained tracking failure or field-of-view exit.
- A new target is initialized when a high-score detection is not covered by any currently tracked target.
- Newly initialized targets are discarded after early tracking failure or if they are absent from the first T_init frames.
- A target is terminated after being untracked for more than T_term successive frames or after exiting the field of view.
4. Experiments
Experiments evaluate STAM on MOT15 and MOT16 using standard tracking metrics, component analyses, and comparisons with online and offline methods. STAM improves tracking accuracy and identity consistency while feature sharing increases speed, though fragmentation remains slightly higher than some offline methods.
- Tracking speed: 0.5 fps versus 0.1 fps demonstrates the speed benefit of feature sharing on MOT15 test sequences.Both measurements use a 2.4GHz CPU and TITAN X GPU.
- Performance analysis: STAM's full configuration improves MOT15 training-sequence MOTA by 8% over the single-object-tracker baseline.The full configuration combines motion modeling with spatial and temporal attention.
- Performance analysis: Spatial and temporal attention each improve MOTA over the motion-model configuration, while their combination performs best and shows complementary benefits.The comparison uses p2 as the motion-model configuration and p3, p4, and p5 as attention variants.
- State-of-the-art comparisons: STAM improves MOT15 MOTA by 4% over MDP, the best peer-reviewed online method, and produces the lowest IDS among compared methods.The comparison uses the benchmark's public detections and contrasts STAM with online and offline trackers.
- State-of-the-art comparisons: On MOT16, STAM achieves the best online performance in MOTA, MT, ML, and FN, with MOTA on par with state-of-the-art offline methods.The table separates online and offline tracking results and marks higher- or lower-is-better metrics.
- State-of-the-art comparisons: STAM produces slightly more Frag than some offline methods, a common defect of online MOT under long-term occlusion and severe camera-motion fluctuation.This is reported as a limitation of the online setting rather than as a general failure of the method.
5. Conclusion
The paper concludes with a dynamic CNN-based online MOT algorithm that shares features and uses ROI-Pooling alongside spatial-temporal attention and a motion model. Experiments on challenging MOT benchmarks demonstrate its effectiveness.
- Conclusion: The proposed dynamic CNN-based online MOT algorithm uses shared CNN features and ROI-Pooling to efficiently support single-object trackers.These components provide individual target features while retaining the merits of single-object tracking.
- Conclusion: Spatial-temporal attention is introduced to alleviate drift caused by frequent occlusions and interactions among targets.The conclusion identifies occlusion and target interaction as the main sources of drift addressed by the mechanism.
- Conclusion: A simple motion model is integrated to use motion information during online multi-object tracking.
- Conclusion: Experimental results on challenging MOT benchmarks demonstrate the effectiveness of the proposed online MOT algorithm.