Source-linked AI summary
Recurrent Autoregressive Networks for Online Multi-Object Tracking
Kuan Fang, Yu Xiang, Xiaocheng Li, Silvio Savarese
TL;DR
Online multi-object tracking requires associating detections with trajectories using tracking history. The paper proposes RANs, which combine external memory of recent inputs with recurrent internal memory to model trajectory dynamics and perform data association. The method achieves top-ranked performance on MOT benchmarks and outperforms state-of-the-art online tracking methods.
Problem
Online multi-object tracking must reliably associate object trajectories with detections in each video frame using tracking history.
Method
RANs model object trajectories with external templates for previous inputs and internal recurrent memory for long-term history and detection likelihoods.
Results
The method outperforms state-of-the-art online tracking methods on the MOT benchmark for pedestrian tracking.
Takeaways & Limitations
Combining external and internal memories supports online multi-object tracking in the paper’s benchmark experiments.
Abstract
from arXiv · showhide
The main challenge of online multi-object tracking is to reliably associate object trajectories with detections in each video frame based on their tracking history. In this work, we propose the Recurrent Autoregressive Network (RAN), a temporal generative modeling framework to characterize the appearance and motion dynamics of multiple objects over time. The RAN couples an external memory and an internal memory. The external memory explicitly stores previous inputs of each trajectory in a time window, while the internal memory learns to summarize long-term tracking history and associate detections by processing the external memory. We conduct experiments on the MOT 2015 and 2016 datasets to demonstrate the robustness of our tracking method in highly crowded and occluded scenes. Our method achieves top-ranked results on the two benchmarks.
1. Introduction
The paper introduces RANs to model object trajectories over time and improve online data association by combining external templates with recurrent internal memory. Experiments report strong performance on MOT benchmarks, including state-of-the-art results among online methods.
- Motivation: Online multi-object tracking must estimate trajectory characteristics over time to associate detections across video frames.
- Motivation: Existing fixed-window templates limit long-term history, while recurrent networks can overfit because tracking datasets are relatively small.
- Method: RANs combine external templates storing previous input features with internal recurrent memory that learns to retrieve and process long-term history.
- Method: The external memory updates in a temporal sliding window according to tracking decisions rather than using an RNN to control data reading.
- Method: Each target is assigned a RAN, and detections are associated using likelihoods from the trajectory distribution modeled by that RAN.
- Results: The method outperforms state-of-the-art online tracking methods on the MOT benchmark for pedestrian tracking.
2. Related Work
Related work frames online tracking as requiring immediate frame-by-frame decisions and reliable feature representations. Prior neural approaches capture similarity or temporal information, while autoregressive models provide related generative modeling foundations.
- Batch Tracking vs. Online Tracking: Batch tracking can use future frames, whereas online tracking uses only previous and current frames for immediate association decisions.
- Batch Tracking vs. Online Tracking: Reliable feature representation is critical in online tracking, where prior methods commonly use hand-crafted features.
- Multi-Object Tracking with Neural Networks: Siamese networks match detection pairs but do not capture long-term object history, while recurrent networks learn data association end-to-end.
- Deep Autoregressive Models: Earlier deep autoregressive models specify conditional dependencies for fixed-dimensional variables, whereas RAN targets temporal or sequential data.
3. Our Model
The model uses paired internal and external memories to model each trajectory’s appearance and motion over time, then applies recurrent probability estimates for online detection association. Multiple trajectory-specific RANs provide scores for bipartite matching, with lost and new trajectories handled during tracking.
- 3.1. Recurrent Autoregressive Networks: RAN models each object trajectory using appearance and motion inputs while maintaining internal and external memories.The external memory stores recent input vectors as templates, while the internal hidden state summarizes how those templates should be combined.
- 3.1. Recurrent Autoregressive Networks: RAN estimates Pr(x_t|x_1:t−1) with an autoregressive model that combines previous inputs using time-varying parameters and Gaussian noise.Unlike traditional autoregressive models with fixed test-time parameters, RAN changes its coefficients and standard deviations as sequence characteristics change.
- 3.1. Recurrent Autoregressive Networks: A GRU updates the internal memory, using update and reset gates to control how much new information is incorporated and how much prior information is forgotten.The candidate hidden state contributes to the new hidden state after these gate operations.
- 3.1. Recurrent Autoregressive Networks: The external memory stores K previous input vectors, whereas the recurrent hidden state encodes long-term information for predicting the next input distribution.The external memory is a sliding window; the internal memory is updated recurrently from incoming trajectory data.
- 3.2. Data Association: For association, each trajectory-specific RAN scores candidate detections from appearance and motion, selects the highest-scoring detection above a threshold, and otherwise marks the trajectory lost.Appearance and motion use sibling internal and external memories, which are updated when a detection is associated.
- 3.3. Tracking Multiple Objects: The multi-object tracker assembles one RAN per object and uses bipartite matching to associate trajectories with detections, initialize new trajectories, and terminate long-lost ones.Trajectories lost for more than t_terminate = 20 time steps are terminated, while unassociated detections initialize new trajectories.
4. Experiments
Experiments on 2DMOT2015 and MOT16 evaluate RAN memory designs, modality choices, memory duration, benchmark performance, and qualitative behavior in crowded scenes. RAN models generally outperform the compared baselines and achieve competitive or state-of-the-art online tracking results.
- Experimental Setup: The experiments use 2DMOT2015 and MOT16, with public and Faster-RCNN detections, and evaluate standard MOT metrics including MOTA, MOTP, IDS, MT, ML, FP, FN, and fragmentation.The datasets contain 14 and 22 videos, respectively, with 1,221 and 1,276 object trajectories.
- Internal and External Memories: RAN models obtain the best MOTA and IDS within each evaluated modality setting, while GRU baselines have the worst results.The performance difference is most apparent for appearance features, whose high dimensionality requires greater memory representation power.
- Qualitative Analysis: RAN parameter weights usually decay over time but vary when occlusions and noisy detections occur, unlike the more regular decay of time-invariant models.The qualitative examples show parameters shifting toward earlier features during overlap or noisy observations and returning toward normal afterward.
- Internal and External Memories: Using both appearance and motion achieves the best performance among the three modality settings.Appearance produces fewer IDS than motion alone, while combining the modalities improves association under overlap and occlusion.
- External-Memory Time Span: After an initial jump, MOTA and IDS improve with longer external-memory spans, fluctuating slightly before converging after 9 steps.Longer spans require more GPU memory during training but have minor influence on test-time runtime.
- Test-Set Evaluation: On 2DMOT2015, RAN achieves state-of-the-art MOTA and IDS among online methods, while on MOT16 it achieves the best MT, FN, and IDS among online methods.On MOT16, RAN outperforms the cited state-of-the-art method in MOTA on 5 of 7 videos and in IDS on all 7.
5. Conclusion
The conclusion presents RAN as an online multi-object tracking framework that combines internal and external memory to model object history. It assigns one RAN to each object, uses likelihoods for data association, and reports benchmark evidence supporting its advantages.
- Conclusion: RAN combines external and internal memories to capture an object's tracking history and characteristics.The external and internal memories are the framework's central design components.
- Conclusion: Each object is represented by a RAN, and detections are associated by computing likelihoods under the distribution modeled by that RAN.This is the paper's stated procedure for extending the framework to multiple objects.
- Conclusion: Experiments on a commonly used multi-object tracking benchmark demonstrate advantages for online multi-object tracking.The conclusion also states that combining internal and external memories may be useful for other sequential data modeling and video analysis tasks.