Source-linked AI summary
MeMOT: Multi-Object Tracking with Memory
Jiarui Cai, Mingze Xu, Wei Li, Yuanjun Xiong, Wei Xia, Zhuowen Tu, Stefano Soatto
TL;DR
Online multi-object tracking must detect objects and associate them across time, including after long gaps. MeMOT addresses this with a Transformer-based spatio-temporal memory and unified decoding, achieving competitive benchmark performance. Its supervised training and memory design remain bounded by annotation scarcity and GPU cost.
Problem
Online MOT traditionally separates object detection from temporal data association, while long-range occlusion makes object-state estimation difficult.
Method
MeMOT jointly performs detection and association using a large spatio-temporal memory, adaptive memory encoding, and Transformer-based hypothesis generation, memory encoding, and decoding modules.
Results
MeMOT achieves state-of-the-art performance among methods with in-network association and competitive performance against post-network association methods on MOT Challenge benchmarks.
Takeaways & Limitations
The unified memory-based design supports object localization and association, including tracking in crowded scenes without post-processing.
Takeaways & Limitations
Supervised training requires annotated tracking videos, while the spatio-temporal memory increases GPU memory cost and limits its temporal length.
Abstract
from arXiv · showhide
We propose an online tracking algorithm that performs the object detection and data association under a common framework, capable of linking objects after a long time span. This is realized by preserving a large spatio-temporal memory to store the identity embeddings of the tracked objects, and by adaptively referencing and aggregating useful information from the memory as needed. Our model, called MeMOT, consists of three main modules that are all Transformer-based: 1) Hypothesis Generation that produce object proposals in the current video frame; 2) Memory Encoding that extracts the core information from the memory for each tracked object; and 3) Memory Decoding that solves the object detection and data association tasks simultaneously for multi-object tracking. When evaluated on widely adopted MOT benchmark datasets, MeMOT observes very competitive performance.
1. Introduction
MeMOT addresses online multi-object tracking by jointly performing detection and association while retaining long-range object history. Its Transformer-based memory architecture uses proposals, track embeddings, and unified decoding, achieving competitive benchmark performance.
- Online MOT must detect objects in individual frames and associate them across time while preserving identities.
- MeMOT jointly performs object detection and data association online using a large spatio-temporal memory of past tracked-object observations.The memory is actively encoded to approximate object states for association.
- MeMOT’s unified decoder outputs both reappearing tracked objects and novel objects first seen in the current frame.
- Hypothesis generation produces proposal embeddings, memory encoding produces track embeddings, and memory decoding solves detection and association simultaneously.The proposal and track embeddings are combined with image features in the decoding module.
- MeMOT achieves state-of-the-art performance among in-network association methods and remains competitive with methods using post-network association.It also outperforms other Transformer-based methods in object detection and data association, with ablations validating its design.
2. Related Work
Prior MOT research includes classical state estimation, CNN-based tracking, and memory networks for prolonged temporal reasoning. MeMOT extends memory-based modeling to robust object association across time.
- Classical Tracking Methods: Classical tracking methods use mathematical and statistical state estimation, including Kalman and particle filters, to handle uncertain object motion and appearance.
- Classical Tracking Methods: Finite-dimensional state representations are difficult for nonlinear, non-Gaussian cases such as occlusion, motivating access to states from multiple frames.
- Memory Networks: Memory networks store and access time-indexed features in prolonged sequences, improving the ability to remember past information in video analysis.
- Memory Networks: MeMOT uses a large spatio-temporal memory to achieve robust object association across time for online MOT.
3. Multi-Object Tracking with Memory
MeMOT performs online object detection and data association jointly by retrieving long-range track information from a spatio-temporal memory. Its Transformer modules generate proposals, encode track history, and decode unified tracking outputs without further post-processing.
- Overview: MeMOT uses a spatio-temporal memory of long-range states to link tracked objects after extended time spans.The memory stores historical states for tracked objects and adaptively retrieves relevant information for association.
- Hypothesis Generation: Hypothesis generation produces Transformer-based proposal embeddings from current-frame image features for new or tracked objects.The encoder processes spatially positioned CNN features, while the decoder uses learnable object queries to produce proposals, objectness scores, and bounding boxes.
- Memory Encoding: Memory encoding combines short-term and long-term attention branches through fusion to form track embeddings and update Dynamic Memory Aggregation Tokens.Short-term attention smooths recent-frame noise, whereas long-term attention extracts relevant history from the memory window.
- Memory Decoding: Memory decoding jointly predicts bounding boxes, objectness, uniqueness, and associations from proposal and track embeddings.Uniqueness identifies novel objects for inclusion and suppresses entries associated with already tracked objects or non-target entries.
- Inference and Supervision: Thresholded decoder outputs inherit existing identities or initialize new tracks, eliminating the need for further post-processing.Ground-truth assignment uses bipartite matching for proposal entries and assigns uniqueness according to whether matched objects were previously seen.
4. Experiments
MeMOT is evaluated on MOT Challenge benchmarks using unified, in-network detection and association, with experiments covering normal and crowded scenarios, visualizations, ablations, and limitations. Results indicate strong association accuracy and reduced identity switches, while memory improves long-range tracking at added training-memory cost.
- Evaluation setup: MeMOT is evaluated on MOT16, MOT17, and MOT20 using CLEAR MOT Metrics and HOTA.Experiments use standard MOT Challenge protocols and compare methods with in-network and post-network association solvers.
- Comparison with state-of-the-art methods: MeMOT achieves state-of-the-art performance among in-network association methods and remains competitive with post-network association methods.The authors report competitive performance on the MOT Challenge benchmarks and stronger detection and association than other Transformer-based methods.
- Comparison with state-of-the-art methods: 63% reduction in IDsw is achieved on MOT20 versus FairMOT, while MeMOT outperforms other Transformer-based methods by 6.7 IDF1 and 5.2 HOTA.FairMOT uses 8× more training data; MOT20 contains crowded scenes and serious occlusions.
- Visualization: MeMOT produces long, consistent trajectories when objects pass frequently and shows stronger detection and association in crowded scenarios.The authors attribute higher identity-switch counts in competing similarity-based methods to small objects, poor lighting, and frequent occlusion.
- Visualization: Memory attention uses pre-occlusion and full-body features to re-link an occluded object, while non-occluded tracking emphasizes short-term memory.For object 55, frames 124 and 121 contribute most to re-linking at frame 129 after occlusion; object 62 attends more to short-term history.
- Ablation studies: Increasing short-term memory from 3 to 5 changes performance little, so Ts=3 is selected as an accuracy-efficiency trade-off.Using only the last two observations slightly decreases performance, with long-term memory compensating for information gaps.
- Ablation studies: Dynamic updating of long-term aggregation queries improves detection and tracking by injecting recent observations into memory retrieval.The ablation reports that firsthand observations enable more detailed information for current association.
- Limitations: MeMOT requires tracking-annotated video datasets and incurs higher GPU memory cost, limiting memory temporal length and motivating efficiency improvements.The limitation is tied to supervised training and the large spatio-temporal memory.
5. Conclusion
MeMOT jointly performs object detection and data association for online MOT by maintaining and actively encoding a large spatio-temporal memory. The paper reports effectiveness for object localization and association in crowded scenes while advocating responsible use within applicable laws and regulations.
- 5. Conclusion: MeMOT jointly performs object detection and data association for online MOT.Its track states are dynamically updated query embeddings, and predictions use attention without post-processing.
- 5. Conclusion: A large spatio-temporal memory stores past observations, which are actively encoded through an attention-based aggregator.
- 5. Conclusion: Extensive experiments validate MeMOT for object localization and association in crowded scenes.
- 5. Conclusion: MOT applications include health monitoring, autonomous driving, and collaborative robots, but use should comply with applicable laws and regulations.
6. Appendix
The appendix describes MeMOT's online workflow: video frames are processed while track-state embeddings are maintained in spatio-temporal memory, and Transformer modules generate, aggregate, and decode queries into trajectories.
- 6. Appendix: MeMOT processes a sequence of video frames while maintaining track states as embeddings in a spatio-temporal memory buffer.The output is a set of trajectories for K objects.
- 6. Appendix: The hypothesis-generation module encodes the current image into frame features and decodes them into proposal embeddings representing hypothetical object instances.
- 6. Appendix: Short-term aggregation queries recent memory with the latest observation, while long-term aggregation uses dynamic memory aggregation tokens over a longer period.The resulting short- and long-term queries are fused into track queries.
- 6. Appendix: Memory decoding uses track queries and frame features to estimate object states and confidence scores for tracked objects and proposals.Tracked states update above 0.6 confidence, new tracks initialize above 0.7, and the short- and long-term windows are 3 and 24 frames.