PaperlayerView exact revision on arXiv

Source-linked AI summary

MeMOTR: Long-Term Memory-Augmented Transformer for Multi-Object Tracking

Ruopeng Gao, Limin Wang

arXiv:2307.15700v3cs.CV

TL;DR

Existing MOT methods often exploit only adjacent-frame information, limiting long-term temporal modeling for target association. MeMOTR introduces per-object long-term memory injection and memory attention to stabilize and distinguish track embeddings. It achieves state-of-the-art MOT performance, especially for association, while retaining limitations in crowded-scene detection and dataset diversity.

  • Problem

    Most previous Transformer-based trackers explicitly exploit features between adjacent frames but do not use long-term temporal information effectively.

  • Method

    MeMOTR maintains long-term memory for each tracked object, injects it into track embeddings, and applies memory attention to produce more distinguishable representations.

  • Results

    MeMOTR achieves state-of-the-art performance on MOT benchmarks, especially in scenes with irregular motion patterns and association challenges.

  • Takeaways & Limitations

    Long-term temporal information can improve the stability, identity representation, and association performance of Transformer-based multi-object trackers.

  • Takeaways & Limitations

    Detection performance remains a drawback in crowded scenarios, where newborn objects may be suppressed by tracked targets during self-attention.

Abstract

from arXiv · show

As a video task, Multiple Object Tracking (MOT) is expected to capture temporal information of targets effectively. Unfortunately, most existing methods only explicitly exploit the object features between adjacent frames, while lacking the capacity to model long-term temporal information. In this paper, we propose MeMOTR, a long-term memory-augmented Transformer for multi-object tracking. Our method is able to make the same object's track embedding more stable and distinguishable by leveraging long-term memory injection with a customized memory-attention layer. This significantly improves the target association ability of our model. Experimental results on DanceTrack show that MeMOTR impressively surpasses the state-of-the-art method by 7.9% and 13.0% on HOTA and AssA metrics, respectively. Furthermore, our model also outperforms other Transformer-based methods on association performance on MOT17 and generalizes well on BDD100K. Code is available at https://github.com/MCG-NJU/MeMOTR.

1. Introduction

MOT must detect objects and preserve their identities, but complex scenes expose weaknesses in association and long-term temporal modeling. MeMOTR addresses this gap with memory-augmented Transformer tracking and evaluates it across challenging benchmarks.

  • Motivation: MOT combines object detection with association to maintain object identities in video streams.It supports applications including action recognition, behavior analysis, autonomous driving, and surveillance.
  • Motivation: Complex scenes with similar appearances and erratic movements make target association difficult for existing tracking methods.Group dancers and sports players are highlighted as challenging examples.
  • Motivation: Most Transformer-based trackers use only adjacent-frame features, leaving long-term temporal information underutilized.The paper motivates richer temporal representations to address analogous appearances, irregular motion, and long-term occlusions.
  • MeMOTR: MeMOTR maintains per-object long-term memory, injects it into track embeddings, and uses memory attention to make target representations more distinguishable.It also adaptively aggregates object features from two adjacent frames.
  • MeMOTR: A light decoder produces semantically specific detection embeddings before joint decoding with track embeddings.This design addresses the semantic information gap between learnable detection queries and object-specific track queries.
  • Evaluation: The method is evaluated on DanceTrack, MOT17, and BDD100K, with particular emphasis on association performance in challenging tracking settings.The paper also reports extensive ablation studies of its design choices.

2. Related Work

Related MOT methods commonly rely on detections, appearance, motion, or progressively decoded queries to associate targets. MeMOTR extends query-based tracking by explicitly modeling longer-term information for stable and discriminative representations.

  • Tracking-by-Detection: Tracking-by-detection obtains trajectories by associating detections in a streaming video.This paradigm has been widely used in MOT.
  • Tracking-by-Detection: Classic pedestrian tracking methods often match targets using appearance features and regular-motion estimates across consecutive frames.Examples include IoU matching with Kalman-filter predictions and cosine-distance matching with ReID features.
  • Tracking-by-Detection: Tracking-by-detection methods have strong detection capabilities but perform less effectively in complex scenarios with irregular movements.Unforeseeable motion can cause trajectory estimation and prediction modules to fail.
  • Tracking-by-Query: Tracking-by-query methods progressively decode track-query locations and typically avoid separate post-processing for association.They use track queries to localize already tracked objects while detection queries handle newborn objects.
  • Long-Term Temporal Modeling: Most query-based methods fuse information from adjacent frames rather than explicitly modeling longer temporal context.MeMOTR addresses this limitation with long-term memory and memory attention for more stable, distinguishable track features.

3. Method

MeMOTR combines separate detection and joint tracking decoders with long-term memory and temporal interaction modules. These components stabilize and distinguish track embeddings while handling newborn, occluded, and tracked objects.

  • Overview: MeMOTR uses a ResNet-50 backbone and Transformer Encoder, then processes detect and track embeddings through detection and joint decoders.The detection decoder produces semantic detect embeddings before concatenation with track embeddings in the joint decoder.
  • Long-Term Memory: The model maintains a long-term memory for each tracked target, initializes newborn-object memory from current outputs, and updates memory with exponentially decaying weights.The memory update rate λ is set to 0.01 in the reported experiments.
  • Temporal Interaction Module: Temporal interaction combines adjacent-frame outputs with adaptive aggregation to enhance target representations under occlusion or blur.A channel-wise weight is generated for current outputs before fusion with previous-frame information.
  • Temporal Interaction Module: A memory-attention layer uses long-term memory to interact across trajectories, then combines the result with memory and an FFN to predict the next track embedding.This design is intended to avoid abrupt embedding changes and improve object association.
  • Inference Details: The model updates track embeddings and memory only when predicted confidence exceeds the update threshold, while inactive tracks persist for a fixed number of missed frames.The thresholds τdet, τtck, and τnext are set to 0.5; Tmiss varies across DanceTrack, MOT17, and BDD100K.

4. Experiments

Experiments evaluate MeMOTR on DanceTrack, MOT17, and BDD100K, emphasizing association under challenging motion, occlusion, and multi-category settings. Ablations show that long-term memory, memory attention, adaptive aggregation, and separated detection decoding contribute to performance, while dataset size and crowded-scene detection remain limitations.

  • Evaluation setup: DanceTrack is prioritized because its varied motions create stronger association challenges than traditional pedestrian-tracking datasets.
  • DanceTrack: 68.5 HOTA and 58.4 AssA are achieved on the DanceTrack test set, with especially strong association performance.The method also reports 71.2 IDF1 and 80.5 DetA in the comparison discussion.
  • Limitations: MOT17 remains constrained by severe overfitting from its approximately 5K-frame training set, while crowded scenes expose detection weaknesses.Additional training data can also introduce bias toward static people, and newborn objects may be suppressed by tracked targets.
  • MOT17: 58.8 HOTA is achieved on MOT17, while AssA and IDF1 surpass MeMOT by 3.2% and 2.5%, respectively.
  • BDD100K: 53.6 TETA and 56.7 mAssocA on BDD100K indicate generalization to multi-category tracking, particularly in association.
  • Ablation study: Adaptive aggregation produces its strongest reported gain when dynamic weighting is used, improving HOTA by 2.6% and AssA by 2.3%.
  • Ablation study: Long-term memory improves AssA by 0.8% without memory attention and 4.3% with memory attention, while memory attention raises HOTA from 62.5 to 63.9.Without long-term memory, memory attention instead reduces AssA by 1.4% and IDF1 by 1.7%.

5. Conclusion

MeMOTR uses long-term memory and memory attention to stabilize and distinguish track embeddings for multi-object tracking. The paper reports state-of-the-art performance on MOT benchmarks, while supplemental DanceTrack comparisons control the underlying DETR framework and training augmentations.

  • MeMOTR builds stable long-term memory for each tracked object and uses it to augment track-embedding representations.
  • A memory-attention layer makes different targets more distinguishable, improving association performance.
  • The approach achieves state-of-the-art performance on MOT benchmarks, especially in association performance.
  • Supplemental DanceTrack comparisons use the same base color for experiments sharing a DETR framework, while most experiments remove MOTR training augmentations.

A. Boosting Tracking Performance

This section describes architectural choices that improve MeMOTR’s detection and tracking performance over the MOTR baseline. The additions include a stronger DETR backbone, proposals, and cumulative component improvements on DanceTrack.

  • Boosting Tracking Performance: MeMOTR is built on DAB-Deformable-DETR rather than Deformable-DETR to improve detection and tracking performance.The authors connect this choice to the stronger detection performance of DAB-Deformable-DETR.
  • Boosting Tracking Performance: Adding YOLOX proposals significantly improves detection and tracking performance simultaneously.Because proposals come from a frozen CNN, this version is not fully end-to-end and is listed as a hybrid architecture.
  • Boosting Tracking Performance: Table 8 summarizes cumulative improvements over MOTR on the DanceTrack validation and test sets.The comparison is intended to verify the effectiveness of the model’s various components.

B. Comparison on Difficult Sequences

On difficult DanceTrack validation sequences, MOTR has weak association despite passable detection, while MeMOTR substantially improves object association performance.

  • Comparison on Difficult Sequences: 23.6 average AssA versus 65.8 average DetA characterizes MOTR’s difficult-sequence performance on DanceTrack.The selected sequences had the lowest MOTR AssA scores, while their detection performance remained passable.
  • Comparison on Difficult Sequences: MeMOTR substantially improves object association performance on these complex sequences.

C. More Visualizations

The supplementary visualizations examine track embeddings across two DanceTrack sequences and report that long-term memory with memory attention produces more stable and distinguishable target representations.

  • More Visualizations: Figure 5 visualizes track embeddings from the first 50 frames of two DanceTrack sequences, using different colors and shapes for target IDs.The upper and lower panels correspond to dancetrack0025 and dancetrack0034, respectively.
  • More Visualizations: Figure 5 identifies the authors’ method specifically in panels 5(d) and 5(h).
  • More Visualizations: The visualizations report that long-term memory and the memory-attention layer learn more stable and distinguishable representations for tracked targets.

D. SOTA Comparison on SportsMOT

On SportsMOT, MeMOTR achieves 70.0 HOTA and 59.1 AssA without additional training data, further confirming its effectiveness.

  • 70.0 HOTA and 59.1 AssA are achieved by MeMOTR on SportsMOT without additional training data.The comparison uses methods trained only on the official SportsMOT training set for fairness.
Loading 2307.15700v3…