PaperlayerView exact revision on arXiv

Source-linked AI summary

Multiple Object Tracking as ID Prediction

Ruopeng Gao, Ji Qi, Limin Wang

arXiv:2403.16848v2cs.CV

TL;DR

Heuristic matching methods can require extensive handcrafted modifications in complex tracking scenarios. MOTIP treats association as in-context ID prediction, using historical trajectory IDs to predict labels for current detections. It reports surpassing state of the art on all benchmarks, while identifying dictionary capacity in crowded scenarios as a limitation.

  • Problem

    Heuristic association methods can require extensive handcrafted modifications when tracking scenarios become complex.

    Source: §1 · p. 2

  • Method

    MOTIP predicts current detections’ ID labels from identity information carried by historical trajectories, treating association as an in-context prediction task.

    Source: §1 · p. 2

  • Results

    MOTIP reports surpassing the state of the art on all benchmarks.

    Source: §6 · p. 8

  • Takeaways & Limitations

    The authors present MOTIP as a simple baseline and say the ID-prediction pipeline has potential for future research.

    Source: §6 · p. 8

    The ID dictionary capacity K may be insufficient in crowded scenarios.

    Source: §5 · p. 8

Abstract

from arXiv · show

Multi-Object Tracking (MOT) has been a long-standing challenge in video understanding. A natural and intuitive approach is to split this task into two parts: object detection and association. Most mainstream methods employ meticulously crafted heuristic techniques to maintain trajectory information and compute cost matrices for object matching. Although these methods can achieve notable tracking performance, they often require a series of elaborate handcrafted modifications while facing complicated scenarios. We believe that manually assumed priors limit the method's adaptability and flexibility in learning optimal tracking capabilities from domain-specific data. Therefore, we introduce a new perspective that treats Multiple Object Tracking as an in-context ID Prediction task, transforming the aforementioned object association into an end-to-end trainable task. Based on this, we propose a simple yet effective method termed MOTIP. Given a set of trajectories carried with ID information, MOTIP directly decodes the ID labels for current detections to accomplish the association process. Without using tailored or sophisticated architectures, our method achieves state-of-the-art results across multiple benchmarks by solely leveraging object-level features as tracking cues. The simplicity and impressive results of MOTIP leave substantial room for future advancements, thereby making it a promising baseline for subsequent research. Our code and checkpoints are released at https://github.com/MCG-NJU/MOTIP.

1. Introduction

Heuristic association methods can require extensive tuning and struggle in complex tracking scenarios. MOTIP reframes association as in-context ID prediction, using historical trajectory IDs to predict labels for current detections and reporting state-of-the-art results across multiple benchmarks.

1. Introduction

  • Heuristic matching rules can struggle with irregular motion and similar appearances, while adapting them requires substantial human analysis and tuning.Earlier methods also relied on trajectory modeling and manually designed matching algorithms.Source: §1 · pp. 1–2
  • The paper asks whether detection and association can remain decoupled while replacing heuristic algorithms with an end-to-end trainable pipeline.The motivation notes that some joint detection-and-tracking approaches can face conflicts between query types or supervision signals.Source: §1 · p. 2
  • Historical trajectory IDs condition predictions for current targets, so association does not require assigning each trajectory a globally fixed label.The authors frame this as in-context ID prediction, which supports generalization to unseen identities.Source: Figure 1 · p. 1, §1 · p. 2
  • MOTIP reports state-of-the-art tracking performance across multiple benchmarks with a minimalist design that uses object-level embeddings and a standard transformer decoder.The method uses Deformable DETR for object-level embeddings and learnable ID embeddings attached to trajectory tokens.Source: §1 · p. 2

2. Related Work

Related work includes tracking-by-detection systems that associate detections with historical trajectories and tracking-by-propagation systems that carry track queries across frames. MOTIP separates detection from association and learns the association decisions rather than relying on handcrafted rules.

2. Related Work

  • Tracking-by-Detection methods associate detections with historical trajectories through post-processing, often using Kalman filters for motion and ReID features for appearance.More recent methods have added complex matching approaches or multimodal information to address manual algorithms' limitations in complex scenarios.Source: §2 · p. 2
  • MOTIP structurally decouples detection and association, making association decisions in an end-to-end pipeline rather than through handcrafted algorithms.Unlike propagation methods, its detection and association occur sequentially in separate modules.Source: §2 · pp. 2–3
  • Tracking-by-Propagation methods represent tracked targets with track queries propagated through video, with later work extending long-term modeling.Some studies identify conflicts in joint detection and tracking and seek to alleviate them.Source: §2 · p. 3

3. Method

MOTIP formulates object association as in-context ID prediction from historical trajectory identity information. Its detector, learnable ID dictionary, and ID Decoder support this approach, with training augmentations and a direct inference procedure.

3.1. In-context ID Prediction

  • MOTIP predicts each detection’s ID from its historical trajectory prompt, keeping predicted labels within the training label range even for unseen trajectories.Unlike fixed-label classification, trajectory IDs indicate consistency rather than specific semantic identities.Source: §3.1 · p. 3

3.2. MOTIP Architecture

  • Deformable DETR supplies object-level features, which MOTIP combines with learnable identity embeddings to form historical tracklets; current detections use a special newborn token.The ID dictionary contains K regular identity tokens and one special token for newborn objects.Source: §3.2 · pp. 3–4
  • The ID Decoder uses historical tracklets as keys and values to decode current detection tracklets and predict their in-context ID labels.A linear classification head predicts the ID, and cross-entropy directly supervises the association task.Source: §3.2 · p. 4

3.3. Training

  • Training combines detection losses with ID cross-entropy and augments trajectories by randomly dropping tokens or swapping ID tokens between trajectories.The augmentations simulate occlusion and possible ID assignment errors during inference.Source: §3.3 · pp. 4–5, Figure 3 · p. 5

3.4. Inference

  • At inference, MOTIP filters detections by confidence, predicts IDs, marks qualifying unassigned detections as newborns, and recycles IDs from expired trajectories.When detections share an ID in one frame, only the one with the highest confidence is retained.Source: §3.4 · p. 5
  • MOTIP does not focus on more complex assignment strategies, such as Hungarian matching or multi-stage matching, which might offer improvements.The authors make this choice to validate the model’s robustness and generalization.Source: §3.4 · p. 5

4. Experiments

MOTIP is evaluated on three challenging tracking benchmarks using HOTA, which balances detection and association accuracy. It reports strong benchmark results and ablations of its design choices and trajectory augmentation.

4.1. Datasets and Metrics

  • Evaluation covers DanceTrack, SportsMOT, and BFT, spanning human, athlete, and bird tracking scenarios with challenges including occlusion, irregular movement, high-speed motion, and similar appearance.HOTA is the main metric; the experiments also list MOTA and IDF1.Source: §4.1 · p. 5

4.2. Implementation Details

  • The default detector is Deformable DETR with a ResNet-50 backbone; training uses sampled video frames and attention masks to enable parallel, GPU-friendly processing.The paper reports training on 8 NVIDIA RTX 4090 GPUs and completing DanceTrack training in less than one day.Source: §4.2 · pp. 5–6

4.3. Comparisons with State-of-the-art Methods

  • On DanceTrack, MOTIP leads CO-MOT by 4.3 HOTA and 6.9 AssA; on SportsMOT, it outperforms prior methods, including joint-training methods in AssA.The SportsMOT model is trained solely on the SportsMOT train set.Source: Table 1 · p. 6, §4.3 · pp. 6–7, Table 2 · p. 6
  • On BFT, MOTIP establishes a state-of-the-art result with 70.5 HOTA and 71.8 AssA.The paper presents this result as evidence of generalization across different scenarios.Source: Table 3 · p. 7, §4.3 · p. 7

4.4. Ablations

  • Ablations report that ID Decoder self-attention is important for tracking, while the Hungarian algorithm provides no considerable benefit by default.The authors say self-attention lets current objects exchange identity information, helping prevent confusion among similar targets.Source: Table 4 · p. 7, §4.4 · p. 7
  • Under the same object features, MOTIP’s ID-prediction formulation outperforms the compared ReID pipelines, including variants with a Hungarian algorithm or trajectory enhancement module.The authors attribute this advantage to greater flexibility in managing historical tracklets.Source: §4.4 · p. 8, Table 5 · p. 8
  • Trajectory augmentation improves performance at λocc = 0.5, but discarding too many tokens at λocc = 1.0 undermines results; λsw = 0.5 gives the best reported performance.The authors use λocc = λsw = 0.5 for final results and keep this setting across datasets, although dataset-specific tuning may do better.Source: §4.4 · p. 8, Table 6 · p. 8

5. Limitations and Discussions

MOTIP is presented as a simple baseline for testing in-context ID prediction, leaving substantial room for architectural and tracking-cue extensions. Its ID dictionary may also be undersized for crowded scenarios.

5. Limitations and Discussions

  • The study prioritizes testing in-context ID prediction over highly customized model designs.The authors identify tailored ID Decoder layers and more sophisticated trajectory modeling as future research directions.Source: §5 · p. 8
  • Adding cues such as motion or depth is proposed as a possible extension to the current approach.The authors describe these as examples of additional tracking cues for future work.Source: §5 · p. 8
  • The ID dictionary capacity K may be insufficient in crowded scenarios, although it can be increased for extreme cases.Token utilization is below 40% in most cases, and the default capacity is intended for general scenarios.Source: §5 · p. 8

6. Conclusion

The paper introduces MOT as in-context ID prediction and presents MOTIP as a simple baseline that surpasses the state of the art on all benchmarks. The supplement covers experimental details, static-image joint-training concerns, and additional results and visualizations.

6. Conclusion

  • MOTIP treats multiple object tracking as in-context ID prediction and simplifies both training and tracking.The authors present it as a simple yet effective baseline design.Source: §6 · p. 8
  • MOTIP surpasses the state of the art on all benchmarks, according to the authors.They suggest the pipeline and method can inspire future research.Source: §6 · p. 8

A. Overview

  • The supplementary material includes experimental details, discussion of static images for joint training, and additional experiments and visualizations.These topics are covered in Appendices B, C, and D, respectively.Source: A. Overview · p. 9

B. Experimental Details

The appendix provides experimental details absent from the main text, covering training, inference, and ablation experiments.

B. Experimental Details

B.1. Training

The appendix describes MOTIP’s training setup and parallelized processing, along with its inference-time ID assignment rule. Training uses random frame intervals and dataset-specific schedules, while inference resolves duplicate IDs by retaining the highest-confidence assignment.

B.1. Training

  • Training samples T + 1 frames with random intervals from 1 to 4 by default.The authors note that continuously increasing the interval can make samples excessively challenging and create a discrepancy with inference videos.Source: §B.1 · p. 9
  • Training runs for 10 epochs on DanceTrack, 13 on SportsMOT, and 22 on BFT, with learning-rate drops scheduled per dataset.The setups use COCO-pretrained weights and dataset-specific detection pretraining; the typical hardware is eight RTX 4090 GPUs with batch size one per GPU.Source: §B.1 · p. 9
  • MOTIP detects targets across all frames at once during training, while attention masks enable parallelism in its ID Decoder.This decoupling is intended to improve GPU utilization and training efficiency; Figure 4 illustrates the process, including two DETR forward passes when gradients are computed for only a subset.Source: §B.1 · p. 9, Figure 4 · p. 9
  • SportsMOT annotations include athletes but exclude referees and spectators, whereas CrowdHuman also annotates crowds outside sports venues.Source: Figure 6 · p. 10

B.2. Inference

  • Inference assigns each detection its highest-confidence ID prediction, then labels lower-confidence duplicates within a frame as newborn objects.The authors state this rule avoids duplicate ID labels within a frame; Figure 5 provides pseudocode for the assignment process.Source: Figure 5 · p. 9, §B.2 · p. 10

B.3. Ablation Studies

The ablations compare in-context ID prediction with ReID and contrastive pipelines, and test one-stage versus two-stage training. Experiments use DanceTrack validation with specified default settings unless otherwise noted.

Default Settings

  • Ablations use DanceTrack training and official validation, with no trajectory augmentation, sequence length T = 19, and inference weights λdet = λnew = 0.5 and λid = 0.1.The defaults set λocc = λsw = 0.0; other unspecified settings follow the referenced defaults.Source: §B.3 · p. 10

Re-ID Pipeline

  • The ReID comparison assigns each training trajectory a fixed class label, uses cross-entropy supervision and a linear classification head, then matches IDs by cosine similarity.This differs from MOTIP’s in-context ID-label procedure.Source: §B.3 · p. 10

Contra Pipeline

  • The contrastive pipeline uses infoNCE supervision for trajectory embeddings and cosine similarity at inference; its similarity threshold is set to 0.1 for ablations.The authors report similar tracking performance below a similarity threshold of 0.5 and tune some hyperparameters for a thorough comparison.Source: §B.3 · p. 10

One-Stage and Two-Stage Training

  • One-stage training jointly supervises the network, whereas two-stage training first trains DETR for detection, freezes it, and then trains association.Freezing DETR is intended to keep its object embeddings consistent for comparing the three pipelines.Source: §B.3 · pp. 10–11

C. Rethinking Joint Training with Images

The authors question joint training with image datasets, citing differences in scenarios and annotations and the simplicity of simulated video clips. They argue that added data and training strategies can bring complexity, especially for end-to-end, long-sequence tracking.

C.1. Inconsistent Scenario Characteristics

  • CrowdHuman scenes can differ from target benchmarks such as DanceTrack and SportsMOT, while image data may help or misdirect training and be redundant when benchmark data is sufficient.CrowdHuman includes atypical low-light and wide-angle scenes; the authors note these scenario inconsistencies had not yet harmed target-benchmark performance.Source: §C.1 · pp. 11–12

C.2. Inconsistent Annotation Standards

  • CrowdHuman annotates all visible people, whereas SportsMOT focuses on athletes, creating ambiguous positive labels for spectators and referees during joint training.The authors argue that model-specific multi-stage training strategies add engineering complexity when sufficient training data is available.Source: §C.2 · p. 12

C.3. Simplistic Video Simulations

  • Random shifting simulates clips through translation and scaling, producing adjacent frames without real-sequence deformation, occlusion, or relative-position changes.The authors warn that this simplified simulation can contaminate training distributions and impair performance, particularly as sampling length increases.Source: §C.3 · p. 12

C.4. Discussions

  • For MOTIP, image-simulated video may provide insufficient benefit for long-sequence training, while joint training can expose annotation inconsistencies and model forgetting.The authors suggest avoiding extra training data when the dataset is sufficiently large, to keep effort focused on tracking challenges.Source: §C.4 · p. 12

D.1. MOT17

On MOT17, MOTIP improves over MOTR in HOTA, while the paper notes comparison caveats and a remaining gap to heuristic methods. Additional analyses cover inference speed, ID Decoder attention, and decoder depth.

D.1. MOT17

  • 59.3 HOTA vs. 57.2 HOTA: MOTIP outperforms MOTR on MOT17; Table 7 compares state-of-the-art methods and marks the best end-to-end result.The paper states that some gray-font results are unfair comparisons because of differences in network structure.Source: Table 7 · p. 13, §D.1 · p. 13

D.2. Inference Speed

  • On a single NVIDIA RTX A5000, MOTIP achieves near-real-time inference at FP16 precision and has similar inference speed to MOTR.The authors say the ID Decoder’s inference cost is negligible relative to Deformable DETR.Source: Table 8 · p. 13, §D.2 · p. 13

D.1. MOT17

  • MOT17 has only 7 video sequences for training; the authors use modern, diverse DanceTrack, SportsMOT, and BFT in the main text to ensure the model is well-trained, but report a significant gap from state-of-the-art heuristic algorithms.They attribute the gap partly to heuristic algorithms’ decade of customization for linear-motion scenarios and call for diverse, large-scale pedestrian tracking datasets.Source: §D.1 · pp. 12–13

D.3. Visualization of ID Decoder

  • Darker cross-attention shades indicate stronger target-to-tracklet responses; after object 5 reappears in frame 642, it selects features from frames 630 and 631, unlike unoccluded objects that select nearer frames.Object 5 is occluded in frames 638–641; targets marked with a red cross are not visible in the current frame.Source: Figure 9 · p. 14, §D.3 · p. 14

D.4. Number of ID Decoder Layers

  • 54.3 to 60.5 HOTA: tracking performance improves as ID Decoder layers increase, but returns diminish and excessive depth can hinder convergence; the default is 6 layers.Table 9 presents the layer ablation and marks the final-experiment choice.Source: Table 9 · p. 14, §D.4 · p. 14

D.5. Previous Results

The paper reports an earlier version at arXiv:2403.16848v1. Its model structure remained unchanged, but codebase and hyperparameter updates improved tracking performance in Section 4.3.

D.5. Previous Results

  • The paper has an earlier version at arXiv:2403.16848v1.The model structure remains unchanged between versions.Source: §D.5 · p. 14
  • Updated code and some hyperparameters improved tracking performance in Section 4.3 relative to the earlier version.Source: §D.5 · p. 14
  • The authors consider comparisons based on either version’s respective code framework reasonable and acceptable.Source: §D.5 · p. 14

Related papers

Loading 2403.16848v2…