PaperlayerView exact revision on arXiv

Source-linked AI summary

DiffMOT: A Real-time Diffusion-based Multiple Object Tracker with Non-linear Prediction

Weiyi Lv, Yuhang Huang, Ning Zhang, Ruei-Sung Lin, Mei Han, Dan Zeng

arXiv:2403.02075v2cs.CV

TL;DR

Non-linear, irregular object motion challenges tracking-by-detection systems that use Kalman Filter prediction, while prior neural approaches have faced accuracy or speed limits. DiffMOT applies decoupled diffusion to motion prediction, conditioning denoising on each object's history and using fewer sampling steps. It outperforms state-of-the-art methods on DanceTrack and SportsMOT, reports 22.7 FPS, and is comparable to state-of-the-art on MOT17.

  • Problem

    Tracking-by-detection with Kalman Filter prediction is suited to pedestrian-dominant scenes but faces complex, non-linear motion; earlier neural approaches have struggled to combine accurate prediction with real-time speed.

    Source: §1 · p. 2

  • Method

    DiffMOT uses a decoupled diffusion-based motion predictor to denoise bounding-box positions conditioned on prior motion trajectories, with one-step sampling.

    Source: §1 · p. 2

  • Results

    DiffMOT exceeds prior trackers on DanceTrack and SportsMOT, with 62.3% and 76.2% HOTA, respectively, and achieves comparable performance on MOT17.

    Source: §1 · p. 2, §5 · p. 8

  • Takeaways & Limitations

    The authors conclude that DiffMOT has potential for realistic applications.

    Source: §5 · p. 8

Abstract

from arXiv · show

In Multiple Object Tracking, objects often exhibit non-linear motion of acceleration and deceleration, with irregular direction changes. Tacking-by-detection (TBD) trackers with Kalman Filter motion prediction work well in pedestrian-dominant scenarios but fall short in complex situations when multiple objects perform non-linear and diverse motion simultaneously. To tackle the complex non-linear motion, we propose a real-time diffusion-based MOT approach named DiffMOT. Specifically, for the motion predictor component, we propose a novel Decoupled Diffusion-based Motion Predictor (D$^2$MP). It models the entire distribution of various motion presented by the data as a whole. It also predicts an individual object's motion conditioning on an individual's historical motion information. Furthermore, it optimizes the diffusion process with much fewer sampling steps. As a MOT tracker, the DiffMOT is real-time at 22.7FPS, and also outperforms the state-of-the-art on DanceTrack and SportsMOT datasets with $62.3\%$ and $76.2\%$ in HOTA metrics, respectively. To the best of our knowledge, DiffMOT is the first to introduce a diffusion probabilistic model into the MOT to tackle non-linear motion prediction.

1. Introduction

DiffMOT targets non-linear, irregular object motion that challenges constant-velocity prediction and prior methods' speed or adaptability. Its diffusion-based tracker uses D2MP for motion prediction and reports strong results on DanceTrack and SportsMOT at real-time speed.

1. Introduction

  • Constant-velocity predictors can miss acceleration, deceleration, and irregular direction changes in less synchronous scenes such as dance and sports.The introduction notes that Kalman Filter prediction fits the linear, constant-speed patterns common in pedestrian-dominant MOT17 [23].Source: §1 · p. 2
  • Earlier neural approaches had not achieved accurate non-linear prediction and real-time speed together, due to limited adaptation or heavy computation.The cited examples include MLP and LSTM methods [5], and optical-flow and transformer-based methods [34] [37].Source: §1 · p. 2
  • DiffMOT uses a diffusion probabilistic model to predict bounding-box positions by denoising, conditioned on previous bounding-box motion trajectories.At prediction time, each object's motion history guides denoising.Source: §1 · p. 2
  • D2MP uses one-step sampling rather than the standard thousand-step diffusion sampling, reducing inference time significantly.The paper reports DiffMOT at 22.7 FPS on an RTX 3090 machine.Source: §1 · p. 2
  • DiffMOT with YOLOX-X combines fast speed and superior performance, while the YOLOX-S version can run faster.Figure 1(b) contrasts it with methods reported to have lower FPS or lower HOTA.Source: §1 · p. 2
  • 62.3% and 76.2% HOTA on DanceTrack and SportsMOT, respectively, with DiffMOT outperforming state-of-the-art methods on both datasets.The tracker runs at 22.7 FPS on an RTX 3090 machine.Source: §1 · p. 2

2. Related Work

Prior MOT motion models include linear Kalman Filter approaches and neural methods for non-linear motion, while diffusion models offer a route to few-step sampling. DiffMOT's D2MP adapts decoupled diffusion to model non-linear motion with the stated aim of combining accuracy and speed.

Motion Model in MOT

  • The Kalman Filter assumes constant velocity and direction over a small time interval, and is used in trackers including SORT, DeepSORT, FairMOT, ByteTrack, and OC-SORT.Source: §2 · p. 2
  • Optical flow [37], LSTM [5], and transformers [34] model motion in different ways, but the paper says none simultaneously achieves accurate non-linear prediction and real-time speed.Source: §2 · p. 2

Diffusion Probabilistic Models

  • Conventional diffusion probabilistic models use standard thousand-step inference sampling, while recent work explores few-step sampling [15].Source: §2 · p. 2
  • DDM [15] splits diffusion into two sub-processes for few-step sampling; DiffMOT adapts this idea to motion prediction, unlike concurrent DiffusionTrack [21], which models relationships between paired boxes.DiffMOT focuses on non-linear motion modeling and learning the entire motion distribution.Source: §2 · p. 3

3. Method

DiffMOT follows a tracking-by-detection pipeline and uses D2MP to predict motion from historical trajectories. D2MP models motion through decoupled diffusion, with a one-step reverse process conditioned on historical information.

3.1. Framework of DiffMOT

  • DiffMOT detects current-frame boxes, predicts object positions from previous-frame trajectories with D2MP, then associates predicted and detected boxes to update trajectories.The framework has detection, motion prediction, and association stages.Source: §3.1 · p. 3

3.2. Decoupled Diffusion-based Motion Predictor

  • D2MP treats motion prediction as generation from a normal distribution conditioned on historical motion, using decoupled diffusion to model dataset-wide motion distributions.It separates the data-to-noise process into data-to-zero and zero-to-noise sub-processes.Source: §3.2 · p. 3

3.2.1 Forward process

  • The forward process represents motion as frame-to-frame bounding-box differences and maps clean motion toward noise through data-to-zero attenuation and zero-to-noise addition.The constant attenuation reaches zero at t = 1, while noise increases to pure normal noise.Source: §3.2.1 · p. 4

3.2.2 Reversed process

  • The reversed process predicts only cΘ, enabling one-step motion reconstruction instead of the thousands of steps typical diffusion models require.The decoupled process's analyticity supports one-step sampling.Source: §3.2.2 · pp. 4–5

3.2.3 Historical Memory Information Network (HMINet)

  • HMINet extracts a condition embedding from previous-frame motion with self-attention, fuses it with noisy motion features, and uses an MLP to predict cΘ.The conditioned input combines bounding boxes and movement information from previous frames.Source: §3.2.3 · p. 5

3.2.4 Training loss

  • Training uses ground-truth c to supervise Θ with a smooth L1 loss.The loss is quadratic for errors below 1 and linear otherwise.Source: §3.2.4 · p. 5

4. Experiments

Experiments report benchmark results on DanceTrack and SportsMOT, comparable MOT17 performance, and ablations and visualizations of motion prediction.

4.1. Datasets and Evaluation Metrics

  • The datasets test different motion regimes: DanceTrack has irregular direction changes, SportsMOT has acceleration and deceleration, and MOT17 is approximately linear.The experiments report HOTA, association and detection metrics, MOTA, and FPS.Source: §4.1 · pp. 5–6

4.2. Benchmark Evaluation

  • DiffMOT scores 62.3% HOTA on DanceTrack and 76.2% HOTA on SportsMOT, achieving the best reported results on both nonlinear benchmarks.The SportsMOT comparison includes detector setups trained on either the training set alone or the training and validation sets.Source: Table 2 · p. 6, §4.2 · pp. 6–7
  • On MOT17, DiffMOT achieves 64.5% HOTA and comparable performance to other state-of-the-art methods in pedestrian-dominant scenarios.The comparison uses the private-detection protocol and the same YOLOX detector for all methods.Source: Table 3 · p. 6, §4.2 · p. 7
  • On DanceTrack, YOLOX-S raises DiffMOT speed to 30.3 FPS, while detector refinements trade increased performance for lower FPS.The authors frame detector choice as a way to flexibly match different real-world application scenarios.Source: Table 4 · p. 7, §4.2 · p. 7
  • DiffMOT exceeds concurrent DiffusionTrack by 8.1% in HOTA on DanceTrack.The authors distinguish their approach by modeling the entire motion distribution and generating future motion from a normal distribution.Source: §4.2 · pp. 6–7

4.3. Ablation Studies

  • D2MP achieves 55.7% HOTA on DanceTrack validation, the best result among the tested motion models.The comparison includes KF, LSTM, Transformer-based models, and IoU-only association.Source: Table 5 · p. 7, §4.3 · p. 7
  • With one-step sampling, D2MP-OB beats D2MP-TB by 11.2% HOTA; 20-step D2MP-TB reaches 54.6% HOTA at 7.5 FPS.D2MP-OB is the one-branch architecture used in DiffMOT; D2MP-TB jointly optimizes cΘ and zΘ.Source: Table 6 · p. 7, §4.3 · pp. 7–8
  • Using I_f−1 as the condition gives the best results, and five historical frames produce the best motion-prediction results.The authors attribute the history-length pattern to insufficient direction from too little history and interference from too much.Source: Table 7 · p. 8, Table 8 · p. 8, §4.3 · p. 8

5. Conclusion

DiffMOT targets nonlinear-motion tracking with a decoupled diffusion predictor. The authors report that it exceeds previous trackers on two nonlinear datasets and is comparable to state-of-the-art methods on MOT17.

5. Conclusion

  • DiffMOT is a diffusion-based tracker designed for objects in nonlinear motion.The paper identifies its decoupled diffusion motion predictor as the design for handling more complex nonlinear patterns.Source: §5 · p. 8
  • DiffMOT exceeds all previous trackers on DanceTrack and SportsMOT.Source: §5 · p. 8
  • On pedestrian-dominant MOT17, DiffMOT achieves performance comparable to state-of-the-art methods; the authors say it has potential in realistic applications.Source: §5 · p. 8

Non-linear Prediction Supplementary Material

The supplement outlines decoupled diffusion and its one-step sampling option, then gives the tracker’s detection, motion-prediction, and association workflow.

A. Preliminary

  • Decoupled diffusion splits its forward process into analytic signal attenuation and an increasing normal-noise process.Source: A. Preliminary · p. 11
  • D2MP uses constant attenuation, f_t=c, and trains neural estimates of f and z against those quantities.The prose identifies z as noise added to x0 and says the reversed process must parameterize both z and f because they are unknown there.Source: A. Preliminary · p. 11
  • Setting ∆t=t=1 permits one-step sampling from x1 toward x0, avoiding iterative generation.Source: A. Preliminary · p. 11

B. Pseudo-code of DiffMOT

  • DiffMOT detects boxes, predicts prior-track boxes with D2MP, then matches high-score detections using re-identification features and IoU and remaining low-score detections using IoU.Matched detections update tracks; unmatched tracks are deleted and unmatched detections initialize new tracks.Source: B. Pseudo-code of DiffMOT · p. 11

C. Benchmark Evaluation on MOT20

The MOT20 supplement reports DiffMOT’s results under the private-detector protocol and includes tracker pseudocode for the inference workflow.

C. Benchmark Evaluation on MOT20

  • On pedestrian-dominant MOT20 test sets, where density is higher and motion more closely approximates linear, DiffMOT under the private-detector protocol achieves 61.7% HOTA, 74.9% IDF1, 60.5% AssA, 76.7% MOTA, and 63.2% DetA.Source: C. Benchmark Evaluation on MOT20 · p. 11
  • The pseudocode takes a video sequence, detector, HMINet model, and score thresholds as inputs and outputs the video’s tracks.Source: C. Benchmark Evaluation on MOT20 · p. 12
  • The association pseudocode matches detections with predicted boxes using re-identification features and IoU, then deletes unmatched tracks and initializes unmatched detections.Source: C. Benchmark Evaluation on MOT20 · p. 12

D. Generalization of D2MP

D2MP trained on DanceTrack generalizes to other datasets with modest metric gaps and reaches state-of-the-art performance on SportsMOT. Figure 6 presents paired KF and D2MP results for DanceTrack sequences.

D. Generalization of D2MP

  • 0.6% and 1.0% lower HOTA and IDF1, respectively, on SportsMOT; DanceTrack-trained D2MP is 2.7% / 0.5% lower on MOT17 and 4.9% / 1.1% lower on MOT20 in HOTA / IDF1.These comparisons are against models trained on the corresponding datasets.Source: D. Generalization of D2MP · p. 12
  • On SportsMOT, DanceTrack-trained D2MP exceeds MixSort-OC by 1.4% HOTA, 0.7% IDF1, and 0.6% MOTA.The passage reports this as state-of-the-art performance on SportsMOT.Source: D. Generalization of D2MP · p. 12
  • Figure 6 pairs KF and D2MP tracking results for each DanceTrack sequence, with same-colored boxes denoting the same ID.Each pair of rows shows one sequence.Source: Figure 6 · p. 13

E. Architecture of D2MP-TB

D2MP-TB uses a two-branch architecture that predicts zΘ and cΘ separately, distinguishing it from D2MP-OB.

E. Architecture of D2MP-TB

F.1. Qualitative Comparison on DanceTrack

Qualitative comparisons show D2MP handling non-linear motion more robustly than KF on DanceTrack and SportsMOT. On SportsMOT, the examples come from moving-camera scenarios.

F.1. Qualitative Comparison on DanceTrack

  • Trackers using KF show discontinuities, lost tracks, and ID switches in DanceTrack examples, while D2MP is more robust to non-linear motion.Reported KF examples include ID changes and a lost track; the comparison attributes these errors to inaccurate trajectory-position prediction.Source: §F.1 · p. 13

F.2. Qualitative Comparison on SportsMOT

  • On SportsMOT, KF often misses object positions during acceleration or deceleration, whereas D2MP predicts positions accurately in the shown scenarios.The examples include football, volleyball, and basketball scenes; the figure cases use moving cameras.Source: §F.2 · p. 13, Figure 7 · p. 14

F.3. Visual Results on MOT17/20

DiffMOT produces appealing visual results on MOT17 and MOT20 despite being designed for non-linear-motion scenes. The authors identify long-term matching as an area for future improvement.

F.3. Visual Results on MOT17/20

  • The authors plan to explore multi-frame trajectory generation to improve the motion model’s capacity for long-term matching.Source: §F.3 · p. 14
  • Inaccurate object-position prediction can lead to new IDs or ID switches.Source: §F.3 · p. 14
  • DiffMOT achieves appealing visual results on MOT17 and MOT20 despite being designed specifically for non-linear-motion scenes.Source: §F.3 · p. 14

G. Illustration of Failure Cases

The failure cases expose two identity-association weaknesses: identity switches when objects cross and difficulty re-associating objects after extended losses.

G. Illustration of Failure Cases

  • Identity switches occur when objects pass through each other; the authors attribute them to the motion model’s lack of velocity-direction constraints.They suggest that constraining predicted boxes by velocity direction could help address this issue.Source: G. Illustration of Failure Cases · p. 14, Figure 10 · p. 16
  • Extended losses can defeat re-association: lost objects may reappear under new IDs, reflecting difficulty recovering long-term lost objects.The examples include an object reappearing as ID 7 after disappearing, and other ID changes.Source: G. Illustration of Failure Cases · p. 14, Figure 10 · p. 16
  • Figure 10’s first and second rows show identity switches and new IDs, respectively.Source: Figure 10 · p. 16

Related papers

Loading 2403.02075v2…