PaperlayerView exact revision on arXiv

Source-linked AI summary

MOTIP2: Spatial Priors for End-to-End Multi-Object Tracking

Benoît Roussel, Damien Bouet, Liming Chen, Pierre Perrault

arXiv:2610.10391v1cs.CV

TL;DR

End-to-end trackers still make spatially implausible association errors, while scarce tracking annotations limit how well they learn spatial constraints. MOTIP2 adds three spatial priors to a real-time DEIM-based tracker, reporting gains across benchmarks and faster matched-accuracy inference than MOTIP.

  • Problem

    End-to-end trackers can make spatially implausible identity associations, while scarce tracking annotations make it difficult to learn spatial constraints from data alone.

    Source: §1 · p. 2

  • Method

    MOTIP2 adds spatial priors at the data, loss, and representation stages to a real-time DEIM-based tracker adapted from MOTIP.

    Source: §1 · pp. 2–3

  • Results

    MOTIP2 reports consistent gains across three benchmarks; with a fixed backbone, its priors add +2.1 HOTA and +3.5 AssA on DanceTrack test.

    Source: §1 · p. 3

  • Takeaways & Limitations

    MOTIP2 runs at real-time rates, over 3× faster than MOTIP at matched accuracy, without a post-hoc association module.

    Source: §1 · p. 3

    MOTIP2’s changes are confined to the detector, ID decoder, and training objective; everything else is inherited unchanged.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

End-to-end multi-object trackers have narrowed the gap with classical tracking-by-detection on association-difficult benchmarks. Yet they still make spatially implausible errors no classical tracker would, such as assigning one identity to objects on opposite sides of the frame. A model could learn to avoid them, but tracking annotations are scarce, so we encode spatial priors explicitly instead, while keeping inference fully end-to-end with no post-hoc association. We propose three spatial priors, at the data, loss, and representation stages. Spatial ID Switches bias trajectory permutations toward spatially overlapping objects, reducing the mismatch between training and inference confusions. Spatial ID Loss scales each identity's penalty by its box distance, so a distant switch costs more than a nearby one. Spatial Anchor gives each track token its frame position, an explicit spatial cue for attention. We instantiate the three priors in MOTIP2, a tracker adapted from MOTIP and built on the real-time DEIM detection transformer. Trained without extra data, its main model, MOTIP2-L, sets a new state of the art: 73.4 HOTA on DanceTrack, 76.0 on SportsMOT, and 71.1 IDF1 on PersonPath22. MOTIP2 is a family of models spanning the speed-accuracy trade-off: a lighter model, MOTIP2-S, matches the original MOTIP at over 3x the speed, and MOTIP2-X reaches 74.8 HOTA on DanceTrack.

1 Introduction

MOTIP2 addresses spatially implausible identity associations with priors at the data, loss, and representation stages. It combines them in a real-time DEIM-based tracker and reports gains across three benchmarks.

  • 3.2× faster than MOTIP at matched accuracy; with a fixed backbone, the priors add +2.1 HOTA and +3.5 AssA on DanceTrack test.MOTIP2 validates the priors with consistent gains across three tracking benchmarks.Source: Figure 1 · p. 2, §1 · p. 3
  • Spatial ID Switches bias tracklet permutations toward spatially overlapping objects, better matching the nearby confusions encountered at inference.This augmentation trains the decoder to look further back in the tracklet when nearby objects are present.Source: §1 · pp. 2–3
  • The three priors restore spatial grounding without adding hand-crafted matching at test time or a post-hoc association module.Source: §1 · p. 3
  • Spatial ID Loss scales each candidate identity’s penalty by its box distance, restoring a geometric cost that cross-entropy omits.A distant jump is penalized more than a nearby confusion.Source: §1 · pp. 2–3
  • Spatial Anchor reinjects the detector’s positional encoding into the ID decoder, giving each track token an explicit spatial cue for association.Source: §1 · pp. 2–3

2 Related Work

Related work contrasts explicit geometric heuristics in tracking-by-detection with learned association in transformer trackers. It also traces how detection transformers retain spatial anchors while query-based tracking can leave them out of association.

Tracking-by-detection

  • Tracking-by-detection combines Kalman motion, IoU costs, and Hungarian matching, but crowding weakens the geometric cues that make these heuristics discriminative.Re-identification scores are commonly gated by motion or overlap; these pipelines also delay confirming tentative tracks until consecutive observations.Source: §2 · p. 3

Detection Transformers

  • Detection transformers progressively ground queries in geometry, and MOTIP2 builds on DEIM for its accuracy at real-time latency.Examples include reference points, 4D anchor boxes, and box-denoising objectives.Source: §2 · p. 3

Tracking-by-query

  • MOTIP separates identity prediction into a dedicated decoder, but passes detector content embeddings without the explicit spatial anchor used inside the detector.FDTA is identified as MOTIP2’s primary baseline; related work also notes that transformer trackers remain slower than tracking-by-detection.Source: §2 · p. 4

3 Spatial Priors for ID Association

MOTIP2 adapts MOTIP with three spatial priors that add geometry at the data, loss, and representation stages, while retaining end-to-end inference without post-hoc association.

3.1 Overview: MOTIP2

  • MOTIP decouples detection from identity prediction with a dedicated ID decoder that predicts identity distributions from a historical context.Inference processes video online, appending assigned embeddings to a FIFO context buffer and using a greedy protocol for unique identity assignments.Source: §3.1 · p. 4
  • MOTIP2 replaces MOTIP’s Deformable DETR detector with DEIM and changes decoder fusion to addition plus RoPE temporal attention.At inference, RoPE timescale is rescaled to match the average training timescale.Source: §3.1 · p. 5

3.2 Spatial ID Switches

  • Spatial ID Switches replace uniform reassignment with noisy GIoU-biased Hungarian permutations, concentrating training switches among nearby objects.The augmentation uses σ=0.5 across benchmarks without per-dataset tuning, and well-separated objects remain virtually untouched.Source: §3.2 · p. 6

3.3 Spatial ID Loss

  • Spatial ID Loss complements spatially blind ID cross-entropy with L1 and GIoU penalties weighted by distance to each candidate identity’s box.Probabilities are renormalized over identities present in the frame plus the newborn class; detached predicted boxes leave detector regression unchanged.Source: §3.3 · pp. 6–7

3.4 Spatial Anchor

  • Spatial Anchor carries detector positional encoding into each ID-decoder token, providing an explicit frame-position cue alongside content.The detector’s positional encoding already steers cross-attention toward image regions, but was dropped when detections were handed to the ID decoder.Source: §3.4 · p. 7

4 Experiments

The experiments evaluate MOTIP2 on three benchmarks, compare accuracy and latency, and ablate its spatial priors and decoder/inference choices. Models are trained per benchmark, with the PersonPath22 model also co-trained with CrowdHuman.

4.1 Experimental Setup

  • Models are trained per benchmark; the PersonPath22 model also uses CrowdHuman, and evaluation reports HOTA alongside MOTA and IDF1.MOTIP2-L is the default for state-of-the-art comparisons, while MOTIP2-S is used for ablations.Source: §4.1 · pp. 7–8

4.2 Comparisons with State-of-the-art Methods

  • 74.8 HOTA for MOTIP2-X on DanceTrack is the best result without extra data; with extra data, MOTIP2-X† reaches 77.1 HOTA.MOTIP2-L reaches 73.4 HOTA, above FDTA’s 71.7; the table identifies the test-set comparison and extra-data protocol.Source: §4.2 · p. 8, Table 1 · p. 9
  • 76.0 HOTA for MOTIP2-L on SportsMOT leads end-to-end trackers, while 71.1 IDF1 on PersonPath22 exceeds TrackFormer’s 57.1 and Path Consistency’s 69.4.On PersonPath22, MOTIP2-L matches Path Consistency’s MOTA at 75.8; prior work there does not report HOTA-family metrics.Source: §4.2 · pp. 9–10, Table 2 · p. 10, Table 3 · p. 10

4.3 Speed and Accuracy

  • MOTIP2-S matches MOTIP’s accuracy at over 3× the speed (27.1 vs 86.9 ms), and its TensorRT FP16 latency reaches 10.1 ms on a T4.The latency breakdown profiles DEIM and the ID decoder separately; decoder latency grows roughly linearly with the number of tracked objects in trajectory history.Source: §4.3 · p. 11, Table 4 · p. 11

4.4 Ablation Studies

  • Removing all three spatial priors costs −2.9 HOTA and −5.8 AssA on PersonPath22, and −1.5 HOTA and −2.5 AssA on DanceTrack.The ablation table uses MOTIP2-S and compares removals against the full model; the data-stage setting differs between benchmarks.Source: §4.4 · p. 12, Table 5 · p. 12
  • −2.1 HOTA follows removing Spatial ID Switches on PersonPath22; spatial sampling recovers +5.5 HOTA over uniform permutations, whose penalty is only −0.8 HOTA on DanceTrack.Removing the augmentation costs −4.4 HOTA on DanceTrack; uniform permutations are harmful on dispersed PersonPath22 objects but mostly nearby swaps among clustered dancers.Source: §4.4 · pp. 12–13
  • Removing Spatial ID Loss costs −1.0 HOTA on DanceTrack, while removing Spatial Anchor costs −0.5 HOTA there and no HOTA on PersonPath22.Spatial ID Loss also costs −0.2 HOTA on PersonPath22; Spatial Anchor reuses detector positional encoding without adding parameters or supervision.Source: §4.4 · p. 13

4.5 Decoder and Inference Ablations

  • RoPE adds +1.0 AssA and +0.3 HOTA at its best timescale, while tracklet confirmation from Ninit=1 to 3 cuts identity switches by 12.5% on PersonPath22 and 28.3% on DanceTrack.The adopted RoPE timescale is α=2.0; tracklet-confirmation accuracy plateaus at Ninit=2, which is the adopted setting.Source: §4.5 · pp. 13–14

5 Conclusion

MOTIP2 uses three spatial priors to address mainly geometric association errors, with their relative value depending on scene geometry. The resulting tracker family spans a speed-accuracy trade-off and reports new state-of-the-art identity association across three benchmarks.

5 Conclusion

  • MOTIP2 offers a family spanning the speed-accuracy trade-off, with variants that set a new state of the art in identity association across all three benchmarks.The conclusion describes the family as ranging from a real-time model to higher-performing variants.Source: §5 · p. 14
  • In wide, crowded scenes, restricting confusions to nearby boxes recovers +5.5 HOTA on PersonPath22 over uniform shuffling.The conclusion says uniform shuffling trains on implausible long-distance swaps and makes training unstable.Source: §5 · p. 14
  • The data-stage prior is decisive in wide, crowded scenes, while the loss- and representation-stage priors matter more when objects look nearly identical.The loss makes confident long-distance errors cost more; the representation prior ties each track to its image position.Source: §5 · p. 14
  • The authors expect simultaneous exit and entry at the same location to be the hardest case, because proximity can draw the leaving identity onto the new object.Source: §5 · p. 14
  • The paper argues that geometric errors are the main association errors separating end-to-end trackers from tracking-by-detection.Source: §5 · p. 14

A Training Details

Training uses a two-stage detector-pretraining and joint-training schedule, with settings varying by model size. Augmentation covers images and trajectories, including spatially weighted identity permutations.

Optimizer Configuration

  • AdamW uses model-size-dependent learning rates, twice the base rate for the ID decoder, model-specific backbone multipliers, and gradient clipping at 0.1.EMA decay is 0.9995; warmup is absent during pretraining and lasts 1000 steps during joint training.Source: A Training Details · p. 23

Learning Rate Schedule

  • Training first pretrains the DEIM detector alone, then jointly trains the detector and ID decoder with a 500-iteration linear warmup.Both stages use a constant learning rate and 40k samples per epoch.Source: A Training Details · p. 23

Batch Size

  • For S/M/L/X, batch sizes are 8 images per GPU during pretraining and 2 clips during joint training; N uses 32 images and 4 clips per GPU, without gradient accumulation.Across four GPUs, these settings give global batches of 32 images and 8 clips for S/M/L/X, and 128 images and 16 clips for N.Source: A Training Details · p. 23

Clip Sampling

  • Clips contain 30 frames, or 60 for SportsMOT, with temporal intervals sampled from 1 to 4; the detector loss uses 4 sampled frames, or 8 for N.The ID decoder receives the full clip.Source: A Training Details · p. 23

Data Augmentation Details

  • Augmentation combines image transformations and multi-scale resizing with trajectory-level Spatial ID Switches weighted by pairwise GIoU affinity and random occlusion masking.Spatial ID Switches and random occlusion masking are applied with probabilities 1.0 and 0.5, respectively; image-level operations include flips, zoom-out, crops, and photometric changes.Source: A Training Details · pp. 23–24

Loss Weights

  • Detection losses weight classification at 1.0, box L1 at 5.0, GIoU at 2.0, FGL at 0.15, and DDF at 1.5; ID cross-entropy has weight 1.0.ID cross-entropy is weighted per sample by predicted-to-ground-truth IoU, emphasizing well-localized detections.Source: A Training Details · p. 24

B Reference Comparison on MOT17

On MOT17, MOTIP2 improves over MOTIP on every reported metric, while association gains remain limited when scaling from L to X.

B Reference Comparison on MOT17

  • MOTIP2 improves over MOTIP on every MOT17 test metric, including +1.0 AssA and +1.4 IDF1 at L.At X, the gains concentrate in detection, with +1.9 DetA.Source: B Reference Comparison on MOT17 · p. 24
  • Scaling from MOTIP2-L to MOTIP2-X leaves AssA flat at 58.0, which the authors attribute to association being data-bound rather than capacity-bound.CrowdHuman supplies detection-only supervision, leaving MOT17’s 7 training sequences as the sole association supervision.Source: B Reference Comparison on MOT17 · p. 24
  • The authors identify this regime as motivation for focusing on PersonPath22, which has 138 training videos.Source: B Reference Comparison on MOT17 · p. 24

C Training Stability of Spatial ID Switches

On PersonPath22, GIoU-based Spatial ID Switches train stably and reach the best HOTA, unlike uniform permutations, which destabilize training. Their sampled swaps are more spatially plausible and produce fewer long-range identity switches at inference.

C Training Stability of Spatial ID Switches

  • GIoU-based sampling stays stable across joint-training epochs and reaches the best HOTA, while uniform permutations diverge and finish below the no-augmentation baseline.The authors link the instability to training on implausible histories that destabilize the jointly trained detector.Source: C Training Stability of Spatial ID… · p. 24
  • Spatial ID Switches focus swaps on overlapping objects, with mean GIoU 0.20/0.15, while uniform permutations mostly swap non-overlapping boxes with GIoU ≤0.At inference, uniform permutations produce 2.2×/1.7× the no-augmentation long-range ID-switch counts; Spatial ID Switches stay close to the no-augmentation level.Source: C Training Stability of Spatial ID… · p. 24

D A Long-Distance Identity Switch

Without spatial priors, an identity jumps across a crowded DanceTrack frame and merges distant dancers; with all three priors, the full model keeps both identities stable.

D A Long-Distance Identity Switch

  • On PersonPath22, GIoU-based ID-switch sampling trains stably to the highest HOTA, while MOTIP’s uniform permutation diverges below no augmentation.Figure 5 compares these variants over joint-training epochs for MOTIP2-S.Source: Figure 5 · p. 25
  • Spatial ID Loss most directly targets this failure by restoring the geometric cost omitted by cross-entropy.With all three spatial priors active, the full model keeps both identities stable.Source: D A Long-Distance Identity Switch · p. 25
  • Identity 2 jumps nearly 975 px between consecutive DanceTrack frames, merging distant dancers despite no occlusion or plausible motion path.The full model keeps both identities stable.Source: D A Long-Distance Identity Switch · pp. 24–25, Figure 6 · p. 26

E Tracklet Confirmation Sweep

Tracklet confirmation reduces identity switches while leaving HOTA essentially unchanged; the sweep selects Ninit=2 as the smallest threshold capturing the gain.

E Tracklet Confirmation Sweep

  • Identity switches fall 12.5% on PersonPath22 (5612 →4911) and 28.3% on DanceTrack (3287→2356) as Ninit rises from 1 to 3.The sweep uses MOTIP2-S on PersonPath22 test and DanceTrack val.Source: E Tracklet Confirmation Sweep · p. 26, Table 9 · p. 27
  • HOTA remains essentially unchanged, with slight IDF1 and MOTA gains, while accuracy plateaus at Ninit=2.Identity-switch reductions are only marginal beyond Ninit=2.Source: E Tracklet Confirmation Sweep · p. 26
  • Ninit=2 is adopted as the smallest threshold that captures the switch-reduction gain.Accuracy plateaus at 2, and identity switches fall only marginally beyond it.Source: E Tracklet Confirmation Sweep · p. 26
  • Ninit is the number of consecutive frames a new tracklet must be matched before confirmation.The sweep tests Ninit on MOTIP2-S, using PersonPath22 test and DanceTrack val.Source: Table 9 · p. 27

F Latency Comparison with Baselines

MOTIP2-S is substantially faster than MOTIP at matched accuracy in the reported latency comparison, while MOTIP2-N is lighter still.

F Latency Comparison with Baselines

Loading 2610.10391v1…