PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond the Image Plane: World-Grounded Queries for Multi-Object Tracking

Orcun Cetintas, Guillem Brasó, Tim Meinhardt, Laura Leal-Taixé

arXiv:2609.00924v1cs.CVcs.AI

TL;DR

Image-plane tracking obscures 3D relationships and limits the evidence available for object association, especially across temporal gaps. PLANET lifts tracking data into 3D, embeds reconstructed geometry into world-grounded queries, adds 3D localization supervision, and uses dual-resolution memory. It achieves state-of-the-art performance across three diverse benchmarks.

  • Problem

    Image-plane projections obscure depth and spatial configuration, while end-to-end trackers have limited temporal context for retaining identity-relevant evidence.

  • Method

    PLANET lifts existing 2D tracking datasets into 3D, forms queries from reconstructed scene geometry with auxiliary 3D localization, and preserves evidence using dual-resolution temporal memory.

  • Results

    PLANET achieves state-of-the-art performance across DanceTrack, SportsMOT, and BFT, including 72.1 HOTA on DanceTrack and a 2.5-point improvement over MOTIP.

  • Takeaways & Limitations

    The results support 3D information as a valuable source of evidence for multi-object tracking across diverse scenarios.

  • Takeaways & Limitations

    End-to-end Transformer trackers commonly operate with short training contexts of only two to ten frames, motivating PLANET’s extended temporal coverage.

Abstract

from arXiv · show

Monocular videos record 3D scenes as sequences of 2D image-plane projections, obscuring depth and spatial relationships. Multi-object trackers localize and associate objects primarily using appearance and geometry observed only in the image plane, inheriting these ambiguities. To address this limitation, we introduce PLANET, an end-to-end multi-object tracker designed to move beyond the image plane. As an enabling step, we lift existing 2D tracking datasets into 3D. We then form world-grounded queries by embedding reconstructed 3D scene geometry into the features and positional encodings used during query formation. An auxiliary 3D location prediction task further encourages the queries to encode object positions during training. A complementary dual-resolution temporal memory preserves this evidence across longer temporal gaps. As a result, PLANET achieves state-of-the-art performance across three diverse benchmarks.

1. Introduction

PLANET addresses the geometric ambiguity of image-plane tracking by incorporating reconstructed 3D scene information into an end-to-end tracker. It also extends temporal access to preserve world-grounded evidence across longer gaps and achieves state-of-the-art performance on three benchmarks.

  • Motivation: Monocular projections obscure depth and spatial configuration, causing overlap and ambiguous physical motion that image-plane trackers inherit.Prior 3D cues aid occlusion reasoning, camera-motion compensation, trajectory forecasting, and association, but their integration into trackers remains underexplored.
  • Approach: PLANET lifts DanceTrack, SportsMOT, and BFT into 3D using dense world-coordinate point maps and pseudo-3D object centers.Depth Anything 3 supplies the reconstructed geometry used in this enabling stage.
  • Approach: 3D embeddings and auxiliary 3D location prediction form world-grounded queries that combine scene geometry with object-level 3D supervision.These queries enrich the representation shared by detection and association.
  • Temporal memory: Dual-resolution temporal memory preserves recent observations densely and older observations sparsely, expanding the accessible horizon from 30 to approximately 200 frames under the same context budget.This keeps spatial evidence available for association after occlusions.
  • Results: PLANET achieves state-of-the-art performance across DanceTrack, SportsMOT, and BFT, while ablations examine its 3D and memory components.The benchmarks cover challenging human motion, sports scenes, and dense non-human motion.

2. Related Work

Prior work either separates detection from association, uses category-specific or external 3D components, or faces short temporal contexts in end-to-end tracking. PLANET instead grounds shared queries in reconstructed geometry and uses dual-resolution memory to extend temporal support within a unified tracker.

  • Tracking paradigms: Tracking-by-detection performs per-frame detection followed by a separately designed association stage using appearance, image-plane motion, confidence, or learned motion cues.End-to-end Transformer trackers jointly optimize detection and identity association through object and track queries.
  • Temporal limitations: End-to-end trackers face a temporal-context bottleneck because longer query histories increase computation and can dilute distant identity-relevant evidence.Recent Transformer trackers commonly train with contexts of only two to ten frames.
  • PLANET: PLANET’s dual-resolution memory retains dense short-term and sparse long-term context, extending temporal support for identity prediction under a fixed context budget.This complements its geometry-grounded shared query representation.
  • 3D cues: Earlier 3D-cue methods support occlusion reasoning or camera-motion compensation but rely on category-specific models or ego-motion assumptions.Other approaches handle geometry through separate forecasting, search, or re-identification modules.
  • 3D cues: Conventional pipelines introduce geometry during downstream association through handcrafted costs, filtering, heuristics, or learned likelihoods rather than unified object representations.This preserves a multi-stage design instead of end-to-end integration.
  • PLANET: PLANET introduces reconstructed scene geometry directly into query formation, combining dense point maps and auxiliary 3D center localization before association.The approach avoids specialized category-specific models, external forecasting, and handcrafted cues.

3. Methodology

PLANET extends end-to-end tracking beyond image-plane evidence by lifting 2D datasets into 3D, grounding queries in scene geometry, and preserving evidence across longer temporal gaps.

  • Method Overview: PLANET builds on MOTIP by using shared detector-query representations for bounding-box prediction and identity association.
  • Lifting 2D Tracking Datasets into 3D: The lifted datasets include reconstructed scene information and additional annotations intended to support future research within and beyond tracking.
  • Lifting 2D Tracking Datasets into 3D: Existing 2D tracking datasets are lifted into 3D using dense world-coordinate point maps and pseudo-3D object centers assigned to annotated 2D boxes.
  • World-Grounded Queries: Geometric query lifting adds dense scene geometry to query formation, while anchored 3D localization trains matched queries to encode object-specific 3D positions.
  • World-Grounded Queries: Together, geometric query lifting and anchored 3D localization form world-grounded queries that combine visual evidence with estimated 3D scene information.
  • Dual-Resolution Temporal Memory: Dual-resolution temporal memory retains recent observations densely and older observations sparsely, extending identity-model context from 30 frames to approximately 200 frames.

4.1. Datasets and Metrics

The evaluation covers human, sports, and dense non-human motion, using HOTA, IDF1, and MOTA to assess tracking quality and identity preservation.

  • Datasets: DanceTrack contains 100 group-dancing videos featuring similar appearances, complex motion, crossovers, and occlusions.
  • Datasets: SportsMOT comprises 240 basketball, volleyball, and football sequences totaling more than 150,000 frames and 1.6 million annotated bounding boxes.
  • Metrics: The reported metrics are HOTA, IDF1, and MOTA, with higher values indicating better performance.
  • Metrics: HOTA combines detection and association accuracy, while IDF1 emphasizes identity preservation through globally assigned identities.
  • Metrics: MOTA combines false negatives, false positives, and identity mismatches, emphasizing detection coverage more than long-term identity preservation.

4.2. Implementation Details

PLANET uses a prior-aligned detector and training setup, while its lightweight geometric additions retain runtime comparable to MOTIP.

  • Model and Initialization: Experiments use Deformable DETR with a ResNet-50 backbone and COCO-pretrained detector weights, matching established prior settings.
  • Training: Training follows MOTIP with AdamW, an initial learning rate of 1 × 10−4, weight decay of 5×10−4, and multiple data augmentations.
  • Training: Each ablation configuration is trained five times and reported as a mean to reduce sensitivity to run-to-run variation.
  • Runtime: PLANET processes video at 12 FPS in FP32 and 20 FPS in FP16, retaining runtime comparable to MOTIP.

4.3. Ablations

The ablations show that geometric query lifting, anchored 3D localization, and dual-resolution temporal memory provide complementary tracking gains. Anchored integration outperforms direct 3D localization, while temporal memory is especially effective with world-grounded queries.

  • Component ablations: Geometric query lifting improves every reported metric over the base model, including a 1.0-point HOTA gain.The comparable association and detection gains indicate benefits to both tracking aspects.
  • Component ablations: Adding anchored 3D localization yields a further 0.4-point HOTA improvement, with a larger association gain than detection gain.The result supports using 3D localization to help resolve occlusion-related association ambiguities.
  • Component ablations: Dual-resolution temporal memory adds another 1.2 HOTA points and improves association while leaving detection accuracy unchanged.The gain reflects access to longer-term temporal information supported by stable 3D spatial cues.
  • Interaction between 3D awareness and temporal memory: For the 3D-aware model, dual-resolution memory improves HOTA by 1.2 points, IDF1 by 2.5 points, and MOTA by 0.4 points.The corresponding 2D-aware gains are smaller: 0.8 HOTA points and 1.5 IDF1 points, with unchanged MOTA.
  • Direct versus anchored 3D localization: Anchored 3D localization outperforms direct 3D localization across all metrics, improving HOTA by 0.6 points and exceeding the baseline by 0.4 points.Anchoring the 3D refinement to the detector’s existing 2D reference point avoids learning 3D localization from scratch.
  • 3D cue integration: World-grounded queries improve HOTA by 1.4 points over 3D foundation features, with gains in both association and detection accuracy.The formulation combines geometric query lifting with anchored 3D localization.

4.4. Benchmark Results

PLANET is evaluated under a benchmark-specific training protocol without external training data on DanceTrack, SportsMOT, and BFT. It achieves state-of-the-art performance across all three benchmarks, including strong gains over MOTIP on DanceTrack and SportsMOT.

  • Evaluation protocol: PLANET is trained exclusively on each corresponding benchmark without pretraining or joint training on related tracking or detection datasets.This follows MOTIP’s evaluation protocol and dataset-specific training scheme.
  • Evaluation protocol: The protocol excludes external training data, although several competing methods use additional supervision.The comparison therefore includes methods trained under less restrictive data settings.
  • Overall results: PLANET achieves state-of-the-art performance on all three benchmarks and outperforms methods with and without external training data.The evaluated benchmarks are DanceTrack, SportsMOT, and BFT.
  • DanceTrack: 72.1 HOTA on DanceTrack exceeds MOTIP by 2.5 points, while IDF1 rises from 74.7 to 78.0.PLANET improves every reported metric over the strongest end-to-end baseline in this setting.
  • SportsMOT: 73.7 HOTA on SportsMOT is 1.1 points above MOTIP, while DetA increases from 83.5 to 86.0 and MOTA from 92.4 to 95.2.The benchmark tests fast, variable player motion and similar appearances across sports scenes.

5. Conclusion

PLANET grounds object queries in reconstructed 3D scene geometry and retains that information across longer temporal horizons. Its ablations and benchmark results support complementary gains from world-grounded queries and dual-resolution temporal memory across diverse tracking scenarios.

  • Conclusion: PLANET moves beyond image-plane reasoning by grounding object queries in reconstructed 3D scene geometry and retaining information across longer temporal horizons.The framework incorporates 3D information directly into an end-to-end 2D tracker.
  • Conclusion: Ablation studies validate each component and show that world-grounded queries and dual-resolution temporal memory provide complementary gains.The conclusion identifies both components as contributing to the overall framework.
  • Conclusion: Benchmark results demonstrate state-of-the-art performance across diverse tracking scenarios, supporting the generality of the approach.The reported conclusion spans the evaluated benchmark settings.
Loading 2609.00924v1…