PaperlayerView exact revision on arXiv

Source-linked AI summary

OmniTracker: Unifying Object Tracking by Tracking-with-Detection

Junke Wang, Zuxuan Wu, Dongdong Chen, Chong Luo, Xiyang Dai, Lu Yuan, Yu-Gang Jiang

arXiv:2303.12079v2cs.CV

TL;DR

Different tracking tasks have led to customized methods, complex training, and redundant parameters. OmniTracker combines tracking and detection in a unified model, achieving state-of-the-art or competitive results across tracking tasks. It uses shared architecture, weights, and inference across five tracking tasks.

  • Problem

    Instance and category tracking use divergent setups and customized methods, resulting in complex training and redundant parameters.

    Source: §1 · p. 1

  • Method

    OmniTracker uses tracking-with-detection, with tracking supplying appearance priors for detection, and shares its architecture, weights, and inference pipeline across tracking tasks.

    Source: §1 · pp. 1–2

  • Results

    Across seven tracking benchmarks, OmniTracker achieves new state-of-the-art or at least competitive results on various tracking tasks.

    Source: §1 · p. 3

  • Takeaways & Limitations

    One shared OmniTracker model addresses SOT, VOS, MOT, MOTS, and VIS with a common architecture, weights, and inference pipeline.

    Source: §1 · p. 2, §5 · p. 12

    On VOS, OmniTracker still trails task-specific models; the authors attribute the gap to their use of compact query-based rather than high-resolution spatial-temporal memory.

    Source: §5 · p. 12

Abstract

from arXiv · show

Visual Object Tracking (VOT) aims to estimate the positions of target objects in a video sequence, which is an important vision task with various real-world applications. Depending on whether the initial states of target objects are specified by provided annotations in the first frame or the categories, VOT could be classified as instance tracking (e.g., SOT and VOS) and category tracking (e.g., MOT, MOTS, and VIS) tasks. Different definitions have led to divergent solutions for these two types of tasks, resulting in redundant training expenses and parameter overhead. In this paper, combing the advantages of the best practices developed in both communities, we propose a novel tracking-with-detection paradigm, where tracking supplements appearance priors for detection and detection provides tracking with candidate bounding boxes for the association. Equipped with such a design, a unified tracking model, OmniTracker, is further presented to resolve all the tracking tasks with a fully shared network architecture, model weights, and inference pipeline, eliminating the need for task-specific architectures and reducing redundancy in model parameters. We conduct extensive experimentation on seven prominent tracking datasets of different tracking tasks, including LaSOT, TrackingNet, DAVIS16-17, MOT17, MOTS20, and YTVIS19, and demonstrate that OmniTracker achieves on-par or even better results than both task-specific and unified tracking models.

1 INTRODUCTION

OmniTracker combines tracking and detection in a shared model for instance and category tracking. Tracking contributes appearance priors to detection, while detected boxes are associated with existing trajectories.

1 INTRODUCTION

  • Instance and category tracking use divergent setups and customized methods, leading to complex training and redundant parameters.Instance tracking specifies targets in the first frame; category tracking detects objects from specified categories and associates them across frames.Source: §1 · p. 1
  • Tracking-with-detection combines tracking-as-detection and tracking-by-detection: appearance priors guide detection, and detected boxes are associated with existing trajectories.The association uses spatial and appearance correlation.Source: Figure 1 · p. 2, §1 · p. 2
  • The Reference-guided Feature Enhancement module uses prior-box RoIAlign features for instance tracking and downsampled previous-frame features for category tracking.The enhanced features are sent to a detector operating on the full image rather than a cropped area.Source: §1 · p. 2
  • OmniTracker supports SOT, VOS, MOT, MOTS, and VIS with shared architecture, model weights, and inference pipeline, using identity embeddings for cross-frame association.The embeddings combine object queries and corresponding RoI features and are supervised with a contrastive ReID loss.Source: §1 · p. 2
  • Across seven tracking benchmarks, OmniTracker achieves new state-of-the-art or at least competitive results on various tracking tasks.The benchmarks include LaSOT, TrackingNet, DAVIS16-17, MOT17, MOTS20, and YTVIS19.Source: §1 · p. 3

2 RELATED WORK

Prior work develops task-specific tracking methods and unified models, while DETR-based detectors provide a basis for OmniTracker. The related work describes differences in task definitions, unification approaches, and detector designs.

2.1 Task-specific Tracking Models

  • SOT tracks a specified arbitrary object, while VOS uses masks and per-pixel predictions; MOT, MOTS, and VIS detect and track objects from categories.MOTS extends MOT with instance masks, and VIS addresses more open scenarios and categories.Source: §2.1 · p. 3

2.2 Unified Tracking Models

  • Unified models vary in scope: UniTrack uses a shared appearance model with multiple heads, UTT unifies SOT and MOT, and Unicorn and UNINEXT pursue broader unification.The passage notes that UniTrack's different heads limit large-scale joint training, UTT does not support per-pixel tasks, and Unicorn uses different inference pipelines and separate box-task and mask-task training.Source: §2.2 · p. 3

2.3 DETR-based Object Detectors

  • OmniTracker builds on Deformable-DETR, which speeds DETR convergence by attending to a small set of sampling points around a reference.The paper selects Deformable-DETR for its simplicity and efficiency and notes that more advanced detectors could further improve performance.Source: §2.3 · pp. 3–4

3 METHOD

The section argues that tracking and detection should reinforce one another, then introduces tracking-with-detection and OmniTracker as its implementation. The method uses reference-frame appearance information to enhance detection features and a Deformable DETR-based detector to predict objects and masks.

3.1 A Brief Review on Previous Tracking Paradigms

  • The section frames prior methods as either prior-guided detection or independent frame detection followed by association, motivating bidirectional tracking and detection.Tracking-as-detection can be hindered by a misestimated search region or false memory match; tracking-by-detection leaves past tracking information out of detection.Source: §3.1 · p. 4
  • Tracking-with-detection lets reference-frame information guide current-frame identification while detections provide candidates for temporal association.The paper contrasts this bidirectional design with tracking-as-detection and tracking-by-detection, which use information unidirectionally.Source: §3.1 · p. 4

3.2 Tracking-with-detection

  • OmniTracker inserts a Reference-guided Feature Enhancement module into a detector to provide appearance priors from prior tracking results.The framework extracts multi-scale backbone features from the current tracking frame.Source: §3.2 · p. 4

3.2.1 Reference-guided Feature Enhancement

  • For instance tracking, RFE uses RoIAlign features of previously tracked boxes; for category tracking, it uses a downsampled previous-frame feature map.The method also describes cross-attention between current-frame features and reference features; location priors are omitted because box coordinates can change with occlusion or movement.Source: §3.2.1 · p. 5

3.2.2 Deformable DETR

  • The Deformable DETR detector decodes box coordinates and category labels from object queries, while an FPN feature map and instance-aware kernels generate masks.For VOS, bounding boxes are calculated from masks.Source: §3.2.2 · p. 5

3.3 Loss Functions

The training objective combines per-frame detection supervision with cross-frame identity supervision. Detection uses set prediction and optimal-transport assignment, while ReID learns discriminative embeddings contrastively.

3.3.1 Per-frame Detection Loss

  • Object detection is formulated as set prediction, with optimal-transport assignment using classification accuracy and box GIoU to assign each ground truth to dynamically selected predictions.The per-frame detection loss combines classification, box, and mask losses; λ1 and λ2 are each set to 2.0.Source: §3.3.1 · p. 5

3.3.2 Contrastive ReID Loss

  • For cross-frame association, RoIAlign features of detector queries are transformed into identity embeddings and trained contrastively using a randomly sampled reference frame.Predictions with the least cost are positives and those with the highest costs are negatives; the total objective is Ldet + β Lreid, with β set to 10 empirically.Source: §3.3.2 · p. 5

3.4 Unified Online Tracking

OmniTracker associates detections with trajectories using appearance embeddings and motion cues in one inference pipeline. It maintains trajectory-level embedding memories, filters candidates by predicted-location overlap, and assigns remaining boxes with the Hungarian algorithm.

3.4 Unified Online Tracking

  • The shared inference pipeline combines localization and appearance information to associate detected boxes with existing trajectories.Each trajectory has a memory bank of historical identity embeddings, whose temporally weighted sum is compared bidirectionally with detected-instance embeddings.Source: §3.4 · pp. 5–6
  • A Kalman filter predicts trajectory locations, and boxes with low IoU against those estimates are filtered out before assignment.The IoU threshold τ is set to 0.25 empirically.Source: §3.4 · p. 6
  • The remaining box assignment uses 1 − S as its cost matrix with the Hungarian algorithm; unmatched boxes above the detection-score threshold start new tracks.For instance tracking, τnew is set to 1.0; SOT has one trajectory and VOS has one trajectory per instance annotated in the first frame.Source: §3.4 · pp. 5–6

4.1 Implementation Details

Implementation uses staged pretraining and joint fine-tuning across tracking datasets, with shared heads except for task-specific classification. Inference uses FIFO trajectory memories, with sizes set by task.

4.1 Implementation Details

  • Training proceeds through COCO detection and segmentation pretraining, pseudo key-reference pairing on COCO, and fine-tuning on tracking datasets with two sampled frames.COCO remains an additional downstream dataset in joint training to make training more stable.Source: §4.1 · p. 6
  • The model iteratively trains on loaders for COCO and five tracking-task groups, sharing all heads except the classification head across tasks.SOT and VOS category annotations are set to 0; the classification head predicts categories, the box head regresses boxes, and the embedding head supports cross-frame association.Source: §4.1 · p. 6
  • Inference implements trajectory memory as a FIFO queue, with memory sizes of 64 for SOT, 64 for VOS, 32 for MOT(S), and 3 for VIS.For instance tracking, the first-frame identity embedding with the highest IoU to the ground-truth box is always reserved during memory updates.Source: §4.1 · pp. 6–7
  • The comparison setup uses Swin-Tiny reproductions of Unicorn and UNINEXT without Objects365 pretraining for a fair comparison.UNINEXT uses DINO and Objects365 pretraining in its reported setup; the reproduced variant also omits Box De-noising.Source: §4.1 · p. 6

4.2 Evaluation on the Instance Tracking Tasks

OmniTracker is competitive with SOT-specific and unified trackers, and its VOS results improve on several unified and multi-task alternatives. The VOS comparison also identifies a remaining gap to task-specific methods and a shortfall against UNINEXT with larger models.

4.2.1 Single Object Tracking

  • OmniTracker exceeds Unicorn by 0.4% on LaSOT and 0.5% on TrackingNet in Pnorm, while remaining competitive with most unified models and SOT-specific trackers.The evaluation covers LaSOT and TrackingNet; the passage also notes stronger detector performance for UNINEXT than OmniTracker.Source: §4.2.1 · p. 7

4.2.2 Video Object Segmentation

  • OmniTracker-L exceeds Unicorn-L by 1.1% on DAVIS 2016 and 1.8% on DAVIS 2017 in J&F, and outperforms UniTrack and Unicorn across model sizes.The comparisons use the DAVIS 2016 and 2017 validation sets; Table 2 compares VOS-specific and unified trackers.Source: §4.2.2 · p. 7, Table 2 · p. 7
  • With larger models, OmniTracker falls slightly short of UNINEXT on VOS, and a gap may remain between unified or multi-task models and task-specific VOS methods.The passage attributes the larger-model comparison to detector performance dominating the VOS task.Source: §4.2.2 · p. 7

4.3 Evaluation on Category Tracking Tasks

On category-tracking benchmarks, OmniTracker surpasses Unicorn on MOT17 and both Unicorn and PointTrackV2 on MOTS20. On YouTube-VIS 2019, it outperforms the compared methods across all reported metrics.

4.3.1 Multiple Object Tracking

  • OmniTracker records 79.1% MOTA and 75.6% IDF1 on MOT17, surpassing Unicorn by 1.9% and 0.1%, respectively.Table 3 compares MOT-specific and unified trackers on the MOT17 test set.Source: §4.3.1 · pp. 7–8, Table 3 · p. 8

4.3.2 Multiple Object Tracking and Segmentation

  • On MOTS20, OmniTracker outperforms PointTrackV2 by 5.2% and Unicorn by 2.2% in sMOTSA.sMOTSA is based on mask overlap, unlike MOTA.Source: §4.3.2 · p. 8, Table 4 · p. 8

4.3.3 Video Instance Segmentation

  • On YouTube-VIS 2019, OmniTracker outperforms the compared methods, including UNINEXT-L-noObjPre, across AP, AP50, AP75, AR1, and AR10.The comparison includes VIS-specific models and UNINEXT; most existing unified tracking models do not support VIS.Source: §4.3.3 · p. 8, Table 5 · p. 8

4.4 Discussion and Analysis

The analysis finds benefits from joint training, ReID loss, and RFE, while also examining backbone choice and hyperparameter sensitivity. RFE’s reported gains differ by task type, and the selected MOT inference threshold affects MOTA.

4.4.1 Joint training v.s. independent training

  • Joint training consistently outperforms separate training and Unicorn across the evaluated tasks, and OmniTracker has higher inference efficiency than Unicorn.The reported Swin-Tiny FPS comparison is 20.9 vs 41.7.Source: §4.4.1 · p. 9

4.4.2 Impact of different components

  • Removing ReID loss significantly reduces results on most tasks; removing history affects instance tracking more for MR, while the Kalman filter is useful for MOT.Without the Kalman filter, OmniTracker’s improvement over Unicorn is even higher, since Unicorn uses no cross-frame association for instance tracking.Source: §4.4.2 · p. 9

4.4.3 Impact of the RFE Module

  • Without RFE, TrackingNet Pnorm falls by 1.6% and MOT17 MOTA by 0.8%; enhanced features make targets more distinguishable.The authors report that RFE improves localization and has a more prominent influence on instance tracking; they suggest it may bring larger gains for open-vocabulary category tracking.Source: §4.4.3 · p. 9, Figure 6 · p. 9, Table 9 · p. 9

4.4.4 Impact of backbone networks

  • ConvNext-L yields better overall results than Swin-L, and OmniTracker still outperforms Unicorn and UNINEXT across all tasks.The comparison uses ConvNext-L to enable a fair comparison with those models.Source: §4.4.4 · p. 10, Table 10 · p. 10

4.4.5 Impact of hyperparameters

  • Changing λ1 and λ2 does not significantly affect results, whereas τ has a marked effect on MOT17 MOTA, with 0.25 performing best.τ affects inference for MOT, MOTS, and VIS.Source: §4.4.5 · p. 10, Table 11 · p. 10

4.5 The possibility to extend OmniTracker

The discussion describes possible extensions using foundation models and multimodal inputs, and illustrates OmniTracker’s tracking behavior and failure cases. The examples include challenges with partial-object annotations, fast motion, and occlusion.

4.5 The possibility to extend OmniTracker

  • The authors identify dense memory for VOS and large-scale datasets as possible ways to extend OmniTracker, while direct integration of SAM2 is difficult.They also suggest multimodal text or contextual inputs for tracking and natural-language descriptions of tracklets; these integrations remain outside the current work’s scope.Source: §4.5 · p. 10

4.5.1 Visualizations

  • OmniTracker is shown tracking objects across instance and category tasks and maintaining tracks in long videos with both slow and rapid movement.The visualizations also compare OmniTracker with TATrack and ByteTrack; the authors report more accurate detection and greater occlusion robustness for OmniTracker.Source: §4.5.1 · p. 11, Figure 9 · p. 11, Figure 10 · p. 11

4.5.2 Failure Case

  • Partial-object annotations can lead OmniTracker to track the complete object, while unusually fast motion or occlusion can disrupt continuity as instance embeddings degrade.The authors attribute the partial-object behavior to co-training on instance and category tracking datasets.Source: §4.5.2 · p. 11, Figure 11 · p. 11

5 CONCLUSION

OmniTracker unifies instance and category tracking in a shared Deformable DETR-based model using tracking-with-detection. The authors report effectiveness across tracking benchmarks, while noting a remaining gap to task-specific models on VOS.

5 CONCLUSION

  • OmniTracker addresses SOT, VOS, MOT, MOTS, and VIS with a shared architecture, model weights, and inference pipeline, combining appearance priors for detection with candidate boxes for association.The conclusion reports effectiveness across a wide variety of tracking benchmarks.Source: §5 · p. 12
  • Despite outperforming Unicorn on VOS, OmniTracker remains behind task-specific models, which the authors relate to their use of high-resolution spatial-temporal memory rather than compact query-based memory.Combining the two memory types is identified as future work.Source: §5 · p. 12
Loading 2303.12079v2…