PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Grand Unification of Object Tracking

Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, Huchuan Lu

arXiv:2207.07078v4cs.CV

TL;DR

Object tracking is fragmented across four tasks, with separate methods that can over-specialize and duplicate parameters. Unicorn uses one network with shared inputs, backbone, embedding, head, and model parameters to unify these tasks. It performs on-par or better than task-specific counterparts across 8 challenging benchmarks and is presented as a step toward a general vision model.

  • Problem

    Most tracking approaches address only one or part of SOT, MOT, VOS, and MOTS, creating task over-specialization and redundant parameters.

  • Method

    Unicorn unifies four tracking tasks with shared inputs, backbone, embedding, head, and model parameters, using target priors and pixel-wise correspondence.

  • Results

    Unicorn performs on-par or better than task-specific counterparts on 8 challenging benchmarks from four tracking tasks.

  • Takeaways & Limitations

    The unified architecture and learning paradigm are presented as a step toward a general vision model.

Abstract

from arXiv · show

We present a unified method, termed Unicorn, that can simultaneously solve four tracking problems (SOT, MOT, VOS, MOTS) with a single network using the same model parameters. Due to the fragmented definitions of the object tracking problem itself, most existing trackers are developed to address a single or part of tasks and overspecialize on the characteristics of specific tasks. By contrast, Unicorn provides a unified solution, adopting the same input, backbone, embedding, and head across all tracking tasks. For the first time, we accomplish the great unification of the tracking network architecture and learning paradigm. Unicorn performs on-par or better than its task-specific counterparts in 8 tracking datasets, including LaSOT, TrackingNet, MOT17, BDD100K, DAVIS16-17, MOTS20, and BDD100K MOTS. We believe that Unicorn will serve as a solid step towards the general vision model. Code is available at https://github.com/MasterBin-IIAU/Unicorn.

1 Introduction

Object tracking is fragmented across SOT, MOT, VOS, and MOTS, encouraging specialized models with limited reuse. Unicorn unifies these tasks in one architecture and learning paradigm, using shared parameters while achieving competitive benchmark performance.

  • Motivation: Four tracking sub-tasks—SOT, MOT, VOS, and MOTS—are typically addressed by separate methods.This fragmentation can produce task over-specialization and redundant parameters.
  • Motivation: SOT and MOT differ in object characteristics and correspondence requirements, creating a central obstacle to unified tracking.SOT follows a given target, whereas MOT detects and associates multiple category-specific instances.
  • Method: Unicorn uses a target prior and pixel-wise correspondence to bridge propagation-based and detection-based tracking tasks.The target prior switches task behavior, while pixel-wise correspondence contains both SOT/VOS and MOT/MOTS correspondence forms.
  • Method: The network takes reference and current frames through a weight-shared backbone, builds correspondence, propagates a target prior, and fuses these into a detection head.The same processing structure supports all four tasks.
  • Results: Unicorn addresses four tracking tasks with the same model parameters and reports on-par or better performance than task-specific counterparts on 8 challenging benchmarks.The paper presents this as a step toward a general vision model.

2 Related Work

Prior work largely specializes tracking models by task, while broader general-vision efforts and partial tracking unifications leave gaps in shared architecture, heads, or parameters. Unicorn is positioned as a single-network alternative spanning all four tracking tasks.

  • Task-specific tracking: SOT, MOT, VOS, and MOTS use task-specific strategies such as search regions, detection-association pipelines, memory networks, and mask extensions.These designs reflect different task requirements but remain fragmented.
  • General vision models: General vision models seek one model that supports multiple vision tasks, using approaches such as multi-task self-training, multi-source supervision, and modality-agnostic representations.The cited works target broader generalization beyond object tracking.
  • Unification in object tracking: Existing tracking unification methods cover only SOT with VOS or MOT with MOTS, while UniTrack leaves tracking heads and detectors unshared.These design choices limit complete parameter sharing and broad data exploitation.
  • Unicorn: Unicorn is presented as a unified architecture with shared inputs, backbone, embedding, and head across four tracking tasks.This contrasts with prior frameworks that retain separate components or detectors.
  • Correspondence learning: Accurate correspondence is central to tracking and related vision tasks, but independent frame embeddings can create ambiguous matches among similar patterns.Attention-based methods address long-range dependencies, while Unicorn applies them to tracking correspondence.

3 Approach

Unicorn processes reference and current frames with shared visual features, constructs pixel- or instance-level correspondence, and uses a target prior with a unified detection head. Full-image inputs support localization across the four tracking tasks.

  • Unified architecture: Unicorn takes the reference frame, current frame, and reference targets to predict tracked-target states with one network.Its three components are unified inputs and backbone, unified embedding, and unified head.
  • Unified inputs and backbone: Full-image inputs replace local search regions, enabling localization of multiple potential targets and re-detection after disappearance.The paper links this design to resistance against tracking failure.
  • Unified inputs and backbone: A weight-sharing backbone extracts feature pyramids from both frames before correspondence computation.The stride-16 feature map is selected for the embedding module to preserve detail while limiting computation.
  • Unified embedding: Pixel-wise correspondence propagates reference targets for SOT/VOS, while instance-level correspondence associates detections with trajectories for MOT/MOTS.Both roles use correspondence between reference and current frames.
  • Unified embedding: Instance-level correspondence is a sub-matrix of pixel-wise correspondence, allowing the four tasks to share an embedding formulation.The instance embeddings are extracted at detected instance centers.
  • Unified embedding: Deformable attention replaces full attention to reduce quadratic memory growth, and enhanced features are upsampled 2× for stride-8 embeddings.The design targets more accurate correspondence with lower memory cost.

Feature Interaction.

The embedding module uses attention-based interaction between reference and current features to produce enhanced embeddings for correspondence learning. Supervision for SOT/VOS compares propagated targets with ground-truth target maps.

  • Feature interaction: The embedding equation applies attention to reference and current features, then upsamples the result to produce enhanced embeddings.This operation forms the feature representations used for correspondence.
  • Feature interaction: For SOT and VOS, the embedding is supervised by the difference between the propagated current target and its ground-truth target map.The target map marks tracked-target regions with one and other regions with zero.

Loss.

Unicorn trains correspondence differently for single-target/segmentation tasks and multi-object tasks, using task-appropriate supervision within one unified objective.

  • For MOT and MOTS, instance correspondence uses a ground-truth matrix whose matched instance entry is 1 and all other entries are 0.
  • For SOT and VOS, Unicorn optimizes correspondence with Dice Loss between predicted and current-frame target maps.
  • The correspondence objective selects Dice loss for SOT/VOS and cross-entropy loss for MOT/MOTS.

3.3 Unified Head

Unicorn uses one detection head for all four tasks by conditioning it on a target prior: propagated target information for SOT/VOS and zero for MOT/MOTS.

  • Unicorn adds a target-prior input to the detector head so one head can detect reference-defined targets and category-defined objects.
  • For SOT and VOS, the target prior is the reshaped propagated target map; for MOT and MOTS, it is set to zero.
  • Feature Fusion: The head fuses FPN features F with the target prior P by broadcast summation before passing the result to the original detection head.
  • Feature Fusion: With a zero prior, MOT and MOTS retain the original FPN features, while the nonzero prior focuses SOT/VOS detection on the tracked target.
  • Feature Fusion: The fusion design preserves a simple detection head and allows Unicorn to exploit pretrained object-detector weights.

3.4 Training and Inference

Unicorn trains in two joint-task stages and uses a lightweight inference path that reuses reference information while avoiding repeated full-network execution.

  • Training first jointly optimizes correspondence and detection on SOT/MOT data, then adds a mask branch for VOS/MOTS while fixing other parameters.
  • During SOT/VOS inference, Unicorn generates the reference target map once, keeps it fixed, and selects the highest-confidence box or mask.
  • Unicorn runs the heavy backbone and correspondence once, then reuses the lightweight head rather than running the whole network repeatedly.

4 Experiments

Across eight benchmarks spanning SOT, MOT, VOS, and MOTS, Unicorn is competitive with or superior to task-specific methods while retaining a unified model and efficient variants.

  • 4 Experiments: Unicorn uses ConvNeXt-Large for state-of-the-art comparisons and evaluates smaller ConvNeXt-Tiny and ResNet-50 backbones in ablations.
  • 4 Experiments: The unified model uses the same parameters across four tasks and is compared with task-specific counterparts on eight tracking datasets.
  • Single Object Tracking: 68.5% Success and 74.1% Precision establish new state-of-the-art results for Unicorn on LaSOT.
  • Single Object Tracking: 83.0% Success and 82.2% Precision make Unicorn the strongest reported method on TrackingNet.
  • Multiple Object Tracking: 0.5% higher MOTA and 0.4% higher IDF1 than the previous SOTA method give Unicorn the best reported MOT17 results.
  • Cross-task results: Unicorn surpasses prior methods on BDD100K MOT, DAVIS, MOTS20, and BDD100K MOTS, including gains of 4.6% mMOTA, 3.2% mIDF1, 3.3% sMOTSA, and 2.2% mMOTSA.
  • Ablations and analysis: Deformable attention, the target prior, and broadcast-sum fusion improve the unified system, while the model remains on-par with independent models and a lightweight variant exceeds 20 FPS.

5 Conclusions

Unicorn unifies four tracking tasks with one approach and the same model parameters, achieving performance on-par or better than task-specific counterparts across eight challenging benchmarks.

  • Unicorn addresses four tracking tasks with a single model using the same model parameters.
  • The method unifies the network architecture and learning paradigm for object tracking.
  • Unicorn performs on-par or better than task-specific counterparts on eight challenging benchmarks.

A Unified Head Architecture

The unified head combines FPN features with a target prior, then predicts boxes or masks through fully convolutional detection and segmentation heads.

  • The unified head takes the original FPN feature F and target prior P as inputs.
  • Broadcast summation fuses the two inputs into a fused feature before prediction.
  • The fused feature passes through detection and instance segmentation heads to predict final boxes or masks.
  • The head is fully convolutional and does not use RoI operations such as RoI Align.

B Training Details

The detailed training procedure is only partially represented here: the training process is split into stages to address severe imbalance between box- and mask-annotated tracking data, while Fig. 4 depicts the unified head architecture.

  • SOT and MOT training data are usually dozens of times more abundant than VOS and MOTS data because mask annotations are more expensive.
  • The training process is divided into two stages to alleviate imbalance when learning from four tracking tasks.
  • Fig. 4 presents the unified head architecture of Unicorn.
Loading 2207.07078v4…