PaperlayerView exact revision on arXiv

Source-linked AI summary

MixFormer: End-to-End Tracking with Iterative Mixed Attention

Yutao Cui, Cheng Jiang, Limin Wang, Gangshan Wu

arXiv:2203.11082v2cs.CV

TL;DR

Designing a simple end-to-end tracker remains challenging amid scale variation, deformation, occlusion, and similar-object confusion. MixFormer unifies feature extraction and target integration with iterative Mixed Attention Modules, achieving state-of-the-art performance on five tracking benchmarks.

  • Problem

    Designing a simple end-to-end tracker remains challenging because of scale variations, object deformations, occlusion, and confusion from similar objects.

  • Method

    MixFormer uses transformer-based Mixed Attention Modules to simultaneously extract target-specific features and integrate information between target templates and search areas.

  • Results

    MixFormer achieves state-of-the-art performance on five tracking benchmarks, including VOT2020, LaSOT, TrackingNet, GOT-10k, and UAV123.

  • Takeaways & Limitations

    The coupled design yields a compact tracking pipeline while enabling target-specific feature extraction and extensive target–search communication.

  • Takeaways & Limitations

    CNN representations may neglect finer structure information and lack global modeling power because they use generic pretraining and local convolutional kernels.

Abstract

from arXiv · show

Tracking often uses a multi-stage pipeline of feature extraction, target information integration, and bounding box estimation. To simplify this pipeline and unify the process of feature extraction and target information integration, we present a compact tracking framework, termed as MixFormer, built upon transformers. Our core design is to utilize the flexibility of attention operations, and propose a Mixed Attention Module (MAM) for simultaneous feature extraction and target information integration. This synchronous modeling scheme allows to extract target-specific discriminative features and perform extensive communication between target and search area. Based on MAM, we build our MixFormer tracking framework simply by stacking multiple MAMs with progressive patch embedding and placing a localization head on top. In addition, to handle multiple target templates during online tracking, we devise an asymmetric attention scheme in MAM to reduce computational cost, and propose an effective score prediction module to select high-quality templates. Our MixFormer sets a new state-of-the-art performance on five tracking benchmarks, including LaSOT, TrackingNet, VOT2020, GOT-10k, and UAV123. In particular, our MixFormer-L achieves NP score of 79.9% on LaSOT, 88.9% on TrackingNet and EAO of 0.555 on VOT2020. We also perform in-depth ablation studies to demonstrate the effectiveness of simultaneous feature extraction and information integration. Code and trained models are publicly available at https://github.com/MCG-NJU/MixFormer.

1. Introduction

MixFormer addresses the complexity of multi-stage tracking by coupling feature extraction with target information integration in an end-to-end transformer framework. Its mixed-attention design supports compact tracking, online template selection, and strong benchmark performance.

  • Motivation: MixFormer replaces the prevailing backbone–integration-module–head pipeline with a compact framework combining a target-search mixed-attention backbone and localization head.The proposed coupled paradigm removes the explicit integration module.
  • Core approach: The framework couples generic feature extraction and target integration to obtain target-specific features and more extensive target–search communication.The motivation includes better handling of target appearance information within a unified process.
  • Core approach: Mixed Attention Modules perform self-attention for feature extraction and cross-attention for communication between target templates and search areas.MixFormer stacks MAMs with progressive patch embedding and adds a simple localization head.
  • Results: 25 FPS on a GTX 1080Ti accompanies MixFormer-L improvements over STARK of 5.0% EAO on VOT2020, 2.9% NP on LaSOT, and 2.0% NP on TrackingNet.The framework is reported to achieve state-of-the-art performance on five challenging benchmarks.
  • Online tracking: Asymmetric attention reduces the cost of multiple-template online tracking, while a score prediction module selects high-quality templates.The online update mechanism is designed to address object deformation and appearance variation.

2. Related Work

Prior tracking methods generally separate feature extraction, target–search integration, and localization, while transformer trackers improve global interaction but retain pipeline limitations. MixFormer instead uses mixed attention to unify feature extraction and information integration.

  • Tracking Paradigm: Prevailing trackers commonly use three stages: a backbone, an integration module, and heads that produce target states.Integration bridges generic feature extraction and target-aware localization.
  • Vision Transformer: Vision Transformer research introduced pure transformer image architectures, with later designs adding local context or multi-stage modeling.Examples include PVT's multi-stage design and CVT's combination of convolutional and transformer modeling.
  • Vision Transformer: Unlike CVT, MAM performs dual self- and cross-attention for feature extraction and target–search information integration using target templates and search regions.The comparison distinguishes MAM's tracking inputs and localization head from CVT's image-classification setting.
  • Attention-based Tracking: Recent tracking transformers improve target–search fusion and end-to-end localization, but earlier approaches still retained separate pipeline elements or post-processing.The related work positions MAM as a unified alternative.

3. Method

MixFormer unifies feature extraction and target-search information integration in iterative Mixed Attention Modules, forming a compact end-to-end tracker with a localization head. Its asymmetric attention and score-based template update support efficient online tracking with multiple templates.

  • Mixed Attention Module (MAM): MixFormer uses MAM to simultaneously extract target-specific features and integrate information between target templates and search regions.MAM performs self-attention and cross-attention within the same module.
  • Mixed Attention Module (MAM): MAM processes separate target and search token sequences, adding depth-wise convolutional projections to model local spatial context and improve efficiency.The projections produce queries, keys, and values for the attention operations and permit down-sampling in key and value matrices.
  • Mixed Attention Module (MAM): Asymmetric mixed attention removes target-to-search cross-attention because distractors may harm templates, reducing computation and preserving template tokens during tracking.The scheme avoids influence from dynamic search regions on template tokens.
  • MixFormer for Tracking: The MixFormer architecture stacks progressive patch-embedding stages and target-search MAM blocks, followed by a localization head that directly predicts bounding boxes.The backbone contains iterative MAMs, while the localization head can use corner-based or query-based prediction without post-processing.
  • MixFormer for Tracking: The localization heads estimate boxes either from top-left and bottom-right corner distributions or through a learnable regression token and fully connected layers.The corner-based head uses convolutional layers, while the query-based head aggregates information from the target and search area.
  • Training and Inference: The score prediction module selects reliable online templates by comparing mined search-region target information with the initial target and rejecting scores below 0.5.The module uses attention blocks, a learnable score token, a three-layer perceptron, and sigmoid activation.

4. Experiments

Experiments evaluate MixFormer across five benchmarks and analyze its architecture, localization head, asymmetric attention, template updates, training choices, and attention behavior. The results show strong benchmark performance, efficient asymmetric attention, and benefits from unified mixed attention and score-based template selection.

  • Benchmark evaluation: MixFormer is evaluated on VOT2020, LaSOT, TrackingNet, GOT10k, and UAV123.
  • Benchmark results: 0.555 EAO is achieved by MixFormer-L on VOT2020, surpassing STARK by 5% EAO.
  • Benchmark results: 79.9% NP is achieved by MixFormer-L on LaSOT, surpassing STARK by 2.9% without multi-layer feature aggregation.
  • Benchmark results: MixFormer-22k and MixFormer-L set new state-of-the-art performance on TrackingNet and outperform other trackers on UAV123, while MixFormer-GOT achieves state-of-the-art performance on GOT-10k.
  • Ablation studies: MixFormer-Base improves over separate-processing trackers by 8.6% and 7.9% with fewer parameters and FLOPs, supporting unified feature extraction and information integration.
  • Ablation studies: More MAMs contribute to higher AUC, and the fully convolutional corner head outperforms the query-based head in MixFormer-Base.
  • Ablation studies: Asymmetric MAM increases running speed by 24% with comparable performance, while score-based template selection achieves the best AUC among tested update strategies.
  • Attention analysis: Attention visualizations indicate progressive background-distractor suppression, adaptive online templates, mutual enhancement among templates, and target-search interactions.

5. Conclusion

MixFormer unifies feature extraction and target integration through iterative mixed attention in a compact end-to-end tracking framework. The paper reports notable improvements over prevailing trackers for short-term tracking and identifies multiple object tracking as future work.

  • MixFormer unifies feature extraction and target integration through iterative mixed attention in a compact end-to-end tracking pipeline.
  • Mixed attention performs both feature extraction and mutual interaction for target templates and search areas.
  • MixFormer shows notable improvement over prevailing trackers for short-term tracking.
  • Extending MixFormer to multiple object tracking is identified as future work.

A. More Results

MixFormer-L performs competitively on OTB-100 and improves over TransT on AUC, while LaSOT evaluation attributes gains to both accuracy and robustness.

  • MixFormer-L surpasses TransT by 1.3% on OTB-100 AUC and is slightly higher than MixFormer.
  • LaSOT results indicate that MixFormer improves through both higher accuracy and greater robustness.
  • Figures 7 and 8 present state-of-the-art comparisons on OTB100 and LaSOT, respectively.

B. More Visualization Results

Attention visualizations on LaSOT car-2 support the earlier analysis and suggest that the final two blocks adapt to bounding-box prediction.

  • The final two attention blocks tend to adapt their features to the bounding-box prediction head.
  • The car-2 visualization reaches the same conclusion as the analysis in Section 4.3.

C. Training Details

Training uses standardized search and template inputs, pretrained CvT backbones, multi-GPU configurations, and gradient clipping.

  • MixFormer uses a 320x320 search region and two 128x128 input images for fair comparison with prevailing trackers.
  • Training uses 8 Tesla V100 GPUs with batch size 32, or 8 2080Ti GPUs with batch size 8 per GPU.
  • CvT21 and CvT24-W pretrained models initialize MixFormer and MixFormer-L, respectively.
  • Gradient clipping is applied during training.
Loading 2203.11082v2…