PaperlayerView exact revision on arXiv

Source-linked AI summary

Divert More Attention to Vision-Language Tracking

Mingzhe Guo, Zhipeng Zhang, Heng Fan, Liping Jing

arXiv:2207.01076v1cs.CV

TL;DR

The paper asks whether Transformer attention is necessary for state-of-the-art tracking, given its training-data and time demands. It introduces CNN-based vision-language tracking with unified-adaptive representations learned by ModaMixer and asymmetrical search, and reports 65.2% SUC on LaSOT while outperforming Transformer trackers.

  • Problem

    Transformer tracking has raised state-of-the-art performance, but relies on more training data and longer training, while existing vision-language trackers lag behind.

    Source: §1 · pp. 1–2

  • Method

    The approach learns unified-adaptive vision-language representations with ModaMixer and asymmetrical search using pure CNN visual feature learning.

    Source: §1 · pp. 1–2

  • Results

    65.2% SUC on LaSOT: the CNN-based VLTSCAR improves its SiamCAR baseline by 14.5% and outperforms Transformer trackers.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results support pure-CNN vision-language tracking as a competitive alternative to Transformer-based tracking.

    Source: §1 · p. 2, §5 · p. 9

    The theoretical explanation assumes empirical risk minimizers for separately trained modality sets, and its proof is deferred to.

    Source: §3.4 · p. 6

Abstract

from arXiv · show

Relying on Transformer for complex visual feature learning, object tracking has witnessed the new standard for state-of-the-arts (SOTAs). However, this advancement accompanies by larger training data and longer training period, making tracking increasingly expensive. In this paper, we demonstrate that the Transformer-reliance is not necessary and the pure ConvNets are still competitive and even better yet more economical and friendly in achieving SOTA tracking. Our solution is to unleash the power of multimodal vision-language (VL) tracking, simply using ConvNets. The essence lies in learning novel unified-adaptive VL representations with our modality mixer (ModaMixer) and asymmetrical ConvNet search. We show that our unified-adaptive VL representation, learned purely with the ConvNets, is a simple yet strong alternative to Transformer visual features, by unbelievably improving a CNN-based Siamese tracker by 14.5% in SUC on challenging LaSOT (50.7% > 65.2%), even outperforming several Transformer-based SOTA trackers. Besides empirical results, we theoretically analyze our approach to evidence its effectiveness. By revealing the potential of VL representation, we expect the community to divert more attention to VL tracking and hope to open more possibilities for future tracking beyond Transformer. Code and models will be released at https://github.com/JudasDie/SOTS.

1 Introduction

The section asks whether Transformer attention is necessary for state-of-the-art tracking and identifies weak vision-language representation as a bottleneck. It proposes CNN-based vision-language tracking with unified-adaptive representations, ModaMixer, and asymmetrical search, reporting competitive results against Transformer trackers.

1 Introduction

  • The paper argues that simple CNN interactions across vision and language offer a Transformer-free route to state-of-the-art tracking.It frames this as an alternative to complex visual attention, which requires more training data and time.Source: §1 · pp. 1, 3
  • Existing vision-language trackers connect vision and language only at result fusion, and their best LaSOT tracker trails a recent Transformer tracker by 10.9% SUC.The section attributes the representation gap to insufficient unified and adaptive multimodal features and to symmetric template and search branches.Source: §1 · p. 2
  • ModaMixer uses language representations to reweight visual channels, then fuses selected and original features into a unified representation.Installed from shallow to deep CNN layers, it improves the pure CNN baseline by 6.9% (50.7%→57.6%).Source: §1 · p. 2
  • Asymmetrical search separately learns distinctive networks for mixed-modality branches, adding 7.6% SUC over ModaMixer (57.6%→65.2%).The search avoids ImageNet retraining and takes 0.625 GPU days on one RTX-2080Ti.Source: §1 · p. 2
  • 65.2% SUC on LaSOT: VLTSCAR improves the SiamCAR baseline by 14.5% and outperforms Transformer trackers.The method also shows similar improvements on four other benchmarks.Source: §1 · p. 2

2 Related Work

Prior work applies Siamese tracking, Transformer feature interactions, language cues, and neural architecture search to tracking. This paper differentiates its approach by learning unified vision-language representations and searching asymmetrical multimodal networks.

Visual Tracking

  • Transformer extensions improve Siamese tracking through visual-feature interactions, while this work explores multimodal tracking with simple CNNs instead [45] [33].Siamese tracking learns a generic matching function.Source: §2 · p. 3

Vision-Language Tracking

  • Earlier vision-language trackers use language to improve robustness or fuse separately produced vision and language results; ModaMixer instead learns unified representations.The paper describes prior vision-language tracking work as.Source: §2 · p. 3

NAS for Tracking

  • Prior tracking NAS searches symmetrical vision-only architectures, whereas this work searches a flexible asymmetrical two-stream network for multimodal tracking.Its search takes 0.625 RTX-2080Ti GPU days, compared with about 40 V100 GPU days for LightTrack.Source: §2 · p. 3

3 Unified-Adaptive Vision-Language Tracking

The section presents a unified-adaptive vision-language tracking framework built around ModaMixer and asymmetrical network search. It describes how these components are integrated and gives a theoretical analysis of multimodal representation learning.

3.1 Modality Mixer for Unified Representation

  • ModaMixer uses language as a selector to reweight visual-feature channels, then adds a visual residual to preserve informative visual details.It averages language-token features, projects them to the visual channel count, and multiplies them with visual features; language semantics can enhance target-specific channels and suppress distractors.Source: §3.1 · p. 4

3.2 Asymmetrical Search for Adaptive Vision-Language Representation

  • ASS searches asymmetric template, search, and ModaMixer components, training a supernet by random sampling and selecting subnetworks using validation SUC.Reusing the SPOS supernet reduces the search pipeline's time complexity to 1/64 of LightTrack's: 0.625 RTX-2080Ti GPU days versus 40 V100 GPU days.Source: §3.2 · pp. 4–5

3.3 Tracking Framework

  • The framework integrates ModaMixer into each stage of both branches, then performs feature matching and target localization as in the baseline.The branches and ModaMixer post-processing layers are asymmetric, allowing adaptation to different depths, inputs, and pure-vision or mixed features.Source: §3.3 · p. 5

3.4 A Theoretical Explanation

  • The theoretical analysis suggests that more modalities could potentially enhance representation quality, while heterogeneous network complexity raises the population-risk upper bound and asymmetry enlarges the optimization space.The authors state that the proof is beyond their scope and refer readers to.Source: §3.4 · p. 6

4 Experiment

The experiments compare VLTSCAR and VLTTT with their baselines and other trackers, then test the contributions and settings of ModaMixer and ASS. They also examine language-input choices and how ModaMixer selects visual features.

4.1 Implementation Details

  • The trackers inherit their matching module and localization head from the corresponding baseline, while ASS searches for subnetworks after supernet training.The VLTSCAR search pipeline takes 15 hours on one RTX-2080Ti GPU.Source: §4.1 · p. 6

4.2 State-of-the-art Comparison

  • 14.5%/10.8%/14.5% SUC gains on LaSOT/LaSOTExt/TNL2K, respectively, let VLTSCAR outperform its SiamCAR baseline across three benchmarks.The comparisons cover LaSOT, LaSOTExt, TNL2K, GOT-10k, and OTB99-LANG.Source: Table 2 · p. 7, §4.2 · p. 7
  • 67.3% SUC on LaSOT with a 2.4% gain over TransT shows VLTTT improves its baseline; it also outperforms TransT in accuracy and speed.VLTSCAR also beats TransT on LaSOT while running at 43 FPS versus 32 FPS and using 2.4 × 10^7 versus 3.8 × 10^7 training pairs.Source: §4.2 · p. 7

4.3 Component-wise Ablation

  • 6.9% SUC gains on LaSOT from adding ModaMixer to SiamCAR support the benefit of unified vision-language representation; ASS gains 7.6% when paired with ModaMixer.On the vision-only baseline, ASS improves LaSOT SUC by 1.4%.Source: §4.3 · pp. 7–8, Table 3 · p. 8

4.4 Further Analysis

  • For training videos without language annotations, replacing language embeddings with pooled template features performs slightly better on average than using zero tensors.Both strategies perform competitively.Source: §4.4 · p. 8, Table 4 · p. 8
  • 3.9% lower SUC and 6.2% lower precision on LaSOT for the symmetrical design support searching asymmetrical networks and ModaMixer post-processing layers.Replacing searched ModaMixer post-processing layers with a fixed ShuffleNet block drops SUC from 63.9% to 59.1%.Source: §4.4 · pp. 8–9, Table 5 · p. 8
  • 65.2% → 53.4% SUC on LaSOT without a language description, while generated captions score 51.6%, makes inference-time language quality consequential.The no-description result remains better than the vision-only baseline, whereas generated captions are reported to bring negative effects.Source: Table 5 · p. 8, §4.4 · p. 9
  • ModaMixer’s highest-scored channels correspond to the target, and fusion enhances target responses while suppressing distractors.Figure 4 shows channel-selection scores and activation maps before and after multimodal fusion.Source: §4.4 · p. 9, Figure 4 · p. 9

5 Conclusion

The paper presents multimodal vision-language tracking with pure CNNs as a path to SOTA tracking without complex Transformers. It attributes the approach to unified-adaptive representations learned with ModaMixer and asymmetrical networks, and reports competitive or better performance than recent SOTAs.

5 Conclusion

  • The paper explores multimodal vision-language tracking with pure CNNs as an alternative path to SOTA tracking without complex Transformers.Its unified-adaptive representation is learned using ModaMixer and asymmetrical networks.Source: §5 · p. 9

A Appendix

The appendix outlines additional analyses of training data, language models, ASS, tracking outcomes, and attribute-based performance.

A Appendix

  • The appendix lists additional analyses of language-annotated training data, language models, ASS efficiency, tracking results and failure cases, activation maps, and attribute-based performance.The listed topics include a comparison of ASS with LightTrack and performance across complex scenarios.Source: A Appendix · p. 10

A.1 Volume of Language-Annotated Training Data

The appendix tests how the amount of language-annotated training data affects LaSOT performance. SUC decreases as the annotated share falls, while total training-data volume remains constant.

A.1 Volume of Language-Annotated Training Data

  • 63.9% → 61.8% → 58.9% SUC on LaSOT as language-annotated training pairs fall from 100% to 75% to 50% shows declining performance.Reduced annotated portions are replaced with data without language annotations, keeping total training-data volume constant.Source: §A.1 · p. 10

A.2 Different Language Models

The section compares BERT and GPT-2 as language models for vision-language representation. GPT-2 decreases performance; the authors suggest BERT's bidirectional learning may capture sentence context better.

A.2 Different Language Models

  • GPT-2 decreases performance relative to BERT in the LaSOT language-model comparison.Table 6(b) evaluates the influence of different language models.Source: §A.2 · p. 10, Table 6 · p. 10
  • The authors suggest BERT's bidirectional learning may capture sentence context better than GPT-2's self-regression.Source: §A.2 · p. 10
  • Table 6 evaluates language-annotated data volumes and language-model choice on LaSOT.Source: Table 6 · p. 10

A.3 Details of The Proposed Asymmetrical Searching Strategy (ASS)

ASS searches an asymmetrical network to adapt the mixed modalities across tracking branches, then retrains the selected subnet. Its pipeline initializes network components, handles absent language annotations, trains sampled subnets, and selects an architecture using validation success.

A.3 Details of The Proposed Asymmetrical Searching Strategy (ASS)

  • ASS has two stages: pretraining searches for an architecture, and retraining optimizes the selected subnet for vision-language tracking.Inference then tracks the target using the retrained network.Source: §A.3 · p. 11
  • ASS initializes visual branches from the SPOS pretrained supernet and initializes the mixed branch from the visual branches' last layer.The authors state this reduces tedious ImageNet training and enables quick reproducibility.Source: §A.3 · p. 11
  • When language annotations are absent, ASS uses either a zero tensor or pooled template visual features as the language representation.The visual-feature strategy is described as the robust setting in the algorithm.Source: §A.3 · p. 11
  • During pretraining, ASS repeatedly samples a subnet, produces predictions, and updates its parameters by gradient descent.Source: §A.3 · p. 11
  • Evolutionary architecture search selects the best subnet using success score on validation data, which is then retrained following baseline-tracker training pipelines.The searched asymmetrical architecture is detailed in Table 7.Source: §A.3 · p. 12, Table 7 · p. 12

A.4 Comparison of ASS and LightTrack [54]

The appendix compares ASS with LightTrack in terms of training-time complexity. It reports that ASS reuses an SPOS pretrained supernet rather than repeating ImageNet pretraining, avoiding that cost.

A.4 Comparison of ASS and LightTrack [54]

  • The authors frame ASS's simplification as a trade-off between training time and performance.Source: §A.4 · p. 12
  • ASS avoids ImageNet pretraining cost by reusing the pretrained SPOS supernet, unlike LightTrack, which trains its backbone on ImageNet twice.Table 8 compares the pipelines in terms of time complexity.Source: §A.4 · p. 12, Table 8 · p. 12

A.5 Visualization of Tracking Result and Failure Case.

The visualizations show VLTSCAR tracking through deformation, occlusion, and similar-object interference, while prolonged full occlusion remains a failure case. Language descriptions alter activation toward or away from targets, and attribute-based evaluation reports advantages on most attributes and supports the role of ModaMixer and ASS.

A.5 Visualization of Tracking Result and Failure Case.

  • VLTSCAR tracks through deformation, occlusion, and similar-object interference, but fails when full occlusion lasts about 100 frames and similar objects distract it.The authors identify global searching as a possible future solution.Source: §A.5 · pp. 12–13, Figure 5 · p. 13

A.6 Activation Analysis of Different Language Descriptions

  • Class-specific and more detailed language descriptions suppress similar-object distractors, while environmental-only description enhances background and suppresses the target.The full description activation map locates the target precisely.Source: Figure 6 · p. 13, §A.6 · p. 14

A.7 Attribute-based Performance Analysis

  • VLTSCAR and VLTTT outperform competing trackers on most LaSOT attributes, and the ablation indicates ModaMixer and ASS are needed for a powerful VL tracker.Figure 7 presents both attribute comparisons and VLTSCAR component ablations.Source: §A.7 · p. 14, Figure 7 · p. 14

Related papers

Loading 2207.01076v1…