PaperlayerView exact revision on arXiv

Source-linked AI summary

Bi-directional Adapter for Multi-modal Tracking

Bing Cao, Junliang Guo, Pengfei Zhu, Qinghua Hu

arXiv:2312.10611v1cs.CVcs.AI

TL;DR

Multi-modal tracking must cope with limited paired data and changing modality dominance, which fixed-role and conventional tuning methods do not address well. BAT uses a frozen foundation model with shared-parameter modality branches and a universal bi-directional adapter for mutual feature prompting. With 0.32M trainable parameters, it reports superior tracking performance against full fine-tuning and prompt-learning methods across RGBT benchmarks.

  • Problem

    Multi-modal tracking lacks sufficient labeled data, while dominant modality relationships change across environments and challenge fixed-role fusion methods.

  • Method

    BAT freezes a pre-trained foundation model and uses shared-parameter modality branches plus a universal bi-directional adapter to cross-prompt RGB and thermal features.

  • Results

    0.32M trainable parameters yield superior tracking performance against full fine-tuning and prompt-learning methods across multiple RGBT tracking datasets.

  • Takeaways & Limitations

    BAT adaptively fuses complementary multi-modal information without presetting a fixed dominant modality.

Abstract

from arXiv · show

Due to the rapid development of computer vision, single-modal (RGB) object tracking has made significant progress in recent years. Considering the limitation of single imaging sensor, multi-modal images (RGB, Infrared, etc.) are introduced to compensate for this deficiency for all-weather object tracking in complex environments. However, as acquiring sufficient multi-modal tracking data is hard while the dominant modality changes with the open environment, most existing techniques fail to extract multi-modal complementary information dynamically, yielding unsatisfactory tracking performance. To handle this problem, we propose a novel multi-modal visual prompt tracking model based on a universal bi-directional adapter, cross-prompting multiple modalities mutually. Our model consists of a universal bi-directional adapter and multiple modality-specific transformer encoder branches with sharing parameters. The encoders extract features of each modality separately by using a frozen pre-trained foundation model. We develop a simple but effective light feature adapter to transfer modality-specific information from one modality to another, performing visual feature prompt fusion in an adaptive manner. With adding fewer (0.32M) trainable parameters, our model achieves superior tracking performance in comparison with both the full fine-tuning methods and the prompt learning-based methods. Our code is available: https://github.com/SparkTempest/BAT.

Introduction

RGB tracking is limited in complex illumination conditions, motivating RGB-T tracking, but existing methods struggle with scarce data and changing modality dominance. BAT addresses this with adaptive bi-directional prompting and lightweight training.

  • Motivation: RGB tracking can fail under illumination variation, while RGB-T tracking combines RGB texture with TIR heat contrast for all-weather robustness.RGB images provide rich color texture in light but fail in dark conditions; TIR offers heat-object contrast but has low resolution and poor texture.
  • Challenges: Existing multi-modal trackers face scale-limited datasets and modality relationships whose dominant correlation changes across environments.These constraints make it difficult to learn effective multi-modal representations from available data.
  • Existing Methods: Full fine-tuning and prompt-tuning methods commonly assume RGB is dominant, overlooking dynamically changing dominant-auxiliary relationships.The cited methods either update the model on generated RGB-T data or freeze the backbone while attaching prompts.
  • BAT: BAT uses two modality-specific branches with fixed foundation-model parameters and a universal bi-directional adapter that transfers prompt information between modalities.The branches learn information from the other modality to integrate with current-modal features.
  • Contributions: 0.32M learnable parameters enable BAT to cross-prompt modalities and adaptively fuse information for robust multi-modal tracking.The framework is presented as an efficient alternative to methods that rely on full fine-tuning or fixed modality roles.
  • Evaluation: BAT was validated against state-of-the-art methods on multiple RGBT tracking-related datasets, with additional analysis of adapter depth and efficiency.The introduction reports experiments covering adapter effects at different layers and more efficient adapter architectures.

Related Works

Multi-modal tracking extends single-modal tracking for difficult conditions, while parameter-efficient tuning adapts frozen backbones with fewer trainable parameters. BAT’s architecture applies this idea through dual-stream RGB and thermal processing.

  • Multi-modal Tracking: Multi-modal tracking addresses cases where single-modal tracking is inadequate, including low illumination, occlusion, and thermal crossover.The related-work discussion motivates combining modalities for situations that challenge individual sensors.
  • Parameter-Efficient Tuning: Full fine-tuning updates all pre-trained model parameters on task-oriented data, making it parameter-inefficient and data-demanding.The paper contrasts this paradigm with newer parameter-efficient approaches.
  • Prompt-Based Methods: Prompt-tuning freezes the backbone and adds learnable tokens or parameters, and prior tracking methods use prompts to adapt pre-trained RGB trackers to multi-modal inputs.Protrack and ViPT are cited as examples of prompt-based adaptation for multi-modal tracking.
  • BAT Architecture: BAT uses parallel RGB and thermal streams with shared parameters, while its adapter performs feature prompting between modalities.The architecture transforms each modality into tokens and processes them through dual-stream transformer encoders.

Methodology

BAT transfers complementary information between RGB and thermal branches through a lightweight, shared-parameter bi-directional adapter while using a pre-trained transformer foundation model. The adapter progressively cross-prompts the modality-specific encoders, whose combined features produce the tracking prediction.

  • Architecture: BAT uses parallel RGB and thermal encoder branches initialized from a pre-trained transformer, with shared parameters across the modality-specific streams.The two modalities are embedded as tokens and processed separately before cross-modal interaction.
  • Cross-modal prompting: The adapter is inserted across transformer layers so each branch integrates modality-specific features with complementary information from the other branch.Feature prompts are exchanged layer by layer, allowing the input modality to adapt as the environment changes.
  • Prediction and objective: The two modal features are added and passed to a prediction head that outputs classification scores, offsets, and normalized bounding boxes under a combined training loss.The loss combines weighted focal classification loss with generalized IoU and L1 bounding-box regression losses.
  • Adapter design: Each adapter down-projects input tokens, applies a linear projection, and up-projects them before feeding the result to the other modality as a feature prompt.This lightweight three-projection structure transfers modality-specific prompts between branches.
  • Adapter integration: The adapter operates in both the multi-head self-attention and MLP stages, with the opposite modality supplying the feature prompt to each branch.The RGB-branch example combines its transformed features with a prompt extracted from the TIR branch.
  • Optimization: Only the newly added adapter is optimized while the transformer encoder and prediction head remain frozen, reducing the number of trainable parameters.The method is designed for efficient transfer of a pre-trained tracker to multi-modal scenarios.

Experimental Setting

The experiments use the RGBT234 and LasHeR multi-modal tracking datasets, with dataset-specific alignment and evaluation choices. BAT is implemented and fine-tuned using a frozen foundation-model initialization for modality-specific branches.

  • Experiments use RGBT234 and LasHeR, evaluating tracking with PR, MPR, SR, and MSR metrics.
  • RGBT234: 234K frames across 234 aligned RGB-infrared sequences characterize RGBT234, which includes 12 tracking attributes and modality-specific ground-truth labels.
  • RGBT234: RGBT234 comparisons use MPR and MSR because RGB and infrared ground truth are not completely consistent.
  • LasHeR: LasHeR aligns the local target area across modalities using labeled matching points, producing consistent RGB and thermal ground truth.
  • Implementation: BAT is trained on four NVIDIA RTX A6000 GPUs with batch size 32, AdamW, weight decay 10^-4, and learning rate 4 × 10^-4.

Comparisons

BAT is compared with 19 competing methods on quantitative benchmarks and qualitative sequences. It achieves the strongest reported results while adapting to changing modality dominance in challenging scenes.

  • Evaluation overview: The study reports overall results in Table 1 and qualitative tracking visualizations in Figure 4, alongside adapter variants in Figure 5.
  • Quantitative comparisons: 86.8% MPR and 64.1% MSR are BAT’s scores, exceeding the runner-up by 2.9% and 2.4%, respectively.
  • Quantitative comparisons: LasHeR exposes larger performance gaps and more challenging attributes than RGBT234, with several previous methods performing unsatisfactorily.
  • Quantitative comparisons: 70.2% PR and 56.3% SR improve on ViPT by 5.1% and 3.8% on LasHeR.
  • Qualitative evaluation: In qualitative sequences, BAT tracks when RGB is unavailable and performs better when RGB and TIR both provide effective information.
  • Qualitative evaluation: BAT extracts target features from both RGB and IR while eliminating RGB interference in transparent-object tracking.

Discussion

The discussion evaluates adapter directions, parameter sharing, and layer placement, then tests BAT across ordinary and extreme LasHeR attributes. Results support dynamically learning modality relevance while retaining parameter efficiency.

  • Adapter variants: BAT-RGB and BAT-TIR significantly improve over their baseline models, while differing by less than 3%.The small directional gap suggests that neither modality is consistently dominant and that dynamic learning can benefit complex conditions.
  • Adapter variants: BAT-Dual uses twice as many parameters as the universal adapter while achieving similar performance.The authors attribute this efficiency to shared foundation-model parameters across the two modality-specific branches.
  • Adapter layer placement: BAT-1 has limited performance because few multi-modal information cross-prompts occur during training, whereas BAT-4 places adapters in middle layers.The supplied passage introduces BAT-4 as the middle-layer variant, with quantitative results reported in Table 3.
  • Attribute-wise comparisons: BAT outperforms competing methods across all 19 LasHeR attributes and performs even better relative to competitors on five extreme attributes.On extreme attributes, BAT exceeds ViPT by over 10% for PR on LI and SR on AIV.
  • Attribute-wise comparisons: BAT’s PR on LI and SR on AIV surpass ViPT by over 10%, supporting its effectiveness under dynamically changing conditions.The extreme attributes are described as more dynamic than other attributes, directly testing changing dominant-auxiliary relationships.

Conclusion

BAT introduces universal bi-directional feature prompting for multi-modal tracking and dynamically extracts complementary information from changing modality relationships. Experiments on multiple RGBT datasets report superiority over competing methods, while broader modalities remain future work.

  • Conclusion: BAT introduces a universal feature prompt-learning paradigm that dynamically extracts complementary information from multiple modalities.The model is presented as a bi-directional adapter for multi-modal tracking.
  • Conclusion: Extensive experiments on multiple RGBT tracking datasets demonstrate BAT’s superiority over competing methods.
  • Conclusion: The method is currently validated for RGB and TIR tracking, with more diverse modalities identified as future work.
Loading 2312.10611v1…