Source-linked AI summary
Multi-Adapter RGBT Tracking
Chenglong Li, Andong Lu, Aihua Zheng, Zhengzheng Tu, Jin Tang
TL;DR
RGBT tracking must exploit complementary RGB and thermal information while addressing neglected shared cues and instance-specific variation. MANet jointly learns these representations with three adapters and a parallel structure, achieving state-of-the-art performance on two benchmark datasets.
Problem
Existing RGBT tracking methods emphasize modality-specific integration but often ignore modality-shared cues and instance-aware information needed for effective multimodal fusion.
Method
MANet uses generality, modality, and instance adapters for shared, modality-specific, and instance-aware feature learning, with a parallel structure reducing computational complexity.
Results
MANet achieves outstanding performance and a new state-of-the-art for RGBT tracking in experiments on two benchmark datasets.
Takeaways & Limitations
The multi-adapter representation jointly captures shared, complementary, and temporally varying target information for RGBT tracking.
Takeaways & Limitations
The authors identify noisy modalities as a remaining issue and plan to introduce modality weights; they also seek real-time performance and extension to more modalities.
Abstract
from arXiv · showhide
The task of RGBT tracking aims to take the complementary advantages from visible spectrum and thermal infrared data to achieve robust visual tracking, and receives more and more attention in recent years. Existing works focus on modality-specific information integration by introducing modality weights to achieve adaptive fusion or learning robust feature representations of different modalities. Although these methods could effectively deploy the modality-specific properties, they ignore the potential values of modality-shared cues as well as instance-aware information, which are crucial for effective fusion of different modalities in RGBT tracking. In this paper, we propose a novel Multi-Adapter convolutional Network (MANet) to jointly perform modality-shared, modality-specific and instance-aware feature learning in an end-to-end trained deep framework for RGBT tracking. We design three kinds of adapters within our network. In a specific, the generality adapter is to extract shared object representations, the modality adapter aims at encoding modality-specific information to deploy their complementary advantages, and the instance adapter is to model the appearance properties and temporal variations of a certain object. Moreover, to reduce computational complexity for real-time demand of visual tracking, we design a parallel structure of generic adapter and modality adapter. Extensive experiments on two RGBT tracking benchmark datasets demonstrate the outstanding performance of the proposed tracker against other state-of-the-art RGB and RGBT tracking algorithms.
1. Introduction
RGBT tracking combines complementary RGB and thermal information, but existing methods largely overlook modality-shared cues and instance-aware variation. MANet addresses this gap with three adapters and a parallel design intended to support effective, efficient tracking.
- RGB and thermal data offer complementary advantages for robust tracking, yet combining them effectively remains an open problem.
- Existing methods emphasize modality weights or robust modality-specific representations while often ignoring modality-shared cues and instance-aware information.
- MANet jointly learns modality-shared, modality-specific, and instance-aware features through generality, modality, and instance adapters.
- The generality adapter extracts shared object representations, while the modality adapter captures complementary modality-specific properties.
- The instance adapter models object appearance properties and temporal variations that create instance-specific tracking challenges.
- MANet uses a parallel generality–modality adapter structure to reduce computational complexity for real-time RGBT tracking.
2. Related Work
Prior RGBT trackers use modality reliability weighting, robust feature construction, and domain-adaptive networks. The related methods motivate MANet’s multi-adapter representation learning and parallel architecture.
- Modality-weighting methods use reconstruction residues or classification scores to guide adaptive fusion, but unreliable reliability estimates can cause target loss.
- Robust representation methods combine heterogeneous RGB–thermal properties and noise-aware weighting through graph learning, sparse representation, or cross-modal ranking.
- MDNet pretrains a CNN backbone for generic target representations and updates fully connected layers online to adapt to temporal target variations.
- MANet extends task-specific adapter learning with three adapter types and a parallel deep architecture designed to reduce computational burden.
3. Multi-Adapter Convolutional Network
MANet combines shared, modality-specific, and instance-aware feature learning through three adapters, while parallelizing the generic and modality adapters to reduce computational cost. Its training treats modality learning as a multi-task problem and uses progressive learning.
- Network Architecture: MANet contains a generality adapter, modality adapter, and instance adapter for jointly learning modality-shared, modality-specific, and instance-aware features.The network is trained end to end for RGBT tracking.
- Generality Adapter: The generality adapter extracts shared object representations, including common cues such as object boundaries and fine-grained textures.These cues are shared across visible and thermal imaging bands.
- Modality Adapter: The modality adapter learns modality-specific representations using small convolutional kernels parallel to the generality adapter, reducing the burden of two-stream CNNs.The kernels can be 3×3 or 1×1, reflecting the assumption that modalities share most parameters.
- Modality Adapter: GA and MA are merged through M = WGA + diagL(WMA), allowing modality-specific adaptation without extra computing costs during training or tracking.Adjusting WMA controls the network’s adaptability to the modality.
- Instance Adapter: The instance adapter models an object’s appearance properties and temporal variations to address instance-specific challenges such as occlusion, deformation, and motion blur.It uses three fully connected layers, with FC6 serving as a binary classification layer.
- Progressive Learning Algorithm: MANet formulates joint modality learning as a multi-task learning problem and trains its adapters with a progressive learning algorithm.The stages separately train GA, MA with GA fixed, and IA using offline and online learning.
4. Online Tracking
During online tracking, MANet samples candidate regions around the previous result, scores them, and selects the highest-scoring candidate. Bounding box regression is trained only in the first frame to improve scale handling and positioning efficiency.
- Candidate Sampling: At each time t, MANet samples 256 candidate regions with a Gaussian centered on the previous tracking result.The candidates are passed through the network to obtain classification scores.
- Candidate Selection: The candidate region with the highest positive score becomes the tracking result X*ᵗ.This selection is expressed by the online tracking maximization formula.
- Bounding Box Regression: Bounding box regression addresses target scale transformation and improves positioning accuracy, but is trained only in the first frame for tracking efficiency.The regression follows the MDNet tracking approach.
5. Performance Evaluation
MANet is evaluated against RGB and RGBT trackers on GTOT and RGBT234, including overall, challenge-based, ablation, qualitative, and efficiency analyses. It achieves strong benchmark performance, while results identify limitations under thermal crossover and no occlusion.
- Evaluation on GTOT dataset: MANet significantly outperforms other trackers on GTOT, reaching 89.4% PR and 72.4% SR.It exceeds SGT by 4.3% in PR and MDNet+RGBT by 8.7% in SR.
- Evaluation on RGBT234 dataset: MANet achieves superior overall performance on RGBT234, with 77.7% PR and 53.9% SR.It exceeds DAT by 4.6% in PR and C-COT by 2.5% in SR.
- Challenge-based performance: MANet beats other methods on most RGBT234 challenges, especially occlusion, low illumination, fast motion, deformation, camera moving, and background clutter.It is overshadowed on thermal crossover and no occlusion.
- Ablation Study: Removing the modality, instance, or generality adapter reduces performance, supporting their roles in modality-specific, temporally adaptive, and generic target representations.The modality adapter captures modality-specific properties, online instance adaptation handles temporal variation, and the generality adapter provides generic target representations.
- Efficiency Analysis: MANet runs at 1.11 FPS, while MANet-NO-MA and MDNet+RGBT run at 1.43 FPS and 1.61 FPS, respectively.Shared generality-adapter weights and parallel generic and modality-specific computations limit additional computational burden while MANet outperforms the compared methods in PR and SR.
6. Conclusion
MANet provides generic, modality-specific, and instance-aware representations for RGBT tracking, while its parallel adapter structure reduces computational burden. Experiments on two benchmark datasets demonstrate effectiveness and efficiency, with future work targeting additional modalities and real-time performance.
- MANet uses three adapter types to extract generic, modality-specific, and instance-aware deep features for varied tracking challenges.The network combines these representations in an RGBT framework for visual tracking.
- A parallel structure reduces MANet's computational burden effectively.
- Experiments on two benchmark datasets demonstrate the proposed tracking method's effectiveness and efficiency.
- Future work will extend the framework to more modalities, including depth, and improve the network toward real-time performance.