Source-linked AI summary
SDSTrack: Self-Distillation Symmetric Adapter Learning for Multi-Modal Visual Object Tracking
Xiaojun Hou, Jiazheng Xing, Yijie Qian, Yaowei Guo, Shuo Xin, Junhao Chen, Kai Tang, Mengmeng Wang, Zhengkai Jiang, Liang Liu, Yong Liu
TL;DR
Existing multimodal trackers face a modality gap and either costly full fine-tuning on scarce data or asymmetric designs that remain dependent on RGB. SDSTrack uses symmetric lightweight adaptation and complementary masked-patch distillation; it achieves state-of-the-art results across several modality combinations and reports strong performance in extreme conditions.
Problem
Multimodal tracking faces a modality gap, while full fine-tuning is costly on scarce multimodal data and asymmetric trackers remain dependent on RGB.
Source: §1 · p. 2, §3.1.1 · p. 4Method
SDSTrack symmetrically fine-tunes lightweight adapters to transfer RGB feature extraction to other modalities, and self-distills complementary masked and clean data.
Source: §3 · p. 4, §3.1.1 · p. 4, §5 · p. 8Results
Takeaways & Limitations
Abstract
from arXiv · showhide
Multimodal Visual Object Tracking (VOT) has recently gained significant attention due to its robustness. Early research focused on fully fine-tuning RGB-based trackers, which was inefficient and lacked generalized representation due to the scarcity of multimodal data. Therefore, recent studies have utilized prompt tuning to transfer pre-trained RGB-based trackers to multimodal data. However, the modality gap limits pre-trained knowledge recall, and the dominance of the RGB modality persists, preventing the full utilization of information from other modalities. To address these issues, we propose a novel symmetric multimodal tracking framework called SDSTrack. We introduce lightweight adaptation for efficient fine-tuning, which directly transfers the feature extraction ability from RGB to other domains with a small number of trainable parameters and integrates multimodal features in a balanced, symmetric manner. Furthermore, we design a complementary masked patch distillation strategy to enhance the robustness of trackers in complex environments, such as extreme weather, poor imaging, and sensor failure. Extensive experiments demonstrate that SDSTrack outperforms state-of-the-art methods in various multimodal tracking scenarios, including RGB+Depth, RGB+Thermal, and RGB+Event tracking, and exhibits impressive results in extreme conditions. Our source code is available at https://github.com/hoqolo/SDSTrack.
1. Introduction
RGB tracking can degrade when imaging quality deteriorates, while multimodal trackers face a trade-off between costly full fine-tuning and asymmetric designs that leave RGB dominant. SDSTrack uses lightweight symmetric adaptation and complementary masked patch distillation; experiments report state-of-the-art performance across several modality combinations and strong results in extreme conditions.
1. Introduction
- RGB trackers can degrade in complex conditions when imaging quality deteriorates, motivating multimodal inputs for more comprehensive information.The introduction identifies safety-sensitive applications such as autonomous driving as a critical context.Source: §1 · p. 1
- Full fine-tuning of RGB-based trackers adds many training parameters and, with scarce multimodal data, can limit generalized representation; prompt tuning is a parameter-efficient alternative.The passage describes prompt tuning as transferring pre-trained RGB trackers to other modalities.Source: §1 · p. 2
- Under modality drop and occlusion, asymmetric trackers’ performance decreases significantly when RGB is dropped, while symmetric trackers remain relatively robust.The toy experiment also reports moderate decreases for asymmetric trackers when other modalities are dropped, and noticeable decreases when both are occluded.Source: §1 · p. 2, Figure 2 · p. 2
- SDSTrack combines parameter-efficient fine-tuning with a symmetric framework and complementary masked patch distillation to address modality imbalance and improve robustness and accuracy.The method transfers RGB feature-extraction capability to other domains and uses random complementary masks during training.Source: §1 · p. 2
- SDSTrack achieves state-of-the-art performance across RGB+Depth, RGB+Thermal, and RGB+Event benchmarks, with impressive tracking results in extreme conditions.The reported experiments span several benchmarks and modality combinations.Source: §1 · p. 2
2. Related Works
Related work covers multimodal tracking methods that fuse or prompt across modalities, alongside parameter-efficient fine-tuning approaches developed for settings with limited training data. The paper positions symmetric, adapter-based tuning as a response to the limitations of existing multimodal tracking approaches.
2.1. Multimodal object tracking
- Visual object tracking locates a target in subsequent frames given its bounding box in the first frame, and multimodal data is explored because RGB reliability can be unstable in challenging scenarios.The related-work passage lists RGB, LiDAR, Depth, Event, and Thermal among modalities used in other vision tasks.Source: §2.1 · p. 2
- Multimodal tracking work includes progressive fusion, mixed-attention fusion, cross-modal masking, weighted-sum fusion, and prompting, with limitations reported for some approaches.ProTrack is described as lacking in-depth multimodal exploration, while ViPT’s asymmetric design is said not to fully exploit other modalities.Source: §2.1 · p. 3
2.2. Parameter-efficient Fine-tuning for Vision Models
- Parameter-efficient fine-tuning freezes the pre-trained model and trains only a few additional parameters, a useful approach when fine-tuning data is limited.Adapter-based tuning inserts trainable adapters, whereas prompt tuning introduces learnable prompt tokens.Source: §2.2 · p. 3
- The related-work discussion identifies a gap in multimodal VOT: ViPT uses prompt tuning but has limited pre-trained knowledge recall across the modality gap and may depend on the primary modality.The passage also states that adapter-based tuning had not yet been used in this field.Source: §2.2 · p. 3
3. Method
SDSTrack adapts a pre-trained RGB tracker with lightweight symmetric adapters and complementary masked-patch distillation. The design transfers features across modalities, fuses them, and trains for robustness to missing or occluded inputs.
3. Method
- SDSTrack combines lightweight multimodal adaptation with complementary masked-patch distillation to transfer RGB features, fuse modalities, and improve robustness and accuracy in extreme conditions.X denotes modalities other than RGB, including Depth, Thermal, and Event.Source: §3 · pp. 3–4
3.1. Symmetric Multimodal Adaptation (SMA)
- Symmetric Multimodal Adaptation freezes the pre-trained model and trains only its adapters, reducing trainable parameters while adapting RGB trackers to multimodal tracking.It comprises cross-modal and multimodal-fusion adaptation.Source: §3.1 · p. 4
3.1.1 Cross Modal Adaptation
- Cross Modal Adaptation transfers RGB feature-extraction capability to other modalities using bottleneck adapters with two fully connected layers, GELU activation, and a residual connection.Adapters are inserted after MSA and parallel to the MLP in ViT blocks.Source: §3.1.1 · p. 4
3.1.2 Multimodal Fusion Adaptation
- Multimodal Fusion Adaptation reuses the last ViT block of each encoder stage to process concatenated RGB and X features, enabling interaction among modality tokens.An attention mask zeros within-modality attention to avoid overfitting and preserve the pre-trained model's modeling capability.Source: §3.1.2 · p. 4
3.2. Complementary Masked Patch Distillation
- Complementary Masked Patch Distillation combines random complementary patch masking with self-distillation to explore modality complementarity and improve robustness in extreme scenarios.The strategy targets trackers' reliance on specific modalities and limited exploration of complementary information.Source: §3.2 · p. 5
3.2.1 Random Complementary Patch Mask (RCPM)
- Random Complementary Patch Mask zeros randomly selected patch embeddings in one modality and masks corresponding portions in the other, keeping at least one modality valid.The strategy is intended to support tracking accuracy and robustness when modalities are compromised.Source: §3.2.1 · p. 5
3.2.2 Self-Distillation Learning (SD)
- Self-distillation uses clean features to supervise masked features through mean squared error between their fusion features at each stage.The clean and masked paths share all network parameters.Source: §3.2.2 · p. 5
3.3. Prediction Head and Supervised Loss
- The prediction head takes the sum of RGB and X search features, and the overall tracking loss combines clean-data, masked-data, and self-distillation losses.The same OSTrack head and loss function supervise clean and masked tracking data.Source: §3.3 · pp. 5–6
4. Experiments
Experiments evaluate SDSTrack on RGB-D, RGB-T, and RGB-E benchmarks, reporting state-of-the-art performance and robustness under challenging conditions. Ablations assess its adaptation and distillation components, with inference speed and visualizations also reported.
4.1. Implementation Details
- Training uses the DepthTrack, LasHeR, and VisEvent training sets; RGB-D, RGB-T, and RGB-E models run for 15, 40, and 50 epochs, respectively.Training uses four NVIDIA 3090Ti GPUs with global batch size 64.Source: §4 · p. 6, §4.1 · p. 6
4.2. Comparison with State-of-the-arts
- DepthTrack exceeds previous state of the art by 2.7% precision, 1.3% recall, and 2.0% F-score; RGBT234 reaches new state-of-the-art 84.8% MPR and 62.5% MSR.Source: §4.2 · p. 6
- LasHeR reaches 66.5% precision and 53.1% success, while VisEvent reaches 76.7% precision and 59.7% success; both are new state of the art.LasHeR precision exceeds the previous best by 1.4%.Source: §4.2 · pp. 6–7, Figure 5 · p. 7, Figure 6 · p. 7
4.3. Exploration Study and Analysis
- Under RGB-frame drops, SDSTrack improves over ViPT by 3.3% RGB-D F-score, 23.2%/16.4% RGB-T precision/success, and 11.6%/7.9% RGB-E precision/success.Source: Table 4 · p. 7, §4.3 · p. 7
- SDSTrack achieves the best performance on challenging attributes including illumination variation, motion blur, and out-of-view scenarios.Source: Figure 7 · p. 7, §4.3 · p. 7
- Adding SMA improves the baseline across DepthTrack, RGBT234, and VisEvent; RCPM and SD provide further gains on reported metrics.The reported gains include 1.9% DepthTrack F-score from RCPM and 2.0% DepthTrack F-score from SD.Source: §4.3 · p. 7, Table 5 · p. 8
- CMA surpasses frozen and fully fine-tuned baselines with few training parameters, while combining CMA and MFA yields the best performance.Source: Table 6 · p. 8, §4.3 · p. 8
- SMA improves robustness, RCPM alone offers limited improvement, and combining RCPM with SD significantly improves robustness; inference runs at 20.86 fps and visualizations show multimodal tracking in challenging situations.Source: Table 7 · p. 8, §4.3 · p. 8, Table 8 · p. 8
5. Conclusion
SDSTrack combines symmetric adapter-based tuning of a pre-trained RGB tracker with complementary patch masking and self-distillation. Experiments on multiple benchmarks demonstrate its effectiveness, while training computational complexity from self-distillation remains a limitation.
5. Conclusion
- SDSTrack uses lightweight adapter-based tuning to transfer a pre-trained RGB tracker’s feature extraction capability to other domains and fuse modalities symmetrically.Source: §5 · p. 8
- Random complementary patch masking and self-distillation of masked and clean data flows are used to enhance robustness in extreme scenarios.Source: §5 · p. 8
- Self-distillation learning introduces additional computational complexity during training, which the authors identify for future exploration.Source: §5 · p. 8