Source-linked AI summary
Template-Search Domain Adaptation via Multi-Stage Feature Alignment for Cross-Modal Object Tracking
Fereshteh Aghaee Meibodi, Amir Mehdi Soufi Enayati, Shadi Alijani, Homayoun Najjaran
TL;DR
When template and search frames come from different sensing modalities, their representation gap can impair tracking, while conventional multi-modal methods assume paired modalities are available together. TSDA-Track aligns features during training using adversarial or contrastive strategies, and evaluations report stronger performance across cross-modal protocols than representative trackers.
Problem
Template and search frames may come from different sensing domains, creating a distribution shift that can impair transformer interaction and target-state estimation.
Source: §1 · p. 2, §3.1 · p. 4Method
TSDA-Track reduces modality-induced template-search discrepancy through training-only feature alignment, using adversarial alignment before interaction or contrastive alignment after the encoder.
Source: §1 · pp. 2–3Results
Across cross-modal switching protocols, explicit alignment outperforms representative single-modal and prior cross-modal trackers, with evaluations on LasHeR, zero-shot RGBT234/GTOT, and Anti-UAV-024.
Source: §1 · p. 3, §6 · p. 13Takeaways & Limitations
The evaluations support feature alignment as an approach to cross-modal tracking that retains shared inference without modality-specific routing.
Source: §6 · p. 13TSDA-Track is architecture-dependent, and extending it to one-stream trackers such as OSTrack or MixFormer requires reformulating alignment.
Source: §5 · p. 12
Abstract
from arXiv · showhide
Visual object tracking typically assumes that the initial template and subsequent search frames share the same sensing modality. In practice, sensor availability or operation may change over time, creating a substantial representation gap between template and search frames. Unlike conventional multi-modal tracking where paired modalities are simultaneously available, cross-modal tracking requires localization when template and search frames originate from different active modalities. Accordingly, we introduce TSDA-Track, a Template-Search Domain Adaptation framework to reduce modality discrepancy during training. We investigate two feature alignment strategies. Pre-AFA TSDA-Track applies adversarial alignment before transformer's template-search interaction to suppress modality-specific bias. Enc-CFA TSDA-Track applies contrastive alignment to encoder representations after interaction to strengthen target-level cross-modal correspondence. Both variants retain a shared inference pipeline without modality-specific branches. Experiments on LasHeR, and zero-shot evaluations on RGBT234 and GTOT under multiple cross-modal protocols demonstrate improvements over representative state-of-the-art trackers. For instance, under the modality-switch protocol on RGBT234, Pre-AFA TSDA-Track achieves an SR/PR of 43.2/56.0, compared with 36.8/50.0 for ToMP-101 baseline. In addition, a study on Anti-UAV-024 further verifies the applicability of TSDA-Track to aerial tracking. Our study highlights the effectiveness of feature alignment domain adaptation for cross-modal tracking.
1 Introduction
Cross-modal tracking addresses the template–search domain gap that arises when template and search frames come from different sensing modalities. TSDA-Track uses training-time feature alignment to reduce that gap, with two alignment strategies and a shared inference pipeline; evaluations report improvements across cross-modal protocols.
1 Introduction
- Single-modal and conventional multi-modal trackers assume consistent template-search modalities, while modality mismatch can weaken feature interaction and degrade localization.Conventional multi-modal tracking typically requires all modalities to be simultaneously available and synchronized.Source: §1 · pp. 1–2
- Pre-AFA applies adversarial alignment before template-search interaction, while Enc-CFA uses contrastive alignment after the encoder to promote target-level cross-modal consistency.The variants suppress modality-specific bias and strengthen target-level correspondence, respectively.Source: §1 · pp. 2–3
- The paper formulates cross-modal tracking from a template-search domain-mismatch perspective and evaluates fixed and temporally dynamic modality-switching protocols.Source: §1 · p. 3
- TSDA-Track aligns modality-induced template-search discrepancy during training, then retains a shared, modality-agnostic inference pipeline without modality estimation, fusion, modality-specific branches, or routing.The alignment branches are removed after training.Source: §1 · p. 3
- Ablations show that cross-modal performance depends on the alignment objective, its strength, and its integration stage.Source: §1 · p. 3
- TSDA-Track shows better overall performance than representative trackers across multiple cross-modal protocols on RGB-T datasets, including LasHeR and zero-shot RGBT234/GTOT evaluations, and is validated on Anti-UAV aerial tracking.Source: §1 · p. 3
2 Related Work
Prior single-object trackers generally assume same-modality template and search frames, while conventional multi-modal trackers rely on synchronized paired modalities. Existing cross-modal methods use modality-aware inference, whereas TSDA-Track targets the within-sequence template-search gap through training-only feature alignment.
Single Object Tracking
- Most single-object trackers assume visually consistent, same-modality template and search frames, despite differing approaches from discriminative and Siamese methods to transformer-based trackers.Source: §2 · p. 3
Multi-Modal and Cross-Modal Object Tracking
- Conventional RGB-T and RGB-X trackers fuse paired modalities, whereas cross-modal methods accommodate switching but commonly rely on modality prediction, adaptive weighting, or state-dependent routing.The proposed TSDA-Track instead uses training-only alignment with a shared inference pipeline.Source: §2 · p. 3
Domain Adaptation for Tracking
- Prior tracking domain adaptation targets shifts across environments, sensing domains, or datasets and models; TSDA-Track instead targets the within-sequence gap between cross-modal template and search frames.Source: §2 · pp. 3–4
3 Proposed Method
TSDA-Track extends ToMP with training-only alignment of template and search features when their modalities differ, while retaining a shared inference pipeline. It offers adversarial alignment before transformer interaction and contrastive alignment after encoder interaction.
3 Proposed Method
- TSDA-Track extends ToMP’s shared template-search pipeline with training-only alignment, while the transformer encoder jointly processes template and search features and the decoder estimates the target state.The alignment branches are removed at inference, with no modality prediction, modality-specific branches, or routing.Source: §3 · p. 4, §3.2 · p. 4
3.1 Cross-Modal Tracking Setting
- Cross-modal tracking addresses template and search frames from different sensing domains, whose distribution shift can impair transformer interaction and target-state estimation.The setting contrasts with standard SOT’s same-modality assumption.Source: §3.1 · p. 4
3.2 Domain Adaptation for Cross-Modal Tracking
- TSDA-Track jointly optimizes either alignment objective with tracking classification and regression losses, then removes the auxiliary alignment loss at inference.The shared pipeline needs no modality prediction, modality-specific branches, or routing decisions.Source: §3.2 · p. 4
3.2.1 Adversarial Feature Alignment
- Pre-AFA applies adversarial alignment to pooled pre-encoder descriptors, with a shared discriminator predicting modality and reversed gradients encouraging modality-invariant features.Pooling rather than modifying spatial feature maps is intended to avoid suppressing location-sensitive cues needed for localization.Source: §3.2.1 · pp. 4–5
3.2.2 Contrastive Feature Alignment
- Enc-CFA applies contrastive alignment to pooled, projected encoder representations, drawing corresponding template-search pairs together while separating non-corresponding tuples.The bidirectional objective is jointly optimized with tracking losses to preserve information needed for classification and bounding-box regression.Source: §3.2.2 · pp. 5–6
3.3 Offline Training
- Training samples use two same-modality templates and a search frame sampled with a 0.5 mismatch probability; cross-modal directions are sampled uniformly.The tracking classification and GIoU losses are augmented by the relevant alignment loss, whose weight controls its contribution.Source: §3.3 · p. 6
3.4 Online Tracking
- At online tracking, alignment branches are disabled and the shared ToMP pipeline uses neither modality labels nor modality-specific inference modules.The initial annotated template is retained while other memory slots are updated with confident predictions, so online memory may contain RGB, thermal, or mixed-modality frames.Source: §3.4 · p. 6
4.1 Implementation Details
The implementation fine-tunes ToMP-101 on LasHeR and evaluates fixed and switching modality-mismatch protocols. The reported setup also specifies the backbone, training schedule, and evaluation hardware.
4.1 Implementation Details
- The ToMP transformer uses six encoder and six decoder layers, and all TSDA-Track variants use ToMP inference settings.The backbone is unfrozen during fine-tuning, and auxiliary alignment losses are enabled after five epochs.Source: §4.1 · p. 6
- Models are fine-tuned from pretrained ToMP-101 on LasHeR’s training split for 30 epochs using 40k subsequences and batch size 8.Training uses an NVIDIA H100 GPU; evaluation uses an NVIDIA TITAN RTX GPU.Source: §4.1 · p. 6
- The three protocols are RGB→T, T→RGB, and Switch, where Switch alternates search modality every 200 frames from an RGB template.Performance is reported as SR0.5 and PR at a 20-pixel center-error threshold.Source: §4.1 · p. 7
4.2 Ablation Study and Analysis
The ablations show that cross-modal robustness depends on the alignment objective, where it is integrated, and its strength. Feature-level analysis also finds that each alignment strategy is most effective at its intended feature stage.
4.2.3 Feature-Level Analysis
- Figure 3 shows strongest pre-encoder feature mixing under Pre-AFA; Rloc is lowest pre-encoder for Pre-AFA (0.435) and post-encoder for Enc-CFA (0.594).Lower Rloc indicates stronger same-object RGB-thermal alignment relative to different-object pairs.Source: §4.2.3 · pp. 7–8
4.2.1 Effect of Alignment Strategy and Stage
- 40.5/53.9 on RGB→T and 43.2/56.0 on Switch: Pre-AFA leads those protocols, while Pre-AFA+Enc-CFA achieves 42.6/53.2 on T→RGB.Cross-modal training alone improves fixed-direction settings but degrades Switch; joint alignment does not consistently improve RGB→T or Switch.Source: Table 1 · p. 7, §4.2.1 · p. 7
4.2.2 Effect of Alignment Strength
- γ = 0.1 consistently outperforms γ = 1.0 for Pre-AFA, while τ = 0.1 performs best across protocols for Enc-CFA.The authors attribute the weaker results at stronger alignment settings to suppression of target-discriminative features and insufficient accommodation of modality and appearance variation, respectively.Source: §4.2.2 · p. 7, Table 2 · p. 8
4.3 Comparison with Representative Trackers
TSDA-Track’s alignment variants perform differently across cross-modal benchmarks: Pre-AFA is strongest on RGBT234 and LasHeR, while Enc-CFA leads on GTOT. Attribute, protocol-curve, and aerial evaluations further examine where these gains hold.
4.3 Comparison with Representative Trackers
- On RGBT234, Pre-AFA achieves the highest zero-shot SR in RGB→T and Switch; ODTrack’s highest T→RGB SR of 44.0 is direction-specific.The comparison uses released checkpoints; other cross-modal tracker comparisons were infeasible because implementations or models were unavailable.Source: §4.3 · pp. 8–9
- Pre-AFA leads on RGBT234 and LasHeR, while Enc-CFA achieves the best GTOT SR0.5 under all three protocols in zero-shot evaluation.GTOT differs in acquisition conditions and modality distribution, including grayscale input.Source: §4.3 · pp. 8–10, Table 4 · p. 9
4.3.1 Attribute-Based Performance on RGBT234.
- Pre-AFA has the strongest overall RGBT234 attribute performance, while low resolution and background clutter remain comparatively difficult.Enc-CFA improves multiple attributes, but its gains are less consistent than Pre-AFA’s.Source: §4.3.1 · p. 10, Figure 5 · p. 11
4.3.2 Success and Precision Plots.
- Across RGBT234 protocols, Pre-AFA shows the most consistent overall success and precision curves; ODTrack’s strength is limited to T→RGB.Enc-CFA improves over ToMP-101 and CM-ToMP-101, while CM-MAFNet’s high precision does not translate into equally reliable bounding-box overlap and scale estimation.Source: §4.3.2 · pp. 10–11, Figure 6 · p. 11
4.3.3 Aerial Cross-Modal Tracking.
- On aerial Anti-UAV-024, Pre-AFA performs best under all protocols, improving SR0.5/PR over CM-ToMP-101 by 6.5/6.6, 5.2/4.9, and 4.4/4.1 points.Enc-CFA ranks second; combining both alignments is inferior to Pre-AFA alone.Source: §4.3.3 · p. 11, Table 5 · p. 12
5 Limitations
The stated limitations concern geometric mismatch and dependence on the ToMP architecture, which constrain bounding-box regression and extension to some tracker designs.
5 Limitations
- TSDA-Track does not explicitly address geometric mismatch, which may limit bounding-box regression under pixel misalignment, scale or aspect-ratio variation, and clutter.The authors identify geometry-aware alignment as future work.Source: §5 · p. 12
- The alignment modules depend on ToMP’s explicit pre- and post-interaction stages, so TSDA-Track is not backbone-agnostic.Applying it to one-stream trackers such as OSTrack and MixFormer requires reformulating alignment because template and search tokens interact early.Source: §5 · p. 12
- The authors identify broader modality-transition benchmarks and extension to RGB-NIR, RGB-depth, and event data as future work.Source: §5 · p. 12
6 Conclusions
The conclusion presents TSDA-Track as a training-only feature-alignment framework with two strategies for cross-modal tracking. Its evaluations support alignment’s effectiveness while showing that the objective and integration stage matter.
6 Conclusions
- TSDA-Track uses training-only adversarial alignment before template-search interaction or contrastive alignment after encoder interaction.Source: §6 · p. 13
- Evaluations on LasHeR, zero-shot RGBT234 and GTOT, and Anti-UAV-024 show improved cross-modal performance while retaining shared inference without modality-specific routing.Source: §6 · p. 13
- Pre-AFA is more consistent under broader appearance variation, whereas Enc-CFA transfers better under acquisition-condition and modality-statistic shifts.The conclusion also states that cross-modal training alone is insufficient for stable performance under modality discrepancy.Source: §6 · p. 13