PaperlayerView exact revision on arXiv

Source-linked AI summary

End-to-End Self-Supervised RGB-T Tracking without Modality Misleading

Shenglan Li, Rui Yao, Kunyang Sun, Hong Jia, Yong Zhou, Javen Qinfeng Shi, Xinyu Zhang

arXiv:2609.37162v1cs.CV

TL;DR

RGB-T tracking methods often require costly aligned box annotations or use two-stage pseudo-label training. ESMTrack learns end-to-end from initial-frame annotations using forward-backward tracking consistency and task-level modality reliability. On five benchmarks, it reports competitive state-of-the-art performance among self-supervised trackers, strong generalization, and fast inference.

  • Problem

    Supervised RGB-T tracking relies on costly modality-aligned box annotations, while many self-supervised methods use two-stage pseudo-label training that is sensitive to label quality and prevents end-to-end optimization.

    Source: §1 · p. 2

  • Method

    ESMTrack uses initial-frame grounding and search-frame temporal triplet losses with forward-backward consistency, plus task-level modality reliability estimation to guide training-time modality decoupling.

    Source: §1 · pp. 3–4

  • Results

    Across five RGB-T benchmarks, ESMTrack achieves competitive state-of-the-art performance among self-supervised trackers, strong generalization, and fast inference.

    Source: §1 · p. 4

  • Takeaways & Limitations

    The framework learns RGB-T tracking without offline pseudo-label generation or dense frame-level bounding-box annotations.

    Source: §1 · p. 4

    The primary performance gap occurs in motion blur and thermal crossover, where GDSTrack leads; motion blur weakens APCE-based modality estimation, while thermal crossover makes modality-aware fusion challenging.

    Source: §3.7 · p. 16

Abstract

from arXiv · show

RGB-T object tracking leverages the complementary characteristics of visible and thermal infrared modalities to improve robustness under adverse conditions. Existing supervised methods typically rely on costly modality-aligned bounding box annotations, while most self-supervised approaches follow a two-stage pseudo-labeling paradigm, making tracker training sensitive to pseudo-label quality and preventing joint end-to-end optimization. In this paper, we propose ESMTrack, a fully end-to-end self-supervised RGB-T tracking framework without offline pseudo-label generation or dense frame-level bounding box annotations. Given only the standard initial-frame annotation used in visual tracking, ESMTrack learns discriminative and temporally consistent representations through two complementary objectives: a grounding triplet loss on annotated initial frames and a cross-frame temporal triplet loss on unlabeled search frames, with reliable samples selected by forward-backward consistency. To address modality dominance bias, ESMTrack employs a three-branch architecture consisting of a fusion branch and two unimodal branches for RGB and thermal inputs. We quantify modality contributions using the Average Peak-to-Correlation Energy by measuring response discrepancies between the fusion and unimodal branches. The resulting reliability estimates guide a training-time modality decoupling mechanism that suppresses dominant-modality shortcuts and adaptively weights cross-modal contrastive learning for task-level alignment. Extensive experiments on five RGB-T tracking benchmarks show that ESMTrack achieves competitive state-of-the-art performance, strong cross-dataset generalization, and real-time inference speed. The source code is available at https://github.com/LiShenglana/ESMTrack.

1 Introduction

ESMTrack replaces two-stage pseudo-label training with end-to-end self-supervised RGB-T tracking from the standard initial-frame annotation. It combines forward-backward tracking consistency with task-level modality reliability estimation to address pseudo-label dependence and modality dominance.

1 Introduction

  • Earlier self-supervised RGB-T methods use two-stage pseudo-label generation and tracker training, leaving training sensitive to pseudo-label quality and preventing end-to-end optimization.The paper notes that errors from saliency detection or optical flow can deviate from target locations and propagate into tracker learning.Source: §1 · pp. 2–3
  • Adaptive Modality Decoupling estimates modality contribution from task-level response quality and suppresses the dominant modality during training to encourage complementary features.At inference, only the learned fusion branch is used.Source: §1 · p. 4
  • Experiments on five RGB-T benchmarks show competitive state-of-the-art performance among self-supervised trackers, strong generalization, and fast inference.The introduction reports these results without listing benchmark-specific metrics.Source: §1 · p. 4
  • ESMTrack learns without offline pseudo-label generation or dense frame-level bounding-box annotations, using only the standard initial-frame annotation.Its forward-backward consistency uses a temporal triplet loss on unlabeled search frames and a grounding triplet loss on annotated initial frames.Source: §1 · pp. 3–4

2 Related works

Related work covers supervised RGB-T fusion and self-supervised tracking, highlighting annotation dependence and challenges in assessing modality reliability. ESMTrack is positioned as an end-to-end, pseudo-label-free RGB-T approach.

2.1 RGB-T Tracking

  • Supervised RGB-T trackers use varied fusion, temporal, and prompt-learning strategies, but commonly estimate reliability from feature activation intensity and depend on costly annotated data.The paper contrasts this with task-level response quality estimation to address modality misleading.Source: §2.1 · pp. 4–5

2.2 Self-supervised Tracking

  • Self-supervised tracking spans cycle consistency, contrastive learning, synthesized data, and pseudo-label generation, while RGB-T methods include S2OTFormer and GDSTrack.The paper identifies pseudo-label-free, end-to-end learning from intrinsic signals in both modalities as needing further investigation.Source: §2.2 · p. 5
  • ESMTrack’s pipeline combines a temporal triplet loss on unlabeled search frames with a grounding triplet loss on the annotated initial frame.Backward IoU filtering supports temporal semantic consistency, while the grounding constraint enhances spatial discriminability.Source: Figure 2 · p. 6

3 Methodology

ESMTrack uses forward-backward tracking consistency and two complementary triplet losses. Its three-branch architecture combines fused and unimodal RGB and infrared features, with shared early layers and separate later layers.

3.1 Framework Overview

  • The framework applies temporal triplet learning to unlabeled search frames and grounding triplet learning to the annotated initial frame.A fusion branch and two unimodal branches feed the adaptive modality mechanism; the fusion branch uses prompt-tuning on a frozen pre-trained foundation model.Source: §3.1 · p. 6

3.2 Multi-Modal Input and Feature Extraction

  • Inputs comprise tightly cropped initial-frame RGB/infrared templates, larger-context initial-frame grounding crops, and RGB/infrared search-frame pairs from subsequent frames.The visible and infrared inputs are tokenized by independent patch-embedding layers into D-dimensional token sequences.Source: §3.2 · p. 7
  • The backbone has fusion, RGB, and infrared branches; the first K transformer layers share weights and use cross-modal prompts, while the final L −K layers have independent parameters.The three branches produce fused and unimodal search-region features for the AMD module; backward tracking uses the same architecture but omits the cross-modal contrastive loss.Source: §3.2 · p. 7

3.3 Adaptive Modality Decoupling with Response-Guided Gating

Adaptive Modality Decoupling estimates modality reliability from tracking-response quality and uses gated feature subtraction to reduce dominant-modality shortcuts. Its gates preserve weak signals and fall back to unsuppressed fusion when both modalities are degraded.

3.3 Adaptive Modality Decoupling with Response-Guided Gating

  • AMD replaces feature activation intensity with task-level tracking confidence to identify and suppress the currently dominant modality during training.This is intended to make the fusion branch mine complementary information and improve robustness to modality degradation.Source: §3.3 · p. 7
  • APCE-based reliability estimation compares response maps from the fusion branch and unimodal branches to assess each modality’s contribution.A higher APCE indicates a more peaked, confident response map; the TIR comparison illustrates how a fusion-versus-unimodal response difference indicates RGB contribution.Source: §3.3 · pp. 7–8
  • Reliability scores gate subtraction of unimodal features from the fused representation, suppressing a modality only when its score exceeds threshold τ.The gate uses a sigmoid, with temperature T set to 0.5.Source: §3.3 · p. 8
  • When a modality is reliable, gated subtraction acts as feature-level dropout; when its quality is low, its gate is zero, and joint degradation leaves fusion unsuppressed.Continuous partial suppression preserves residual gradient signal, while the joint-degradation fallback avoids acting on unreliable APCE estimates.Source: §3.3 · pp. 8–9

3.4 Triplet Loss for Grounding and Search Frames

ESMTrack uses complementary triplet losses to learn from annotated grounding frames and unlabeled search frames without offline pseudo-labels. Backward-tracking IoU filters unreliable search samples, while the losses use ground-truth regions and cross-sequence negatives to preserve localization and object identity.

3.4 Triplet Loss for Grounding and Search Frames

  • Grounding and search losses address distinct supervision conditions: annotated frames provide spatial supervision, while search frames enforce cross-frame identity consistency.The grounding loss uses spatially shifted box regions as hard negatives; the temporal loss uses negatives from other object identities.Source: §3.4 · p. 9
  • For unlabeled search frames, the cross-frame temporal triplet loss links predicted-location features to the corresponding grounding-frame ground-truth representation and repels other object identities.Negatives are formed by rolling the batch of positive features by one position.Source: §3.4 · pp. 9–10
  • Only search samples with backward-tracking IoU above τ contribute, filtering cases where the tracker has drifted to an incorrect object.The backward prediction is compared with the grounding-frame ground-truth box.Source: §3.4 · p. 9
  • The grounding triplet loss pulls predicted-region features toward the ground-truth object region and away from a nearby spatially perturbed hard negative.Its anchor and negative features are detached, so the loss updates only the predicted-region branch.Source: §3.4 · p. 10

3.5 Training and Inference

Training combines localization and contrastive objectives, including task-level weighted cross-modal alignment and augmented-view consistency. At inference, only the fusion branch is retained for online single-pass tracking.

3.5 Training and Inference

  • Ground-truth grounding frames alone provide localization supervision through GIoU, ℓ1 box regression, and predicted Gaussian-heatmap Focal losses.Their respective weights are 2.0 and 5.0 for GIoU and ℓ1 regression; the passage gives no weight for Focal loss.Source: §3.5 · p. 11
  • The cross-modal alignment loss is a bidirectional InfoNCE objective on search frames, weighted by modality importance to constrain task-level representation learning.The multi-view instance contrastive loss applies NT-Xent between fused features from independently augmented views of a grounding frame.Source: §3.5 · p. 11
  • RGB-TIR cross-view contrastive learning applies NT-Xent to region-level features from augmented grounding-frame views to pull together the same object across modalities.The region features are obtained by masked average pooling over encoder output tokens.Source: §3.5 · p. 11
  • At inference, the tracker operates online in a single pass, retaining only the fusion branch and discarding unimodal branches and suppression operations.The template is encoded once; each predicted box sets the next frame’s search region, with no additional post-processing.Source: §3.5 · p. 11

3.6 Datasets and Settings

The evaluation spans five RGB-T benchmarks with different scales and settings, using precision, normalized precision, and success metrics. Training uses LasHeR, with triplet losses activated in stages.

3.6 Datasets and Settings

  • LasHeR comprises 1,224 sequence pairs with 19 challenge attributes and is identified as the primary testbed for evaluating self-supervised RGB-T trackers.The five benchmarks also include small-scale GTOT, RGBT210, RGBT234, and aerial VTUAV.Source: §3.6 · p. 12
  • Performance is evaluated with PR, NPR, and SR; the final success score on each of the five datasets is the area under the success-rate curve.The center-distance threshold is 5 pixels for GTOT and 20 pixels for RGBT210, RGBT234, LasHeR, and VTUAV-ST.Source: §3.6 · pp. 12–13
  • Training uses the LasHeR dataset for 20 epochs with AdamW, learning rate 2.5 × 10^-4, weight decay 10^-4, and batch size 8.The backbone learning rate is one tenth of the head learning rate.Source: §3.6 · p. 13
  • Triplet losses are activated progressively: grounding-frame triplet loss after epoch 10 and search-frame triplet loss after epoch 15.Source: §3.6 · p. 13

3.7 Main Results

ESMTrack ranks among the top two methods across five benchmarks and performs best on LasHeR, while training end-to-end without offline pseudo-label generation. Results also show strong performance across challenge attributes, with speed and response-map analyses supporting its practical evaluation.

3.7 Main Results

  • On LasHeR, ESMTrack surpasses GDSTrack by 1.4% PR, 3.4% NPR, and 0.6% SR, and ranks first or second on the remaining benchmarks.On VTUAV-ST it ranks second with 59.7% PR and 48.3% SR; on GTOT it records 76.8% PR and 64.0% SR, ranking second.Source: §3.7 · p. 15
  • On RGBT234 and RGBT210, ESMTrack leads all methods in SR at 49.6% and 47.4%, while its PR trails GDSTrack by 0.3 and 0.5 percentage points.Figure 6 reports second-best PR and best SR on RGBT234; Figure 7 reports best PR, NPR, and SR on LasHeR.Source: §3.7 · pp. 15, 17, 18, Figure 6 · p. 17, Figure 7 · p. 17
  • Across five benchmarks, ESMTrack ranks among the top two methods and achieves the best performance on LasHeR without offline pseudo-label generation.It surpasses GDSTrack on LasHeR by 1.4% PR, 3.4% NPR, and 0.6% SR.Source: §3.7 · p. 15
  • On RGBT234, ESMTrack ranks first on BC, FM, SV, PO, and DEF under both PR and SR; on LasHeR it leads nine attributes.On LasHeR, GDSTrack leads ESMTrack on heavy occlusion, total occlusion, motion blur, camera motion, and fast-and-long-term attributes.Source: §3.7 · pp. 15–16, Table 2 · p. 16
  • ESMTrack runs at 50.3 fps, 31.7% faster than S2OTFormer and 33.8% faster than GDSTrack.It is trained end-to-end on raw RGB-T data, avoiding the offline label-generation stage of pseudo-label methods.Source: §3.7 · p. 17
  • Visualizations show competing trackers drifting on challenging RGBT234 sequences, while AMD sharpens target responses and removes diffuse background activations across five LasHeR scenarios.The AMD analysis reports increased APCE after suppression in all five scenarios.Source: §3.7 · p. 18

3.8 Ablation Study

Ablations favor asymmetric negative sampling and show gains from modality-aware suppression and complementary losses, while other design choices depend on dataset or loss weighting.

Component and suppression ablations

  • AMD improves performance across benchmarks, and AMD outperforms random suppression; grounding and temporal triplet losses add further gains in several settings.The grounding loss supplies spatial discrimination from annotated initial frames; the temporal loss improves GTOT and RGBT234, with some marginal trade-offs on RGBT210 and LasHeR.Source: §3.8 · pp. 18–21, Table 3 · p. 19, Table 5 · p. 19

Trainable parameter configuration

  • Training normalization layers and the CLS token alongside task-specific modules improves results on RGBT234, RGBT210, and LasHeR, while task-specific-only training scores higher on GTOT.The second configuration is adopted for the final model.Source: Table 4 · p. 19, §3.8 · pp. 19–20

Reliability metric

  • APCE performs best on RGBT234, RGBT210, and LasHeR, while negative entropy is comparable on GTOT; the authors adopt APCE for all experiments.The analysis attributes Max Response’s GTOT degradation to response saturation and negative entropy’s weakness on complex scenes to multi-peaked response maps.Source: Table 6 · p. 20, §3.8 · p. 21

Negative sampling strategies

  • Shifted grounding with Cross Sequence search negatives achieves the best aggregate performance across benchmarks.The grounding and temporal objectives provide complementary spatial precision and cross-instance identity discrimination; homogeneous sampling configurations perform worse.Source: Table 7 · p. 20, §3.8 · p. 23
  • Cross Sequence search negatives consistently outperform Shifted search negatives across grounding strategies and all four benchmarks.With Cross Sequence grounding, GTOT improves from PR 75.2%, SR 62.3% to PR 80.3%, SR 66.2%; RGBT234 improves from PR 63.2%, SR 44.1% to PR 67.5%, SR 47.0%.Source: Table 7 · p. 20, §3.8 · pp. 21–22
  • Cross-sequence grounding performs best on GTOT, while shifted grounding performs best on RGBT234, RGBT210, and LasHeR.The authors associate this pattern with GTOT’s well-separated sequences and the greater diversity and visual similarity of objects in the other benchmarks.Source: Table 7 · p. 20, §3.8 · pp. 22–23

Pseudo-label filtering

  • Backward IoU filtering improves RGBT234 PR and SR by 3.9% and 2.5% over no filtering, and by 1.3% and 1.0% over forward APCE filtering.The comparison supports backward tracking consistency as a more reliable sample-selection criterion than confidence-based heuristics.Source: Table 8 · p. 20, §3.8 · pp. 23–24

Cross-view loss

  • Joint cross-modal and cross-view losses improve performance over the cross-modal loss alone across all four benchmarks.On LasHeR, the combination improves PR, NPR, and SR by 3.8%, 3.4%, and 2.6%, respectively.Source: Table 9 · p. 21, §3.8 · p. 24

3.9 Failure Cases

The reported failure cases expose limits under sustained extreme scale compression and complete occlusion, although tracking recovers after occlusion resolves.

crouch

  • Sustained extreme scale compression can make APCE-based modality estimates unreliable when both RGB and IR responses deteriorate, followed by tracking drift.In crouch, the prediction offset appears at frame #328 and grows into severe drift at frame #331.Source: §3.9 · p. 24

diamond

  • Complete occlusion can prevent re-localization, while the tracker re-acquires the object after the occlusion resolves.In diamond, the tracker fails under deep occlusion at frame #118 and recovers at frame #238.Source: §3.9 · p. 25

3.10 Analysis of Hyperparameters

The hyperparameter ablations favor full weighting for cross-modality and temporal losses, but show dataset-specific and non-monotonic behavior for grounding-loss weight.

Grounding triplet loss weight

  • λ = 0.4 gives the highest GTOT result among settings (PR 81.5%, SR 67.7%) but lowers RGBT234 to PR 65.4%, SR 45.2%; λ = 1.0 gives the best aggregate performance.The authors adopt λ = 1.0; λ = 0.8 substantially lowers results on both GTOT and RGBT234.Source: §3.10 · pp. 25, 27, Table 12 · p. 26

Cross-modality loss weight

  • β = 1.0 achieves the best results on all three reported datasets: GTOT PR 79.3%, SR 65.1%; RGBT234 PR 69.1%, SR 48.5%; LasHeR PR 47.0%, NPR 42.5%, SR 35.1%.Intermediate β values can fall below or match the β = 0 baseline on GTOT; the authors use β = 1.0 in all experiments.Source: §3.10 · p. 25, Table 11 · p. 26

Temporal triplet loss weight

  • γ = 1.0 improves over all baselines and achieves the best results on all datasets, whereas intermediate weights frequently underperform γ = 0 on RGBT234 and LasHeR.The authors attribute this pattern to noise in search-frame anchors derived from predictions and set γ = 1.0.Source: Table 13 · p. 26, §3.10 · p. 27

4 Conclusion

The conclusion presents ESMTrack as an end-to-end self-supervised RGB-T tracker that avoids search-frame boxes and offline pseudo-label generation. It reports benchmark performance and identifies modality-reliability estimation as future work.

Conclusion

  • ESMTrack requires neither bounding-box annotations on search frames nor offline pseudo-label generation.Its grounding triplet uses initial-frame ground truth, while its temporal triplet uses backward-tracking IoU to filter samples.Source: §4 · p. 27
  • The conclusion reports state-of-the-art performance on multiple RGB-T benchmarks and says experiments validate the proposed components.Source: §4 · p. 27
  • More principled modality-reliability estimation, including gradient-based importance scores or causal attribution, is identified as future work.Source: §4 · p. 27

Related papers

Loading 2609.37162v1…