PaperlayerView exact revision on arXiv

Source-linked AI summary

Challenge-Aware RGBT Tracking

Chenglong Li, Lei Liu, Andong Lu, Qing Ji, Jin Tang

arXiv:2007.13143v1cs.CV

TL;DR

RGBT tracking must handle modality-shared and modality-specific challenges while learning target appearance with insufficient training data. CAT uses challenge-aware branches, cross-modality guidance, adaptive aggregation, and staged training; experiments on three benchmark datasets show promising efficiency and effectiveness against state-of-the-art trackers.

  • Problem

    RGBT tracking lacks complete modeling of target appearance changes under diverse shared and modality-specific challenges, especially with insufficient training data.

  • Method

    CAT uses shared branches for modality-shared challenges, independent branches with gated guidance for modality-specific challenges, and adaptive residual aggregation in the backbone.

  • Results

    Extensive experiments on three benchmark datasets show promising performance in both efficiency and effectiveness against state-of-the-art methods.

  • Takeaways & Limitations

    Parallel and hierarchical challenge-aware branches represent challenge-specific target changes with low computational complexity.

Abstract

from arXiv · show

RGB and thermal source data suffer from both shared and specific challenges, and how to explore and exploit them plays a critical role to represent the target appearance in RGBT tracking. In this paper, we propose a novel challenge-aware neural network to handle the modality-shared challenges (e.g., fast motion, scale variation and occlusion) and the modality-specific ones (e.g., illumination variation and thermal crossover) for RGBT tracking. In particular, we design several parameter-shared branches in each layer to model the target appearance under the modality-shared challenges, and several parameterindependent branches under the modality-specific ones. Based on the observation that the modality-specific cues of different modalities usually contains the complementary advantages, we propose a guidance module to transfer discriminative features from one modality to another one, which could enhance the discriminative ability of some weak modality. Moreover, all branches are aggregated together in an adaptive manner and parallel embedded in the backbone network to efficiently form more discriminative target representations. These challenge-aware branches are able to model the target appearance under certain challenges so that the target representations can be learnt by a few parameters even in the situation of insufficient training data. From the experimental results we will show that our method operates at a real-time speed while performing well against the state-of-the-art methods on three benchmark datasets.

1 Introduction

RGBT tracking must represent target appearance despite modality-shared and modality-specific challenges, while existing methods do not fully model these changes. CAT addresses this gap with challenge-aware branches, guidance, adaptive aggregation, and staged training for limited data.

  • RGBT tracking remains difficult because illumination variation, thermal crossover, and occlusion alter target appearance.
  • CAT uses challenge annotations to learn robust target representations under different challenges despite insufficient training data.
  • Modality-shared challenges use convolutional branches across modalities, whereas modality-specific challenges use separate structures for each modality.
  • A guidance module transfers complementary modality-specific information, while gated transforms enhance weak modalities and suppress noisy propagation.
  • Challenge-aware branches are adaptively aggregated and embedded in parallel within the backbone as residual information requiring few parameters.
  • A three-stage training scheme separately trains challenge branches, guidance modules, and adaptive aggregation because challenge annotations are unavailable during testing.

2 Related Work

RGBT tracking methods have progressed from fusion and representation techniques to dominant deep-learning approaches. Related work includes multimodal representations, fusion, hybrid architectures, attribute branches, and residual adapters.

  • Deep-learning RGBT trackers include multimodal representation, multimodal fusion, and hybrid models.
  • Prior work uses two-stream extraction, adaptive fusion, multilayer aggregation, attention, multi-adapter representations, and multilevel fusion strategies.
  • Multi-task learning commonly shares early layers and creates multiple branches for different tasks or attribute representations.
  • Residual adapter modules provide another approach for building networks with a high degree of parameter sharing.

3 Challenge-Aware RGBT Tracker

The proposed tracker is a challenge-aware network built around annotated challenge categories and a lightweight two-stream backbone. Its architecture distinguishes modality-shared from modality-specific challenges and supports online tracking.

  • The architecture uses a challenge-aware neural network followed by a progressive learning algorithm and online tracking procedure.
  • Fig. 2 identifies the guidance module, adaptive aggregation layer, and five challenges: IV, TC, FM, OCC, and SV.
  • The framework models annotated RGBT challenges with multiple branches separated into modality-shared and modality-specific structures.
  • The backbone is a lightweight two-stream CNN that extracts RGB and thermal target features in parallel.

Two-stream CNN backbone

The tracker uses parallel RGB and thermal streams within a lightweight CNN, then applies fully connected layers and losses for foreground–background discrimination. Its challenge taxonomy distinguishes shared from modality-specific factors.

  • Two-stream CNN backbone: Three fully connected layers accommodate appearance changes across instances, videos, and frames.
  • Two-stream CNN backbone: Softmax cross-entropy and instance embedding losses perform binary foreground–background classification.
  • Two-stream CNN backbone: The framework considers IV, FM, SV, OCC, and TC, separating FM, SV, and OCC as modality-shared from IV and TC as modality-specific.

Modality-shared branches

The tracker uses shared branches to model challenge effects across modalities and modality-specific branches to capture each modality’s distinct appearance. A guidance module transfers discriminative features between modalities, while adaptive aggregation combines branch outputs.

  • Unlike FiLM, the guidance module uses feature shift, with gating and point-wise rather than channel-wise linear transformation.
  • Modality-shared branches model target appearance under one challenge across all modalities.
  • Modality-specific branches model target appearance separately for each modality to account for heterogeneity.
  • The guidance module transfers discriminative features from one modality to another when one modality is weaker.For illumination variation, thermal data can guide enhanced RGB target representations.
  • Concatenation followed by a 1×1 convolution adaptively aggregates branch outputs and reduces their dimensionality.

Hierarchical challenge-aware architecture

The architecture is hierarchical because different challenges can be represented most effectively at different convolutional layers. Challenge-aware branches are therefore inserted throughout the backbone.

  • Challenge-aware branches are added to each convolutional layer, forming a hierarchical challenge-aware backbone.Thermal crossover may appear in shallow layers, occlusion in middle layers, and fast motion in deep layers in some scenarios.

3.2 Training Algorithm

Training uses three stages to prevent inappropriate loss propagation, separately train modality-specific guidance, and learn adaptive aggregation without test-time challenge annotations.

  • A three-stage training algorithm addresses shared optimization, guidance-module training, and unavailable test-time challenge annotations.
  • Stage I: Train all challenge-aware branches: Stage I trains all challenge-aware branches using challenge-based data while guidance and aggregation modules are removed.The pretrained VGG-M backbone is fixed during this stage.
  • Stage II: Train all guidance modules: Stage II trains modality-specific guidance modules separately after the challenge branches have been trained.
  • Stage III: Train all adaptive aggregation modules: Stage III learns adaptive aggregation modules and the classifier using challenging and non-challenging frames while fine-tuning the backbone.The previously trained challenge branches and guidance modules are fixed.

3.3 Online Tracking

Online tracking samples candidate regions around the previous result, scores them with the trained network, selects the highest-scoring positive candidate, and refines its bounding box.

  • The initial frame supplies 500 positive and 5000 negative samples for 50-epoch adaptation of the fully connected layers.Positive and negative samples have IoUs greater than 0.7 and less than 0.3, respectively.
  • At each frame, 256 candidate regions are sampled by a Gaussian distribution around the preceding tracking result.
  • The candidate with the highest positive score becomes the tracking result, followed by bounding-box regression refinement.

4 Performance Evaluation

CAT is evaluated on GTOT, RGBT210, and RGBT234 using PR/SR-based comparisons, ablations, and runtime analysis. It delivers competitive or state-of-the-art accuracy while operating at about 20 FPS.

  • Experimental setting: CAT is evaluated on GTOT, RGBT210, and RGBT234 against several state-of-the-art trackers using dataset-specific PR/SR metrics.RGBT234 uses MPR and MSR to compute final PR and SR from the smaller modality-specific scores.
  • GTOT evaluation: 5.0%/4.8% gains in PR/SR over RT-MDNet+RGBT are achieved on GTOT.CAT nearly reaches 100% PR at the 20-pixel threshold and is about 20 times faster than MANet in SR comparison.
  • RGBT210 evaluation: 0.6% higher PR but 2.2% lower SR than mfDiMP is reported on RGBT210.The authors attribute the SR difference partly to mfDiMP’s IoU loss and much larger synthetic training set, compared with CAT’s GTOT training data.
  • RGBT234 evaluation: CAT achieves the best PR and SR scores on RGBT234, exceeding DAFNet, MANet, and RT-MDNet+RGBT by reported PR/SR gains.The gains are 0.8%/1.7% over DAFNet, 2.7%/2.2% over MANet, and 7.0%/7.8% over RT-MDNet+RGBT.
  • Ablation study: Guidance-module variants and comparisons with FiLM support the effectiveness of the proposed guidance module.The guidance module achieves superior performance over FiLM on GTOT and RGBT234.
  • Runtime analysis: 20 FPS is achieved by CAT, compared with nearly 30 FPS for RT-MDNet+RGBT, while CAT reports better tracking performance across the three benchmarks.The tracker is implemented in PyTorch 0.4.0 and Python 2.7 on an Intel Xeon E5-2620 v4 CPU with an RTX 2080Ti GPU.

5 Conclusion

The paper concludes that parallel and hierarchical challenge-aware branches learn target-appearance representations under different challenges with low computational complexity, including when training data are insufficient.

  • Conclusion: Parallel and hierarchical challenge-aware branches model target-appearance changes under specific challenges while maintaining low computational complexity.Experiments on three benchmark datasets support the method’s reported effectiveness and efficiency against state-of-the-art trackers.
Loading 2007.13143v1…