PaperlayerView exact revision on arXiv

Source-linked AI summary

RGBT Tracking via Multi-Adapter Network with Hierarchical Divergence Loss

Andong Lu, Chenglong Li, Yuqing Yan, Jin Tang, Bin Luo

arXiv:2011.07189v3cs.CV

TL;DR

RGBT tracking lacks fully exploited representations of shared and modality-specific information despite complementary RGB and thermal cues. MANet++ jointly learns these cues and target-specific dynamics using three adapters and hierarchical divergence loss, with experiments demonstrating effectiveness and efficiency on benchmark datasets.

  • Problem

    RGBT tracking underutilizes complementary RGB and thermal information, while existing methods incompletely exploit modality-shared and modality-specific cues.

  • Method

    MANet++ uses generality, modality, and instance adapters with hierarchical divergence loss to jointly learn shared, modality-specific, and instance-aware representations.

  • Results

    Extensive experiments on benchmark datasets demonstrate the effectiveness and efficiency of the proposed tracking method.

  • Takeaways & Limitations

    The proposed architecture provides a shared-parameter design for learning complementary RGBT representations and quality-aware modality fusion.

Abstract

from arXiv · show

RGBT tracking has attracted increasing attention since RGB and thermal infrared data have strong complementary advantages, which could make trackers all-day and all-weather work. However, how to effectively represent RGBT data for visual tracking remains unstudied well. Existing works usually focus on extracting modality-shared or modality-specific information, but the potentials of these two cues are not well explored and exploited in RGBT tracking. In this paper, we propose a novel multi-adapter network to jointly perform modality-shared, modality-specific and instance-aware target representation learning for RGBT tracking. To this end, we design three kinds of adapters within an end-to-end deep learning framework. In specific, we use the modified VGG-M as the generality adapter to extract the modality-shared target representations.To extract the modality-specific features while reducing the computational complexity, we design a modality adapter, which adds a small block to the generality adapter in each layer and each modality in a parallel manner. Such a design could learn multilevel modality-specific representations with a modest number of parameters as the vast majority of parameters are shared with the generality adapter. We also design instance adapter to capture the appearance properties and temporal variations of a certain target. Moreover, to enhance the shared and specific features, we employ the loss of multiple kernel maximum mean discrepancy to measure the distribution divergence of different modal features and integrate it into each layer for more robust representation learning. Extensive experiments on two RGBT tracking benchmark datasets demonstrate the outstanding performance of the proposed tracker against the state-of-the-art methods.

I. INTRODUCTION

MANet++ addresses underused RGBT information by jointly learning modality-shared, modality-specific, and instance-aware representations with a multi-adapter framework and hierarchical divergence loss.

  • RGBT tracking remains limited by underuse of complementary visible and thermal information.
  • MANet++ jointly learns modality-shared, modality-specific, and instance-aware representations in an end-to-end framework.
  • The generality adapter extracts shared representations, while parallel modality adapters learn multilevel modality-specific features with fewer parameters.The modality adapter shares most parameters with the generality adapter and uses small added blocks.
  • The instance adapter captures target appearance changes and temporal variations through quality-aware fusion of RGB and thermal features.
  • Hierarchical divergence loss jointly enhances shared and modality-specific features by reducing divergence in the generality adapter and increasing it in the modality adapter.The loss is embedded at multiple layers and uses multiple kernel maximum mean discrepancy for robustness to outliers.
  • Experiments on three RGBT tracking benchmarks suggest excellent performance against state-of-the-art methods.Compared with MANet, MANet++ adds hierarchical divergence loss and dynamic fusion while avoiding the earlier two-stage learning risk of over-fitting.

II. RELATED WORK

Prior RGBT trackers use sparse, graph-based, or deep-learning approaches, with deep models commonly emphasizing either modality-specific or shared representations.

  • Early RGBT tracking methods use pixel-level fusion, sparse representation, manifold ranking, or dynamic graphs.
  • Two-stream deep networks extract modality-specific features and combine them for tracking.
  • Single-network approaches extract shared RGB and thermal features or recursively fuse multilevel multimodal features.
  • Existing deep methods may overlook cross-modal collaboration, shared information, or modality-specific information, leaving representation learning incomplete.

III. MULTI-ADAPTER CONVOLUTIONAL NETWORK

MANet++ combines generality, modality, and instance adapters in a shared architecture that separates common and modality-specific cues before instance-aware fusion.

  • Network Architecture: MANet++ processes RGB and thermal images through generality and modality adapters, combines their features, and extracts candidate features with RoIAlign.
  • Generality Adapter: The generality adapter uses the first three VGG-M layers to extract shared object representations across modalities.It uses dilated convolution in the third layer and adaptive RoIAlign for spatial alignment.
  • Modality Adapter: The modality adapter adds small parallel convolutional blocks to generality-adapter layers to encode modality-specific information efficiently.Its parameter count is 20% of the generality adapter, with independent-component and max-pooling layers added afterward.
  • Modality Adapter: The parallel design decomposes each modality’s parameters into shared generality and modality-specific components.
  • Instance Adapter: The instance adapter uses modality-specific fully connected layers, modality weights, feature concatenation, and fusion for target classification and regression.During online tracking, its instance layer is rebuilt for each sequence and updated to adapt to target changes.

B. Loss Function

The network combines hierarchical divergence, binary classification, and instance embedding losses to learn shared, modality-specific, and target-discriminative representations. MK-MMD aligns shared-feature distributions while separating modality-specific distributions across layers.

  • Loss components: The network uses hierarchical divergence, binary classification, and instance embedding losses.The hierarchical divergence loss is based on MK-MMD; binary classification separates target from background, while instance embedding separates targets across domains.
  • Hierarchical divergence loss: MK-MMD measures distribution differences between RGB and thermal features while tolerating outliers.The method treats modalities as samples from different distributions because Euclidean distance is sensitive to outliers.
  • Hierarchical divergence loss: The hierarchical loss minimizes divergence between modality-shared features and maximizes divergence between modality-specific features at each layer.Layer-wise estimates are computed for generality-adapter and modality-adapter outputs.
  • Binary classification loss: Binary cross-entropy drives target-background discrimination using scores from individual modalities and their fusion.The network records separate binary classification scores for RGB, thermal, and fused outputs.
  • Instance embedding loss: Instance embedding separates targets from different videos, enhancing current-domain target scores and suppressing scores in other domains.The loss operates only on positive samples and can improve discrimination between similar targets during testing.

C. One-stage Learning Algorithm

Training uses shared RGB-Thermal ground truths in a one-stage end-to-end offline procedure, while online adaptation uses sampled positives and negatives with complementary update schedules. The hierarchical divergence term uses a fixed multi-kernel Gaussian configuration.

  • Offline training: The whole network is trained end-to-end in one offline stage using shared ground truths and stochastic gradient descent.The datasets provide high-aligned common ground truths for both modalities.
  • Hierarchical divergence configuration: The hierarchical divergence loss uses 11 Gaussian kernels with bandwidths σu = 2^(u−6) for u = 1,...,11.The kernels are linearly combined, with weights set as in the referenced MK-MMD formulation.
  • Online training: Online training collects 500 positive and 5000 negative samples to adapt a new instance branch for each test sequence.The instance branch is fine-tuned from first-frame information and used to maintain robustness and adaptability.
  • Online training: Online updates combine short-term updates triggered by low target scores with long-term updates performed every 10 frames.The score threshold for short-term updates is set to zero.

IV. ONLINE TRACKING

During tracking, the generality and modality adapters remain fixed while an instance adapter scores Gaussian-sampled candidate regions and fuses modality-aware features. The highest-scoring candidate becomes the tracking result, with regression and update rules applied conditionally.

  • Online tracking pipeline: The tracker fixes the generality and modality adapters and replaces the instance adapter’s final fully connected layer for each RGBT sequence.Initial training uses the first RGB-Thermal frame pair and its ground-truth bounding box.
  • Candidate scoring: The tracker samples 256 candidate regions around the previous result and scores them through the instance adapter.Candidate regions are generated by Gaussian sampling centered on the previous tracking result.
  • Candidate scoring: Modality weights are computed from single-modality scores before re-encoding and concatenating modality features for fused classification.The fused representation passes through FCfusion and FCinstance to obtain each candidate’s classification score.
  • Result selection: The candidate with the highest score is selected as the tracking result.The selection is performed after obtaining the fused classification scores.
  • Update rules: Bounding-box regression adjusts position and scale when the positive score exceeds 0.5, while short-term updates begin when it falls below 0.Long-term updates occur at 10-frame intervals.

V. PERFORMANCE EVALUATION

The evaluation compares MANet++ with popular RGB and RGBT tracking algorithms and tests the effectiveness of its major components.

  • Comparative evaluation: MANet++ is evaluated against current RGB and RGBT tracking algorithms.The comparison is conducted in the performance evaluation experiments.
  • Component analysis: Ablation experiments verify the effectiveness of the proposed algorithm’s major components.The evaluation includes component-level verification in addition to comparisons with existing trackers.

A. Evaluation Data and Metrics

The evaluation uses three RGBT benchmarks with dataset-specific challenge settings and tracking metrics. GTOT and RGBT234 use PR/SR measures, while VOT-RGBT2019 follows the VOT protocol with EAO, robustness, and accuracy.

  • Datasets: The paper evaluates MANet++ on GTOT, RGBT234, and VOT-RGBT2019.GTOT contains 50 aligned RGB-thermal sequences, RGBT234 contains 234 sequences and about 200,000 frames, and VOT-RGBT2019 contains 60 sequences and 20,083 frames.
  • Datasets: GTOT contains 50 spatially and temporally aligned RGB-thermal sequences with bounding-box annotations and seven challenge attributes.The sequences span different scenes and conditions.
  • Metrics: GTOT and RGBT234 use precision rate and success rate, with representative PR thresholds of 5 and 20 pixels, respectively.SR is the area under the overlap-success curve.
  • Datasets: RGBT234 contains 234 aligned sequences, about 200,000 frames, and labeled challenges including motion blur, camera movement, illumination, deformation, and occlusion.Its longest sequence contains about 4,000 frames, and each modality has ground truths.
  • Metrics: VOT-RGBT2019 uses the VOT protocol and reports expected average overlap, robustness, and accuracy.Lost targets are re-initialized with the corresponding ground truth.

B. Evaluation on GTOT Dataset

On GTOT, MANet++ is compared with RGB and RGBT trackers using PR and SR. It improves PR over several baselines, remains close to MANet in SR, and runs substantially faster than MANet.

  • RGB tracker comparison: MANet++ outperforms 11 RGB trackers on GTOT, including ECO, DAT, RT-MDNet, C-COT, ACT, SiamDW, SRDCF, BACF, ACFN, DSST, and MDNet.The comparison uses PR and SR results shown in Fig. 3.
  • RGB tracker comparison: MANet++ improves PR/SR over MDNet by 8.9%/9.0%, over DSST by 11.6%/15.7%, and over DAT by 13.0%/10.5%.These are the reported GTOT differences for the RGB-tracker comparison.
  • RGBT tracker comparison: MANet++ exceeds MANet, FANet, and DAPNet by 0.7%, 1.0%, and 1.9% in PR, respectively.The paper also compares against extended RGB methods and other RGBT trackers.
  • RGBT tracker comparison: MANet++ has slightly lower SR than MANet and FANet, but its PR is 0.901 versus MANet’s 0.894.The reported SR values for MANet++ and MANet are 0.723 and 0.724, respectively.
  • Efficiency: MANet++ runs eight times faster than MANet on GTOT.The authors attribute the reported effectiveness to the added hierarchical divergence loss and RoIAlign layer.

C. Evaluation on RGBT234 Dataset

On RGBT234, MANet++ is evaluated against RGB and RGBT trackers across overall, challenge-based, and visual comparisons. It achieves the highest reported overall PR and SR and performs strongly across difficult conditions.

  • Overall performance: MANet++ outperforms all compared RGB and RGBT trackers on RGBT234 in all reported metrics.The comparison includes 11 RGB trackers and 10 state-of-the-art RGBT trackers.
  • Overall performance: MANet++ achieves 80.0% PR and 55.4% SR on RGBT234.It gains 6.9% PR over DAT, 4.0% SR over C-COT, and 2.3%/1.5% PR/SR over MANet.
  • Challenge-based performance: MANet++ performs best in most labeled challenges, including partial and heavy occlusion, low illumination, low resolution, thermal crossover, deformation, fast motion, scale variation, camera movement, and background clutter.The RGBT234 challenge set contains 12 labeled attributes.
  • Challenge-based performance: MANet++ improves PR by 6% over second place in the low-resolution challenge and by about 3% in partial occlusion, low illumination, thermal crossover, deformation, camera movement, and background clutter.Compared with MANet, it shows notable improvements in partial occlusion, low illumination, and thermal crossover.
  • Visual comparison: Visual comparisons show MANet++ handling occlusion, thermal crossover, high illumination, background clutter, and modality-specific illumination noise across four sequences.In the elecbike10 sequence, it suppresses RGB illumination noise when illumination returns to normal.

D. Evaluation on VOT2019-RGBT Dataset

On VOT-RGBT2019, MANet++ is compared with several state-of-the-art trackers and shows strong performance, while the RoIAlign variant performs comparably to mfDiMP and surpasses other listed methods.

  • Overall comparison: MANet++-RoIAlign has comparable performance to mfDiMP and outperforms MANet, FANet, and MaCNet on VOT-RGBT2019.The comparison attributes this result to the added HD loss and IC layer.
  • Overall comparison: MANet++ significantly outperforms RT-MDNet+RGBT and FANet on VOT-RGBT2019.However, its performance is inferior to MANet in the reported comparison.
  • Cross-dataset results: MANet++ improves performance by 2.3%/1.5% in PR/SR on RGBT234.The result is reported as a final comparison with MANet’s performance.
  • Ablation study: Ablation results attribute performance gains to the IC layer, hierarchical divergence loss, and quality-aware modality fusion.The study reports that HD loss helps adapters learn modality-shared and modality-specific features, while fusion improves tracking performance.

F. Impact of Fusion Position

The experiments examine fusion placement and efficiency-related components of MANet++. Fusion at the first fully connected layer gives the best performance, while RoIAlign substantially improves candidate processing speed.

  • Fusion position: MANet++ achieves the best performance among the tested fusion positions on RGBT234.MANet++early fuses at the last convolution layer, whereas MANet++late fuses at the second fully connected layer.
  • Efficiency analysis: MANet++ runs about 1.34 times faster than FANet and has comparable or higher performance on VOT-RGBT2019.The implementation comparison also reports faster tracking than MaCNet.
  • Efficiency analysis: RoIAlign extracts candidate features directly from input-image feature maps, greatly improving tracking speed.Without RoIAlign, all 256 candidates pass through the network separately, which is time consuming.
  • Architecture: The proposed network combines generality, modality, and instance adapters, sharing most parameters between the first two for efficiency.Its instance adapter uses quality-aware fusion of modality features.
Loading 2011.07189v3…