PaperlayerView exact revision on arXiv

Source-linked AI summary

Duality-Gated Mutual Condition Network for RGBT Tracking

Andong Lu, Cun Qian, Chenglong Li, Jin Tang, Liang Wang

arXiv:2011.07188v3cs.CV

TL;DR

RGBT trackers often fail to fully exploit low-quality modalities because they mix useful discriminative features with noise. DMCNet uses duality-gated mutual conditioning and conditional optical-flow-based resampling, and experiments across four benchmarks show outstanding performance against state-of-the-art methods.

  • Problem

    Existing RGBT tracking methods do not fully exploit low-quality modalities, which contain both noisy information and discriminative features useful for target localization.

  • Method

    DMCNet mutually uses one modality's discriminative information to guide target-appearance learning in the other, gates conditions to reduce noise, and resamples candidates during detected sudden camera motion.

  • Results

    83.9%/59.3% in PR/SR on RGBT234, with gains of 1.6%/1.8% over CMPP, 3.5%/3.2% over CAT, and 4.9%/2.0% over JMMAC.

  • Takeaways & Limitations

    The method fully uses discriminative information from RGB and thermal modalities while suppressing noise and achieves outstanding performance across four RGBT benchmarks.

Abstract

from arXiv · show

Low-quality modalities contain not only a lot of noisy information but also some discriminative features in RGBT tracking. However, the potentials of low-quality modalities are not well explored in existing RGBT tracking algorithms. In this work, we propose a novel duality-gated mutual condition network to fully exploit the discriminative information of all modalities while suppressing the effects of data noise. In specific, we design a mutual condition module, which takes the discriminative information of a modality as the condition to guide feature learning of target appearance in another modality. Such module can effectively enhance target representations of all modalities even in the presence of low-quality modalities. To improve the quality of conditions and further reduce data noise, we propose a duality-gated mechanism and integrate it into the mutual condition module. To deal with the tracking failure caused by sudden camera motion, which often occurs in RGBT tracking, we design a resampling strategy based on optical flow algorithms. It does not increase much computational cost since we perform optical flow calculation only when the model prediction is unreliable and then execute resampling when the sudden camera motion is detected. Extensive experiments on four RGBT tracking benchmark datasets show that our method performs favorably against the state-of-the-art tracking algorithms

I. INTRODUCTION

The paper targets noisy but discriminative low-quality modalities and introduces DMCNet to exploit their useful information while suppressing noise. It also adds conditional resampling for sudden camera motion and reports strong performance across four RGBT benchmarks.

  • Motivation: Low-quality modalities contain both substantial noise and discriminative features, but existing trackers do not exploit this potential well.Noise can degrade tracking accuracy and robustness, while useful features can support target localization.
  • Method: DMCNet uses a mutual condition module to let each modality guide target-appearance learning in the other modality.The approach uses discriminative information from one modality as a condition for feature learning in the other.
  • Method: A duality-gated mechanism improves condition quality and suppresses noise in both conditions and condition-guided features.The paper reports reduced noise and stronger target-feature discriminability, especially in low-quality modalities.
  • Resampling: DMCNet resamples candidate target regions along estimated camera motion when sudden motion causes unreliable tracking predictions.Optical flow is computed only on local regions when failure from sudden camera motion is detected, limiting computational overhead.
  • Evaluation: Experiments on GTOT, RGBT210, VOT2019-RGBT, and RGBT234 show outstanding performance against state-of-the-art trackers.The evaluation covers four RGBT tracking benchmark datasets.

II. RELATED WORK

Prior RGBT trackers emphasize adaptive modality fusion, mutual learning, or cross-modal guidance, while DMCNet combines a backbone, duality-gated mutual conditioning, and resampling. Its inputs are aligned RGB-thermal candidate patches processed through a convolutional backbone and classification head.

  • RGBT Tracking Methods: Recent RGBT methods commonly learn modality weights to adaptively fuse RGB and thermal features.Examples include modality-aware attention and quality-aware feature aggregation approaches.
  • Mutual Learning: Mutual-learning approaches train multiple branches collaboratively, often using mimicry losses based on Kullback-Leibler divergence.The surveyed methods use collaborative supervision to improve branch learning.
  • DMCNet Architecture: DMCNet consists of a backbone network, a duality-gated mutual condition module, and a resampling module.The architecture is presented as the overall framework for the tracking method.
  • Backbone Architecture: The backbone receives aligned RGB-thermal candidate patches resized to 107×107 and computes prediction scores for candidate anchor boxes.Candidate patches are cropped from aligned image pairs before network inference.
  • Backbone Architecture: The backbone uses the first three convolutional layers of VGG-M followed by a binary classification head with fully connected layers.The fully connected layers have output dimensions 512, 512, and 2.

B. Duality-Gated Mutual Condition Module

The module uses bidirectional conditional feature learning so each modality guides the other, while duality-gated transformations suppress noisy information in conditions and propagated features.

  • Bidirectional Conditional Learning: The mutual feature modulation performs RGB-to-T and T-to-RGB transformations simultaneously in a bidirectional manner.The symmetric T-to-RGB structure is intended to mine discriminative information from both low- and high-quality modalities.
  • RGB-to-T Feature Modulation: RGB-to-T modulation uses RGB discriminative features to guide thermal target-feature learning, while addressing limited condition diversity and possible RGB noise.Multi-scale scaling conditions are generated from RGB features, and fused RGB–thermal information forms shifting conditions.
  • Duality-Gated Mechanism: Gates filter noisy scaling conditions and fused features before information propagates across modalities.G1 and G2 mitigate noise in multi-scale scaling conditions and fused features, while corresponding gates are used symmetrically in the two directions.
  • Duality-Gated Mechanism: In example frames, gating suppresses glare-related RGB responses while enhancing useful thermal target information during cross-modal propagation.The reported examples attribute single-modality noise suppression and discriminative-feature enhancement to G1 and G2.

C. Re-Sampling Module

The resampling module addresses abrupt camera motion by using optical-flow-guided candidate regions when the model prediction is unreliable, improving target coverage without relying only on Gaussian sampling.

  • Motivation: Abrupt camera motion can move targets outside Gaussian-sampled search windows, causing tracking failure.Expanding the search region or performing global search introduces more background and increases computational cost or model-drift risk.
  • Optical-Flow-Guided Resampling: The method starts fast optical-flow estimation when the model state is unreliable and uses camera-motion estimation to guide resampling.The resampling scheme is designed to cover target regions missed by the original sampling process.
  • Optical-Flow-Guided Resampling: The scheme empirically samples 16 candidate regions opposite the camera-motion direction, using quarter-target-width or quarter-target-height steps.Candidate scores are combined with Gaussian-sampling results to obtain the final prediction.

D. Network Training

Training initializes the backbone from a pretrained MDNet model and optimizes the whole RGBT network with SGD and softmax cross-entropy over 200 epochs.

  • Initialization: The backbone is MDNet, pretrained through multi-domain learning on ImageNet, while modality-specific convolutional subnetworks are randomly initialized.The modality-specific subnetworks run in parallel with the backbone.
  • Optimization: The whole network is trained on an RGBT dataset for 200 epochs using SGD and softmax cross-entropy loss.Different subnetworks use different learning rates, with a smaller rate assigned to the pretrained backbone.

IV. ONLINE RGBT TRACKING

The online tracker generates candidate regions, scores them, and averages the top five candidates. When unreliable predictions indicate abrupt camera motion, it resamples candidates and combines the resampled and Gaussian-sampling results.

  • Online tracking: The tracker samples candidate regions around the previous target state and scores them with the trained network.Gaussian sampling generates candidates from the previous state, while the network computes positive and negative scores for each candidate.
  • Online tracking: The current target state is computed as the mean of the five candidates with the highest scores.The top-five candidate set is averaged to produce the tracking result for the current frame.
  • Camera-motion detection: If the mean of the top five scores is below zero, the tracker estimates camera motion using the average moving vector [dx, dy].Abrupt motion is detected when the vector amplitude exceeds threshold u.
  • Camera-motion compensation: When abrupt camera motion is detected, the tracker performs resampling and computes scores for the new candidate set.The resampled candidates are evaluated separately to compensate for target displacement caused by camera motion.
  • Camera-motion compensation: The final prediction uses the higher-scoring candidate from Gaussian sampling and resampling, followed by short-term and long-term model updates.Algorithm 1 combines both candidate sources before updating the tracking model.

V. PERFORMANCE EVALUATION

DMCNet is evaluated against existing RGBT and RGB trackers on four RGBT tracking benchmarks using a PyTorch-based experimental setup.

  • Evaluation scope: The evaluation compares DMCNet with existing RGBT and RGB trackers on GTOT, RGBT210, RGBT234, and VOT-RGBT2019.Experiments use a PyTorch 1.0+ environment with eight NVIDIA GeForce GTX 2080Ti GPUs.

A. Evaluation Setting

The evaluation uses four challenging aligned RGB–thermal datasets and precision and success rates to measure tracking performance.

  • Datasets: GTOT, RGBT210, RGBT234, and VOT-RGBT2019 provide the benchmark datasets for evaluation.The datasets differ in numbers of video pairs, frames, and tracking attributes.
  • Metrics: Precision rate measures the fraction of frames whose predicted position lies within a predefined pixel threshold of the ground truth.The representative PR thresholds are 5 pixels for GTOT and 20 pixels for RGBT234.
  • Metrics: Success rate is used alongside precision rate to evaluate RGBT tracking performance.Both metrics are reported for comparisons across the RGBT tracking datasets.

B. Evaluation on GTOT Dataset

On GTOT, DMCNet achieves performance comparable to state-of-the-art trackers while providing better tracking efficiency; the broader evaluation also reports clear gains on RGBT234.

  • GTOT comparison: DMCNet is compared with 11 state-of-the-art trackers on GTOT using precision and success evaluation curves.The comparison includes RGBT trackers and an MDNet+RGBT extension.
  • GTOT comparison: DMCNet achieves comparable performance with state-of-the-art methods on GTOT.The paper specifically notes comparable success-rate performance and better tracking efficiency.
  • Evaluation curves: The reported evaluation curves present representative precision-rate and success-rate scores in their legends.Figure 6 covers GTOT, RGBT210, and RGBT234.
  • Cross-dataset comparison: On RGBT234, DMCNet clearly outperforms CMPP.The authors present this comparison as evidence of effectiveness and efficiency.

C. Evaluation on RGBT210 Dataset

DMCNet is evaluated against state-of-the-art RGBT trackers on RGBT210 and RGBT234, including overall and challenge-based comparisons. The method reports strong quantitative and qualitative performance across adverse tracking conditions.

  • Overall comparison: DMCNet uses only GTOT’s 50 training sequences, whereas mfDiMP uses a generated RGBT dataset containing 9,335 sequences.The comparison highlights the different training-data scales.
  • Overall comparison: On RGBT234, DMCNet achieves 83.9% PR and 59.3% SR, outperforming CMPP, CAT, and JMMAC by the reported PR/SR margins.The gains are 1.6%/1.8% over CMPP, 3.5%/3.2% over CAT, and 4.9%/2.0% over JMMAC.
  • Qualitative comparison: DMCNet more robustly localizes targets than competing trackers under camera motion, low resolution, low illumination, clutter, and heavy occlusion.The qualitative examples cover four sequences and compare RGB and thermal frames with colored tracking rectangles.
  • Challenge-based performance: DMCNet performs best under most listed challenge conditions, including occlusion, low illumination, thermal crossover, fast motion, camera motion, and background clutter.The challenge-based evaluation compares multiple RGBT trackers across attribute-specific subsets.

E. Evaluation on VOT-RGBT2019 Dataset

The paper analyzes DMCNet through VOT-RGBT2019 evaluation, low-quality-modality tests, visual comparisons, parameter analysis, and ablations. These experiments examine robustness, noise suppression, input complementarity, and component contributions.

  • VOT-RGBT2019 evaluation: DMCNet has comparable VOT-RGBT2019 performance to mfDiMP and outperforms MANet and MaCNet under the VOT protocol.The evaluation uses EAO, R, and A metrics and compares seven RGBT tracking algorithms.
  • VOT-RGBT2019 evaluation: DMCNet is lower than mfDiMP in the R metric but exceeds FSRPN by 2.43% in EAO.The paper attributes the mfDiMP difference partly to training-data scale and its IoU network.
  • Low-quality modalities: Removing low-quality data decreases performance on both GTOT-V1 and RGBT234-V1, indicating that DMCNet leverages effective features from those modalities.The paper reports that directly discarding low-quality data causes a significant performance drop.
  • Noise suppression: With DMC suppresses more noise than without DMC in low-quality modalities, including cases where one modality is low-quality and the other provides effective features.The visualization compares input rows and modality-specific feature maps with and without DMC.
  • Complementary modalities: DMCNet outperforms RGB-only, thermal-only, and direct average-fusion variants on the evaluated RGBT234 and GTOT datasets.The results support adaptive fusion rather than directly averaging the two modalities.
  • Ablation study: The ablation results support the effectiveness of the proposed components, while the resampling strategy has a smaller time-cost impact than the DMC module.The DMC module is reported to affect speed most, and the resampling module has little impact on time cost.

VI. CONCLUSION

The paper concludes that DMCNet uses discriminative information from all RGB and thermal modalities, especially low-quality ones, through mutual conditioning and duality gating. Experiments on four benchmarks report outstanding performance against state-of-the-art methods.

  • Conclusion: DMCNet transforms effective RGB and thermal information into mutual conditions that enhance the discriminative ability of both modalities.A duality-gated mechanism improves the quality of the generated conditions.
  • Conclusion: Experiments on four RGBT benchmark datasets report outstanding performance compared with state-of-the-art methods.The conclusion identifies the method’s benchmark-level outcome without specifying a single metric.
  • Conclusion: Future work will explore external knowledge to strengthen duality-gated mutual conditions for more robust RGBT tracking.This is stated as the paper’s proposed future direction.
Loading 2011.07188v3…