PaperlayerView exact revision on arXiv

Source-linked AI summary

Jointly Modeling Motion and Appearance Cues for Robust RGB-T Tracking

Pengyu Zhang, Jie Zhao, Dong Wang, Huchuan Lu, Xiaoyun Yang

arXiv:2007.02041v1cs.CV

TL;DR

RGB-T tracking must combine complementary modalities while remaining reliable when appearance cues fail under occlusion or camera motion. JMMAC jointly models appearance and motion using weighted late fusion, target and camera motion modules, and tracker switching; experiments on three datasets report significantly better performance than competing algorithms.

  • Problem

    RGB-T tracking needs effective fusion of RGB and thermal information and motion cues because appearance can become unreliable under occlusion and camera motion.

  • Method

    JMMAC uses MFNet for global and local weighted late fusion of RGB and thermal responses, and models target and camera motion with dynamic tracker switching.

  • Results

    Experiments on GTOT, RGBT234, and VOT19-RGBT report significantly better performance than competing algorithms.

  • Takeaways & Limitations

    Combining appearance and motion cues provides a robust RGB-T tracking framework across three recent tracking benchmarks.

Abstract

from arXiv · show

In this study, we propose a novel RGB-T tracking framework by jointly modeling both appearance and motion cues. First, to obtain a robust appearance model, we develop a novel late fusion method to infer the fusion weight maps of both RGB and thermal (T) modalities. The fusion weights are determined by using offline-trained global and local multimodal fusion networks, and then adopted to linearly combine the response maps of RGB and T modalities. Second, when the appearance cue is unreliable, we comprehensively take motion cues, i.e., target and camera motions, into account to make the tracker robust. We further propose a tracker switcher to switch the appearance and motion trackers flexibly. Numerous results on three recent RGB-T tracking datasets show that the proposed tracker performs significantly better than other state-of-the-art algorithms.

I. INTRODUCTION

RGB-T tracking combines complementary visible and thermal information, but appearance cues can fail under occlusion, camera motion, and other complex variations. JMMAC therefore jointly models appearance and motion cues for robust tracking.

  • Motivation: RGB and thermal modalities provide complementary strengths for illumination robustness, foreground-background separation, and thermal-crossover handling.Thermal information is less sensitive to illumination, while visible information is more discriminative under normal conditions and during thermal crossover.
  • Motivation: Full occlusion makes appearance information meaningless, while camera motion, blur, and target displacement can make appearance models ineffective.These conditions motivate target prediction and camera-motion compensation.
  • Proposed framework: JMMAC jointly models appearance and motion information through a framework designed for robust RGB-T tracking.The paper presents this joint modeling objective as its central contribution.
  • Proposed framework: MFNet performs late fusion by learning global and local weights for combining RGB and thermal responses into a robust response map.The method uses global and local multimodal fusion networks to infer fusion weights.
  • Proposed framework: The motion component exploits camera and target motion and dynamically switches between appearance and motion cues.This supplements appearance tracking when appearance information is unreliable.
  • Results: Experiments on three recent RGB-T tracking benchmarks report significantly better performance than competing algorithms.The cited contribution states the comparison across three benchmarks without specifying metric values.

B. Multimodal Fusion

The framework uses late fusion to combine RGB and thermal response maps, with MFNet learning modality-aware and distractor-suppressing weight maps. This appearance model is integrated into a broader tracking pipeline that also mines motion cues.

  • Multimodal Fusion: The JMMAC pipeline applies camera-motion compensation, aggregates modality responses with MFNet, and uses motion prediction for target tracking.The framework contains multimodal fusion and motion-mining components.
  • Multimodal Fusion: Late fusion combines response maps from separate RGB and thermal trackers through a weighted linear combination.The base tracker creates separate ECO trackers and corresponding response maps for the two modalities.
  • Multimodal Fusion: MFNet learns a pixel-wise fusion weight map WF bounded between zero and one, and the fused response peak determines the target location.The network is trained offline and applied directly during tracking without online fine-tuning.
  • Multimodal Fusion: Global MFNet estimates whole-context modality importance, while local MFNet suppresses distractors within individual modalities.The local network acts as an attention mechanism and produces a finer weight map.
  • Multimodal Fusion: The fused response is trained by minimizing squared Euclidean distance to a desired response whose peak denotes the target location.The desired response Y is defined as a two-dimensional Gaussian map.

B. Motion Modeling for Robust RGB-T Tracking

JMMAC explicitly models target and camera motion because appearance information becomes unreliable under complex variations. Its motion modules support target prediction, camera compensation, and switching between appearance and motion trackers.

  • B. Motion Modeling for Robust RGB-T Tracking: Appearance cues become unreliable under extreme illumination, low resolution, camera motion, and occlusion, motivating explicit motion modeling.The paper divides motion information into target motion and camera motion.
  • B. Motion Modeling for Robust RGB-T Tracking: TMP switches between appearance and motion trackers by combining MAX-PSR with template matching.The figure caption identifies this combination as the basis for the switching scheme.
  • B. Motion Modeling for Robust RGB-T Tracking: The framework handles target motion with target motion prediction and camera motion with camera motion estimation.These are the two dedicated motion modules in JMMAC.

1) Target Motion Prediction:

The target motion prediction component supplements unreliable appearance tracking with motion-based prediction and a switcher that selects the more reliable cue. It evaluates appearance and motion results using response-map reliability and template similarity.

  • Target Motion Prediction:: The motion tracker predicts target motion when appearance information is unreliable, particularly under low resolution and occlusion.
  • Target Motion Prediction:: MAX-PSR combines the response-map peak-to-sidelobe ratio with its maximum value to estimate appearance-tracker reliability.
  • Target Motion Prediction:: The switcher compares appearance and motion trackers using response-map reliability and template similarity to select the more suitable cue.
  • Target Motion Prediction:: Template matching uses the initial target template and tracker-provided target regions to calculate separate similarity scores for appearance and motion results.
  • Target Motion Prediction:: DDIS matching is used because its feature-match diversity measure is described as robust to deformation, background clutter, and occlusion.

2) Camera Motion Estimation:

Camera motion is estimated in the 2D image plane from thermal reference and search images, then used to compensate camera movement and stabilize the search region.

  • Camera Motion Estimation:: The method models camera motion in the 2D image plane, assuming target movement and target-depth variation are small.
  • Camera Motion Estimation:: SIFT extracts key points, while M-estimator sample consensus matches them and excludes outliers before obtaining the transformation matrix.
  • Camera Motion Estimation:: The transformation function maps reference-image key-point coordinates to corresponding search-image coordinates using parameter matrix O.
  • Camera Motion Estimation:: The resulting transformation matrix compensates camera motion, and the CME scheme facilitates obtaining a stable search region in an example with large camera motion.

1 Calculate the tracking reliability q via equation (4)

The tracking reliability calculation supports switching between appearance and motion trackers while coordinating filter and motion-tracker updates.

  • 1 Calculate the tracking reliability q via equation (4): The switcher obtains appearance and motion similarities before applying threshold conditions to choose a tracker.
  • 1 Calculate the tracking reliability q via equation (4): Appearance tracking is selected when q and sA satisfy the specified high- or low-reliability conditions, including a sufficient advantage over sM.
  • 1 Calculate the tracking reliability q via equation (4): Motion tracking is selected when the switcher’s appearance-based conditions are not met.
  • 1 Calculate the tracking reliability q via equation (4): When the motion tracker is used, filter updating is skipped to avoid degradation from corrupted samples during unreliable appearance tracking.
  • 1 Calculate the tracking reliability q via equation (4): MFNet training uses 200 × 200 RGB and thermal patches, learning rates of 1e−5 and 1e−7, and batch size 8.

IV. EXPERIMENTS

The experiments evaluate JMMAC against competing RGB-T trackers on GTOT, RGBT234, and VOT19-RGBT using dataset-specific tracking benchmarks and metrics.

  • IV. EXPERIMENTS: JMMAC is evaluated on three RGB-T datasets: GTOT, RGBT234, and VOT19-RGBT.
  • IV. EXPERIMENTS: GTOT contains 50 grayscale-thermal sequences with seven annotated challenging attributes, while RGBT234 contains 234 sequences and more than 234,000 frames.
  • IV. EXPERIMENTS: The evaluation uses success and precision plots for GTOT and RGBT234, as shown in Figures 6 and 7.
  • IV. EXPERIMENTS: The comparisons include eight competing RGB-T methods, including MANet, FANet, ECO+RGBT, TODA, DAPNet, SiamFC+RGBT, SGT, and CMR.

A. Quantitative Evaluation

JMMAC achieves leading quantitative performance across the evaluated RGB-T tracking benchmarks, with especially strong results on GTOT and VOT19-RGBT.

  • GTOT: 73.2% success and 90.1% precision make JMMAC the best-performing method on GTOT.It improves over MANet by 0.8% in success and 0.7% in precision.
  • GTOT: JMMAC outperforms eight state-of-the-art competitors by a large margin on GTOT attribute-based evaluation.The evaluation uses maximum success rate and maximum precision rate across challenging attributes.
  • RGBT234: JMMAC is superior to all compared algorithms on the large-scale RGBT234 dataset.The reported comparisons cover challenging factors including occlusion, low illumination, deformation, scale variation, motion blur, and camera motion.
  • VOT19-RGBT: 26.8% relative improvement in EAO over SiamDW T gives JMMAC the best EAO, accuracy, and robustness on VOT19-RGBT.The comparison also reports more accurate results and fewer failure times than the second-ranked tracker.

B. Qualitative Evaluation

Qualitative VOT19-RGBT examples show JMMAC maintaining accurate object localization across camera motion, low resolution, scale variation, and occlusion.

  • Visual comparisons: JMMAC accurately captures targets in camera-motion, low-resolution, and occlusion cases while also refining bounding-box tightness.The examples include Baby, Car37, Caraftertree, and Greyman sequences.

C. Ablation Analysis

Ablation and robustness analyses attribute JMMAC’s gains to multimodal fusion and motion modules, while showing competitive fusion quality and approximately 4 FPS operation.

  • Component effectiveness: Each JMMAC component substantially contributes to performance, with global and local fusion networks both improving the baseline.The final model combines multimodal fusion, camera motion estimation, target motion prediction, and bounding-box refinement.
  • Image fusion: MFNet achieves competitive image-fusion performance with real-time speed and runs seven times faster than FusionGAN.Evaluation uses eight metrics, including EN, MI, Qabf, FMI, Nabf, SSIM, MS SSIM, and FPS.
  • Fusion analysis: MFNet outperforms alternative fusion methods by a large margin on VOT19-RGBT.The comparison includes both early- and late-fusion methods and weighted response fusion.
  • Parameter robustness: The tracker switcher remains robust across a large parameter range under individual perturbations.Parameters are enumerated with interval 2, except tdiff, whose interval is 1.
  • Speed analysis: Approximately 4 FPS is achieved, with MFNet, target motion prediction, and camera motion estimation adding no significant speed decline.The main computational cost comes from the deep-feature appearance tracker; YOLOv2 box refinement is negligible.

V. CONCLUSION

The paper concludes that JMMAC robustly combines appearance and motion cues for RGB-T tracking, improving tracking and image-fusion performance across multiple evaluations.

  • Conclusion: JMMAC combines MFNet-based appearance fusion with camera-motion and target-motion cues to improve robustness when appearance information is unreliable.Experiments on GTOT, RGBT234, and VOT19-RGBT report better performance than state-of-the-art algorithms.
  • Conclusion: MFNet improves tracking accuracy and also achieves competitive performance on the image-fusion task.The method infers fusion weight maps for RGB and thermal modalities to produce a more reliable response map.
Loading 2007.02041v1…