Source-linked AI summary
Multi-Modal Fusion for End-to-End RGB-T Tracking
Lichao Zhang, Martin Danelljan, Abel Gonzalez-Garcia, Joost van de Weijer, Fahad Shahbaz Khan
TL;DR
RGB-T tracking seeks to combine RGB and TIR because their information is complementary, but prior methods lack suitable fusion schemes and large aligned training data. The paper builds an end-to-end DiMP-based tracker with pixel-, feature-, and response-level fusion using synthetic paired data. The proposed fusion improves single-modality baselines, with the best VOT-RGBT2019 result reaching an EAO of 0.391 and state-of-the-art performance on RGBT210.
Problem
RGB-T tracking lacks a specific end-to-end fusion scheme and large-scale aligned multi-modal training data.
Method
The paper extends DiMP with pixel-level, feature-level, and response-level RGB-T fusion and trains it using a large-scale paired synthetic RGB-T dataset.
Results
The proposed fusion improves single-modality tracking, achieves an EAO of 0.391 on VOT-RGBT2019, and sets a new state-of-the-art on RGBT210.
Takeaways & Limitations
End-to-end modality fusion enables the tracker to use complementary RGB and TIR information and improves benchmark RGB-T tracking performance.
Abstract
from arXiv · showhide
We propose an end-to-end tracking framework for fusing the RGB and TIR modalities in RGB-T tracking. Our baseline tracker is DiMP (Discriminative Model Prediction), which employs a carefully designed target prediction network trained end-to-end using a discriminative loss. We analyze the effectiveness of modality fusion in each of the main components in DiMP, i.e. feature extractor, target estimation network, and classifier. We consider several fusion mechanisms acting at different levels of the framework, including pixel-level, feature-level and response-level. Our tracker is trained in an end-to-end manner, enabling the components to learn how to fuse the information from both modalities. As data to train our model, we generate a large-scale RGB-T dataset by considering an annotated RGB tracking dataset (GOT-10k) and synthesizing paired TIR images using an image-to-image translation approach. We perform extensive experiments on VOT-RGBT2019 dataset and RGBT210 dataset, evaluating each type of modality fusing on each model component. The results show that the proposed fusion mechanisms improve the performance of the single modality counterparts. We obtain our best results when fusing at the feature-level on both the IoU-Net and the model predictor, obtaining an EAO score of 0.391 on VOT-RGBT2019 dataset. With this fusion mechanism we achieve the state-of-the-art performance on RGBT210 dataset.
1. Introduction
RGB tracking can fail under low illumination, rain, and smog, while RGB and TIR provide complementary information. The paper addresses limited end-to-end RGB-T research by evaluating fusion architectures and generating paired training data.
- RGB tracking performs poorly in low illumination, rain, and smog, whereas thermal infrared sensors provide a more stable signal.
- RGB supplies high-frequency texture and rich object representations, while TIR is resistant to illumination and shadows and separates warm objects from colder backgrounds.
- Prior multi-modal trackers largely use sparse or hand-crafted features, while end-to-end fusion remains unexplored because the fusion location and training data are unclear or unavailable.
- The paper proposes pixel-level, feature-level, and response-level fusion architectures built around the DiMP RGB tracker.
- The authors generate a large-scale paired synthetic RGB-T dataset and evaluate the approach on VOT-RGBT2019 and RGBT210.
- The multi-modal fusion tracker achieves an EAO score of 0.391 on VOT-RGBT2019 and a 55.5% success rate on RGBT210.
2. Related work
Existing tracking research relies heavily on RGB data and, for TIR and RGB-T tracking, often uses hand-crafted or off-the-shelf features. This paper motivates end-to-end RGB-T learning using synthetic paired data and modality fusion.
- Single modality tracking: Deep RGB tracking progressed from hand-crafted features to end-to-end training as large-scale datasets such as GOT-10k became available.
- Single modality tracking: Siamese networks learn similarity metrics offline, while correlation-filter trackers emphasize discriminative filtering and efficient tracking.
- TIR trackers: Top-performing TIR trackers still commonly use hand-crafted features, although synthetic TIR data from RGB datasets has enabled end-to-end deep tracking.
- Modality fusion tracking: Earlier RGB-T methods combine modalities through spatiograms, sparse representations, adaptive weights, or modality ranking.
- Modality fusion tracking: The paper identifies insufficient training data and the lack of a specific fusion scheme as barriers to end-to-end RGB-T learning.
3. Baseline RGB tracker
The baseline is DiMP, an end-to-end tracker that predicts a discriminative target model from training samples and estimates boxes using an IoU-Net. Its RGB-T extension fuses modality features within these components.
- DiMP is an end-to-end trainable tracker that embeds online target-model learning and includes a feature extractor, model predictor, and IoU-Net.
- Feature extractor: DiMP uses ImageNet-trained ResNet-18 or ResNet-50 backbones and fine-tunes them during end-to-end training.
- Feature extractor: The feature extractor produces training samples from image patches and bounding-box centers, split between training and test portions of each segment.
- Model predictor: The model predictor maps the training sample set Strain to a filter f, which is evaluated on test samples to compute classification loss.
- Bounding box estimation: IoU-Net predicts the IoU between image features and candidate boxes, after which box estimation maximizes the predicted IoU.
- Multi-modal fusion: In feature-level fusion, block3 and block4 features feed IoU modulation and prediction, while block4 features feed the model predictor for the final response map.
4. End-to-end multi-modal tracking
The paper addresses native fusion and data-scarcity barriers in end-to-end RGB-T tracking by evaluating fusion at multiple network stages and generating aligned synthetic training pairs.
- Motivation: End-to-end RGB-T tracking must address both the absence of native fusion components in RGB trackers and the lack of large-scale paired training data.These constraints complicate extending RGB trackers to RGB-T inputs and training feature representations end to end.
- Fusion architectures: The paper evaluates pixel-level, feature-level, and response-level fusion architectures within the DiMP tracking framework.The architectures place fusion at the network input, an intermediate representation stage, or the response level.
- Feature-level fusion: Feature-level fusion processes RGB and TIR images through separate feature extractors, concatenates their deep representations, and feeds them to the IoU and model predictors.This delays fusion until a more semantically aware stage of the network.
- Pixel-level fusion: Pixel-level fusion concatenates RGB and TIR images along the channel direction before feature extraction.The first feature-extractor layer is expanded from 7 × 7 × 3 × 64 to 7 × 7 × 4 × 64.
- RGB-T data generation: The training data pipeline uses an RGB tracking dataset and image-to-image translation to generate aligned synthetic TIR images with identical bounding boxes.This produces paired RGB-T data for end-to-end training of the proposed fusion architectures and fine-tuning pre-trained DiMP models.
5. Experiments
Experiments evaluate RGB-T fusion mechanisms in DiMP on VOT-RGBT2019 and RGBT210, using synthetic paired training data and repeated stochastic runs. Feature-level fusion across both the model predictor and IoU-Net performs best, improving over single-modality and baseline trackers across the benchmarks.
- Training data: Synthetic paired TIR images are generated from GOT-10k RGB training data using pix2pix for end-to-end fusion training.The reduced GOT-10k training set contains 9,335 videos and 1,403,359 frames; pix2pix is trained using 87K aligned RGB-T image pairs.
- Single-modality baselines: 1% absolute gain results from fine-tuning the single-modality network for TIR over the pre-trained networks.Fine-tuning on RGB also improves the pre-trained model, but less than fine-tuning with TIR.
- Fusion analysis: 0.389 EAO results when fused features feed both the model predictor and IoU-Net, a 5.8% absolute gain over the best single-modality model.Feature-level fusion for IoU-Net alone reaches 0.366, while the final selected variant uses fused features for both components.
- Benchmark results: 6.4% absolute EAO improvement over DiMP is obtained on VOT-RGBT2019, while RGBT210 gains are 6.7% in precision rate and 4.2% in success rate.On RGBT210, mfDiMP achieves state-of-the-art performance and surpasses other trackers on all but one annotated attribute; several challenging attributes show about 10% Success Rate gains.
6. Conclusions
The proposed end-to-end multi-modal fusion tracker improves over single-modality tracking and achieves state-of-the-art results on both evaluated datasets.
- The proposed fusion tracker significantly improves the DiMP baseline over single-modality tracking.
- The tracker sets new state-of-the-art results on both VOT-RGBT2019 and RGBT210.
- The study introduces three end-to-end fusion architectures: pixel-level, feature-level, and response-level fusion.
- A large-scale paired synthetic RGB-T dataset supports end-to-end training of the proposed fusion tracker.