Source-linked AI summary
Borrow from Anywhere: Pseudo Multi-modal Object Detection in Thermal Imagery
Chaitanya Devaguptapu, Ninad Akolekar, Manuj M Sharma, Vineeth N Balasubramanian
TL;DR
Thermal object detection is constrained by relatively small thermal datasets despite the richer resources available in RGB imagery. The paper generates pseudo-RGB images from thermal inputs and combines them with thermal data in a multimodal detector, outperforming baselines and doing so with as little as one quarter of the thermal training set. Performance remains challenging for small, nearby, heavily occluded, or crowded objects.
Problem
Thermal object detection has relatively limited training data compared with RGB detection, motivating ways to use RGB-domain resources without paired multimodal examples.
Method
The framework translates thermal images into pseudo-RGB images and processes the thermal and pseudo-RGB inputs through two branches of a multimodal Faster R-CNN.
Results
The framework outperforms baselines across FLIR ADAS and KAIST settings and can surpass full-training-set baseline accuracy using only one quarter of the thermal training set.
Takeaways & Limitations
RGB-domain features can improve thermal detection without paired training examples, providing a strategy for learning from fewer thermal examples.
Takeaways & Limitations
MMTOD finds detection challenging for very small distant objects, adjacent objects merged into one detection, heavy occlusion, and crowds.
Abstract
from arXiv · showhide
Can we improve detection in the thermal domain by borrowing features from rich domains like visual RGB? In this paper, we propose a pseudo-multimodal object detector trained on natural image domain data to help improve the performance of object detection in thermal images. We assume access to a large-scale dataset in the visual RGB domain and relatively smaller dataset (in terms of instances) in the thermal domain, as is common today. We propose the use of well-known image-to-image translation frameworks to generate pseudo-RGB equivalents of a given thermal image and then use a multi-modal architecture for object detection in the thermal image. We show that our framework outperforms existing benchmarks without the explicit need for paired training examples from the two domains. We also show that our framework has the ability to learn with less data from thermal domain when using our approach. Our code and pre-trained models are made available at https://github.com/tdchaitanya/MMTOD
1. Introduction
Thermal object detection matters for safer sensing because thermal cameras remain useful under illumination changes, while large-scale RGB datasets offer stronger detection resources than thermal datasets. The proposed pseudo-multimodal framework transfers RGB-domain features through generated pseudo-RGB inputs without requiring paired modalities.
- 1. Introduction: Thermal cameras can support autonomous driving and surveillance because they are robust to illumination changes and useful during day and night.The paper also describes them as low-cost, nonintrusive, and small.
- 1. Introduction: 75% of 5,987 U.S. pedestrian fatalities occurred after dark in 2016, highlighting the risks faced by current autonomy sensors without thermal imaging.
- 1. Introduction: Large-scale RGB datasets have enabled strong object detectors, but fewer publicly available thermal datasets restrict comparable progress in thermal imagery.
- 1. Introduction: The proposed pseudo-multimodal detector uses an RGB branch pretrained on large datasets and finetuned on pseudo-RGB images generated from thermal inputs, alongside a thermal branch.
- 1. Introduction: The framework borrows high-level RGB features without paired examples or a synchronized multimodal dataset, using any available large-scale RGB detection dataset.
2. Related Work
Prior thermal detection work includes handcrafted, machine-learning, and deep-learning methods, while multimodal methods combine visual and infrared information in several architectures. The paper positions its approach as a pseudo-multimodal alternative that uses translated inputs to access RGB-domain knowledge.
- 2. Related Work: Earlier thermal detection methods used templates, candidate filtering, AdaBoost, and other conventional approaches, but assumed conditions such as humans being hotter than the background.
- 2. Related Work: Deep learning methods have outperformed earlier approaches on thermal classification and detection by learning more descriptive features.
- 2. Related Work: Thermal deep-learning research has included compact CNNs, boosted classifiers, and architectures for face identification, pedestrian behavior recognition, and low-resolution target recognition.
- 2. Related Work: Existing multimodal approaches fuse visual and infrared information using channel features, boosted trees, separate proposal networks, or fused deep features.
3. Methodology
MMTOD borrows RGB-domain knowledge for thermal detection by translating each thermal image into pseudo-RGB and processing both modalities in a two-branch detector.
- Unpaired Image-to-Image Translation: CycleGAN and UNIT provide unpaired thermal-to-RGB generators, avoiding the need for paired multimodal training examples.The thermal-to-RGB generator parameters are denoted W_T 2R in the methodology.
- Pseudo Multi-modal Object Detection: The framework generates a pseudo-RGB image from each thermal input and feeds both images into separate thermal and RGB branches.The branches are initialized with domain-specific pretrained models.
- Pseudo Multi-modal Object Detection: The branch feature maps are stacked, transformed with a 1 × 1 convolution, and passed into the remaining Faster R-CNN network.The 1 × 1 convolution connects the two modality-specific feature streams to the detector head.
- MMTOD Training: Training minimizes the object-detection loss while updating the RGB branch, thermal branch, detector head, convolution, and translation-generator weights.Algorithm 1 lists these components as trainable during the proposed methodology.
- Pseudo Multi-modal Object Detection: The I2I generator is updated jointly with detector components because translated images remain perceptually different from natural RGB images.Joint optimization learns pseudo-RGB representations that better support borrowing RGB-domain features for thermal detection.
4. Experiments
Experiments evaluate pseudo-multimodal detection on FLIR ADAS and KAIST, using unpaired thermal data, RGB pretraining, and image-to-image translation. MMTOD consistently improves detection over thermal Faster R-CNN baselines, including with fewer thermal examples.
- Datasets and Experimental Setup: The experiments use thermal images from FLIR ADAS and KAIST, while RGB branches are initialized from PASCAL VOC or MS-COCO detectors.The method intentionally does not use paired RGB images, including for KAIST despite its aligned RGB and thermal data.
- Image-to-Image Translation: CycleGAN and UNIT generate pseudo-RGB images, although their translations remain perceptually far from natural RGB images.Despite the domain gap and missing chrominance information, the generated images add value to detection.
- Thermal Object Detection Results: MMTOD outperforms the baseline across all classes on both FLIR ADAS and KAIST.Tables 1 and 2 compare class AP and mAP against baseline detectors; KAIST contains only the Person class.
- Thermal Object Detection Results: At least 7 mAP points are gained on FLIR when combining MS-COCO initialization with UNIT translation.MS-COCO initialization performs better than PASCAL-VOC, and UNIT performs better than CycleGAN in the FLIR experiments.
- Thermal Object Detection Results: MMTOD surpasses the FLIR benchmark of 0.58 mAP at IoU 0.5 despite using the older Faster R-CNN detector.The benchmark uses RefineDetect-512, whereas the proposed method uses Faster R-CNN.
5. Discussion and Ablation Studies
The discussion shows that MMTOD retains its advantage with substantially fewer thermal training examples and at reduced image resolution, while revealing concrete failure cases.
- Learning with limited examples: Both MMTOD-UNIT and MMTOD-CG trained on FLIR (1/2) outperform baselines trained on the complete FLIR dataset.The comparison is reported in Table 4.
- Learning with limited examples: With only one quarter of the thermal training examples, MMTOD surpasses baseline accuracy achieved using the full training set.The authors attribute this result to borrowing helpful features from the RGB domain.
- Learning with limited examples: The limited-data experiments indicate that MMTOD can learn effectively from fewer thermal examples, which is useful given the scarcity of large-scale thermal datasets.The test set remains unchanged in these experiments.
- Effect of Image Resolution: MMTOD significantly improves object-detection performance on 400 × 400 FLIR images.The authors identify extension to even lower resolutions as future work.
- Missed Detections: MMTOD struggles with very small distant objects, nearby objects merged into one detection, and heavy occlusion or crowds.These cases are identified through analysis of missed detections shown in Figure 7.