Source-linked AI summary
Learning Cross-Modal Deep Representations for Robust Pedestrian Detection
Dan Xu, Wanli Ouyang, Elisa Ricci, Xiaogang Wang, Nicu Sebe
TL;DR
The paper tackles pedestrian detection under adverse illumination when surveillance systems primarily provide RGB data. It learns RGB-to-thermal representations from paired images, transfers them to an RGB detector, and reports state-of-the-art KAIST performance and competitive Caltech results.
Problem
Pedestrian detection remains difficult under adverse illumination, while most surveillance systems use RGB sensors and few labeled multimodal datasets are available.
Method
A two-phase framework learns a non-linear RGB-to-thermal mapping with RRN, then transfers its representations to the RGB-based multi-scale MSDN detector.
Results
The framework outperforms the state-of-the-art on KAIST, is competitive on Caltech, and achieves a 64.01% miss-rate on Caltech-All.
Takeaways & Limitations
Thermal data can provide training supervision for RGB pedestrian detection without being required during testing or requiring thermal-domain pedestrian annotations.
Takeaways & Limitations
The demonstrated application is pedestrian detection, while extending the cross-modality framework to other tasks remains a future research direction.
Abstract
from arXiv · showhide
This paper presents a novel method for detecting pedestrians under adverse illumination conditions. Our approach relies on a novel cross-modality learning framework and it is based on two main phases. First, given a multimodal dataset, a deep convolutional network is employed to learn a non-linear mapping, modeling the relations between RGB and thermal data. Then, the learned feature representations are transferred to a second deep network, which receives as input an RGB image and outputs the detection results. In this way, features which are both discriminative and robust to bad illumination conditions are learned. Importantly, at test time, only the second pipeline is considered and no thermal data are required. Our extensive evaluation demonstrates that the proposed approach outperforms the state-of- the-art on the challenging KAIST multispectral pedestrian dataset and it is competitive with previous methods on the popular Caltech dataset.
1. Introduction
The paper addresses pedestrian detection with RGB cameras under adverse illumination by learning cross-modal representations from RGB-thermal pairs. Thermal information supervises feature learning during training, while testing uses RGB images alone.
- RGB surveillance systems still face illumination variation, shadows, and low external light, motivating robust pedestrian detection without relying on additional sensors.
- The method learns a non-linear mapping from RGB to thermal images using multispectral pairs without pedestrian bounding-box annotations.Pair-wise RGB-thermal information provides weak supervision for cross-modal representation learning.
- The learned representations are transferred into a second deep architecture that operates on RGB data and models multi-scale information for detection.
- At test time, pedestrian detection requires only color images, so thermal data are not needed in deployment.The framework uses the auxiliary thermal modality as training supervision rather than as a test-time input.
- The framework is presented as beneficial for distinguishing hard negative samples, such as electric poles or trees resembling pedestrians, in color images.
- The evaluation reports state-of-the-art performance on KAIST and competitiveness with previous methods on Caltech.
2. Related Work
Prior pedestrian detectors improved through deep networks, specialized handling of occlusion, speed, and domain adaptation, but generally did not combine multimodal data with adverse-illumination detection. Cross-modal feature transfer remained relatively underexplored.
- Pedestrian Detection: Deep networks learn discriminative pedestrian features directly from raw pixels, while other methods address occlusion, visibility, target relations, or detection speed.
- Pedestrian Detection: Earlier multimodal pedestrian approaches used additional thermal or depth features, whereas the paper targets adverse illumination with a CNN-based approach.
- Domain Transfer: Scene-specific transfer methods adapt source models to target scenes using contextual cues, sample reweighting, or distribution-matching selection.
- Learning Cross-modal Deep Representations: Cross-modal representation learning has been studied with deep networks, but jointly learning and transferring such features has rarely been investigated.
- Learning Cross-modal Deep Representations: Related work includes hallucinating data from other modalities, but the paper distinguishes its CNN-based cross-modal approach from those methods.
3. Learning and transferring cross-modal deep representations
The framework uses two convolutional networks: RRN learns RGB-to-thermal representations from paired data, and MSDN transfers them into RGB pedestrian detection. Training proceeds in reconstruction and detection phases, while inference uses RGB only.
- 3.1. Overview: The Region Reconstruction Network (RRN) learns a non-linear RGB-to-thermal mapping from pedestrian proposals in paired RGB-thermal images without human annotation.RRN uses thermal images as deep supervision while requiring only pair-wise information.
- 3.2. Region Reconstruction Network: Pedestrian proposals from a generic detector provide training regions containing true and false positives, avoiding reliance on carefully annotated ground-truth boxes.
- 3.2. Region Reconstruction Network: RRN receives an RGB image and pedestrian proposals, extracts ROI features, and reconstructs thermal regions through learned deconvolutional upsampling.The ROI feature maps are 512 × 7 × 7 and are upsampled to 50×50 reconstruction outputs.
- 3.3. Multi-Scale Detection Network: The Multi-Scale Detection Network (MSDN) fuses feature maps from two parallel sub-networks to detect pedestrians from RGB images.
- 3.3. Multi-Scale Detection Network: Sub-Net B receives the transferred convolutional parameters from RRN, embedding cross-modal mid-level representations into MSDN.
- Training: Training has two phases: RRN is trained on multispectral data, then MSDN parameters are optimized with RGB images and target-domain pedestrian annotations.
- Detection: During detection, MSDN outputs class scores and bounding-box offsets from RGB images and proposals, followed by non-maximum suppression.
4. Experiments
Experiments evaluate the cross-modality framework on KAIST and Caltech, showing improved pedestrian detection, especially under difficult illumination, while requiring only RGB images at test time.
- Datasets: The evaluation uses the KAIST multispectral and Caltech pedestrian datasets, with KAIST testing divided into reasonable all, day, and night settings.KAIST contains aligned RGB-thermal pairs captured across day and night; Caltech evaluation uses Caltech-All and Caltech-Reasonable settings.
- Results on KAIST multispectral dataset: CMT-CNN significantly outperforms its evaluated variants across all three KAIST test sets, indicating that cross-modality learning improves detection accuracy.Its lower miss rate than CMT-CNN-SA-SB also indicates the gain is not explained only by the increased number of parameters.
- Results on KAIST multispectral dataset: Qualitative results show that CMT-CNN reconstructs thermal regions from RGB detections and better rejects hard negatives such as tree foliage than CMT-CNN-SA.The reconstruction network uses the top nine ACF detection windows, while the detection comparison highlights fewer false pedestrian detections for foliage.
- Results on KAIST multispectral dataset: CMT-CNN improves on KAIST over the ACF-RGBT+HOG baseline while using only color images at test time, unlike the baseline's use of RGB and thermal data.The improvement is more significant on Reasonable night than on Reasonable day, supporting particular usefulness under dark illumination.
- Results on Caltech pedestrian dataset: On Caltech, thermal knowledge transfer improves accuracy over CMT-CNN-SA-SB initialized with KAIST RGB data, although the gain is less pronounced than on KAIST.The authors associate the smaller Caltech improvement with generally better illumination conditions in that dataset.
5. Conclusions
The paper introduces a deep cross-modality framework that transfers information from RGB-thermal training data into RGB-only pedestrian detection. It demonstrates robust detection under challenging illumination and identifies broader multimodal applications as future work.
- Conclusions: The proposed architecture pre-trains a sub-network on paired RGB and thermal images to learn cross-modal representations, then transfers that knowledge to detection from raw RGB images.This enables accurate detection under challenging illumination conditions without requiring thermal data at test time.
- Conclusions: Extensive experiments on the KAIST multispectral and Caltech pedestrian datasets demonstrate the effectiveness of the proposed approach.The conclusion frames these publicly available benchmarks as evidence for robust pedestrian detection under adverse illumination.
- Conclusions: The framework is specifically evaluated for pedestrian detection, while extending cross-modality learning to depth reconstruction for RGBD recognition remains a future research direction.The authors identify applications such as RGBD object and action detection and recognition.