PaperlayerView exact revision on arXiv

Source-linked AI summary

Drone-based RGB-Infrared Cross-Modality Vehicle Detection via Uncertainty-Aware Learning

Yiming Sun, Bing Cao, Pengfei Zhu, Qinghua Hu

arXiv:2003.02437v2cs.CVcs.LGeess.IV

TL;DR

The paper addresses difficult drone-based vehicle detection in complex and low-light aerial scenes, where RGB and infrared modalities each contain useful but uncertain information. It constructs the DroneVehicle dataset and proposes UA-CMDet with uncertainty-aware weighting and illumination-aware fusion, reporting improved cross-modal detection performance, including gains from IA-NMS.

  • Problem

    Drone-based vehicle detection is challenging in complex, low-light aerial scenes because RGB and infrared images contain complementary but also redundant or misleading information.

  • Method

    The paper constructs DroneVehicle and proposes UA-CMDet, using UAM to quantify modality uncertainty and IA-NMS to fuse modality-specific detections.

  • Results

    IA-NMS improves mAP by 0.76% with concatenate fusion and by 0.46% with element-wise-add fusion under the reported experimental conditions.

  • Takeaways & Limitations

    The dataset and framework support cross-modal vehicle detection across complex aerial scenes from day to night.

Abstract

from arXiv · show

Drone-based vehicle detection aims at finding the vehicle locations and categories in an aerial image. It empowers smart city traffic management and disaster rescue. Researchers have made mount of efforts in this area and achieved considerable progress. Nevertheless, it is still a challenge when the objects are hard to distinguish, especially in low light conditions. To tackle this problem, we construct a large-scale drone-based RGB-Infrared vehicle detection dataset, termed DroneVehicle. Our DroneVehicle collects 28, 439 RGB-Infrared image pairs, covering urban roads, residential areas, parking lots, and other scenarios from day to night. Due to the great gap between RGB and infrared images, cross-modal images provide both effective information and redundant information. To address this dilemma, we further propose an uncertainty-aware cross-modality vehicle detection (UA-CMDet) framework to extract complementary information from cross-modal images, which can significantly improve the detection performance in low light conditions. An uncertainty-aware module (UAM) is designed to quantify the uncertainty weights of each modality, which is calculated by the cross-modal Intersection over Union (IoU) and the RGB illumination value. Furthermore, we design an illumination-aware cross-modal non-maximum suppression algorithm to better integrate the modal-specific information in the inference phase. Extensive experiments on the DroneVehicle dataset demonstrate the flexibility and effectiveness of the proposed method for crossmodality vehicle detection. The dataset can be download from https://github.com/VisDrone/DroneVehicle.

I. INTRODUCTION

Drone-based vehicle detection remains difficult in complex, low-light aerial scenes, while existing RGB-only or non-aerial cross-modal datasets do not address full-time oriented vehicle detection. The paper introduces DroneVehicle and UA-CMDet, which combine RGB and infrared information while accounting for modality-specific uncertainty.

  • Motivation: Complex backgrounds and varying illumination make aerial vehicle detection challenging, especially in low-light scenes.RGB-only methods cannot cope with low-light complex scenarios, motivating full-time aerial-view detection methods.
  • Research gap: Existing cross-modal detection methods focus on multispectral pedestrian detection in urban street scenes and do not handle oriented vehicles across aerial scenarios.The paper identifies the lack of large-scale full-time aerial RGB-Infrared vehicle datasets as a central obstacle.
  • Dataset: DroneVehicle contains 28,439 RGB-Infrared image pairs and 953,087 annotated objects across urban roads, residential areas, parking lots, and day-to-night conditions.The dataset uses oriented bounding-box annotations and is presented as the first and largest full-time drone-based RGB-Infrared cross-modality dataset.
  • Motivation: RGB and infrared images provide complementary information but also introduce modality-specific redundancy and false vehicle-like objects.Infrared can improve localization under low light, whereas missing color information and thermal crossover can produce confusing rectangles and ghost shadows.
  • Method: UA-CMDet combines RGB and infrared information through an uncertainty-aware module and illumination-aware cross-modal non-maximum suppression.The UAM uses RGB illumination and cross-modal IoU to calculate modality uncertainty weights, while IA-NMS fuses modality-specific detection results.
  • Evaluation: Qualitative comparisons show that the proposed method improves detections where RGB suffers from poor lighting and infrared contains ghost shadows or confusing rectangles.The method integrates effective information from both modalities and achieves better results than single-modality baselines in the illustrated cases.

II. RELATED WORK

The paper reviews natural-scene vehicle detection datasets and highlights their broad utility alongside limitations for aerial, cross-modality vehicle detection.

  • Existing Natural Scene Datasets: PASCAL VOC and MS COCO are natural-scene benchmarks, while KITTI and BDD100K represent driving-scene datasets.These datasets provide object and vehicle annotations across different scene complexities and scales.
  • Existing Natural Scene Datasets: PASCAL VOC contains 16,551 images, 20 categories, and vehicle annotations for cars and buses.The dataset includes 3,267 annotated car boxes and 822 annotated bus boxes.
  • Existing Natural Scene Datasets: MS COCO contains more than 328,000 images and 2.5 million manually annotated instances across 80 categories, including car, bus, and truck.Compared with PASCAL VOC, it has more complex backgrounds, more objects, and smaller objects.
  • Existing Natural Scene Datasets: BDD100K provides 1,841,435 annotated bounding boxes for 100,000 key frames from 100,000 videos across 10 categories.Its vehicle-related categories are bus, truck, and car.

B. Existing Aerial View Datasets

Existing aerial datasets support vehicle detection through diverse scenes, scales, modalities, and annotations, while DroneVehicle emphasizes cross-modal drone imagery and oriented vehicle boxes.

  • Existing Aerial View Datasets: Aerial vehicle datasets include UA-DETRAC, DLR 3K, VEDAI, COWC, CARPK, UAVDT, and VisDrone.They span vehicle detection and tracking, oriented boxes, infrared subsets, overhead imagery, parking lots, and attribute annotations.
  • Existing Aerial View Datasets: VEDAI contains more than 3,700 annotated objects in over 1,200 images, including large- and small-size color and infrared subsets.Its vehicle categories are annotated in aerial imagery.
  • Existing Aerial View Datasets: CARPK consists of 1,448 drone images containing 89,777 annotated cars captured from different parking lots.UAVDT instead provides about 80,000 frames and 14 annotated attributes.
  • Vehicle Detection: The reviewed vehicle detectors include RetinaNet, Faster R-CNN, Deformable RoI Pooling, and Mask R-CNN.These methods address dense detection, proposal-based detection, changing aerial orientations and scales, and segmentation-assisted detection.
  • Existing Aerial View Datasets: DroneVehicle selects five vehicle categories and annotates 953,087 oriented bounding boxes for object instances.The categories are car, truck, bus, van, and freight car.

B. Data Preprocessing

DroneVehicle preprocessing cleans and aligns drone-captured RGB-infrared pairs, uses oriented annotations for aerial objects, and documents broad coverage across modalities, lighting, viewpoints, and heights.

  • Data Preprocessing: Poor-quality images such as blurred samples are discarded, remaining images are manually checked, and resolution is standardized to 840 × 712.This preprocessing produces data without annotations before the annotation procedure.
  • Data Preprocessing: Affine transformation and region cropping compensate for pixel misalignment between RGB-infrared pairs caused by unstable drone attitudes.The calibration aims to ensure that most cross-modal pairs are aligned.
  • Data Annotation: Oriented bounding boxes represent aerial objects using four vertex coordinates because UAV camera angles make precise rectangular calibration difficult.The annotation is designed to capture varied object orientations compactly.
  • Statistics and Attributes: DroneVehicle contains 56,878 images and oriented annotations for five categories, with cars receiving 389,779 RGB and 428,086 infrared annotations.The five categories are car, truck, bus, van, and freight car.
  • Statistics and Attributes: The dataset includes 14,478 day, 5,468 night, and 8,493 dark-night images, with night plus dark-night data close to the day-data volume.Infrared images yield more annotations in dark scenes, while RGB images often provide more complete annotations in some daytime scenes.
  • Statistics and Attributes: DroneVehicle covers 15,475 vertical-view pairs and 12,964 pairs across 15°, 30°, and 45° angles at heights of 80m, 100m, and 120m.Each image averages 16.76 vehicles, with a maximum of 206.

IV. METHOD

UA-CMDet combines separate RGB, infrared, and fused-feature branches with uncertainty-aware weighting based on cross-modal annotation agreement and modality-specific missing information.

  • Method: UA-CMDet contains a cross-modality detector, an uncertainty-aware module, and illumination-aware cross-modal non-maximum suppression.IA-NMS integrates modality-specific outputs during inference.
  • Method: The framework uses three branches for RGB, infrared, and fused features, with each branch predicting classification confidence scores and bounding-box coordinates.The branches are post-processed through IA-NMS to produce final detections.
  • Uncertainty-Aware Module: The uncertainty-aware module quantifies cross-modal uncertainty in a task-driven manner rather than through a generic mathematical distribution.It provides uncertainty information to the detection process.
  • Uncertainty-Aware Module: Cross-modal IoU measures localization differences between polygonal ground-truth boxes in paired RGB and infrared images.The module uses this measure to match corresponding boxes and identify missing boxes in each modality.
  • Uncertainty-Aware Module: RGB ground-truth boxes fill missing infrared annotations, while infrared boxes fill missing RGB annotations when visibility differs between modalities.Newly added boxes receive uncertainty weights, whereas original infrared boxes have weight 1.

B. Uncertainty-Aware Cross-modality Detector

UA-CMDet combines RGB, infrared, and fused detection branches with uncertainty-aware weighting. Its UAM recalibrates regression losses using modality-specific uncertainty information.

  • Architecture: UA-CMDet modifies RoITransformer into a cross-modality detector with RGB, infrared, and fusion branches.The detector includes a cross-modal fusion module for joint learning of fused and modality-specific knowledge.
  • Architecture: RGB and infrared feature maps are concatenated, reduced with a 1×1 convolution, and passed to separate detection heads.The three branches produce outputs that are combined during inference, while modality-specific heads are retrained during training.
  • Uncertainty weighting: UAM supplies uncertainty weights to recalibrate object bounding-box regression in the RPN and RoI Transformer modules.The uncertainty weight ω is associated with predicted bounding-box regression targets and differs by modality branch.
  • Training objective: The RGB branch uses a multi-task loss that jointly trains classification and bounding-box regression.Classification uses cross-entropy, and the task-loss balance parameter λ is set to 1 in all experiments.
  • Training objective: The total UA-CMDet loss is a weighted sum of the RGB, infrared, and fusion branch losses, with α, β, and γ all set to 1.These trade-off settings keep the respective modal weights consistent.

C. Illumination-Aware Cross-modal NMS

UA-CMDet retains independent RGB and infrared heads because fusion can lose modality-specific information. IA-NMS integrates the three branch outputs while accounting for RGB sensitivity to illumination.

  • Motivation: Independent RGB and infrared detection heads are retained during training and inference to preserve modality-specific information lost during fusion.The outputs of the RGB, infrared, and fusion branches are integrated during inference.
  • Motivation: Cross-modal NMS is needed because softmax classification probabilities can be inflated, making false positives from one modality disrupt fusion.The text identifies this issue as especially acute in cross-modal detection.
  • IA-NMS: IA-NMS processes the RGB, infrared, and fusion detection boxes and scores to integrate predictions from all three branches.The branch outputs are represented as separate box and score lists before cross-modal suppression.
  • IA-NMS: IA-NMS is designed to reduce RGB prediction interference in dark scenarios, using an NMS threshold Nl of 0.1.The strategy leverages RGB sensitivity to lighting and NMS removal of redundant boxes.

V. EXPERIMENT

The experiment section reports ablation studies, analyzes operations affecting performance, compares mAP with state-of-the-art methods, and visualizes detection results.

  • Experimental program: Experiments begin with ablation studies on DroneVehicle and analyze operations affecting the proposed method’s performance.The section then reports mAP comparisons with state-of-the-art methods and discusses visualized detection results.

A. Experimental Setting

The experiments use a ResNet-FPN backbone, SGD training, parameter settings for uncertainty and alignment, and comparisons against seven state-of-the-art detectors.

  • Implementation: The implementation uses ResNet-FPN with pretrained ResNet-50 initialization, random horizontal flipping, and 12 SGD training epochs.Training uses learning rate 0.005, batch size 2, weight decay 0.0001, and momentum 0.9.
  • Dataset statistics: The DroneVehicle object statistics are reported after dataset splitting, with R denoting RGB and I denoting infrared.The table distinguishes object counts by modality after the split.
  • Hyperparameters: The RGB uncertainty weight is selected as 0.1, the infrared reference weight is 1, and the alignment threshold μ is 0.8.The RGB value is chosen from experiments ranging from 0.01 to 0.5; the infrared branch compensates for missing objects.
  • Baselines: Performance is compared with seven detectors, including RetinaNet, Faster R-CNN variants, Mask R-CNN variants, and RoITransformer.The compared methods include oriented bounding-box and deformable RoI pooling variants.

3) Partition Protocol:

The DroneVehicle dataset is partitioned into training, validation, and testing sets, and mAP evaluates vehicle detection using an IoU threshold above 0.5. Ablations show that UAM and cross-modal fusion improve detection performance over the baseline.

  • Partition Protocol: 17,990 training, 1,469 validation, and 8,980 testing RGB-Infrared image pairs define the DroneVehicle partition.The training set contains 286,793 RGB vehicles and 316,411 infrared vehicles.
  • Partition Protocol: mAP evaluates test-set bounding-box quality, counting predictions as true positives when IoU with the nearest ground truth exceeds 0.5.
  • Partition Protocol: UAM increases mAP by 1.45% in RGB modality and 0.37% in infrared modality over the corresponding baseline.The ablation adds UAM directly to the RoITransformer baseline and trains RGB and infrared detectors separately.
  • Partition Protocol: Without IA-NMS, UA-CMDet remains 13.89% higher than the baseline with UAM in RGB modality and 3.73% higher in infrared modality.Replacing the cross-modal fusion operation with element-wise addition lowers mAP by 0.23% relative to original UA-CMDet.
  • Partition Protocol: CMDet improves mAP over single-modality baselines by 14.67% RGB and 3.43% infrared with element-wise addition, versus 14.62% and 3.38% with concatenation.

3) Illumination-Aware NMS:

IA-NMS improves cross-modal detection under matched experimental conditions, while the ablations show incremental contributions from SMO, MA, and IA. UA-CMDet achieves the highest reported mAP against compared methods, with category-specific gains varying by modality.

  • Illumination-Aware NMS: 0.76% and 0.46% mAP gains result from IA-NMS with concatenation and element-wise-add fusion, respectively.The comparisons use identical experimental conditions and corresponding cross-modal fusion branches.
  • Illumination-Aware NMS: More than 6% AP improvement for bus occurs with IA-NMS under both concatenation and element-wise-add fusion.IA-NMS also improves AP for many categories.
  • Illumination-Aware NMS: SMO, MA, and IA are the three operations used to quantify modality uncertainty and are evaluated through ablation experiments.
  • Illumination-Aware NMS: In RGB ablations, SMO raises mAP by 0.42%, MA adds 0.61%, and IA adds a further 0.42% over the model without IA.
  • Illumination-Aware NMS: With element-wise-add fusion, SMO plus MA increases mAP by 0.24% and IA adds 0.20%; concatenation yields gains of 0.07% and a further IA improvement.
  • Illumination-Aware NMS: UA-CMDet achieves the highest mAP among compared methods, exceeding RoITransformer by 16.10% in RGB and 4.86% in infrared modality.
  • Illumination-Aware NMS: Compared with the highest category accuracies, UA-CMDet improves four RGB categories by more than 16% and all infrared categories by more than 3.5%.RGB gains include car 18.99%, freight car 17.72%, truck 16.03%, and bus 16.53%; infrared car accuracy decreases.

2) Qualitative evaluation:

Qualitative evaluations compare UA-CMDet with a RoITransformer baseline across RGB and infrared imagery, including night and daytime scenarios. The visualizations report improved handling of uncertain locations, infrared “ghost shadows,” and confusing objects.

  • Qualitative evaluation: UA-CMDet is qualitatively evaluated against RoITransformer on DroneVehicle in both RGB and infrared modalities.
  • Qualitative evaluation: UA-CMDet addresses uncertain vehicle locations in poorly lit RGB images and correctly detects infrared “ghost shadows” and confusing rectangle objects.
  • Qualitative evaluation: Night and daytime visualizations place RGB results in the first and third rows, with corresponding infrared results in the second and fourth rows.
  • Qualitative evaluation: Red dashed boxes mark errors and missed detections, while solid red boxes magnify areas of interest; the reported UA-CMDet rows accurately detect each object in both modalities.
  • Qualitative evaluation: DroneVehicle is presented as the first and largest cross-modal dataset enabling vehicle detection in complex aerial scenes.
  • Qualitative evaluation: UA-CMDet combines uncertainty information in a unified cross-modal framework, using UAM and IA-NMS, and experiments validate its effectiveness against state-of-the-art methods.
  • Qualitative evaluation: The work focuses on uncertainty quantification for RGB and infrared modalities, while long-tail data distribution remains a future concern.
Loading 2003.02437v2…