PaperlayerView exact revision on arXiv

Source-linked AI summary

Early versus Late RGB-Infrared Fusion for Counter-UAV Detection: A Comparative Study

Alberto De Zan, Denis Tavaris, Gian Luca Foresti, Ivan Scagnetto

arXiv:2610.29828v1cs.CV

TL;DR

RGB and infrared sensors have complementary strengths, but Counter-UAV fusion must account for image alignment and operational conditions. The paper compares early fusion of RGB–IR inputs with late fusion of separate detector outputs; early fusion has better overall performance, while late fusion can accommodate misalignment.

  • Problem

    RGB detection is vulnerable in low light and adverse weather, while IR has lower spatial resolution and higher cost; Counter-UAV sensor fusion remains underexplored.

    Source: §I · p. 1

  • Method

    The study compares a single detector on fused RGB–IR inputs with separate RGB and IR detectors whose bounding-box outputs are merged.

    Source: §I · p. 1

  • Results

    Early fusion achieved better overall performance than late fusion, and RGT reached 56.2% mAP@50-95 among the reported fusion strategies.

    Source: A. Early Fusion · p. 6, §V · p. 9

  • Takeaways & Limitations

    Early fusion is presented as the better general-use strategy, while late fusion may be advantageous at night or when sensors or frames are misaligned.

    Source: 2) Late-Fusion (Decision-Level) · p. 3, §VI · p. 10

    Early fusion is harder to implement and run because it requires a dedicated model trained on fused data and accurate pre-alignment.

    Source: §VI · p. 10

Abstract

from arXiv · show

This paper presents a novel strategy for UAV detection through visible (RGB) and infrared (IR) fusion, in order to improve accuracy and precision in harsh environments and weather conditions. In this research, two opposite approaches have been studied and implemented: the first concerns image-level (early) fusion, which merges two different spectra into a single image to which one detector is applied. On the other hand, the second approach, called decision-level (late) fusion, focuses on blending together the output bounding boxes of two different detection pipelines, aiming to create a common result that can be applied to both spectra simultaneously. Both strategies have been extensively tested in a dedicated environment, which included alternating low and high brightness scenes, different backgrounds (sky and ground), distances (5, 25 and 60 m), altitudes and zoom levels (1X, 2X and 4X). These tests allowed for the assessment of actual detection capabilities and system reliability in real-world applications. Furthermore, the two fusion strategies were evaluated and compared over a set of aligned images in order to highlight the better approach of the two. The results indicate that early fusion achieves better raw performance, since it incorporates the thermal information from the beginning of the training process. However, late fusion presents an innovative and useful detection workflow that can be applied in cases of sensor or frame misalignment. Both positive and negative aspects have also been pointed out and discussed, in order to give a more comprehensive overview. Finally, the paper outlines implementation considerations and practical recommendations based on operational priorities. Overall, the findings contribute to the research in RGB-IR fusion modalities, with the goal of enhancing UAV detection capabilities.

I. INTRODUCTION

UAV detection must contend with security concerns and sensor trade-offs: RGB offers high spatial resolution but struggles in low light and adverse weather, while IR works in low light but is costly and lower-resolution. The section traces detection approaches and motivates RGB–IR fusion as an underexplored Counter-UAV setting with practical sensor-alignment constraints.

I. INTRODUCTION

  • RGB cameras provide high spatial resolution at lower cost, but low light, adverse weather, and cluttered backgrounds can hinder detection; IR works in low light but has lower thermal and spatial resolution and higher cost.The paper identifies combining the sensors’ strengths as a motivation for fused UAV detection.Source: §I · p. 1
  • Counter-UAV fusion remains underexplored, and many existing approaches require substantial sensor adjustments before fusion can be applied.Source: §I · p. 1

A. Detection Techniques

  • Classical image-processing and feature-based UAV detectors can be computationally efficient but often lack robustness to dynamic backgrounds, illumination changes, and clutter.Deep learning advanced detection; two-stage detectors can be computationally costly, while one-stage detectors such as YOLO offer higher inference speed.Source: A. Detection Techniques · p. 2
  • Transformer-based and multimodal methods broaden UAV perception, while many UAV detectors still rely on visible imagery, motivating complementary sensing for low-light and adverse-weather conditions.RGB–IR and other multimodal approaches have been explored for operational Counter-UAV detection.Source: A. Detection Techniques · p. 2

B. Fusion Techniques

The section reviews RGB–IR fusion as a response to complementary sensor strengths and summarizes early-, mid-, and late-fusion approaches alongside practical alignment and deployment constraints.

B. Fusion Techniques

  • RGB and IR have complementary characteristics, making their integration a promising direction for operational Counter-UAV detection.Source: B. Fusion Techniques · p. 2
  • Image-fusion methods use decomposition, contrast enhancement, and transformer approaches, but paired-image alignment is often required and may be unavailable because of sensor limitations.Other methods integrate image registration, and some approaches address spatially misaligned but temporally synchronized images.Source: B. Fusion Techniques · p. 2
  • Despite advances in feature-level, Bayesian, and RGB-T fusion, publicly available UAV detectors using multimodal fused data remain limited under misalignment and real-time deployment constraints.Source: B. Fusion Techniques · p. 2

1) Early-Fusion (Pixel-Level):

Early fusion combines RGB and infrared information at the pixel level before detection. The section describes prior fusion architectures and the alignment and channel-format considerations involved in constructing fused inputs.

1) Early-Fusion (Pixel-Level):

  • Pixel-level fusion concatenates RGB and IR images into a four-channel input for a conventional object detector.Source: 1) Early-Fusion (Pixel-Level) · p. 2
  • Prior image-level fusion methods include autoencoder-based and illumination-aware architectures, while some studies found pixel-level fusion underperformed deeper fusion variants in pedestrian detection.The cited passage also notes ways to improve pixel-level fusion.Source: 1) Early-Fusion (Pixel-Level) · p. 3
  • Early fusion requires aligned RGB–IR pairs; the described channel-stacking approach adds 8-bit IR thermal information to RGB channels, producing a 32-bit image.The passage identifies PNG as necessary when more than three channels are needed because JPG supports at most three.Source: 1) Early-Fusion (Pixel-Level) · p. 3

2) Late-Fusion (Decision-Level):

Late fusion runs separate RGB and IR detectors, then combines their bounding boxes. The section presents confidence-weighted coordinate fusion as more tolerant of misalignment, with localization precision as a trade-off.

2) Late-Fusion (Decision-Level):

3) Mid-Fusion (Feature-Level):

Mid-fusion merges separately extracted RGB and infrared feature maps before detection, preserving modality-specific inputs without combining images or final detections. It was not implemented because it would require a custom detection architecture, preventing comparable metrics in this study.

3) Mid-Fusion (Feature-Level):

  • Mid-fusion extracts RGB and infrared features separately, then merges feature maps by concatenation or summation before the final detection layers.This places it between early fusion, which merges images, and late fusion, which combines detection outputs.Source: 3) Mid-Fusion (Feature-Level) · p. 3
  • Keeping original channels preserves modality-specific information, while avoiding statistical combination of final detection results makes mid-fusion more flexible to inputs and detection goals.Source: 3) Mid-Fusion (Feature-Level) · p. 3
  • Mid-fusion was not implemented because it requires a custom architecture unlike the tested state-of-the-art YOLO design, and did not permit the same strategy-comparison metrics.Source: 3) Mid-Fusion (Feature-Level) · p. 3

III. LOGICAL PROPOSED ARCHITECTURES

The study uses YOLO-based detection pipelines and a real-world multimodal dataset to test the fusion methods. The paired data include aligned RGB–IR images of still and moving drones, with a 75/25 training-validation split.

III. LOGICAL PROPOSED ARCHITECTURES

  • The proposed architecture is based on YOLO, with separate pipelines developed for the two fusion types.Source: §III · p. 3
  • A real-world multimodal dataset of still and moving drones was built because acquired data better reflect operational conditions than synthetic data.Source: §III · p. 3
  • The dataset includes drone rotation, shifts, and pose changes across roll, pitch, and yaw, with 400 aligned image pairs split into 300 training and 100 validation images.Source: §III · p. 3

A. Early Fusion

Early fusion stacks aligned RGB and thermal channels into a single input for YOLOv8s, using red, green, and thermal (RGT) channels to retain thermal information without modifying the detector. RGT leads the proposed channel combinations on reported mAP metrics, while comparison with FCDFusion also highlights a trade-off between detection metrics and computational complexity.

A. Early Fusion

  • Early fusion requires spatially aligned RGB and infrared frames, which are transformed and overlapped by stacking channels into a single multimodal input.Source: A. Early Fusion · p. 4
  • RGT replaces the blue channel with thermal information, reducing potentially redundant spectral information while retaining a three-channel input compatible with YOLOv8s.The camera uses far-infrared, distinguishing RGT from color-infrared imagery based on near-infrared.Source: A. Early Fusion · p. 5
  • RGT, RBT, and GBT each retain thermal information while omitting one RGB channel, enabling an ablation study of channel selection.Source: A. Early Fusion · p. 5, Figure 10 · p. 6
  • 56.2% mAP@50-95 for RGT is the highest among the proposed fusion strategies; RGT mAP@50 is 97.8%, versus 89.8% for Infrared.The comparison uses YOLOv8s trained for 300 epochs on the same dataset while varying channels and fusion strategies.Source: Table II · p. 6, A. Early Fusion · p. 6
  • FCDFusion exceeds RGT in precision and mAP@50-95, while RGT has higher recall and mAP@50; both achieve an F1-Score of 95.2%.FCDFusion requires a dedicated fusion network, whereas RGT fuses at frame level without one during inference.Source: A. Early Fusion · p. 6
  • The reported evaluation is preliminary and could be extended with a larger fused dataset and other object-detection models.Source: A. Early Fusion · p. 6

B. Late Fusion

Late fusion processes RGB and IR detections independently, then combines nearby bounding boxes using confidence-weighted averaging. A distance threshold governs fusion; boxes outside it remain separate, allowing the approach to handle misaligned frames.

B. Late Fusion

  • Nearby RGB and IR box centers are fused using confidence-weighted averaging; boxes beyond the distance threshold are preserved independently.The fused center is confidence-weighted, while its width and height take the maximum dimensions of the original boxes.Source: B. Late Fusion · pp. 6–8
  • Figure 11 depicts one bounding box shared by both images and notes that the approach can work with misaligned frames.The figure caption states this capability based on the experiments.Source: Figure 11 · p. 7
  • The fixed threshold is δ = 50 pixels at 640 × 640 input resolution, serving as a correspondence tolerance for parallax and residual registration errors.The paper treats it as predefined rather than an optimized detection hyperparameter.Source: B. Late Fusion · p. 7
  • The method preserves original detections when centers fall outside the threshold, including detections that occur in only one modality.When two boxes are fused, the maximum original width and height avoid excessive shrinking of the combined box.Source: B. Late Fusion · pp. 7–8

IV. EXPERIMENTAL TESTS

The experimental tests compare Early and Late Fusion using performance metrics and visual examples. The analysis covers both quantitative and qualitative evidence.

IV. EXPERIMENTAL TESTS

  • The tests analyze and compare real-world results for both Early and Late Fusion after model performance metrics are computed.The section describes the analysis as both visual and empirical.Source: §IV · p. 8
  • Quantitative evaluation includes average detection precision, recall, and F1-score.The passage also mentions qualitative visualizations through image samples.Source: §IV · p. 8
  • The evaluation uses image samples alongside quantitative metrics to assess the two fusion strategies.Source: §IV · p. 8

A. Acquisition Scenarios

Acquisition scenarios use real-world video frames and evaluate detection across varying environments and zoom levels. The reported observations include conditions associated with confidence changes and a computational cost trade-off for Late Fusion.

A. Acquisition Scenarios

  • The external-test review uses confidence scores for an initial qualitative assessment and presents additional quantitative metrics in a dedicated table.The review discusses benefits and limitations through frames from the collected videos.Source: A. Acquisition Scenarios · p. 8
  • Sky backgrounds generally yield higher confidence, while wooded clutter, leaf openings, summer heat, and brightness challenge detection or add infrared noise.The passage also reports that airborne targets are better identified than stationary or sharply changing targets.Source: A. Acquisition Scenarios · p. 8
  • The drone was detected at all zoom levels, including great distances; 1X detection was sometimes intermittent, whereas 2X and 4X worked reliably, making the system more stable.Tests were limited to a maximum distance of 60 meters and altitude of 20 m. Zoom was used in borderline cases with wooded backgrounds and a drone camouflaged among leaves.Source: A. Acquisition Scenarios · p. 8
  • Outdoor tests cover different times and conditions, moving-target altitudes and distances, tree and sky backgrounds, and three zoom levels.Figure 13 identifies the takeoff area and target-drone flight area; Table III summarizes the experimental evaluation.Source: Table III · p. 8, Figure 13 · p. 8
  • Late Fusion requires two 21.5 MB detection models at 640×640 input, resulting in approximately twice the model and inference requirements of RGT.Source: A. Acquisition Scenarios · p. 8
  • Figure 14 lays out ground and sky comparisons by time and zoom, with Early Fusion RGT in the first row and Late Fusion results in the third and fourth rows.Its caption identifies 1X, 2X, and 4X and times 9:00, 12:00, and 15:00.Source: Figure 14 · p. 9

V. RESULTS

Early-Fusion achieved better overall performance than Late-Fusion, which nevertheless remained competitive. The comparison used validation metrics from a pool of 400 spatially aligned images aggregated across varied acquisition conditions.

V. RESULTS

  • Validation compared both modalities on 400 spatially aligned images aggregated across backgrounds, altitudes, lighting, and other operating conditions.Results were presented in aggregate because individual scenario datasets were considered too small for meaningful comparison.Source: §V · p. 9
  • Late-Fusion independently processes each RGB-IR pair and combines detections into fused bounding boxes for comparison with ground-truth annotations.Early-Fusion predictions are compared with corresponding annotations using an overlap mechanism; both approaches are evaluated using average precision, recall, and F1-score.Source: §V · p. 9
  • Early-Fusion achieved the best overall performance, while Late-Fusion remained competitive in combining complementary information for aerial target detection.Table IV summarizes the mean metrics for both strategies.Source: §V · p. 9, Table IV · p. 9

VI. CONCLUSION AND FUTURE WORK

Early fusion is favored for general use, while late fusion may suit conditions where RGB information is lost. The conclusion also notes implementation demands and the need for broader multimodal data.

VI. CONCLUSION AND FUTURE WORK

  • Early fusion is identified as best for general use, while late fusion might be advantageous at night when the RGB counterpart is completely lost.The paper links early fusion to channel blending and notes that IR could support detection when RGB is lost.Source: §VI · p. 10
  • Early fusion is harder to implement and run because it requires training a dedicated model on fused data and accurate pre-alignment.Source: §VI · p. 10
  • The proposed solution is designed for drone-to-drone detection, and adapting it to other detection pipelines requires more multimodal data.Future work also considers newer sensors and image types such as multispectral or hyperspectral imagery, as well as radar data.Source: §VI · p. 10

Related papers

Loading 2610.29828v1…