PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal RGB-Infrared Combination for UAV-Based Wildfire Segmentation: A Comparative Study on FLAME3

Matheus F. Kovaleski, Luís Garrote, Cristiano Premebida, Jérôme Mendes, João Ruivo Paulo

arXiv:2609.01390v1cs.CV

TL;DR

Wildfire segmentation needs robust use of visual and thermal information, yet the effects of modality choice and fusion timing remain insufficiently compared. This paper evaluates RGB and infrared baselines alongside three fusion strategies across U-Net, DeepLabV3+, and SegFormer on FLAME 3, finding stronger thermal-based performance and especially promising feature-level fusion with SegFormer.

  • Problem

    The study addresses limited understanding of modality selection and fusion timing because prior work rarely compares unimodal and multiple fusion strategies under one protocol.

  • Method

    The paper compares RGB-only and infrared-only baselines with early, middle, and late fusion across U-Net, DeepLabV3+, and SegFormer using FLAME 3 multimodal data.

  • Results

    Thermal infrared imagery provides substantially more discriminative information than RGB alone, while SegFormer with middle fusion achieves the best overall performance.

  • Takeaways & Limitations

    Feature-level fusion with transformer-based architectures is the most promising direction identified for exploiting complementary RGB–IR information.

  • Takeaways & Limitations

    Evaluation is restricted to FLAME 3, so the observed trends require assessment on additional UAV wildfire datasets and environmental and sensor conditions.

Abstract

from arXiv · show

Unmanned Aerial Vehicles (UAVs) have emerged as a promising platform for firefighting operations due to their flexibility, low operational cost, and ability to acquire high-resolution imagery in locations that may be difficult or dangerous to access using conventional methods. Recent advances in deep learning have significantly improved the capabilities of UAV-based wildfire monitoring systems. The present work investigates RGB-infrared fusion for binary wildfire segmentation on the FLAME3 dataset. In this Study, RGB and Infrared baselines are compared with three representative fusion strategies across three segmentation architectures, including U-Net, DeepLabV3+, and SegFormer. The key motivation of this work is to analyze the contribution of each modality, evaluate the impact of fusion timing, and examine how different network architectures exploit multimodal information for UAV wildfire delineation. The findings indicate that thermal information plays a dominant role in UAV segmentation and that feature-level multimodal fusion combined with transformer-based architectures offers the most promising direction for future research.

I. INTRODUCTION

UAV wildfire monitoring benefits from flexible, high-resolution imaging, but RGB-only segmentation can struggle under adverse conditions. This study systematically compares modalities, fusion timing, and architectures on FLAME 3 to clarify multimodal segmentation performance.

  • Motivation: UAVs provide flexible, low-cost, high-resolution imagery for monitoring difficult-to-access wildfire locations.Compared with satellite systems, UAVs offer higher spatial resolution and lower latency for detailed fire observation.
  • Motivation: RGB imagery supplies contextual and spatial detail but may degrade under smoke, illumination changes, shadows, and low-contrast fire boundaries.Thermal infrared imagery captures heat emissions and is relatively robust to lighting changes.
  • Dataset opportunity: FLAME 3 provides synchronized visible-spectrum and radiometric thermal UAV imagery for investigating multimodal wildfire segmentation.Its radiometric thermal measurements enable analysis of how thermal information complements RGB observations.
  • Research gap: Existing studies provide limited evidence about how fusion strategies affect segmentation because they often examine one approach or architecture.Unified comparisons of RGB-only, infrared-only, and multimodal methods remain scarce.
  • Study design: The study compares RGB-only and infrared-only baselines with early, middle, and late fusion across U-Net, DeepLabV3, and SegFormer.The evaluation targets modality contributions, fusion timing, and architectural use of multimodal information.

II. RELATED WORK

UAV datasets and deep learning have advanced wildfire analysis by enabling high-resolution imagery and learned segmentation, while challenging fire conditions continue to motivate multimodal approaches.

  • UAV sensing: UAVs enable fine-resolution, flexible observation of fire fronts, smoke plumes, and burned areas in near real time.Their operational advantages support wildfire monitoring and emergency response.
  • Datasets: FLAME 3 extends earlier wildfire benchmarks with synchronized RGB and radiometric thermal imagery for multimodal learning.The dataset supports analysis based on calibrated thermal measurements.
  • Deep learning: Deep learning segmentation models generally outperform traditional computer vision methods by learning visual representations directly from aerial imagery.Wildfire segmentation remains difficult because of small flames, smoke occlusions, illumination changes, and heterogeneous backgrounds.

B. Deep Learning Architectures for Wildfire Segmentation

Wildfire segmentation research spans encoder–decoder CNNs, multi-scale convolutional models, and transformer-based architectures, while RGB–thermal fusion introduces complementary information and unresolved design choices.

  • CNN architectures: U-Net is a common low-cost baseline that preserves spatial information through skip connections.Prior studies report competitive wildfire segmentation performance across datasets and imaging conditions.
  • CNN architectures: DeepLabV3+ uses atrous convolutions and ASPP to extract multi-scale context while preserving spatial resolution.It has achieved higher segmentation accuracy than conventional encoder–decoder networks in aerial wildfire imagery.
  • Transformer architectures: Transformer-based and hybrid CNN–Transformer models capture long-range dependencies and may exploit multimodal information differently from CNNs.FireFormer combines convolutional feature extraction with transformer contextual modeling and achieved competitive FLAME performance.
  • Multimodal sensing: RGB provides texture, shape, and contextual cues, whereas thermal infrared highlights active fire under adverse illumination or partial smoke occlusion.These complementary properties motivate multimodal wildfire segmentation.
  • Open comparison: The literature lacks unified comparisons of RGB-only, infrared-only, and multiple fusion strategies across architectures.This study evaluates early, middle, and late fusion with U-Net, DeepLabV3, and SegFormer on FLAME 3.

III. METHODOLOGY

The study uses FLAME 3 synchronized RGB, thermal, and radiometric data, with a common preprocessing and partitioning protocol for segmentation experiments.

  • Dataset: FLAME 3 supplies synchronized RGB imagery, thermal imagery, and radiometric temperature measurements from UAV wildfire samples.The dataset includes samples acquired under different wildfire conditions.
  • Data sources: Each sample includes visible-spectrum RGB images, colorized thermal JPEG files, and Celsius radiometric temperature TIFF maps.Only samples containing all three valid file types were included, and images were resized to 512 × 512 pixels.
  • Preprocessing: RGB images were normalized with ImageNet statistics, while thermal images were independently normalized to preserve temperature-related information.
  • Experimental protocol: The dataset was randomly split into 70% training, 15% validation, and 15% testing samples.The same partition was used across experiments to support fair comparisons.

B. Mask Generation

FLAME 3 fire masks are derived from radiometric temperature measurements, then used to compare five RGB, infrared, and fusion input configurations.

  • Mask Generation: Binary fire masks are generated from radiometric Celsius TIFF images through temperature thresholding.The resulting masks serve as ground truth for all segmentation experiments.
  • Mask Generation: An 80°C threshold labels pixels as fire, based on the FLAME 3 temperature-based fire/no-fire screening criterion.The resulting binary mask is denoted M.
  • Mask Generation: The thresholding procedure produces objective fire-region annotations from measured thermal information and avoids variability associated with manual labeling.
  • Fusion Strategies: Five input configurations are evaluated: RGB-only, IR-only, early fusion, middle fusion, and late fusion.RGB and IR provide unimodal baselines, while fusion timing is varied for multimodal comparisons.
  • Fusion Strategies: Early fusion concatenates RGB and thermal inputs into a six-channel tensor, whereas middle fusion integrates intermediate modality-specific feature maps before decoding.
  • Fusion Strategies: Late fusion combines final segmentation logits from independent RGB and thermal networks through a learnable fusion layer.

D. Segmentation Models

The study evaluates three segmentation architectures selected for complementary handling of spatial detail, contextual information, and feature representation.

  • Segmentation Models: Three semantic segmentation architectures are evaluated: U-Net, DeepLabV3, and SegFormer.
  • Evaluation: Model selection is based on the highest validation Intersection over Union (IoU).
  • Evaluation: Evaluation uses standard segmentation metrics after converting probabilities into binary predictions with a 0.5 threshold.Reported results are obtained on the held-out test set.
  • U-Net: U-Net is an encoder–decoder convolutional architecture using skip connections to preserve spatial information during reconstruction.Its localization capability supports segmentation of small and sparse targets.

2) DeepLabV3:

DeepLabV3 and SegFormer provide contrasting architectural mechanisms for contextual and global feature representation within the comparative segmentation study.

  • DeepLabV3: DeepLabV3 uses atrous convolutions and Atrous Spatial Pyramid Pooling to capture multi-scale contextual information.
  • DeepLabV3: The study adopts DeepLabV3 with a MobileNet backbone because of its balance between computational efficiency and segmentation performance.
  • SegFormer: SegFormer combines hierarchical self-attention representations with a lightweight decoder to capture long-range dependencies and preserve multi-scale features.
  • Comparative Analysis: The architectures are compared for how convolutional and transformer-based models exploit visible, thermal, and fused multimodal information.
  • Experimental Design: All experiments use the same training protocol to ensure a fair comparison across architectures and fusion strategies.

IV. EXPERIMENTAL RESULTS

The experiments compare RGB-only and infrared-only baselines with early, middle, and late fusion across architectures using a consistent evaluation protocol on FLAME 3.

  • Experimental Results: The evaluation analyzes how visible and infrared modalities contribute to fire segmentation and how fusion strategies affect multiple architectures.
  • Experimental Results: RGB-only and IR-only inputs are compared with early, middle, and late fusion strategies.
  • Experimental Results: Fusion timing is assessed through precision, recall, F1-score, and IoU.
  • Experimental Protocol: The same dataset partitions, preprocessing pipeline, and optimization settings are maintained across architectures and fusion configurations.This supports a consistent comparison.
  • Results Table: Table I reports wildfire segmentation performance on the FLAME 3 test set, with bold values marking the best result for each metric.

B. Evaluation Protocol

Models were evaluated on FLAME 3 using Precision, Recall, F1-score, and IoU to compare RGB-only, IR-only, and three RGB–IR fusion strategies across architectures. Infrared generally outperformed RGB, while fusion effects depended on the architecture and fusion stage.

  • Evaluation setup: Evaluation used Precision, Recall, F1-score, and IoU on the FLAME 3 test set to compare modalities and fusion strategies.The comparison covered RGB-only and infrared-only baselines alongside early, middle, and late fusion.
  • U-Net: U-Net’s RGB-only configuration had low recall and IoU, indicating that visible imagery alone missed many fire pixels.RGB achieved moderate precision, but its low recall indicated poor fire-region detection.
  • U-Net: 0.75 F1-score and 0.62 IoU were achieved by U-Net’s IR-only configuration, while fusion strategies reached similar F1-scores of 0.73–0.74 without clearly surpassing IR-only.This result identifies infrared as more discriminative than RGB for U-Net on FLAME 3.
  • U-Net: U-Net early fusion produced the highest precision but lower recall, whereas late fusion offered a more balanced precision–recall trade-off near the IR-only setup.Overall, RGB did not provide a consistent additional gain when fused with IR for U-Net.
  • DeepLabV3+: DeepLabV3+ RGB-only input performed poorly, with low recall and IoU, confirming that RGB alone was insufficient for reliable segmentation on FLAME 3.The pattern was similar to U-Net’s RGB-only result.
  • DeepLabV3+: DeepLabV3+ IR-only achieved an F1-score of 0.69 and IoU of 0.55, while late fusion reached 0.74 F1-score and 0.58 IoU; middle fusion performed substantially worse.The results show that DeepLabV3+ performance depended strongly on fusion stage.

3) SegFormer Results:

SegFormer achieved the strongest overall results among the evaluated architectures, with infrared providing a substantial advantage over RGB-only input. Middle and late fusion were most effective, and middle fusion offered the best overall trade-off.

  • SegFormer: SegFormer’s RGB-only configuration performed poorly, although it achieved the best RGB-only result among the three architectures with an F1-score of 0.24 and IoU of 0.14.This remained substantially weaker than SegFormer’s IR-based configurations.
  • SegFormer: 0.81 F1-score and 0.69 IoU were achieved by SegFormer with IR-only input, substantially improving over RGB-only performance.The result confirms the strong relevance of infrared information for wildfire segmentation.
  • SegFormer: SegFormer’s strongest fusion results came from middle and late fusion, with middle fusion achieving the highest IoU and F1-score and late fusion achieving the highest recall.These strategies fused modalities after modality-specific feature extraction.
  • Cross-model analysis: RGB-only segmentation performed poorly across all architectures, with consistently low recall indicating that visible imagery missed most fire pixels.The evaluated results suggest RGB alone was not sufficiently reliable for active wildfire segmentation.
  • Cross-model analysis: Infrared was the dominant modality across architectures, substantially outperforming RGB-only models in recall, F1-score, and IoU.The paper attributes this to infrared capturing fire-related information not easily detected from visible imagery alone.
  • Cross-model analysis: Fusion effectiveness depended on architecture and strategy: U-Net did not clearly improve over IR-only, DeepLabV3+ favored late fusion, and SegFormer favored middle and late fusion.Transformer-based architectures may benefit from modality-specific feature extraction before fusion.
  • Cross-model analysis: SegFormer with RGB–IR middle fusion provided the best trade-off across precision, recall, F1-score, and IoU, while IR-only models remained strong across architectures.RGB therefore needs careful incorporation through an appropriate fusion strategy.

V. CONCLUSION

The study systematically compared unimodal baselines and RGB–infrared fusion across U-Net, DeepLabV3+, and SegFormer on FLAME 3. Thermal infrared was more discriminative than RGB, while fusion performance depended on architecture and strategy, with SegFormer middle fusion performing best overall.

  • Contributions: The study compared unimodal baselines and multimodal fusion approaches across U-Net, DeepLabV3+, and SegFormer using FLAME 3.The comparison was conducted under a unified experimental protocol.
  • Findings: Thermal infrared provided substantially more discriminative information than RGB, with RGB-only configurations underperforming and infrared-only inputs consistently achieving strong results.These findings emphasize thermal information for active-fire delineation in UAV imagery.
  • Findings: Fusion effectiveness depended on both architecture and strategy; early fusion did not consistently improve over infrared-only baselines, while SegFormer middle fusion achieved the best overall performance.The results suggest transformer architectures can exploit RGB–IR information better when modality-specific features are learned before fusion.
  • Limitation: Evaluation was restricted to FLAME 3, so the observed trends require assessment on additional UAV wildfire datasets with different environmental and sensor conditions.This is the study’s stated generalization boundary.
  • Future work: Future work may examine additional fusion mechanisms, uncertainty-aware segmentation, radiometric temperature integration, and lightweight multimodal architectures for real-time UAV deployment.These directions extend the study’s multimodal wildfire-segmentation focus.
Loading 2609.01390v1…