PaperlayerView exact revision on arXiv

Source-linked AI summary

RGB-to-IR image translation for infrared vehicle detection in unseen UAV domains

Thijs A. Eker, Ella P. Fokkinga, Jan Erik van Woerden, Elfi I. S. Hofmeijer, Sebastiaan P. Snel, Klamer Schutte, Friso G. Heslinga

arXiv:2609.02556v1cs.CV

TL;DR

Annotated infrared data are scarce, while RGB-to-IR translation must contend with thermal properties that RGB imagery does not directly reveal. The paper trains and compares generative translators on source pairs, uses them to augment detection training in unseen target domains, and finds synthetic infrared consistently improves over RGB and grayscale baselines, although real target infrared remains better.

  • Problem

    Annotated infrared data are scarce, and thermal appearance is not uniquely inferable from RGB imagery because traits such as engine heat are unobservable.

  • Method

    The study trains supervised GAN, ControlNet diffusion, and LoRA-based translation approaches on paired source RGB–IR data, then augments unseen-target detector training with generated infrared images.

  • Results

    Synthetic infrared consistently outperforms RGB and grayscale baselines for vehicle detection in unseen UAV infrared domains.

  • Takeaways & Limitations

    RGB-to-IR translation can recover a substantial part of lost cross-domain detection performance when target-domain RGB data exist but infrared training data are scarce.

  • Takeaways & Limitations

    Real target-domain infrared data substantially outperform synthetic alternatives, and evaluation covers only two unseen UAV target domains.

Abstract

from arXiv · show

Synthetic training data is crucial for developing vision AI when real-world data is scarce, as in thermal infrared (IR) aerial vehicle detection. While abundant UAV RGB imagery motivates RGB-to-IR translation for data augmentation, unobservable thermal traits (e.g., engine heat) make learning transferable mappings challenging. This work investigates whether modern generative translators can overcome this cross-modal gap to improve infrared vehicle detection on unseen UAV target domains. Translators are trained on paired RGB-IR source datasets and applied to RGB training images from held-out target datasets to generate synthetic IR data. Evaluated methods include supervised GANs, ControlNet-based diffusion models, and foundation-model editing via LoRA. The resulting synthetic IR imagery is used to train RF-DETR vehicle detectors, which are evaluated on unseen IR target test splits across five aerial datasets, with Kust4K and VTUAV serving as target domains. Synthetic IR consistently outperforms RGB and grayscale baselines. Stable Diffusion 3.5 with ControlNet yields the best results, improving mAP from 50.8 to 60.1 on Kust4K and from 25.6 to 38.4 on VTUAV compared to models trained only on source-domain IR data. Increasing output diversity via multiple seeds (+1.1 mAP) and prompt variations (+3.3 mAP) provides additional gains on VTUAV. Although a performance gap to real target IR data remains, generative RGB-to-IR translation effectively mitigates IR data scarcity and improves cross-domain aerial vehicle detection.

1. INTRODUCTION

Infrared vehicle detection is constrained by scarce annotated data and a difficult RGB-to-IR gap because thermal appearance is not uniquely determined by visible imagery. This work tests cross-domain generative translation as augmentation for detection in unseen infrared UAV domains.

  • Motivation: Limited annotated infrared datasets constrain development of high-performance infrared object detectors.Synthetic data is proposed as one response to infrared data scarcity.
  • Motivation: RGB-to-IR translation can leverage abundant RGB imagery and its scene geometry to expand infrared training data.Simulation is difficult, whereas real RGB imagery captures representative scene variability.
  • Research gap: Thermal appearance cannot be inferred deterministically from RGB because engine heat, reflections, material retention, and environmental history are largely unobservable.These factors make transferable RGB-to-IR mapping fundamentally challenging.
  • Approach: The study trains translation models on paired data from multiple source datasets and applies them to RGB images from previously unseen target datasets.Generated infrared images are evaluated as added detector-training data for unseen infrared domains.
  • Study scope: The experiments compare grayscale, GAN, and diffusion-based translation methods for improving vehicle detection on unseen infrared datasets.The study evaluates both cross-domain translation feasibility and its practical value for air-to-ground detection.

2. RELATED WORKS

Prior work addresses infrared UAV domain shifts, RGB-to-IR translation, controllable synthesis, and downstream perception. This study differs by withholding target-domain infrared data from translator training while testing synthetic imagery in unseen UAV domains.

  • Infrared UAV perception: Infrared UAV datasets span diverse environments, viewpoints, altitudes, illumination, sensors, and weather, creating substantial domain shifts.Reported shifts occur both within and between infrared UAV datasets.
  • RGB-to-IR translation: Limited and costly infrared collection and annotation motivate using abundant RGB imagery and annotations to generate synthetic infrared training data.Prior RGB-to-IR work includes GAN-based methods such as pix2pixHD and aerial approaches targeting realistic or diverse infrared generation.
  • Generative methods: Diffusion and foundation-model approaches introduce ControlNet conditioning, semantic guidance, vision-language guidance, or LoRA adaptation for cross-spectral translation.These methods extend RGB-to-IR synthesis beyond earlier GAN-based approaches.
  • Controllability: Diffusion translation treats RGB-to-IR mapping as one-to-many, allowing different plausible outputs from the same RGB image through random seeds.Additional conditioning can steer stochastic variation toward particular infrared appearances.
  • Downstream perception: Downstream studies report benefits for detection, segmentation, and tracking, but some adapt translators using target-domain infrared data.This work instead withholds target-domain infrared imagery from translator training and adaptation in unseen UAV domains.

3. METHODS

The workflow trains RGB-to-IR translators on aligned source-domain pairs, applies them to RGB images from unseen target domains, and uses the generated infrared data to augment detector training. Detection is then measured on real target-domain infrared imagery.

  • Dataset preparation: Five paired RGB–infrared UAV datasets are collected and aligned before translation and detection experiments.The workflow standardizes the paired data for subsequent processing.
  • Translation: Translation models are trained on source-domain RGB–infrared pairs and applied to unseen target-domain RGB training images.The models generate synthetic infrared training data without target-domain infrared translation training.
  • Detection evaluation: Generated infrared imagery augments training of an infrared vehicle detector, which is evaluated on real infrared images from the target domains.This separates synthetic-data generation from target-domain infrared evaluation.

3.1 Datasets

The study builds an aligned paired RGB–infrared dataset from five UAV datasets, using DroneVehicle, Caltech, and M3OT as sources and Kust4K and VTUAV as target domains. Dataset-specific cropping, correction, sampling, masking, alignment, and annotation verification prepare the data for detection experiments.

  • Dataset composition: Five datasets—DroneVehicle, Caltech Aerial RGB-Thermal, M3OT, Kust4K, and VTUAV—are converted into aligned RGB, infrared, and vehicle-annotation pairs.Vehicle categories are mapped to a single detection class.
  • Alignment: Alignment uses cross-modal correspondences and RANSAC-estimated affine transformations, with low-quality pairs discarded under dataset-specific thresholds.Rejected pairs commonly come from low-light RGB imagery with insufficient reliable correspondences.
  • Experimental roles: The source datasets are DroneVehicle, Caltech, and M3OT, while Kust4K and VTUAV are reserved for cross-domain evaluation.Table 1 defines source and target roles after preprocessing and alignment.
  • DroneVehicle: DroneVehicle is center-cropped and aligned, retaining 23,529 paired samples with predominantly small vehicle bounding boxes.Its stored imagery is 840 × 712 pixels before cropping to 640 × 512 pixels.
  • Caltech: Caltech retains 223 paired samples after alignment and manual correction, adding diversity through different backgrounds and environmental conditions.Vehicle segmentation masks are converted to bounding boxes and manually verified.
  • M3OT: M3OT is subsampled every tenth frame after infrared distortion correction, alignment, and masking of improperly annotated regions.The preprocessing retains 952 paired samples.
  • Kust4K: Kust4K undergoes infrared radial-distortion correction and alignment, retaining 2,279 paired samples.Its UAV viewpoint is visually similar to DroneVehicle and M3OT.
  • VTUAV: VTUAV uses sequence sampling, alignment, GroundingDINO pre-labeling, and manual correction, resulting in 1,028 training pairs and 549 test pairs.Its lower-altitude imagery has larger apparent objects and different viewing geometry from the source datasets.

3.2 RGB-to-IR translation methods

The study compares grayscale, supervised GAN, ControlNet diffusion, and LoRA-based foundation-model approaches for RGB-to-IR translation. ControlNet uses RGB images as spatial conditioning, while alternative methods learn paired RGB–IR mappings or edit images with LoRA.

  • Translation approaches: Four translation families are evaluated: grayscale conversion, supervised GAN translation, ControlNet-conditioned diffusion, and instruction-based foundation-model editing.The comparison includes both generative methods and a non-generative grayscale baseline.
  • ControlNet-conditioned diffusion: ControlNet uses the RGB image as conditioning input and specifies the target modality with the prompt “Aerial infrared photo.”Stable Diffusion 3.5 Medium and FLUX.1-dev are used as pretrained ControlNet backbones, with only ControlNet parameters optimized.
  • Translation diversity: Multiple random seeds generate alternative infrared variants for the same RGB image, enabling an evaluation of whether greater training-data diversity improves detection.The experiment compares one generated variant with three variants for both SD and FLUX.
  • Prompt conditioning: SD-ControlNet additionally uses dataset-specific prompts to encode source-dataset imaging characteristics and learn dataset-dependent RGB-to-IR mappings.Training prompts identify the source dataset; target-domain images are translated using corresponding prompt styles.
  • Paired translation and LoRA editing: pix2pixHD directly maps aligned paired RGB images to corresponding infrared images, whereas FLUX-LoRA adapts a foundation model using paired source-domain RGB–IR images and low-rank adaptation.FLUX-LoRA uses the fixed instruction “convert this RGB drone image to thermal infrared” and freezes the base model and text encoder while optimizing LoRA parameters.

3.3 Experimental setup

The experiments train translators and a baseline detector on three source UAV datasets, then use RGB images from two held-out target datasets to augment infrared detector training. Performance is measured with RF-DETR on real target-domain infrared imagery using COCO-style mAP50:95.

  • Datasets: DroneVehicle, M3OT, and Caltech provide paired source data, while Kust4K and VTUAV are held out as close-domain and far-domain targets.VTUAV contains imagery captured from substantially lower flight altitudes than the source datasets.
  • Baseline: The source-only baseline trains exclusively on source infrared imagery and evaluates transfer to source tests, Kust4K, and VTUAV without target-domain training data.This establishes performance when no target-domain data are used for detector training.
  • Augmentation: Target-domain RGB training images are translated into synthetic infrared data and added to source infrared training data, with RGB and grayscale augmentation as simpler comparisons.Grayscale preserves scene geometry without requiring a learned translation model.
  • Additional experiments: The study also combines grayscale and SD-ControlNet imagery and includes real target infrared data as an upper-bound experiment.The combined settings test whether grayscale and generated infrared representations provide complementary benefits.
  • Object detector: RF-DETR Medium is fine-tuned as a single-class vehicle detector, with repeated runs using three random seeds and proportional epoch reductions for expanded datasets.The detector uses pretrained RF-DETR Medium weights and DINOv2 as its vision backbone.
  • Evaluation: Evaluation uses COCO-style mAP50:95 on real infrared target test imagery, averaged across IoU thresholds from 0.50 to 0.95 and three detector seeds.The metric is reported for Kust4K and VTUAV target domains.

4. RESULTS

Synthetic translations preserve scene geometry while varying thermal appearance, and SD-ControlNet provides the strongest learned augmentation results. Gains are largest on the far-domain VTUAV dataset, although real target infrared data remains an upper bound and FLUX-LoRA can misalign annotations.

  • Qualitative translation results: All generative methods preserve the spatial structure of RGB inputs, keeping vehicles, roads, vegetation, and buildings aligned.The grayscale baseline remains unrealistically dark in a low-light Kust4K example because it preserves visible-spectrum intensities.
  • Qualitative translation results: Diffusion seeds change thermal appearance while preserving geometry, especially the relative temperature of roads and vehicles.Different seeds can produce warmer backgrounds, cooler vehicles, or increased vehicle-to-background contrast.
  • Object detection: +6.3 and +4.5 mAP result from adding target-domain RGB images on Kust4K and VTUAV, while grayscale augmentation yields total gains of +8.5 and +8.1 mAP.Both alternatives use the same target-domain RGB training images.
  • Object detection: +9.2 mAP on Kust4K and +11.7 mAP on VTUAV are obtained by single-variant SD-ControlNet over the source-only baseline.Three seeds raise the VTUAV gain to +12.8 mAP, and dataset-specific prompts raise it to +15.0 mAP without additional Kust4K benefit.
  • Limitations: FLUX-LoRA can shift vehicle positions during translation, causing transferred RGB annotations to become misaligned and reducing detector performance below the source-only baseline.This spatial inconsistency limits its usefulness for annotation-preserving augmentation.
  • Object detection: Source-domain test performance remains largely stable across target-domain augmentation experiments, with synthetic variants causing at most a 0.4 mAP reduction.The source-only baseline reaches 63.3±0.1 mAP, while real target infrared data reaches 63.8±0.0 mAP.
  • Combined augmentation: Combining grayscale with SD-ControlNet improves over grayscale alone in every setting, reaching 62.4 mAP on Kust4K and approximately 40 mAP across VTUAV combinations.On VTUAV, the similar combined results suggest the main benefit comes from combining representations rather than adding more generative variation.

5. DISCUSSION

Synthetic infrared data improves unseen-domain vehicle detection beyond RGB and grayscale by combining target-domain scene content with source-learned infrared appearance. Gains are larger in the far-domain VTUAV setting, while translation remains constrained by missing thermal information, source-data imbalance, and limited evaluation scope.

  • Benefits of target-domain data under domain shift: Grayscale already provides substantial benefit by exposing detectors to target-domain geometry, viewpoints, backgrounds, object scales, and annotation distributions.This indicates that part of the domain gap is unrelated to infrared appearance.
  • Benefits of target-domain data under domain shift: Synthetic infrared improves detection beyond RGB and grayscale, with SD-ControlNet gaining +9.3 mAP on Kust4K and +12.8 mAP on VTUAV.The improvement over grayscale is +0.8 mAP on Kust4K and +4.7 mAP on VTUAV.
  • Role of infrared appearance and generative diversity: Random-seed diversity provides limited benefit: VTUAV rises from 37.3 to 38.4 mAP, whereas Kust4K changes negligibly from 60.0 to 60.1 mAP.The larger VTUAV gain suggests multiple plausible appearances are more useful when domain gaps increase.
  • Role of infrared appearance and generative diversity: 40.6 mAP on VTUAV follows from dataset-specific prompt styles, exceeding the 38.4 mAP obtained with three random seeds.Prompt conditioning selects among dataset-associated infrared styles at inference time.
  • Benefits of target-domain data under domain shift: Combining grayscale with SD-translated infrared is more effective than either representation alone, particularly on VTUAV.Grayscale supplies target-domain scene content without translation artifacts, while SD-ControlNet adds source-learned infrared-like appearance.
  • Limitations and future work: RGB-to-IR translation cannot uniquely infer thermal appearance, because engine heat, temperatures, and environmental thermal history are absent from RGB observations.The source infrared collection is also relatively small, imbalanced, and dominated by DroneVehicle.
Loading 2609.02556v1…