PaperlayerView exact revision on arXiv

Source-linked AI summary

Training with synthetic data for drone detection in thermal imagery

Tanel Liiv, Sander Soodla, Nzamba Bignoumba, Alma M. Liezenga, Toomas Pruuden

arXiv:2608.17799v1cs.CVcs.AIcs.ETcs.RO

TL;DR

LWIR drone detection is difficult because thermal imagery offers limited texture and annotated data are scarce. This paper evaluates synthetic-first pre-training with real-data fine-tuning and finds that synthetic data help, but small amounts of real LWIR data remain necessary for the best performance.

  • Problem

    LWIR drone detection is difficult because reduced texture, sensor noise, weak thermal contrast, and scarce annotated datasets limit reliable detection.

  • Method

    The study uses synthetic IR scene generation for detector pre-training, followed by targeted fine-tuning on limited real LWIR data.

  • Results

    0.723 mAP@50:95 was achieved by RF-DETR-L with multi-modality fine-tuning, while synthetic pre-training followed by real-data fine-tuning improved performance over limited real-only training.

  • Takeaways & Limitations

    Synthetic data are useful for few-shot real-data fine-tuning but do not replace in-domain LWIR data, which aligns representations with sensor-specific image statistics.

  • Takeaways & Limitations

    The validation setting covers limited sensors, environments, target ranges, drone types, bird distributions, and deployment scenarios, so operational robustness remains unestablished.

Abstract

from arXiv · show

Ground-to-Air (G2A) drone detection in medium- and long-wave infrared (MWIR/LWIR) imagery is challenging due to reduced texture information, sensor noise, weak thermal contrast, and the scarcity of annotated data. This work investigates a synthetic-first training strategy that combines synthetic scene generation with fine-tuning on real data. We show that synthetic data provides an effective basis for learning initial object representations, while real in-domain thermal imagery is still essential for reliable deployment. Even small amounts of real IR data substantially reduce domain gaps. Our experiments indicate that dataset alignment has a stronger impact on performance than model scale. Finally, our analysis of the dataset suggests that semantic alignment in feature space is the strongest predictor of model performance, while radiometric properties such as entropy and dynamic range also contribute to detection robustness. This work provides a foundation for combining synthetic and real IR data for effective G2A drone detection.

1. INTRODUCTION

The study addresses the challenges and annotation scarcity of G2A drone detection in LWIR imagery through synthetic-first training with targeted real-data fine-tuning. It also analyzes synthetic–real dataset alignment across radiometric, representation, and structural levels.

  • Challenges: LWIR G2A drone detection is difficult because of reduced texture, sensor noise, radiometric variability, and weak thermal contrast.These modality-specific limitations distinguish infrared detection from visible-spectrum object detection.
  • Data scarcity: Limited, costly, and error-prone annotation of real LWIR data makes conventional large-scale supervised learning difficult.Small airborne targets are time-consuming to annotate, and real-world data acquisition is constrained by operational requirements.
  • Synthetic data: Synthetic IR data offers controllable scene variation and automatic ground-truth annotations, but synthetic-only detectors often face a domain gap on real imagery.Simulation can control geometry, pose, environmental conditions, and scene composition, while differences from real data limit transfer.
  • Training strategy: The proposed synthetic-first strategy uses structured synthetic scene generation for scalable detector pre-training followed by targeted fine-tuning on real LWIR imagery.The simulation pipeline models drone geometry and pose variation, approximates thermal-like target appearance, and uses diverse environments as backgrounds.
  • Dataset analysis: The dataset analysis framework evaluates synthetic–real alignment through pixel-level radiometric characteristics, detection-model feature-space representations, and structural diversity.It links these dataset properties to downstream detection performance to assess how composition influences robustness.
  • Contributions: The work contributes a structured IR synthetic-data pipeline combining IR-like backgrounds, Blender-based scene construction, sensor-noise post-processing, and automatic annotations.It also evaluates synthetic pre-training with limited real LWIR fine-tuning across multiple detector families.

2. RELATED WORK

Related work frames LWIR G2A drone detection as a problem spanning small-object detection, thermal dataset curation, synthetic-data generation, and domain adaptation. It emphasizes that synthetic data offers controllability and annotation benefits, but thermal image formation and the reality gap impose important limitations.

  • LWIR and small-object detection: CNN- and Transformer-based detectors, including Faster R-CNN, YOLO, and RF-DETR, have advanced object detection and enabled real-time applications, while LWIR remains constrained by lower spatial resolution and signal-to-noise.Transformer architectures have also shown promise for automatic target detection.
  • LWIR and small-object detection: LWIR small-object detection is challenged by low signal-to-noise ratio, clutter, weak target structure, varying thermal signatures, and drones appearing as few-pixel, low-texture silhouettes.Dataset studies also find that data and annotation quality and curation matter more than dataset volume.
  • Synthetic data for detection: Synthetic data reduces dependence on scarce real datasets by providing perfect annotations and controllable variation in object placement, backgrounds, rendering, distractors, post-processing, and camera settings.Synthetic-to-real performance depends strongly on training design, augmentation, domain randomization, and architecture choice.
  • Synthetic thermal data generation: Semantic and wavelength-aware visible-to-thermal translation methods preserve scene structure, while physics-aware studies model emissivity, heat distribution, optics, atmospheric attenuation, sensor response, correction, compression, and AGC.Thermal tone mapping from 16-bit to 8-bit can materially affect object detection performance.
  • Reality gap and evaluation: The reality gap arises from limited diversity, poor domain fit, and low fidelity, with LWIR discrepancies occurring at pixel and spatial-frequency levels because renderers differ from microbolometers in heat dissipation, dynamic-range compression, and sensor noise.The cited work presents several metrics for measuring this gap.

3. METHODOLOGY

The methodology uses a sequential pipeline that generates and post-processes synthetic IR scenes before training representative object detectors for G2A drone detection. Training includes synthetic pre-training, real-IR fine-tuning, and evaluation on real-world IR data.

  • Methodology pipeline: The pipeline generates synthetic IR scenes, applies post-processing, and trains state-of-the-art and legacy detectors for G2A drone detection.The methodology is illustrated in Figure 1.
  • Synthetic data generation: Synthetic datasets approximate military operational conditions because sufficiently large, sensor-diverse, consistently annotated public IR datasets remain limited.The process begins with high-resolution IR background generation and scene creation.
  • Synthetic data generation: The background-generation process collected 2,000 web images using automated Pexels and Pixabay searches across environments, weather conditions, and camera angles.Duplicate images were excluded during collection.
  • Post-processing: Post-processing addresses overly smooth synthetic signals by adding sensor-relevant distortions, including Gaussian filtering with a 5 x 5 kernel, additive Gaussian noise, and Poisson noise.These effects could be applied simultaneously or exclusively.
  • Detector models: Evaluation spans one-stage YOLOv13, DETR-style RF-DETR, and two-stage Faster R-CNN detectors, covering three major object-detection paradigms.The models were selected to provide representative evaluation across these paradigms.
  • Training protocol: Training includes 20K synthetic IR samples followed by fine-tuning on either 100 real IR multicopter images or 200 real IR multicopter and bird images, with validation on 3K real IR multicopter samples.The two fine-tuning configurations are termed single-modality and dual-modality fine-tuning.

4. DATA

The study evaluates real and synthetic thermal datasets through pixel-level radiometry, structural diversity, and semantic feature-space alignment. Real imagery differs substantially from synthetic data in image statistics, target complexity, backgrounds, and distractor-induced feature distributions.

  • Dataset diagnostics: Dataset diagnostics span pixel-level radiometry, Kernel DINO Distance for semantic alignment, and Vendi Score for structural diversity.These metrics are diagnostic tools for interpreting downstream detection performance rather than formal causal predictors.
  • Pixel-level differences: 1.242 Cohen’s d marked the largest measured real–synthetic effect, with real imagery showing higher Histogram Total Variation linked to AGC-like dynamic-range stretching.The synthetic pipeline did not yet reproduce this full-image radiometric behavior.
  • Pixel-level differences: Over four times higher Sobel Gradient Variance in real detections reflected uneven heating, sensor noise, and optics absent from comparatively uniform synthetic meshes.The reported effect size was d = 1.224, and real-dataset differences also showed sensor hardware and AGC dependence.
  • Feature-space diversity: Synthetic full-image diversity was inflated by highly textured backgrounds, while target-crop Vendi Scores aligned more closely with real validation targets.This indicates that background complexity, rather than drone appearance, drove the full-image diversity discrepancy.
  • Distractor effects: RGB birds measurably increased object-level KDD, whereas IR birds caused a much smaller shift relative to their fine-tuning set and remained closer to validation alignment.The results identify cross-modal distractors as a source of feature-space displacement.

5. RESULTS

Synthetic pre-training followed by real LWIR fine-tuning improves few-shot drone detection, with RF-DETR-L achieving the best reported mAP@50:95 under multi-modality fine-tuning. However, real LWIR fine-tuning remains necessary for sensor-domain alignment, while RGB bird distractors have conditional benefits.

  • Overall performance: 0.723 mAP@50:95 was achieved by RF-DETR-L with multi-modality fine-tuning, narrowly exceeding single-modality fine-tuning at 0.713.Because repeated-seed experiments were not performed, the small difference should be interpreted cautiously.
  • Pre-training composition: Adding RGB bird data to synthetic pre-training improved mAP@50:95 across all model architectures compared with synthetic-only pre-training.The combined synthetic and RGB bird data nevertheless increased the KDD distance from real-world IR multicopter data.
  • Sensor-domain alignment: Except for Faster R-CNN, no synthetic pre-training strategy exceeded the real-only baseline, indicating that real LWIR fine-tuning is necessary for sensor-domain alignment.Synthetic backgrounds increased latent-feature diversity but did not provide accurate thermal signals.
  • Few-shot detection: Synthetic pre-training followed by real-data fine-tuning improved few-shot (100 samples) LWIR drone detection across the evaluated detector families versus the real-only baseline.All models improved in mAP@50:95 under single-modality fine-tuning.
  • Bird distractors: RF-DETR variants consistently gained slightly from RGB bird distractors, whereas YOLOv13-s gained slightly and YOLOv13-n, YOLOv13-l, YOLOv13-xl, and Faster R-CNN did not benefit.Birds can act as aerial-object hard negatives, but cross-modal mismatch may hinder modality-specific LWIR representation learning during fine-tuning.

6. DISCUSSION

Synthetic IR-like pre-training improves few-shot LWIR drone detection when followed by real-data fine-tuning, but does not replace in-domain data. The discussion attributes this limitation to incomplete physical and sensor realism and identifies restricted validation and experimental design as priorities for future work.

  • Main findings: Synthetic pre-training followed by fine-tuning on a small real LWIR dataset improved mAP@50:95 over training only on limited real data, but did not replace in-domain LWIR data.The strategy was evaluated for ground-to-air drone detection under severe real-data scarcity.
  • Main findings: The synthetic pipeline captures object morphology but incompletely reproduces target heat structure, sensor noise, optics, compression, and camera-side processing in real LWIR imagery.The dataset analyses indicate that morphological diversity alone is insufficient.
  • Main findings: Bird imagery is a conditional robustness mechanism: it can provide aerial morphology and hard negatives, but benefits vary across detector families and scales, while RGB birds introduce cross-modal divergence.Several RF-DETR variants benefited from bird augmentation, but the effect was not universal.
  • Main findings: RF-DETR achieved the strongest absolute results, although architecture-native, non-harmonized training recipes make explanations involving long-range feature interactions and target–background context tentative.The proposed explanation is plausible rather than definitive.
  • Limitations and future work: The study’s limitations include non-physical LWIR simulation, narrow validation coverage, non-harmonized architecture comparisons without repeated seeds, and too few configurations for robust metric–accuracy correlations.Future work calls for physically grounded thermal rendering, broader real-world evaluation, controlled ablations, repeated runs, learning curves, and more configurations.

APPENDIX A: MODEL TRAINING AND HYPERPARAMETER CONFIGURATIONS

Appendix A presents the complete training hyperparameter matrix used in the experiments, supporting reproducibility and future comparative studies. Table 2 details the configurations used to benchmark the models.

  • The complete matrix of training hyperparameters is provided to ensure experiment reproducibility.
  • The hyperparameter configurations are intended to facilitate future comparative studies.
  • Table 2 details the hyperparameter configurations used for benchmarking models.

APPENDIX B: DATASET CONFIGURATIONS … C.1.4 Dynamic Range Utilization

The appendix defines real LWIR dataset configurations and a scarcity-aware 100-sample fine-tuning subset, then evaluates synthetic-to-real alignment across radiometry, semantics, texture, gradients, and dynamic range. The analysis uses normalized and sampling-based comparisons to reduce confounding from dataset size and image quantization.

  • APPENDIX B: DATASET CONFIGURATIONS: Real-world validation sequences came from,,, and, while fine-tuning distributions aggregated,, and [57].All publicly available LWIR datasets were provided as processed 8-bit images rather than raw radiometric data.
  • APPENDIX B: DATASET CONFIGURATIONS: A 100-sample fine-tuning subset was selected from the full real-world set using K-means sampling in DINOv2 feature space.Validation and fine-tuning sets intentionally shared some drone models while using unseen sequences and backgrounds.
  • APPENDIX C: DATASET ANALYSIS METHODOLOGY: Dataset alignment was quantified at pixel-level radiometry, representation-space semantics, and structural diversity, with normalization and sampling used to reduce dataset-volume effects.Luminance histograms were normalized by pixel counts before aggregation.
  • C.1 Level I: Pixel-Level Radiometric and Textural Alignment: The pixel-level domain gap reflects differences in heat dissipation, dynamic-range compression, and sensor noise between rendering engines and real microbolometers.Images were converted to single-channel 8-bit grayscale; Local Weber Contrast and Sobel gradient metrics were computed within target bounding boxes.
  • C.1.1 Radiometric Intensity, Quantization, and Contrast Alignment: Mean Luminance Distribution differences were compared with Earth Mover’s Distance (EMD), using normalized dataset-level histograms and the exact closed-form 1D formulation.For 1D luminance histograms, EMD is computed as the L1 distance between their Cumulative Distribution Functions.
  • C.1.2 Signal-to-Noise and Texture Analysis: Gray-Level Co-occurrence Matrix (GLCM) statistics measured real LWIR micro-texture, including Homogeneity, Correlation, and Entropy averaged across four orientations.The orientation averaging was used to ensure rotation invariance.
  • C.1.3 Intensity Gradient Distribution: Sobel gradient-magnitude variance within target bounding boxes assessed competing spatial-frequency effects from sensor blur, thermal diffusion, heat sources, and microbolometer noise.Real imagery can soften boundaries while introducing localized intensity fluctuations absent from relatively uniform synthetic meshes.

C.2 Transitioning to Semantic Evaluation: Feature Extraction · C.3 Level II: Representation-Space Alignment (Kernel DINO Distance) · C.4 Level III: Structural Diversity

The paper evaluates semantic alignment and structural diversity using standardized DINOv2 object-crop embeddings. It uses Kernel DINO Distance for domain gaps and Vendi Score for effective sample-size diversity.

  • C.2 Transitioning to Semantic Evaluation: Feature Extraction: DINOv2 ViT-L/14-reg provides a universal, model-agnostic feature baseline across detection architectures, with object crops isolating the domain gap from background environments.Full-image embeddings largely reflect background because targets occupy a small fraction of each frame.
  • C.2 Transitioning to Semantic Evaluation: Feature Extraction: 56×56 crop resolution anchors semantic evaluation to target data rather than artificial scaling artifacts caused by severe upscaling noise.Stretching natively small synthetic targets by an order of magnitude would emphasize interpolation noise over true morphological features.
  • C.2 Transitioning to Semantic Evaluation: Feature Extraction: Three-channel grayscale inputs are ImageNet-normalized, embedded through the global semantic [CLS] token, and L2-normalized before metric computation.The [CLS] token aggregates overarching semantic context, while normalization aligns inputs with DINOv2 pre-training and standardizes comparisons.
  • C.3 Level II: Representation-Space Alignment (Kernel DINO Distance): Kernel DINO Distance quantifies semantic domain gaps using an MMD-based estimator that avoids FID’s Gaussian-distribution assumption and sample-size bias.KDD uses DINO features because Inception-V3 features can miss complex semantic similarities and align poorly with human judgments.
  • C.4 Level III: Structural Diversity: Vendi Score measures dataset diversity by estimating effective sample size from eigenvalue entropy of a feature-similarity matrix.The calculation uses standardized DINOv2 embeddings with a linear kernel because Vendi depends strongly on feature representation.
  • C.4 Level III: Structural Diversity: The diversity analysis addresses raw Vendi-score bias from unequal dataset sizes through a mitigation designed for uneven comparisons.Raw Vendi scores can require truncation when dataset sizes differ.

APPENDIX D: PIXEL-LEVEL METRIC RESULTS

Pixel-level analysis reveals substantial radiometric differences between synthetic and real thermal imagery, including quantization and intra-domain distribution shifts. Valid target-level analysis requires excluding undersized bird targets, while effect sizes identify quantization, texture, and dynamic-range metrics as major differences.

  • Dataset-level radiometry: Real LWIR imagery exhibits comb-like luminance quantization from AGC, whereas synthetic images show continuous histograms.AGC maps 14-bit or 16-bit raw radiometric measurements into an 8-bit display range.
  • Dataset-level radiometry: Real-world datasets show distinct intensity peaks and dynamic-range utilization, demonstrating substantial intra-domain radiometric variance.The Real Validation, FTMIR, and FTMIR + FTBIR distributions differ visibly, with pronounced quantization spikes after adding the bird dataset.
  • Target-level metrics: Target-level histogram and GLCM calculations enforce a 400-pixel minimum to prevent finite-sample artifacts.Natively small real IR birds fall below this threshold and are excluded from these analyses to preserve statistical integrity.
  • Effect sizes: The largest reported effects are Full: Quantization HTV 1.242 and Crop: Sobel Gradient Variance 1.224.Other isolated effects include Crop: GLCM Entropy 0.974, Full: Dynamic Range Used 0.899, and Crop: Dynamic Range Used 0.763.

APPENDIX E: SEMANTIC GAP AND DATASET DIVERSITY RESULTS

The appendix evaluates semantic alignment and dataset diversity using Kernel DINO Distance (KDD) and Vendi Score for full frames and object crops. KDD is lower-is-better, while higher Vendi indicates greater diversity, with desirable diversity depending on target-domain alignment.

  • Semantic Gap and Dataset Diversity: KDD and Vendi Score were computed for both full frames and object crops to assess feature-space alignment and dataset diversity.All KDD values are scaled by 1000 for readability.
  • Semantic Gap and Dataset Diversity: KDD measures domain gap, with lower values indicating better alignment.The analysis uses Kernel DINO Distance as the feature-space alignment metric.
  • Semantic Gap and Dataset Diversity: Vendi measures diversity, with higher values indicating greater diversity, although desirable values depend on alignment with the target domain.The table jointly reports domain-gap and diversity results.

APPENDIX F: VENDI SCORE SENSITIVITY ANALYSIS … APPENDIX H: MODEL PARAMETER COUNT, COMPUTATIONAL COMPLEXITY AND INFERENCE SPEED COMPARISON

The appendices show that Vendi-score ordering is preserved under fixed-volume subsampling, fine-tuning improves performance across models, and computational comparisons cover accuracy, complexity, and inference speed. Results include model-specific performance values and standardized NVIDIA T4 benchmarking conditions.

  • APPENDIX F: VENDI SCORE SENSITIVITY ANALYSIS: Across N∈{1000,500,100}, fixed-volume sampling isolates structural diversity while averaging three evaluation seeds for large datasets.The N=100 tier directly incorporates the 100-sample FTMIR dataset.
  • APPENDIX F: VENDI SCORE SENSITIVITY ANALYSIS: The full-frame relative distances and dataset ordering remain precisely preserved from the unsampled baseline through the N=100 ceiling.Reported Vendi scores include Synthetic (SIR) 22.51 and Real Validation 23.72 at their native dataset sizes.
  • APPENDIX G: MODEL PERFORMANCE: Fine-tuning generally improves recall, precision, mAP@50, and mAP@50:95 across models, indicating model-agnostic benefits and supporting synthetic scenes as pre-training data.The appendix notes a few exceptions among fine-tuning cases.
  • [20 K] SIR + RBRGB: RF-DETR under [20 K] SIR + RBRGB reports four performance values of 0.771, 0.955, 0.846, and 0.501.These values are reported for the RF-DETR configuration in the supplied passage.
  • [20 K] SIR + RBRGB: YOLOv13 under [20 K] SIR + RBRGB reports four performance values ranging across supplied rows from 0.610, 0.897, 0.721, and 0.430 to 0.787, 0.942, 0.885, and 0.637.Additional supplied YOLOv13 rows report 0.787, 0.924, 0.876, and 0.622, and 0.760, 0.947, 0.863, and 0.616.
  • APPENDIX H: MODEL PARAMETER COUNT, COMPUTATIONAL COMPLEXITY AND INFERENCE SPEED COMPARISON: Appendix H compares mAP@50:95 with parameter count, GFLOPs, latency, and frame rates across training configurations.Latency was measured on an NVIDIA T4 using TensorRT 11 in FP16 through trtexec, averaged over 100 runs after an initial GPU warm-up.
Loading 2608.17799v1…