PaperlayerView exact revision on arXiv

Source-linked AI summary

ThermalGen: Style-Disentangled Flow-Based Generative Models for RGB-to-Thermal Image Translation

Jiuhong Xiao, Roshan Nayak, Ning Zhang, Daniel Tortei, Giuseppe Loianno

arXiv:2509.24878v1cs.CVcs.RO

TL;DR

Synchronized, calibrated RGB-thermal pairs are scarce despite their importance to sensor fusion and cross-modality tasks. ThermalGen uses flow-based translation with RGB conditioning and disentangled style embeddings, supported by curated and newly introduced datasets. It reports comparable or superior translation performance to GAN- and diffusion-based methods, while performance on FLIR remains below state-of-the-art benchmarks.

  • Problem

    Synchronized, calibrated RGB-thermal pairs are scarce and costly to collect, although they are needed for training cross-modality and sensor-fusion systems.

    Source: §1 · p. 1

  • Method

    ThermalGen is an adaptive flow-based RGB-to-thermal model that conditions on an RGB image and dataset-specific style embeddings.

    Source: §1 · p. 2, §3.2 · pp. 4–5

  • Results

    ThermalGen achieves comparable or superior translation performance to GAN- and diffusion-based methods across multiple benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    ThermalGen demonstrates cross-dataset generalization across thermal sensors, viewpoints, and environmental conditions, with disentangled style embeddings highlighted as significant to translation robustness.

    Source: §6 · p. 9

    On FLIR, blurred distant objects and extreme lighting remain challenges, and the reported improvement does not yet surpass state-of-the-art benchmarks.

    Source: §5.4 · p. 9

Abstract

from arXiv · show

Paired RGB-thermal data is crucial for visual-thermal sensor fusion and cross-modality tasks, including important applications such as multi-modal image alignment and retrieval. However, the scarcity of synchronized and calibrated RGB-thermal image pairs presents a major obstacle to progress in these areas. To overcome this challenge, RGB-to-Thermal (RGB-T) image translation has emerged as a promising solution, enabling the synthesis of thermal images from abundant RGB datasets for training purposes. In this study, we propose ThermalGen, an adaptive flow-based generative model for RGB-T image translation, incorporating an RGB image conditioning architecture and a style-disentangled mechanism. To support large-scale training, we curated eight public satellite-aerial, aerial, and ground RGB-T paired datasets, and introduced three new large-scale satellite-aerial RGB-T datasets--DJI-day, Bosonplus-day, and Bosonplus-night--captured across diverse times, sensor types, and geographic regions. Extensive evaluations across multiple RGB-T benchmarks demonstrate that ThermalGen achieves comparable or superior translation performance compared to existing GAN-based and diffusion-based methods. To our knowledge, ThermalGen is the first RGB-T image translation model capable of synthesizing thermal images that reflect significant variations in viewpoints, sensor characteristics, and environmental conditions. Project page: http://xjh19971.github.io/ThermalGen

1 Introduction

Synchronized, calibrated RGB-thermal pairs are scarce and costly, motivating synthesis from abundant RGB images. ThermalGen combines flow-based translation and style conditioning with new paired datasets, and is reported to perform comparably or better than GAN- and diffusion-based baselines across benchmarks.

1 Introduction

  • Paired RGB-thermal data is limited, difficult to generalize across applications, and costly to collect in synchronized and calibrated form.Synthesized pairs can align RGB and thermal details, use abundant public RGB imagery, and simulate varying thermal characteristics and environments.Source: §1 · pp. 1–2
  • ThermalGen is an adaptive flow-based RGB-to-thermal model with RGB conditioning and a style-disentangled mechanism for generating images across varied sensors, viewpoints, and environments.Its architecture supports joint training with additional datasets.Source: §1 · p. 2
  • The authors introduce three satellite-aerial paired datasets and curate public satellite-aerial, aerial, and ground RGB-thermal datasets for large-scale training and evaluation.The new datasets are DJI-day, Bosonplus-day, and Bosonplus-night, with variation in time of day, sensor type, and geographic region.Source: §1 · p. 2
  • ThermalGen achieves comparable or superior performance to GAN- and diffusion-based baselines across multiple benchmarks.The paper also describes it as the first model capable of generating high-fidelity thermal imagery across diverse sensor types, viewpoints, and environmental conditions.Source: §1 · p. 2

2 Related Works

RGB-to-thermal translation must infer thermal cues absent from RGB while handling limited paired data and sensor or viewpoint shifts. Prior work spans GANs, diffusion, and synthetic-data applications, while ThermalGen targets broader dataset coverage and style-conditioned generation.

2 Related Works

  • Synthetic thermal data has been used for cross-modal alignment, object detection, satellite-thermal retrieval, and homography estimation.Examples include XoFTR, AVIID, STGL [57], and STHN [58] [56].Source: §2 · p. 3
  • RGB-to-thermal translation is challenged by scarce calibrated pairs, missing thermal cues in RGB, and sensor and viewpoint differences between training and testing.Unsupervised approaches such as CycleGAN also struggle to bridge the data gap.Source: §2 · p. 3
  • Earlier RGB-to-thermal methods mainly used GANs, while recent work explored diffusion [62] [38]; narrow training datasets limit generalization for most methods.ThermalGen evaluates satellite-aerial, aerial, and ground data and uses dataset-specific style embeddings to generate target styles without retraining.Source: §2 · p. 3
  • General image-translation methods include diffusion, flow, latent-space, and style-transfer approaches, but RGB and thermal modality differences complicate direct weight sharing and cross-dataset translation.These challenges limit the applicability of RGB style-transfer methods to robust RGB-to-thermal translation.Source: §2 · p. 3

3 Methodology

ThermalGen learns a flow-based process for synthesizing thermal images from RGB images, conditioned on dataset-specific style. Its latent-generation design combines RGB conditioning with learnable style embeddings to support varied RGB-T domains.

3.1 Preliminaries

  • A neural network learns the latent velocity field, which is integrated in reverse from terminal noise to recover a denoised thermal latent.The process uses a time-dependent interpolation between data and Gaussian noise.Source: §3.1 · pp. 3–4

3.2 RGB-Image-Conditioning and Style-Disentangled Generative Model

  • ThermalGen models the thermal-image distribution conditioned jointly on an RGB image and a dataset-specific style embedding.The paper defines RGB-T style as the mapping between RGB and thermal images influenced by sensor, viewpoint, and environmental conditions.Source: §3.2 · p. 4
  • Thermal flow operations use compressed encoder latents, which a thermal decoder converts back into images.Table 1 reports 200k samples used for large-scale training.Source: §3.2 · p. 4, Table 1 · p. 5
  • Learnable style embeddings use adaLN-Zero conditioning, including an unconditional embedding, and can be extended by adding embeddings for new datasets or styles.Random selection between conditional and unconditional embeddings supports classifier-free guidance and dataset-unconditional generation.Source: §3.2 · p. 5
  • RGB information enters the SiT through either cross-attention or concatenation; concatenation also supports fine-tuning from pretrained SiT weights.Cross-attention uses the RGB latent as query and the noised thermal latent as key and value.Source: §3.2 · p. 5

4 Experiment Setup

The experiments curate and preprocess paired RGB-T data spanning satellite-aerial, aerial, and ground settings. Evaluation compares ThermalGen with GAN- and diffusion-based baselines using image-quality metrics and fixed image sizes and denoising steps.

Datasets

  • Data preparation standardizes formats, normalizes thermal images to 8-bit, aligns pairs, and removes unusable data and invalid thermal regions.The study reports using over ten publicly available paired RGB-T datasets.Source: §4 · pp. 5–6
  • The curated benchmark covers paired satellite-aerial, aerial, and ground datasets, including three newly collected satellite-aerial datasets with varied sensors, acquisition times, and geographic coverage.The satellite-aerial collections pair satellite RGB with aerial thermal imagery; the figure illustrates sensor, temporal, and geographic variation.Source: §4 · p. 6, Figure 3 · p. 6

Metrics

  • Evaluation uses PSNR, SSIM, FID, and LPIPS to assess fidelity, structural similarity, distribution similarity, and perceptual similarity, respectively.FID uses 2048-dimensional Inception v3 features; LPIPS uses AlexNet features.Source: §4 · p. 6

Baselines

  • Baselines include pix2pix, CycleGAN, pix2pixHD, VQGAN, DDIM, and DiffV2IR, with models trained on the curated collection except pretrained DiffV2IR models.The comparison tables cover satellite-aerial and aerial translation; the aerial table notes that DiffV2IR models use the LLVIP test set for training.Source: §4 · p. 7, Table 2 · p. 7, Table 3 · p. 7

Implementation Details

  • Training randomly samples batches across datasets and uses random 256×256 crops; evaluation resizes images to 256×256 and uses T = 50 denoising steps.The style embedding dimensionality is 1024.Source: §4 · p. 7

5 Results

ThermalGen outperforms GAN- and diffusion-based baselines across satellite-aerial, aerial, and ground datasets, while ablations identify effective design choices. Dataset-specific challenges remain, including low-contrast imagery, difficult lighting, and train-test distribution gaps.

5.1 Cross-dataset RGB-T Image Translation Performance

  • ThermalGen outperforms GAN- and diffusion-based baselines across satellite-aerial, aerial, and ground datasets, with strong FID and LPIPS results on six named benchmarks.Its advantage is especially notable over fine-tuned DiffV2IR models, which often perform well on individual datasets but struggle to generalize across imaging conditions.Source: §5.1 · p. 7

5.2 Ablation Studies

  • SiT-L and SiT-XL outperform SiT-B in FID across M3FD, MSRS, and AVIID, while SiT-XL/2 and smaller patches yield the best reported results.The passage attributes the gains to larger transformer sizes and finer patch granularity.Source: §5.2 · p. 8
  • Concatenating RGB latents yields better overall FID performance than using them as query inputs in cross-attention across five evaluation datasets.Source: §5.2 · p. 8
  • For distinctive styles, CFG at scale 2.0 outperforms dataset-specific embeddings alone, while unconditional embeddings degrade FID; gains are marginal on M3FD and MSRS.The analysis states that style embeddings encode dataset-specific RGB-T relationships.Source: §5.2 · p. 8

5.3 Qualitative Results

  • ThermalGen produces high-fidelity thermal images matching ground-truth distribution characteristics across ground, aerial, and satellite-aerial domains; DDIM often generates samples unconditioned on RGB.The qualitative comparison reports distortions from Pix2Pix and Pix2PixHD, grayscale-like outputs from CycleGAN, grid artifacts from VQGAN, and overly sharp boundaries from DiffV2IR.Source: §5.3 · pp. 8–9

5.4 Limitations and Discussions

  • FID falls from 161.22 to 116.46 on Boson-night at CFG 8.0 and from 70.09 to 63.43 on FLIR at CFG 4.0.Boson-night's extremely low-contrast imagery hampers learning and produces dark, blurred outputs; FLIR issues include blurred distant objects and over- or underexposure.Source: §5.4 · p. 9, Table 5 · p. 9
  • LLVIP's limited scene diversity and mostly static backgrounds create a train-test distribution shift, with t-SNE analysis indicating a gap primarily attributed to camera differences.The passage identifies expanding the training dataset as the most effective solution for bridging the gap and improving generalization.Source: §5.4 · p. 9
  • Despite dataset-specific challenges, CFG adjustment consistently improves performance and supports the style-disentangled approach, while cross-domain generalization remains an avenue for further enhancement.Source: §5.4 · p. 9

6 Conclusions

ThermalGen combines flow-based generation, RGB conditioning, and style disentanglement, and the paper reports cross-dataset generalization across sensors, viewpoints, and environmental conditions. The conclusion also identifies synthetic-data integration as future work and notes a risk of synthetic outputs being mistaken for genuine thermal data.

6 Conclusions

  • ThermalGen is an adaptive flow-based RGB-T translation model with RGB conditioning and style disentanglement, evaluated across diverse datasets including three newly introduced satellite-aerial datasets.Source: §6 · p. 9
  • Figure 6 compares methods across ground, aerial, and satellite-aerial datasets, while Figure 7 illustrates DDIM outputs that can resemble training-distribution samples rather than reflect the RGB input.Source: Figure 6 · p. 10, Figure 7 · p. 10
  • Future work will systematically incorporate synthesized thermal data from multiple sources to advance cross-modal model training.Source: §6 · p. 10
  • Synthetic thermal outputs may be misinterpreted as genuine thermal data in safety-critical contexts, despite their use in validating thermal-data realism.Source: §6 · p. 10

NeurIPS Paper Checklist

The checklist identifies the paper’s contributions as an RGB-T translation model, new datasets, and a focused scope. It reports reproducibility details and several disclosure practices, while noting that experiments lack error bars.

NeurIPS Paper Checklist

A Satellite-Aerial Dataset Collection Process

The collection process aligns satellite RGB images with UAV-captured thermal imagery through orthomosaic construction, spatial matching, filtering, and grid-based patch extraction.

A Satellite-Aerial Dataset Collection Process

B Dataset Preprocessing Details

The appendix describes dataset-specific preprocessing and training configurations for the RGB-T experiments.

B Dataset Preprocessing Details

C Additional Training Details

  • Training uses one NVIDIA A100 or H100 GPU and runs both the thermal encoder-decoder and flow models for 200k steps with specified batch sizes and AdamW settings.The encoder-decoder uses batch size 16, learning rate 6 × 10^-5, and weight decay 1 × 10^-3; the flow model uses batch size 64, learning rate 1 × 10^-4, and no weight decay.Source: C Additional Training Details · p. 21

D Dataset Parameters Comparison

The dataset comparison highlights broader geographic coverage, greater thermal-sensor diversity, and both daytime and nighttime imagery in the authors’ datasets relative to Boson-night.

D Dataset Parameters Comparison

  • Compared with Boson-night, the authors’ datasets extend geographic coverage and thermal-sensor diversity and include both daytime and nighttime imagery.Source: D Dataset Parameters Comparison · p. 21, Table 6 · p. 21
  • Table 6 compares area coverage, satellite map sources, thermal sensors, collection years, and numbers of surveyed regions.Source: Table 6 · p. 21

E Thermal Auto-encoder Training Details and Reconstruction Performance

The thermal auto-encoder is trained with reconstruction and latent-space regularization, with optional GAN fine-tuning that trades slightly lower PSNR for improved FID; reported results use the decoder without GAN loss. Fixed style embeddings outperform CLIP-based embeddings on two Bosonplus datasets and are comparable on FLIR, while avoiding CLIP feature computation.

E Thermal Auto-encoder Training Details and Reconstruction Performance

  • Only two regions of Bosonplus-day and Bosonplus-night will be released for compliance.Source: E Thermal Auto-encoder Training… · p. 21
  • ThermalGen reports results using the decoder without GAN loss; optional fine-tuning of final decoder layers with GAN loss enhances FID scores but slightly negatively affects PSNR metrics.Training uses L1 and LPIPS reconstruction losses plus KL regularization; after 300 epochs (200k steps), the final decoder layers are optionally fine-tuned for 75 epochs using GAN loss plus those losses.Source: E Thermal Auto-encoder Training… · p. 22

F Comparison of Style and CLIP Embeddings

G Evaluation Dataset Comparison

The evaluation datasets are compared by diurnal setting, thermal sensor, viewpoint, and environment. t-SNE visualizations show feature discrepancies across sensor, viewpoint, and diurnal pairs, while the authors report that ThermalGen adapts to these shifts.

G Evaluation Dataset Comparison

Related papers

Loading 2509.24878v1…