PaperlayerView exact revision on arXiv

Source-linked AI summary

Text2Thermal: Physics-Aware Thermal Image Synthesis from Textual Priors

Tayeba Qazi, Brejesh Lall, Prerana Mukherjee

arXiv:2609.03585v2cs.CV

TL;DR

RGB-to-thermal translation is ambiguous because a single visible input can correspond to many valid thermal outputs, forcing deterministic translators to collapse this uncertainty. Text2Thermal uses language as a structured prior, supplying ambiguous thermal factors explicitly rather than inferring them from RGB. Text2Thermal achieves the best FID among thermal image synthesis methods across M3FD and FLIR and generates thermal images from text alone.

  • Problem

    RGB-to-thermal translation is ambiguous because a single visible input can correspond to many valid thermal outputs, forcing deterministic translators to collapse this uncertainty.

  • Method

    Text2Thermal uses language as a structured prior, supplying ambiguous thermal factors explicitly rather than inferring them from RGB.

  • Results

    Text2Thermal achieves the best FID among thermal image synthesis methods across M3FD and FLIR and generates thermal images from text alone.

  • Takeaways & Limitations

    Language conditioning enables thermal images to be generated from text alone by adapting a large-scale text-to-image backbone to the thermal domain.

  • Takeaways & Limitations

    The physics encoded in the captions is soft because the descriptions are not exact radiometric measurements.

Abstract

from arXiv · show

Thermal infrared imaging offers reliable perception in darkness and adverse weather, but thermal datasets remain scarce, motivating extensive work on translating abundant RGB images into thermal. Such translation is fundamentally ill-posed as thermal appearance is governed by surface emissivity and object temperature, neither of which is observable in the visible spectrum, so a single RGB image is consistent with many valid thermal outputs. We argue that language offers a natural means of resolving this ambiguity, and propose **Text2Thermal**, a framework for physics-aware thermal image synthesis from textual priors. Rather than inferring the unobservable radiometric factors from RGB, we supply them explicitly through thermally grounded captions encoding material, weather, time-of-day, and heat-emission state, and adapt a pre-trained Stable Diffusion backbone to the thermal domain. Because the radiometric content is determined entirely by the prompt, Text2Thermal synthesizes thermal imagery without requiring a registered RGB image at inference. Where spatial guidance is desired, an optional control signal imparts scene geometry without disturbing the prompt-specified radiometry. On M3FD and FLIR, Text2Thermal achieves state-of-the-art FID among thermal image synthesis methods, and we additionally report results on the FMB dataset, while offering text-level control that translation-based approaches cannot provide.

1. Introduction

Text2Thermal addresses the ill-posedness of RGB-to-thermal translation by supplying thermally relevant factors through language and adapting a pretrained diffusion model for text-to-thermal synthesis.

  • Motivation: Thermal appearance depends on surface emissivity and absolute temperature, which are not observable in visible images, making RGB-to-TIR mapping fundamentally many-to-many.A deterministic translator must arbitrarily resolve the multiple valid thermal outputs consistent with one RGB image.
  • Approach: Text2Thermal encodes material, weather, time-of-day, and heat-emission state in structured captions instead of inferring these factors from RGB.The framework treats language as a structured prior carrying thermal physics through the text channel.
  • Capabilities: At inference, the model requires only text, unlike translation methods that need a registered visible frame for every generated image.An optional control branch adds scene geometry when paired spatial information is available, supporting layout-aligned synthesis.
  • Approach: Text2Thermal adapts a pretrained Stable Diffusion backbone to thermal imagery using parameter-efficient finetuning and physics-aware captions.The method transfers semantic knowledge from large-scale visible pretraining while prompting thermal content explicitly.
  • Results: Text2Thermal achieves the best reported FID among thermal image synthesis methods on M3FD and FLIR.The framework also evaluates FMB and reports prompt-retention, spatial-control, and caption-field ablation findings.
  • Results: A leave-one-attribute-out ablation finds that every caption field contributes thermal information, with material among the most costly attributes to remove.This result is consistent with material’s role in setting emissivity.

2. Related work

Prior thermal synthesis methods largely rely on visible inputs or manually authored physical simulations, while text-conditioned thermal generation requires captions grounded in infrared emission factors.

  • Thermal synthesis: Simulation-based methods explicitly model infrared emission but require manually authored scene descriptions, limiting scalability to the diversity needed for perception training.Their physical grounding does not remove the burden of authoring varied scenes.
  • Thermal synthesis: Data-driven approaches progressed from adversarial translation with auxiliary priors to diffusion models, but GAN training remains prone to instability and mode collapse.The related methods learn visible-to-infrared mappings from paired data.
  • Thermal synthesis: Existing simulation and data-driven thermal methods generally assume visible imagery is available at inference, although visible light cannot recover emissivity or absolute temperature.Thermal signals combine emitted, reflected, and atmospheric radiation, so visible appearance is insufficient to determine thermal output.
  • Text-conditioned generation: Adapting visible text-to-image models to thermal imagery is difficult because no web-scale thermal-caption corpus exists and RGB-trained text priors lack heat-radiation knowledge.Parameter-efficient adaptation addresses the absence of a comparable thermal corpus without retraining the full model.
  • Spatial control: Spatial-control methods inject auxiliary maps such as edges, depth, or segmentation, but the best spatial modality for thermal generation remains an open question.Text supplies semantics but not layout, motivating explicit spatial conditioning.
  • Thermal-aware captions: Thermal-aware captions must describe material emissivity, ambient conditions, and object heat-emission state rather than only visible appearance.Text2Thermal adopts a schema grounded in observed thermal appearance and analyzes which attribute classes carry the most thermal information.

3. Methodology

Text2Thermal adapts latent diffusion to synthesize thermal images from thermally grounded text, optionally adding spatial guidance without changing prompt-specified radiometry.

  • 3.1.1. Stable Diffusion: The diffusion backbone encodes text through CLIP cross-attention and generates images by iteratively denoising latent representations.Stable Diffusion compresses images into latents, predicts noise during reverse diffusion, and uses classifier-free guidance to strengthen prompt adherence.
  • 3.3.2. Training with Additional Control: Conditional Text2Thermal adds a trainable control branch that injects RGB-derived spatial information while leaving the pretrained diffusion path frozen.Segmentation, edge, and depth maps can be extracted from RGB frames, encoded at latent resolution, and used as optional inference conditions.
  • 3.2. Problem Formulation: RGB-to-thermal translation is ill-posed because identical visible images can correspond to different thermal appearances determined by unobserved physical factors.Material, temperature, and environmental radiation affect thermal measurements but are not observable in the visible spectrum.
  • 3.2. Problem Formulation: Thermally grounded prompts encode environmental context, objects, materials, and heat-emission states so the generator receives physically meaningful radiometric priors.These attributes respectively describe weather and time of day, visible objects, emissivity-related surface composition, and object temperature.
  • 3.3.1. Unconditional Text2Thermal Training: Text2Thermal adapts a pretrained Stable Diffusion latent model to thermal imagery using frozen backbone weights and low-rank updates.The approach retains the pretrained compositional prior while retargeting the output distribution to the thermal domain.
  • 3.3.3. Inference: The method separates geometry from radiometry: RGB contributes object extents, boundaries, and layout, while the physics-aware prompt determines thermal content.This division enables generation without visible-spectrum input and supports rendering with or without spatial guidance.

3.4. Evaluating semantic fidelity

The paper evaluates semantic fidelity by re-captioning generated thermal images and comparing those captions with ground-truth thermal captions using BERTScore.

  • 3.4. Evaluating semantic fidelity: FID and CLIP Score alone cannot verify whether individual prompt attributes appear in the generated image.FID compares feature distributions without scene-request awareness, while CLIP Score compresses prompts and images into single embeddings.
  • 3.4. Evaluating semantic fidelity: Generated images are re-captioned and compared with corresponding ground-truth thermal captions using BERTScore.The caption comparison is designed to evaluate whether requested thermal attributes are recovered in the generated output.
  • 3.4. Evaluating semantic fidelity: The evaluation targets material, heat-emission state, and environmental context rather than visible appearance.Agreement between generated and ground-truth captions therefore indicates that physically meaningful attributes survive image synthesis.
  • 3.4. Evaluating semantic fidelity: The experiments use datasets whose images are annotated with thermal-aware captions, with dataset splits reported in Table 1.The table covers the experimental split organization, while every image receives a thermal-aware caption.

4. Experimental results and analysis

The experiments evaluate Text2Thermal on three RGB–thermal benchmarks using benchmark-specific and zero-shot protocols, measuring distributional fidelity, text alignment, and spatial reconstruction. Text2Thermal achieves the best reported FID on M3FD and FLIR while preserving prompt semantics and benefiting from spatial conditioning.

  • 4.1. Experimental setup: The evaluation uses R2T2 pretraining, benchmark-specific retraining, and zero-shot transfer to assess performance across FLIR, M3FD, and FMB.Unconditional synthesis lacks pixel-aligned ground truth, so it is evaluated with distributional and text-based measures rather than reference-based pixel metrics.
  • 4.2.1. Comparison with existing methods: 72.67 FID on FLIR and 78.18 on M3FD are the best reported results among compared thermal image synthesis methods.Text2Thermal also reports results on FMB, where no published comparison results exist.
  • 4.3. Semantic fidelity of the generated images: 0.9052 unconditional and 0.9092 RGB-conditional average BERTScore F1 indicate close agreement between generated and ground-truth thermal descriptions.The conditional model improves average F1 by 0.0040, showing that spatial guidance preserves rather than dilutes semantic fidelity.
  • 4.4.1. Effect of Proposed Components: 287.14 baseline FID falls to 109.27 with the thermal-adapted UNet and 72.67 after adding the RGB control branch.CLIP Score remains 0.19 in both settings, indicating improved distributional fidelity without changing prompt adherence.
  • 4.4.2. Spatial Conditioning Modality: 72.67 FID for RGB conditioning outperforms edge, depth, and segmentation maps at 99.30–105.74.Composite conditions degrade FID, with the best RGB-plus-Depth result reaching 93.90 while improving SSIM only marginally.

5. Discussion

The discussion examines parameter efficiency, zero-shot transfer, caption ablations, and failure cases. Text2Thermal remains competitive with translation methods despite not requiring a paired visible frame at inference, but generated images can omit prompted objects or hallucinate spatial structures.

  • 5.1. Parameter efficiency: 3.19 M updated parameters yield 109.27 FID, while the control configuration reaches 72.67 FID with 361.28 M trainable parameters.The unconditional model updates 1.4% of the parameters optimized by PID and TherA, while the conditional model achieves the table’s best FID at comparable training cost.
  • 5.2. Zero-shot transfer: Zero-shot Text2Thermal achieves the best FID on M3FD and second-best on FLIR, with CLIP Scores of approximately 0.18.These results are obtained without dataset-specific fine-tuning, whereas translation baselines receive a paired visible frame at inference.
  • 5.4. Caption ablation: The thermal caption schema encodes weather, colour, material, and heat-emission state as distinct attribute classes for leave-one-attribute-out analysis.Removing an attribute at training and inference measures the degradation associated with its thermal information content.
  • 5.3. Failure Cases: Generated thermal images may omit prompted objects, such as pedestrians, or hallucinate structures transferred from the spatial condition.The failure examples include a hallucinated tower carried over from the spatial input.

6. Conclusion

Text2Thermal synthesizes thermal images from physics-aware text prompts by adapting a large text-to-image backbone, achieving strong benchmark results while exposing prompt-level control. The conclusion also identifies soft radiometric supervision, caption-data requirements, and object-level generation errors as limitations.

  • 6. Conclusion: Text2Thermal generates thermal images from text alone by adapting a large-scale text-to-image backbone to the thermal domain.The framework can render imagery with or without spatial guidance at no additional training cost.
  • 6. Conclusion: Best FID among reported thermal image synthesis methods is achieved on M3FD and FLIR, while only a small fraction of backbone parameters is updated.Results are also reported on FMB.
  • 6. Conclusion: BERTScore comparisons with ground-truth captions confirm that synthesized imagery retains thermal content specified in the prompt.The evaluation uses re-captioned generated images compared against ground-truth captions.
  • 6. Conclusion: Removing any attribute class degrades generation quality, with material causing the largest decline, consistent with its role in setting emissivity.This supports the conclusion that prompts function as thermal physics-aware specifications rather than merely stylistic cues.
  • 6. Conclusion: Limitations include soft caption-based physics, dependence on paired RGB-TIR data for caption construction, and occasional object omission or structural hallucination.The authors suggest calibrated radiometric supervision and finer object-level control as directions for future work.
  • 6. Conclusion: The framework lowers barriers to infrared work by enabling thermal training data without a thermal sensor and providing direct prompt control over generated conditions.The authors position this as a broader consequence of text-conditioned thermal synthesis.

CRediT authorship contribution statement

The CRediT statement assigns contributions across conceptualization, methodology, software, validation, analysis, investigation, data curation, writing, visualization, supervision, and project administration.

  • CRediT authorship contribution statement: Tayeba Qazi contributed to conceptualization, methodology, software, validation, analysis, investigation, data curation, original drafting, and visualization.
  • CRediT authorship contribution statement: Brejesh Lall contributed to review and editing, supervision, and project administration.
  • CRediT authorship contribution statement: Prerana Mukherjee contributed to review and editing and supervision.

Declaration of competing interest

The authors declare no known competing financial interests or personal relationships that could have influenced the reported work.

  • Declaration of competing interest: The authors declare that they have no known competing financial interests or personal relationships relevant to the reported work.

Declaration of generative AI and AI-assisted technologies in the writing process

The authors used Claude to assist with language editing and formatting, then reviewed and edited the manuscript and retained responsibility for its content.

  • Declaration of generative AI and AI-assisted technologies in the writing process: Claude was used during manuscript preparation for language editing and formatting assistance.
  • Declaration of generative AI and AI-assisted technologies in the writing process: The authors reviewed and edited the assisted content as needed and took full responsibility for the published article.

Data availability

All datasets used in the work are publicly available.

  • All datasets used in this work are publicly available.

Appendix A. Additional Implementation Details

Appendix A summarizes implementation and training details for unconditional and conditional Text2Thermal models in Table A.1.

  • Entries marked — indicate details that do not apply to the corresponding setting.
  • Table A.1 summarizes implementation and training details for the unconditional and conditional Text2Thermal models.
  • Settings shared by both models span the two table columns.
Loading 2609.03585v2…