Source-linked AI summary
SynThermFace: Amplifying Limited Paired Data for Visible-Thermal Face Recognition via Synthetic Data Generation
Anjith George, Adam Unal, Sebastien Marcel
TL;DR
Cross-spectral face recognition needs paired visible–thermal data, but collecting it at scale is difficult and costly. SynThermFace adapts a diffusion generator on limited real pairs, uses it to create larger paired datasets, and adapts a visible face-recognition model with PACT. PACT improves over evaluated CFR baselines, larger generated datasets provide further gains, and Tufts results provide evidence of transfer to an unseen database.
Problem
Paired visible–thermal datasets are scarce because synchronized multimodal collection is costly and difficult, limiting cross-spectral face-recognition training.
Method
SynThermFace adapts a diffusion model on limited real pairs, generates paired visible–synthetic thermal data from visible faces, and uses PACT to adapt a pretrained visible recognizer.
Results
PACT improves over evaluated CFR adaptation baselines, larger generated paired datasets provide further gains, and Tufts evaluation provides evidence of transfer to an unseen database.
Takeaways & Limitations
Generated visible–synthetic thermal pairs provide useful downstream supervision for cross-spectral recognition within the evaluated PACT pipeline.
Takeaways & Limitations
Generated images may inherit biases and artifacts from the diffusion model, fine-tuning pairs, and recognition model, while identity preservation is difficult to measure directly.
Abstract
from arXiv · showhide
Face recognition (FR) is a widely used modality for biometric authentication, but conventional models rely on visible-spectrum imagery and degrade when high-quality RGB images cannot be captured. Cross-spectral face recognition addresses this limitation by matching visible images with other modalities such as thermal imagery, enabling more reliable performance in low-light, nighttime, and unconstrained conditions. However, progress is limited by the scarcity of paired visible-thermal data, which is difficult and costly to collect at scale. We propose SynThermFace, a framework that amplifies limited real visible-thermal supervision into larger paired adaptation datasets for cross-spectral face recognition. A diffusion model is first adapted using a limited set of paired visible--thermal images and then used to generate large-scale paired visible--synthetic thermal data from existing real or synthetic visible face datasets. The generated pairs are used to adapt a pretrained visible-spectrum face recognition model into a CFR model. Unlike synthesis-based approaches that require image translation at test time, the proposed method shifts generation to the training stage and performs inference with a single forward pass through the adapted recognition model. Under the same MCXFace real-pair protocol, PACT improves over the evaluated CFR adaptation baselines, isolating the effect of the proposed adaptation objective. Training PACT on larger generated paired datasets provides additional improvements over both the unadapted model and the real-pair PACT configuration. Cross-database evaluation on the Tufts dataset provides evidence that the learned representation transfers to an unseen database. The source code and trained models will be made publicly available.
1 Introduction
Cross-spectral face recognition can use thermal imagery when visible capture is unreliable, but progress is constrained by scarce paired visible–thermal data. SynThermFace amplifies limited real pairs into larger synthetic paired datasets and adapts recognition models so synthesis occurs during training rather than inference.
- Motivation: Thermal imaging supports face recognition in low-light or no-light conditions, while cross-spectral matching can reuse visible-spectrum galleries.Thermal sensors capture emitted heat, enabling visible-to-thermal matching without separate thermal enrollment.
- Motivation: Large-scale paired visible–thermal datasets are scarce because collection requires specialized sensors and synchronized multimodal capture.This scarcity leaves models operating under limited-data conditions and can produce poor performance.
- Approach: SynThermFace adapts a diffusion model on limited real pairs, then generates thermal counterparts for large real or synthetic visible-face datasets.The resulting visible–synthetic thermal pairs provide larger paired adaptation sets for cross-spectral recognition.
- Approach: PACT combines symmetric thermal–visible contrastive alignment with visible-domain preservation regularization for cross-modal adaptation.The strategy is designed to adapt recognition models while preserving visible-domain knowledge.
- Evaluation: The work evaluates generated adaptation sets from real and synthetic visible sources and assesses transfer to the unseen Tufts database.Source code and trained models are also planned for public release.
- Deployment: Synthesis is performed only during training, enabling deployed recognition with a single model forward pass and no test-time image translation.The framework therefore shifts generative computation out of inference.
2 Related Work
Related work spans modality-invariant representation learning, common-space projection, synthesis-based translation, and synthetic face generation. SynThermFace addresses the specific practical gap of scarce paired visible–thermal data by shifting generation to training-time paired-data amplification.
- Heterogeneous Face Recognition: Heterogeneous face recognition methods include modality-invariant features, common-space projections, and synthesis-based image translation.These categories address the modality gap between visible, near-infrared, and thermal domains through different representation or translation strategies.
- Synthetic Data for Face Recognition: Synthetic visible-spectrum face datasets have been explored to supplement real data and address legal, privacy, and ethical challenges.GAN, diffusion, and hybrid pipelines target identity preservation, diversity, and realism.
- Cross-Spectral Synthetic Data: ThermVision-DB generates synthetic LWIR thermal faces and videos but does not demonstrate paired RGB–thermal model training or cross-spectral translation.Its stated targets include controlled identity, gender, and other attributes for privacy-preserving thermal analysis.
- Motivation: Cross-spectral synthetic data is motivated primarily by the scarcity, cost, and scaling difficulty of paired visible–thermal collection.Specialized sensors and synchronized capture limit identity and diversity compared with large-scale RGB collection.
- Positioning: SynThermFace shifts generative cost to training by constructing paired visible–thermal data from available visible faces for cross-spectral model adaptation.This differs from image-translation approaches that translate non-RGB inputs during inference.
Proposed Method
The method uses a three-stage pipeline: adapt a diffusion model on limited paired visible–thermal data, generate paired synthetic thermal data, and adapt a pretrained visible face-recognition model with PACT.
- Diffusion data generation: A diffusion-based visible-to-thermal model is trained on a small paired dataset, then applied to large-scale visible face data to create paired visible–synthetic thermal samples.The editing setup preserves input facial geometry and pose while changing the imaging modality.
- Diffusion data generation: Identity preservation adds a proxy thermal-domain loss comparing generated and ground-truth thermal embeddings from a frozen EdgeFace-Base model.The loss combines with the standard flow-matching objective, with λid controlling its contribution.
- PACT adaptation: The recognition stage fine-tunes a lightweight EdgeFace backbone pretrained on a large-scale RGB face dataset using either real or synthetically generated paired data.PACT is designed to reduce the modality gap without discarding identity-discriminative visible-domain structure.
- PACT adaptation: PACT adapts a pretrained EdgeFace model by combining symmetric, identity-aware multi-positive cross-modal contrastive learning with visible-domain preservation against a frozen teacher.The contrastive objective jointly supports thermal-to-visible and visible-to-thermal retrieval, while preservation limits drift from RGB pretraining.
- PACT adaptation: For each anchor, all opposite-modality samples sharing its identity in the mini-batch are treated as positives, including the paired sample.The symmetric loss averages thermal-to-visible and visible-to-thermal directional terms and reduces similarity to different identities.
4 Experiments
The experiments evaluate PACT and synthetic visible–thermal adaptation across within-database, ablation, and cross-database settings. Results show gains from the adaptation objective and larger generated paired datasets, while sensor and database shift remains challenging.
- Within-database evaluation: PACT fine-tuned on real MCXFace pairs achieves 3.04% EER and 96.49% Rank-1 accuracy, improving over evaluated CFR baselines under the same real-data protocol.This setting isolates the contribution of the PACT adaptation loss from synthetic-data effects.
- Synthetic adaptation data: PACT with fully synthetic Digi2Real pairs reaches 1.20% EER and 99.50% Rank-1 accuracy, while CASIA-derived pairs reach 0.99% EER and 99.75% Rank-1 accuracy.The generated pairs improve performance across the reported metrics relative to real-pair PACT.
- Synthetic adaptation data: The synthetic adaptation pipeline reuses limited real MCXFace supervision to construct larger paired datasets from real or synthetic visible sources.CASIA-Synthetic-Thermal uses real CASIA-WebFace visible images, whereas Digi2Real-Synthetic-Thermal uses synthetic visible images; both use generated thermal counterparts.
- Ablations: Balanced non-zero NCE and preservation-loss weights produce stable performance near EER ≈1.0%, with equal 1.0/1.0 weighting adopted.The NCE loss improves cross-modal alignment, and the preservation loss further improves reported metrics by limiting drift from the pretrained identity representation.
- Cross-database evaluation: On Tufts, PACT with fully synthetic data reduces EER from the 43.41% EdgeFace baseline to 13.91% and reaches 56.37% Rank-1 accuracy.All adapted models improve over the baseline, and the results provide evidence of transfer beyond MCXFace identities and acquisition conditions.
- Limitations: The experiments do not independently establish physical thermal realism or exact identity preservation, evaluating generated data through downstream cross-spectral recognition utility instead.The generated images may inherit biases and artifacts from the diffusion model, paired fine-tuning data, and recognition model used for identity supervision.
5 Conclusions
SynThermFace amplifies limited real visible–thermal supervision through offline synthetic paired generation. PACT improves CFR adaptation over evaluated baselines, with further gains from larger generated datasets and evidence of transfer to an unseen database.
- SynThermFace uses offline synthetic paired generation to amplify limited real visible–thermal supervision for cross-spectral face recognition.