Source-linked AI summary
ReFlowSET: Representation-Aligned Latent Flow Matching for SAR-to-EO Image Translation
Jeonghyeok Do, Seungchul Lee, Munchurl Kim
TL;DR
SAR-to-EO translation must bridge an inherently ambiguous mapping, while existing latent methods often fix the autoencoder without auditing modality-specific reconstruction fidelity. ReFlowSET selects its codec through a joint SAR–EO reconstruction audit, trains a smaller conditional DiT with dual-stream conditioning and training-only EO representation alignment, and achieves leading perceptual and distributional performance on QXS-SAROPT and SAR2Opt.
Problem
SAR backscatter and EO reflectance describe different physical properties, while existing latent approaches typically use a predetermined codec without examining modality-specific reconstruction fidelity.
Method
ReFlowSET selects a codec through joint SAR–EO reconstruction auditing, then trains a conditional DiT from scratch with dual-stream SAR conditioning and training-only alignment to frozen clean-EO representations.
Results
ReFlowSET achieves the best FID and DISTS on both benchmarks and the best LPIPS on SAR2Opt among evaluated methods.
Takeaways & Limitations
Codec selection and representation-aligned flow matching provide a latent SET framework with leading perceptual and distributional performance without inference-time alignment overhead.
Takeaways & Limitations
ReFlowSET does not maximize PSNR or SSIM, which are sensitive to ambiguity and local misalignment in paired SAR–EO observations.
Abstract
from arXiv · showhide
SAR-to-EO image translation aims to generate electro-optical (EO) imagery from synthetic aperture radar (SAR) observations. Existing latent diffusion approaches typically inherit a predetermined autoencoder, although reconstruction fidelity can vary substantially across codecs and modalities. Because the latent codec affects the round-trip preservation of both SAR conditions and EO targets, codec selection constitutes a fundamental design choice; nevertheless, existing methods largely rely on codecs pretrained on natural images. To remedy this, we introduce ReFlowSET, a conditional latent flow-matching framework that selects its codec through a joint SAR--EO reconstruction audit. Rather than inheriting a heavyweight pretrained generator, ReFlowSET trains a substantially smaller conditional DiT from scratch in the selected latent space, using dual-stream SAR conditioning followed by joint feature refinement. To provide semantic guidance for this from-scratch training, intermediate noisy-EO features are aligned with clean target-EO representations extracted by a frozen vision foundation model. This alignment is used only during training and introduces no additional inference cost. Experiments on QXS-SAROPT and SAR2Opt demonstrate state-of-the-art performance across diverse perceptual fidelity and distributional metrics. Code and pretrained weights are publicly available at https://github.com/KAIST-VICLab/ReFlowSET.
1. Introduction
SAR-to-EO translation seeks interpretable EO imagery from SAR observations, but their different physical properties make the mapping inherently ambiguous. ReFlowSET addresses this with reconstruction-guided codec selection, a smaller from-scratch conditional DiT, and training-only EO representation alignment.
- SAR offers reliable observation across illumination and most weather conditions, but speckle and geometric distortions make it less intuitive than EO imagery.
- The SAR-to-EO mapping is inherently ambiguous because SAR backscatter and EO reflectance describe fundamentally different physical properties.
- Existing latent methods reduce computation but typically adopt the pretrained Stable Diffusion VAE as a fixed codec without examining its reconstruction fidelity.
- ReFlowSET audits modality-specific reconstruction ceilings, selects a codec with strong joint SAR–EO fidelity, and trains a substantially smaller conditional DiT from scratch.
- Intermediate noisy-EO features are aligned with clean-EO representations from a frozen vision foundation model, adding neither annotations nor inference-time overhead.
- Evaluations on QXS-SAROPT and SAR2Opt report leading performance in perceptual similarity and distributional realism.
2. Related Work
Prior SAR-to-EO work spans GANs, pixel-space and latent diffusion, and bridge-based translation, while ReFlowSET treats the autoencoder endpoint as an explicit design variable and uses flow matching with training-only guidance.
- Earlier image-translation methods used conditional or cycle-consistent GANs, with later models improving high-resolution, spatially controlled, or non-bijective synthesis.
- SAR-specific GANs incorporated atrous context or EO segmentation labels to reduce blur and semantic errors.
- Diffusion approaches addressed pixel-space denoising and one-step distillation, while cBBDM and C-DiffSET operated in the SD2.1 latent space.
- ReFlowSET makes the autoencoder endpoint an explicit design variable and combines flow matching with label-free, training-only representation guidance.
3. Analysis of VAE Reconstruction Performance
The paper evaluates frozen autoencoders by their modality-wise round-trip reconstruction ceilings, treating codec choice as consequential for attainable SAR conditioning and EO targets. FLUX.2 provides the strongest reported reconstruction profile and is retained without its heavyweight generator.
- A round-trip reconstruction upper-bound is defined as the fidelity of decoding an image after encoding it with a frozen autoencoder.
- Higher EO and SAR reconstruction ceilings indicate less target distortion and greater retention of source structure for conditional generation, respectively.
- Figure 1 compares six pretrained autoencoders across SAR and EO imagery from four datasets using encode–decode PSNR and representative reconstructions.
- FLUX.2 achieves the highest PSNR in six of eight dataset–modality settings and raises the mean EO ceiling by 7.49 dB over SD2.1.
- The selected FLUX.2 autoencoder is retained for SAR and EO, while its associated multi-billion-parameter pretrained generator is excluded to avoid substantially higher computational cost.
4. ReFlowSET
ReFlowSET performs conditional latent flow matching with a dual-stream DiT that separately processes SAR and noisy-EO latents before joint refinement. Training aligns intermediate noisy-EO features with clean EO representations from a frozen vision foundation model, while the alignment components are removed at inference.
- 4.1. Conditional Latent Flow Matching: ReFlowSET encodes paired SAR and EO observations with a selected frozen encoder and trains a conditional DiT to predict latent flow velocities.The SAR latent remains a persistent condition during flow-matching training.
- 4.1. Conditional Latent Flow Matching: Independent noise is transformed toward the EO latent along a linear probability path, and the learned ODE is integrated from t = 0 to 1 for decoding.Starting from independent noise prevents the target velocity from becoming an analytic function of the model inputs.
- 4.2. Dual-Stream SAR Conditioning: The model uses independently parameterized SAR and noisy-EO streams for the first r blocks, then concatenates and jointly processes their features to predict the velocity field.This differs from a single-stream baseline that concatenates the latents only at the input.
- 4.3. Training-Only EO Representation Alignment: Because the DiT is trained from scratch without pretrained-generator semantic priors, intermediate noisy-EO features are aligned with clean EO representations from a frozen vision foundation model.A trainable projector maps DiT features into the frozen representation space, and cosine distance supplies the alignment objective.
- 4.4. Experiments: Table 1 compares methods on QXS-SAROPT and SAR2Opt using identical training splits and test pairs after retraining baselines from official code.The table reports quantitative comparisons under a common evaluation protocol.
- 4.3. Training-Only EO Representation Alignment: The total objective combines flow matching with representation alignment, while the vision foundation model and projector are used only during training and discarded at inference.The fixed alignment weight is λ = 0.5, with no test-time overhead.
5. Experiments
Experiments evaluate ReFlowSET on two paired SAR-to-EO benchmarks against diverse baselines, ablations, and qualitative comparisons. Results highlight strong perceptual and distributional performance, while analyzing codec choice, fusion, conditioning, alignment, and pixel-fidelity limitations.
- Experimental Setup: QXS-SAROPT contains 20,000 paired 256-pixel tiles, while SAR2Opt contains 2,077 paired 600-pixel images with specified training and testing splits.All baselines are retrained on the same training splits and evaluated on identical test items.
- Evaluation: Table 1 evaluates distributional realism with FID, perceptual fidelity with DISTS and LPIPS, and pixel fidelity with SSIM and PSNR.
- Comparison with the State of the Art: 7.7% and 12.3% relative gains are reported for FID and DISTS on SAR2Opt, respectively, alongside the lowest LPIPS.FID improves from 71.8 to 66.3 and DISTS from 0.211 to 0.185.
- Comparison with the State of the Art: Replacing the SD2.1 codec with FLUX.2 improves all five metrics on both benchmarks, reducing FID from 25.5 to 19.1 on QXS-SAROPT and from 84.5 to 66.3 on SAR2Opt.The comparison supports reconstruction-guided codec selection within the ReFlowSET framework.
- Comparison with the State of the Art: ReFlowSET does not maximize PSNR or SSIM because pixel-aligned metrics are sensitive to ambiguity and local misalignment in paired SAR–EO observations.The study therefore emphasizes perceptual and distributional metrics while reporting pixel fidelity for completeness; Figure 3 shows sharper structures and more coherent land-cover appearance than competing models.
- Ablation and Efficiency Analysis: The ablation compares SAR conditioning, stream fusion, and EO representation alignment using compact 12-block models, while the final model uses an 8-and-16 block allocation.The table reports deployed parameters, peak training memory, and per-image inference latency; training-only alignment modules are excluded from deployed parameters.
- Ablation and Efficiency Analysis: Channel-wise fusion avoids the token-wise fusion variant’s increase in training VRAM from 10.63 to 13.57 GB and latency from 1.34 to 2.06 s/image, with virtually no FID benefit from token-wise fusion.Under the dedicated SAR stream, adding EO representation alignment reduces FID from 71.851 to 70.436 without changing deployed parameters or inference latency.
6. Conclusion
ReFlowSET combines reconstruction-guided codec selection, latent flow matching, delayed SAR–EO fusion, and training-only EO representation alignment for SAR-to-EO translation. It achieves strong benchmark performance without inference-time overhead from the alignment branch, while identifying cross-sensor generalization and unsupported EO-like structures as future directions.
- ReFlowSET combines reconstruction-guided codec selection, conditional latent flow matching, delayed SAR–EO fusion, and training-only EO representation alignment.
- ReFlowSET achieves the best FID and DISTS on both benchmarks and the best LPIPS on SAR2Opt among the evaluated methods.
- The alignment branch introduces no inference-time overhead.
- Future work will examine cross-sensor generalization and suppression of unsupported EO-like structures.