PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning the Target Priors Before Image Translation: A Decoupled Training Paradigm for Cross-Modal Image Translation in Remote Sensing

Keyan Hu, Mingtao Wang, Ziyu Zhou, Tiandong Shi, Haifeng Li, Ji Qi, Chao Tao

arXiv:2608.28517v1cs.CV

TL;DR

Remote-sensing cross-modal translation must preserve source content while matching a target distribution, yet scarce paired training entangles target-prior learning with cross-modal dependence. LTP-BIT learns the target prior first from unpaired imagery, then uses P-DART for parameter-efficient source-conditioned adaptation, achieving state-of-the-art results with limited task-specific and paired data.

  • Problem

    Existing methods jointly learn target priors and cross-modal dependence from scarce paired data, although only dependence learning intrinsically requires cross-modal correspondence.

  • Method

    LTP-BIT pretrains a target-domain generative prior on large-scale unpaired imagery, freezes its backbone, and learns source-conditioned control through P-DART.

  • Results

    LTP-BIT achieves state-of-the-art performance across SAR-to-RGB and NIR-to-RGB benchmarks using 9.81% task-specific parameters, with near-full-data fidelity on QXS-SAROPT at 25% paired data.

  • Takeaways & Limitations

    Prior matching and scaling primarily improve target-domain realism, whereas instance fidelity depends more strongly on conditional adaptation.

  • Takeaways & Limitations

    Effective decoupling still requires adequate target-domain support and source-conditioned controllability without eroding the pretrained prior.

Abstract

from arXiv · show

Cross-modal image translation in remote sensing must preserve source-observed content while matching the target-domain distribution. Existing methods jointly learn the target prior and cross-modal dependence from scarce paired data, overlooking a key asymmetry: only the latter intrinsically requires cross-modal correspondence. We formalize this distinction through conditional-score and denoising-risk analyses and propose Learning the Target Priors Before Image Translation (LTP-BIT), a prior-first paradigm that decouples the two learning tasks. LTP-BIT first learns a target-domain generative prior from large-scale unpaired imagery, then retains the pretrained backbone weights and learns source-conditioned control through P-DART, a parameter-efficient dual-stream architecture. Controlled experiments show that prior matching and scaling primarily improve target-domain realism, whereas instance fidelity relies more strongly on conditional adaptation. LTP-BIT achieves state-of-the-art performance across SAR-to-RGB and NIR-to-RGB benchmarks using only 9.81% task-specific parameters. On QXS-SAROPT, it retains near-full-data instance fidelity with only 25% of the paired samples.

1 Introduction

LTP-BIT frames remote-sensing translation as two coupled needs: target-domain prior learning and source-conditioned dependence learning. It separates them by pretraining a target prior on unpaired imagery, then adapting it with limited paired data through P-DART.

  • Motivation: Paired translation must model both plausible target images and source correspondence, but scarce pairs make the task underdetermined.Different sensors capture distinct physical properties, so one source observation can correspond to multiple plausible target images.
  • Decoupling challenge: Separately trained components are not directly composable because unconditional marginals do not identify cross-modal dependence and frozen priors shift correction into conditioning.The paired-stage correction includes both cross-modal conditioning and prior-mismatch compensation.
  • Proposed framework: LTP-BIT first learns a target-domain generative prior from large-scale unpaired imagery, then freezes its backbone and learns source-conditioned adaptation from paired data.The framework also studies pretraining scale, model capacity, and target-domain composition.
  • Proposed framework: P-DART provides source control through asymmetric interaction between generation and reference streams while optimizing only the reference branch and LoRA adapters.
  • Results: 9.81% task-specific parameters suffices for state-of-the-art SAR-to-RGB and NIR-to-RGB performance, while QXS-SAROPT retains near-full-data fidelity with 25% paired samples.

2 Related Work

Related work addresses remote-sensing translation through task-specific mappings, unpaired constraints, auxiliary observations, pretrained knowledge, and conditional adaptation of generative models.

  • Image-to-Image Translation: Paired methods use adversarial, transport, or denoising objectives, while semi-supervised and unsupervised methods exploit unpaired data through consistency, cycle, latent-space, or contrastive constraints.
  • Remote Sensing Cross-Modal Image Translation: Remote-sensing methods expand observations, encode inductive biases, or transfer pretrained knowledge to address cross-modal underdetermination.
  • Conditional Adaptation of Pretrained Generative Models: Pretrained generative models incorporate conditions through residual injection, key-value attention, or joint attention.

3 Method

LTP-BIT separates target-domain prior learning from source-conditioned adaptation: it pretrains a target prior on unpaired imagery, then freezes or selectively adapts it using paired data. Analyses and controlled experiments distinguish distributional realism, local compatibility, target-manifold coverage, and pixel-wise fidelity.

  • 3.1 Conditional Score Decomposition: The conditional score decomposes into a target-domain prior term and a source-dependent likelihood term, motivating prior-first training.The denoising-risk formulation further separates cross-modal information from residual prior correction.
  • 3.1 Conditional Score Decomposition: LTP-BIT pretrains a target-domain generative prior on unpaired imagery, then freezes pretrained backbone weights while learning source-conditioned control from limited paired data.The framework varies pretraining scale, model capacity, and target-domain composition to assess prior suitability.
  • 3.2 Target Prior Coverage: Under fixed conditional adaptation, prior-backbone and pretraining-data scaling produce closely aligned improvements in prior-generation and translation FID.The comparison varies prior size and nested pretraining pools while holding conditional capacity or the adaptation protocol fixed.
  • 3.2 Target Prior Coverage: Better target-distribution fit does not necessarily improve pixel-wise fidelity: CMMD improves overall with scaling, while PSNR varies non-monotonically with model size and decreases with more pretraining data.This motivates evaluating local structural compatibility and target-manifold coverage in addition to aggregate distributional metrics.
  • 3.2 Target Prior Coverage: Prior TFRR and Translation TFRR increase together with pretraining scale and remain strongly correlated across and within scales.TFRR measures the fraction of spatially aligned DINOv3 patch-feature discrepancy removed by unconditional or conditional recovery.
  • 3.2 Target Prior Coverage: Across expanding pretraining pools, Coverage and distribution-level metrics improve, whereas Density and instance-level fidelity do not improve consistently.At fixed pool size, increasing the proportion of QXS targets improves Coverage, Density, and translation metrics overall.
  • 3.3 Conditional Control: Jointly scaling P-DART and the prior backbone from S to XL consistently decreases CMMD and increases PSNR, unlike prior scaling with fixed conditioning capacity.The result indicates that stronger priors require commensurate cross-modal conditioning capacity.

4 Experiments

Experiments evaluate LTP-BIT across three remote-sensing translation datasets, comparing pretrained-prior sources, adaptation strategies, and paired-data budgets. Results show advantages from target-domain priors, parameter-efficient P-DART adaptation, and scarce-pair supervision.

  • Datasets and evaluation: The evaluation covers single-polarization SAR-to-RGB, multi-polarization SAR-to-RGB, and NIR-to-RGB translation on QXS-SAROPT, SpaceNet6, and Chesapeake.QXS-SAROPT contains 20,000 registered 256 × 256 SAR–RGB pairs, with 16,000 for training and 4,000 for testing.
  • Datasets and evaluation: Performance is assessed with PSNR, SSIM, LPIPS, FID, and CMMD, separating content fidelity from target-domain realism.CMMD is additionally reported because it provides more stable estimates on small evaluation sets.
  • Overall results: LTP-BIT achieves strong overall performance across all three datasets, with consistent advantages in perceptual similarity and distributional realism.Tables 4 and 5 report the cross-dataset comparison, while Table 6 evaluates full fine-tuning against frozen-base adaptation.
  • Comparison with existing methods: Methods pretrained on target-domain imagery generally outperform adaptations from generic-domain priors, demonstrating the importance of prior–target alignment.The comparison includes methods without pretrained generative priors, generic-domain priors, and remote-sensing optical priors.
  • Paired-sample scale: RS-XL outperforms IN-XL at every paired-data budget, with the largest margins at 1K samples, showing particular benefit when paired supervision is scarce.Both priors improve PSNR and CMMD as paired data increase under matched trainable parameters and 5,040 training steps.
  • Adaptation strategy: P-DART uses about one-ninth as many trainable parameters as full fine-tuning while reducing FID and CMMD, improving LPIPS, and retaining close PSNR and SSIM.Against parameter-matched LoRA and DoRA, P-DART improves PSNR, SSIM, and LPIPS with similar FID and CMMD.

5 Conclusion

LTP-BIT decouples target-domain prior learning from cross-modal dependence learning. Its results support scalable target priors and scarce-pair adaptation through parameter-efficient conditioning.

  • Conclusion: LTP-BIT decouples target-domain prior learning from cross-modal dependence learning.The framework learns a target prior before paired cross-modal adaptation.
  • Conclusion: Prior mismatch burdens conditional adaptation, while prior scaling primarily improves target-domain realism and needs sufficient conditioning capacity to improve instance fidelity.P-DART provides generation-aware control of the pretrained DiT prior.
  • Conclusion: LTP-BIT achieves state-of-the-art performance across SAR-to-RGB and NIR-to-RGB benchmarks with only 9.81% task-specific parameters.The method retains near-full-data fidelity on QXS-SAROPT using 25% of the paired samples.
  • Conclusion: Target generative priors can serve as independently scalable components, allowing scarce paired supervision to focus on cross-modal correspondence.This conclusion follows the reported decoupling and paired-data efficiency results.

Supplementary Material

The supplementary analysis decomposes conditional prediction into target-marginal and source-dependent components, then relates pretrained-prior mismatch to the residual required during paired adaptation.

  • Assumptions and notations: The analysis defines X and Y as source and target observations and Z = E(Y) as the target latent under a fixed encoder.The noisy latent state is defined as Z_t = α_t Z + σ_t ε with Gaussian noise.
  • Conditional score decomposition: Bayesian score decomposition separates the target marginal score learned from target images from the source–target dependence learned from paired data.The first term depends only on target images, whereas the second captures cross-modal dependence.
  • Prediction field decomposition: The flow-matching prediction field similarly decomposes into a target marginal field and a source-conditioned component.The common term cancels when fields are subtracted, leaving the marginal and conditional contributions.
  • Population risk decomposition: Relative to a pretrained prior, the optimal paired-adaptation modification combines the source-conditioned component with the residual between task and prior marginal fields.Closer agreement between pretrained and task priors reduces the residual prior correction.
  • Population risk decomposition: This decomposition motivates P-DART for source-conditioned control and backbone LoRA for adapting the residual prior correction.The paired objective jointly supervises the complete modification required for adaptation.

B.1 Two-Stage Training Pipeline

The two-stage pipeline prepares a remote-sensing-compatible codec, learns a target-domain prior from unpaired imagery, and then performs paired conditional adaptation while preserving the pretrained backbone.

  • Motivation and prior architecture: Mainstream remote-sensing generative models largely reuse Stable Diffusion weights or its traditional diffusion architectures.Existing improvements emphasize domain adaptation, conditional control, data construction, and generation scale more than generative-network representation learning.
  • Codec preparation: RAE-RS lowers Git-10M reconstruction errors and FID relative to RAE-PT but leaves ship, pier, vehicle, building, and roof details blurry.On ImageNet, RAE-RS increases DC error and produces brightness, color, contrast, and fine-detail degradation.
  • Codec preparation: RAEv2-Mix is trained on 2.5 million mixed remote-sensing and natural images to address local-detail and cross-domain degradation.The corpus includes one million high-information remote-sensing images, one million random remote-sensing images, and 500,000 ImageNet images.
  • Codec preparation: RAEv2-IN achieves reconstruction FIDs of 9.39 on Git-10M and 2.49 on ImageNet, versus 14.99 and 4.32 for RAE-PT.Mixed-domain training further reduces Git-10M reconstruction FID to 3.83 and improves reconstruction clarity for small objects and structural details.
  • Two-stage training pipeline: Stage I freezes the RAEv2-Mix codec and trains a DiTDH target-domain prior in a 16 × 16 × 1024 latent space.The RAEv2 representation encoder aggregates multi-level features from DINOv3-L without parameter updates.
  • Two-stage training pipeline: Stage II uses P-DART with source-conditioned reference and generation streams for paired conditional adaptation.Reference-stream states provide conditioning while the generation stream retains the pretrained sequence structure.
  • Training procedure: The training procedure initializes a fixed pretrained backbone alongside trainable P-DART and LoRA components before optimizing paired adaptation.The algorithm samples paired task data and updates the adaptation modules over training iterations.

B.2 Prior Scaling Protocols

The scaling studies vary prior data, model capacity, and continued-training controls while standardizing paired adaptation protocols. Expanded-pool and fixed-pool trajectories are compared at matched cumulative training volumes.

  • Evaluation Protocol: All scaling experiments use 256 × 256 inputs with spatial-resolution conditions, while baseline methods receive aligned paired-data exposure.Baseline computational costs and optimization trajectories remain specific to their official implementations.
  • Model-Capacity Scaling: S, M, L, and XL priors are independently pretrained on the same D1M dataset for 15 epochs under identical optimization settings.The model-scale study reduces DiT backbone depth, width, and attention heads for smaller tiers.
  • Model-Capacity Scaling: The P-DART Decoder keeps a 512-dimensional, four-head reference stream and decoder-side LoRA rank 256 across prior tiers.Encoder-side LoRA is disabled, and the DDT head remains invariant, keeping conditional structure and trainable-parameter count fixed.
  • Prior Data Scaling: D50K, D250K, D500K, and D1M form strictly nested pools for sequential prior training initialized from preceding EMA weights.The optimizer and learning-rate scheduler are re-initialized at each expansion stage.
  • Fixed-Pool Controls: Expanded-pool training and fixed-pool controls share parent checkpoints and are matched by cumulative processed images at 40M, 90M, and 190M milestones.The fixed-pool branch provides the continued-training comparison for the expanded data trajectory.

C.1 Unpaired Target-Domain Data Construction

The unpaired target-domain corpus is built from remote-sensing imagery using resolution-aware curation, quality control, redundancy suppression, and information-density sampling. Benchmark construction standardizes paired patch preparation and dataset partitioning.

  • Source Corpus: Git-10M provides more than ten million remote-sensing image-text pairs with geographic and spatial-resolution metadata for corpus construction.The pipeline uses its RGB imagery and Ground Sample Distance metadata.
  • Resolution-Aware Curation: Each source image is center-cropped to 256 × 256 and assigned to one of nine GSD categories spanning 0.5 to 128 m/pixel.Fixed sampling quotas preserve coverage across spatial resolutions.
  • Quality Control: Screening removes invalid padding, severe brightness clipping, blur, and occlusion before dHash and DINOv2 analyses suppress duplicate and visually similar patches.Information density combines local entropy, spatial frequency, edge density, and local intensity variation.
  • Nested Pools: The resulting RS-1M inventory contains one million RGB patches, from which strictly nested D50K, D250K, D500K, and D1M pools are constructed.Nesting preserves sample identity across prior-scaling experiments.

D.1 Local Feature-Structure Compatibility Analysis

The analysis tests whether a prior’s local target-feature recovery corresponds to recovery by its SAR-conditioned translation model. It uses matched perturbations, shared feature representations, and common aggregation procedures.

  • Experimental Design: A single-step recovery experiment compares frozen-prior and SAR-conditioned translation recovery under identical target images, perturbations, timesteps, and noise.The comparison probes compatibility between prior-side and translation-side local feature recovery.
  • Representation Setup: Target latents are encoded from paired targets and decoded back into reference images so feature distances remain within one codec representation space.This avoids including the codec’s reconstruction bias in recovery error.
  • Metric: TFRR measures the fraction of target-feature deviation removed by one-step recovery; zero means unchanged distance, while negative values indicate increased deviation.Higher TFRR denotes greater elimination of perturbation-induced feature deviation.
  • Perturbation Binning: The perturbation x-axis uses a shared median pre-recovery distance at t⋆ = 0.9801, while normalized perturbation degree only determines bins and coordinates.Individual normalized degrees can exceed 100% even though 100% represents the global median distance.
  • Evaluation: Four DiTDH-L priors trained on 50K, 250K, 500K, and 1M target-domain samples are evaluated on 4,000 QXS test pairs across 50 fixed timesteps.Shared quantile bins are averaged first within images and then across images, with bins below average distance 0.05 excluded.

D.2 Target Feature Manifold Coverage Analysis

The manifold analysis evaluates target-prior coverage and density alongside paired translation performance under standardized generation and feature protocols. Controls vary continued training and target-domain enrichment while isolating conditional adaptation.

  • Evaluation Protocol: Each prior generates 5,000 unconditional 256 × 256 images using 1 m/pixel conditioning, 50 sampling steps, and seed 42.Evaluation uses the same QXS optical reference set of 4,000 images.
  • Manifold Metrics: Coverage measures the proportion of real local neighborhoods reached by generated features, whereas Density measures generated-feature overlap within those neighborhoods.Coverage lies in [0, 1], while Density is unbounded and is reported as a raw value.
  • Continued Training: Fixed-pool continued-training controls compare 50K, 250K, and 500K groups at cumulative processed-image counts of 40M, 90M, and 190M.These controls use the same evaluation protocol as the expanded-pool experiment.
  • Target-Domain Enrichment: With a frozen prior backbone, increasing QXS target samples expands manifold coverage and local overlap, improves distribution-level translation metrics, and leaves instance-level metrics largely stable.The control separates prior manifold structure from backbone adaptation strategy.
  • Caveat: Coverage comparisons are constrained to fixed N = 4,000, M = 5,000, neighborhood scale, and a shared feature extractor.The absolute Coverage value also depends on real-sample count, generated-sample count, and neighborhood scale.
  • Interpretation: Prior TFRR and Translation TFRR rise together, while greater task-relevant coverage and density accompany stronger distribution-level translation performance.These patterns support aligning prior pretraining with downstream target domains and scaling conditional pathways with the prior backbone.

E.1 Effect of Backbone Prior Initialization

Remote-sensing prior initialization improves perceptual and distributional quality relative to ImageNet, while curriculum timing and conditional adaptation affect reconstruction and fine-structure fidelity.

  • Backbone Prior Initialization: 0.0405 dB higher PSNR and reductions of 3.6781 FID, 0.0193 CMMD, and 0.0124 LPIPS accompany RS-1M initialization versus ImageNet.SSIM decreases by 0.0045, indicating improved perceptual and distributional quality with comparable reconstruction fidelity.
  • Backbone Prior Initialization: RS-1M better preserves building arrangements, vessel locations, local boundaries, and coherent optical appearance, while ImageNet retains a slight SSIM advantage.The qualitative comparison agrees with RS-1M’s higher PSNR and lower LPIPS, FID, and CMMD.
  • Curriculum Endpoint: The 20-epoch curriculum endpoint improves distributional consistency across datasets, whereas QXS reconstruction metrics favor the 80-epoch alternative.On QXS, the 20-epoch setting reduces FID from 17.1970 to 16.5363 and CMMD from 0.2046 to 0.2009.
  • Fine-Tuning Strategies: Addition and concatenation with full fine-tuning recover less faithful structures than LoRA and DoRA, while Ours-XL reconstructs road geometry and parking-lot markings most accurately.LoRA and DoRA preserve overall scene layout but still blur or omit fine structures.

F Qualitative Results and Analysis

Across QXS, SpaceNet6, and Chesapeake, the proposed model balances preservation of source structures with plausible target-domain appearances despite modality-specific ambiguities.

  • SAR-to-Optical Translation: QXS and SpaceNet6 require SAR-to-optical translation under nondeterministic color and texture correspondence, speckle noise, layover, and complex scattering.Reliable structures include buildings, roads, shorelines, ships, and field boundaries.
  • NIR-to-RGB Translation: Chesapeake requires NIR-to-RGB translation because one NIR band does not uniquely determine visible-spectrum colors across vegetation, water, bare land, and built-up areas.The source and target images are geometrically well aligned, but spectral differences remain ambiguous.
  • Overall Qualitative Findings: Qualitative comparisons across all three benchmarks show the method consistently balancing source-structure preservation and target-domain realism.The visualizations compare predictions from the proposed model and baseline methods.
Loading 2608.28517v1…