PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Translation Priors: Bayesian Imaging with Cross-Modality Image Translation

Evan Bell, Jiaming Liu, Yifan Chen, Yu Sun

arXiv:2608.28872v1eess.IVcs.CVcs.LG

TL;DR

Measurement-driven reconstruction using cross-modality generative models as priors remains largely unexplored, despite the value of complementary modalities. GTP adapts diffusion-based translation models with measurement likelihood guidance, and experiments show benefits over conventional reconstruction and diffusion-based inverse solvers across CT/MRI and PET/CT tasks.

  • Problem

    Measurement-driven reconstruction using cross-modality generative models as priors remains largely unexplored, despite the value of exploiting complementary modalities.

    Source: §1 · p. 1, §2.4 · p. 6

  • Method

    GTP adapts a diffusion-based translation prior toward posterior sampling by incorporating measurement likelihood through gradient- or proximal-based updates.

    Source: §1 · p. 3, §3.1 · p. 6

  • Results

    Across limited-angle and sparse-view CT with MRI guidance and low-dose PET with CT guidance, GTP shows benefits over conventional reconstruction and diffusion-based inverse solvers.

    Source: §6 · p. 18

  • Takeaways & Limitations

    Its ground-truth-free intrinsic-bias estimate offers a reference-free criterion for assessing recovered-posterior accuracy.

    Source: §6 · p. 18

Abstract

from arXiv · show

The ability to leverage images from co-available modalities to inform target-domain reconstruction is highly desirable in imaging algorithms. In this work, we introduce Generative Translation Priors (GTP)--a Bayesian framework that transforms diffusion-based image-to-image translation models into cross-modality image priors for ill-posed imaging inverse problems. GTP incorporates target-domain measurements through likelihood guidance, steering the translation process toward the desired posterior distribution. The framework is grounded in a theoretical analysis of the resulting posterior dynamics, which reveals an intrinsic bias introduced by likelihood guidance. We further characterize this bias and derive a ground-truth-free formulation for its estimation, enabling it to serve as a practical metric for assessing posterior sampling quality. Building on this analysis, we derive two discretized GTP algorithms based on gradient and proximal likelihood guidance, respectively. We validate GTP on computed tomography reconstruction with magnetic resonance side information, and on positron emission tomography reconstruction with computed tomography side information. Experiments demonstrate that GTP effectively incorporates complementary cross-modality information and achieves high-fidelity reconstruction even under severely undersampled measurements.

1 Introduction

Generative Translation Priors (GTP) uses diffusion-based image translation as a cross-modality image prior and incorporates measurement likelihood during sampling to steer reconstruction toward a posterior. The work also analyzes the bias induced by this guidance, develops two discretized algorithms and ground-truth-free bias estimators, and evaluates GTP on CT and PET tasks with cross-modality guidance.

1 Introduction

  • GTP addresses the need for expressive, generalizable, and theoretically principled cross-modality priors, amid limitations in existing methods and single-modality Bayesian reconstruction.The introduction identifies hand-crafted or modality-pair-specific dependency models and the lack of a principled Bayesian integration of image translation models as gaps.Source: §1 · pp. 1–2
  • GTP adapts pre-trained diffusion-based translation dynamics with a time-varying measurement likelihood, steering samples toward the desired posterior while preserving learned inter-modal dependencies.The framework treats image translation models as expressive cross-modality priors within a Bayesian reconstruction framework.Source: §1 · pp. 2–3, Figure 1 · p. 2
  • Analysis of the likelihood-guided SDE reveals an intrinsic bias that can be estimated without ground truth to assess and select likelihood guidance schemes.The analysis also extends to single-modality diffusion from a Gaussian distribution toward the image distribution.Source: §1 · p. 3
  • Grad-GTP and Prox-GTP impose measurement likelihood through gradient- and proximal-based updates, corresponding to forward and backward Euler discretizations, respectively.Ground-truth-free bias estimators are derived for both methods, and numerical results show a strong correlation between estimated bias and posterior sampling accuracy.Source: §1 · p. 3
  • GTP is validated on sparse-view and limited-angle CT reconstruction with MRI guidance and PET reconstruction with CT guidance, including Poisson shot noise in the PET-CT experiment.The introduction reports high-quality reconstruction across these inverse problems.Source: §1 · p. 3

2 Background

The background reviews Bayesian inverse problems and diffusion priors, then surveys generalized diffusion for image translation and existing cross-modality imaging methods to establish the context for GTP.

2.1 Inverse Problems & Bayesian Inference

  • Bayesian inverse problems recover images from measurements using a likelihood and prior; MAP estimation balances data fidelity against regularization, while posterior sampling characterizes reconstruction uncertainty.Proximal methods can solve objectives with structured or nondifferentiable terms, with the proximal operator interpreted as a backward gradient step.Source: §2.1 · pp. 3–4

2.2 Diffusion Models as Learned Image Priors

  • Diffusion models learn reverse dynamics from clean images to Gaussian noise and approximate the score with a neural network; posterior guidance adds a likelihood score that is generally intractable to compute.Existing approaches approximate that score under simplifying assumptions or use empirical measurement updates, but their connection to posterior diffusion dynamics remains only partially understood.Source: §2.2 · pp. 4–5

2.3 Image Translation via Generalized Diffusion

  • Generalized diffusion models transport between arbitrary source and target distributions, unlike standard diffusion’s Gaussian endpoint, and GTP applies to these models and standard diffusion regardless of training paradigm or supervision.Examples include diffusion bridges and stochastic interpolants; DDBMs use Doob’s h-transform to enforce both endpoint distributions.Source: §2.3 · p. 5

2.4 Related Cross-Modality Imaging Methods

  • Existing cross-modality methods use joint optimization, supervised image regression, or generative translation, but generative models typically omit measurement information, leaving measurement-driven reconstruction with cross-modality priors largely unexplored.Joint optimization models shared structure between modalities; supervised methods learn source-to-target mappings from paired data, while GAN- and diffusion-based generative approaches are generally designed for image synthesis.Source: §2.4 · p. 6

3 Generative Translation Priors

Generative Translation Priors adapt a translation diffusion to sample from an imaging posterior by introducing likelihood guidance. The section characterizes the resulting intrinsic bias, relates it to posterior-sampling accuracy, and develops two discretized algorithms with bias-estimation rules.

3.1 Construction of GTP from Translation Diffusions

  • Likelihood guidance adapts a calibrated prior translation process toward the posterior, but generally introduces an intrinsic bias that prevents exact realization of the desired posterior path.The likelihood potential must equal the negative log-likelihood at the terminal time, while its earlier evolution can be chosen freely; the bias vanishes only for evolutions that eliminate the residual terms.Source: §3.1 · pp. 6–7

3.2 Theoretical Analysis of the Intrinsic Bias

  • The final KL sampling error is controlled by accumulated intrinsic bias, making bias a criterion for evaluating and selecting likelihood-evolution schemes.A ground-truth-free bias formulation uses quantities available during sampling; choosing α_t = σ_t^2/2 removes the Laplacian and gradient-norm terms, leaving a score-network-computable inner product.Source: §3.2 · pp. 8–9

3.3 Discretized Algorithms: Grad-GTP & Prox-GTP

  • Grad-GTP uses Euler–Maruyama with explicit likelihood guidance, whereas Prox-GTP incorporates the likelihood through a proximal update.Both fix α_t = σ_t^2/2 and accumulate bias as a running estimate of the KL-error bound; for Prox-GTP, a Moreau-envelope approximation evaluates bias from proximal iterates without an explicit likelihood-potential gradient.Source: §3.3 · pp. 9–10

4 Numerical Validations of the Bias Theory

In a known-posterior Gaussian imaging test, Prox-GTP’s intrinsic bias tracked reference-based MMD and NLL across likelihood strengths, identifying both under- and over-guidance and the regime where the recovered posterior best matched ground truth.

Experimental Setup

  • The test used a closed-form Gaussian posterior derived from a grayscale 32×32 Gaussian prior estimated from 70,000 FFHQ images, with 100 measurements and noise σ=0.01.The ground-truth image was sampled from that prior, and measurements were generated using a Gaussian random matrix with normalized rows.Source: §4 · p. 10

Validation Results

  • Figure 2 shows that Prox-GTP’s intrinsic bias closely tracked MMD and NLL across likelihood strengths, reaching its minimum when the recovered posterior best matched ground truth.Unlike MMD and NLL, the intrinsic bias requires no ground-truth reference.Source: §4 · pp. 11–12
  • Small λ left samples near the prior, intermediate λ minimized intrinsic bias, MMD, and NLL together, while large λ over-weighted measurements and degraded uncertainty structure.At large λ, the recovered posterior’s standard deviation lost meaningful spatial structure.Source: §4 · p. 12

5 Experiments on Cross-Modality Imaging

The experiments evaluate paired-trained DDBM translation priors for MRI-guided sparse-view and limited-angle CT and CT-guided low-dose PET, including Poisson measurements. GTP benefits most when target measurements leave substantial structure unobserved, while also recovering fine anatomical detail.

5.1 CT Reconstruction with MRI Guidance

  • GTP better preserves subject-specific kidney anatomy than unconditional methods, while JTV’s MR-guided reconstructions are smoother and contain less anatomical detail.The learned cross-modal relationship helps recover anatomy more faithfully than the hand-crafted joint regularizer.Source: Figure 3 · p. 13, §5.1 · p. 16
  • At 16 and 32 views, GTP and leading baselines achieve comparable performance; the translation prior provides substantial complementary anatomical information under severe undersampling, with its benefit diminishing as CT measurements become more informative.Source: §5.1 · p. 16
  • At 90° limited-angle CT, GTP recovers major structures, especially bones, that methods without MR guidance fail to reconstruct reliably.JTV also uses MR guidance, but remains limited in recovering fine structures in missing-angle regions.Source: Figure 4 · p. 14, §5.1 · pp. 16–17
  • At 30° coverage, Prox-GTP exceeds all diffusion baselines by more than 7 dB PSNR; at 90°, it exceeds DDS by 1.92 dB PSNR.GTP has the highest PSNR across all four tested angular ranges, with larger advantages as coverage decreases.Source: §5 · p. 14, §5.1 · p. 16

5.2 PET Reconstruction with CT Guidance

  • At 5 × 10^5 PET counts, Prox-GTP recovers a small tumor lesion as a distinct hot spot, unlike the less pronounced OSEM and PET-DDS reconstructions.The visual comparison highlights local lesion detail that is less apparent in global image-quality metrics.Source: Figure 5 · p. 16, §5.2 · p. 18
  • At 5 × 10^5 and 10^6 counts, Prox-GTP beats PET-DDS by 1.10 and 2.24 dB PSNR; at 10^5, PET-DDS leads by 0.40 dB PSNR and 0.001 SSIM.Both diffusion methods outperform OSEM, especially at lower counts; GTP also improves substantially over the unguided translation model.Source: Table 3 · p. 17, §5.2 · p. 18

6 Conclusion … A.2 Proof of Proposition 2

GTP uses time-varying measurement likelihoods to guide cross-modal image-prior sampling, whose dynamics introduce an intrinsic bias that can be estimated without ground truth. The paper presents two discretized algorithms and proofs of their posterior-dynamics results.

6 Conclusion

  • GTP incorporates a time-varying measurement likelihood into generative-translation sampling, and the resulting dynamics introduce intrinsic sampling bias that can be estimated without ground truth.The bias estimate provides a reference-free criterion for assessing recovered-posterior accuracy.Source: §6 · p. 18
  • Experiments show Grad-GTP and Prox-GTP benefit limited-angle and sparse-view CT with MRI side information and low-dose PET with CT side information, outperforming conventional reconstruction and diffusion-based inverse solvers.The conclusion reports validation on these challenging cross-modality imaging tasks.Source: §6 · p. 18

A Technical Proofs

  • The proofs assume sufficiently regular probability densities and a function g(x, t) continuously differentiable in x and t, and define expectations on a probability space.The appendix also introduces KL divergence notation.Source: A Technical Proofs · p. 29

A.1 Proof of Proposition 1

  • The Proposition 1 proof derives q_t’s evolution PDE, substitutes p_t = e^g q_t, rearranges terms into an SDE’s Fokker–Planck equation, and restores the normalizing constant for π_t.These steps establish the normalized-density evolution used in the proposition.Source: §A.1 · pp. 30–31

A.2 Proof of Proposition 2

  • The Proposition 2 proof compares the SDE’s Fokker–Planck equation with equation (3), uses ∇log q_t(x) = −∇g(x, t) + ∇log p_t(x), and simplifies the bias by removing Laplacian and squared-norm terms.The proof then restores the normalizing constant for π_t.Source: §A.2 · pp. 31–32

A.3 Proof of Theorem 1 · B Validation of bias theory for Grad-GTP

The proof tracks the time evolution of KL(π̂_t ∥ π_t) and integrates its bound to obtain a nonnegative measure of accumulated bias.

A.3 Proof of Theorem 1

  • The proof characterizes the time evolution of KL(π̂_t ∥ π_t), building on related results in and a more general formulation in.The cited prior results are identified as Lemma A.3.2 of and Lemma A.4 of.Source: §A.3 · p. 32
  • The transport terms cancel in the KL-evolution analysis, as shown in.Source: §A.3 · p. 32
  • The derivation combines two intermediate results to obtain the next step of the proof.Source: §A.3 · p. 32
  • The proof then uses a stated fact in its derivation.Source: §A.3 · p. 33
  • Integrating (12) over t ∈ [0, T], dropping the nonpositive Fisher information term, and taking absolute values yields the result.Source: §A.3 · p. 33
  • The absolute value makes the resulting quantity a nonnegative measure of accumulated bias.Source: §A.3 · p. 33

B.1 Numerical experiments … C Implementation details for main experiments

The Grad-GTP experiment supports intrinsic bias as a ground-truth-free indicator of posterior quality, while the implementation sections describe adjustments for numerical stability and normalizing-constant estimation.

B.1 Numerical experiments

  • The Grad-GTP experiment uses 200 samples and 200 discretization steps, with guidance strength λ multiplying the step size and gradient steps rescaled by ∥∇g∥2.The setup uses the same ground-truth and measurement generation and likelihood as the Prox-GTP experiment.Source: §B.1 · p. 33
  • Intrinsic bias, MMD, and ground-truth NLL reach their minima at the same guidance strength, supporting intrinsic bias as a ground-truth-free posterior-quality criterion.Intrinsic bias upper bounds the KL divergence between recovered and true posteriors.Source: §B.1 · p. 33

B.2 Modifying the diffusion coefficient

  • For some likelihoods, the default diffusion-coefficient schedule σt may be too large and cause numerical challenges; simply rescaling g may prevent qt(x) from reaching the true posterior.Source: §B.2 · p. 33
  • Changing σt is theoretically justified by stochastic interpolants, which preserve marginal distributions under arbitrary σt schedules when the drift is modified accordingly.The experiments retain the unmodified DDBM solver, though manual drift modification and other numerical schemes are possible.Source: §B.2 · p. 34

B.3 Computing the normalizing constant

  • Estimating intrinsic bias at each step requires estimating the normalizing constant’s change through ∂tlog Zt; the method uses a harmonic mean estimator.Source: §B.3 · p. 34
  • The harmonic mean estimator is straightforward to compute from generated-sample batches, but it may have high variance.Source: §B.3 · p. 34

C.1 Training GTPs

GTP translation models and diffusion baselines use variance-preserving denoising diffusion bridge models (VP-DDBMs), trained with a shared U-Net setup. The experiments use preprocessed MR→CT and CT→PET datasets with held-out slices for tuning and testing.

DDBM configuration

  • The translation models and diffusion baselines are VP-DDBMs, which translate between source and target distributions from t = 1 to t = 0 and also support unconditional generation from a Gaussian source.The authors state this design aligns with their theoretical framework and offers strong image-to-image translation and unconditional generation performance.Source: §C.1 · p. 34

Architecture and training

  • Both translation and unconditional diffusion models use a shared diffusion U-Net, trained for 250,000 steps with AdamW at a 10^-4 learning rate and an exponential moving average decay of 0.9999.The model uses 64 base channels, channel multipliers, and attention at 16 × 16 resolution; evaluation uses the end-of-training averaged checkpoint.Source: §C.1 · p. 34

Data preprocessing and selection for MR→CT

  • For MR→CT, SynthRad volumes were body-cropped, CT and MR intensities were rescaled to [−1, 1], and 30 test slices were sampled from artifact-free middle slices of random test volumes.The 2023 and 2025 datasets were split approximately 80/10/10 for training, validation, and testing; five validation images were used for hyperparameter tuning.Source: §C.1 · p. 35

Data preprocessing and selection for CT→PET

  • For CT→PET, registered FDG-PET/CT Lesions volumes were cropped to 256 × 256, CT intensities were rescaled to [−1, 1], and PET values were normalized per slice before training rescaling.The 900 volumes were split into 800/50/50 for training, validation, and testing; three validation slices tuned hyperparameters, and 50 quality-filtered test images were selected, including 30 with annotated lesions.Source: §C.1 · p. 35

C.2 Implementation details of baseline methods for CT reconstruction … D Additional experimental results

The appendix specifies implementations and tuning procedures for CT baselines and GTP variants, PET baselines, and PET Prox-GTP. It also reports that intrinsic bias tracks posterior-quality measures and detects measurement under- or overweighting.

C.2 Implementation details of baseline methods for CT reconstruction

  • FBP used LEAP’s default filters, TV used LEAP’s ASD-POCS with smoothed TV, and JTV used a PDHG solver with task-tuned regularization and side-gradient scaling.TV used δ=0.001, 30 subsets, and 50 inner iterations; JTV hyperparameters α and η were selected by grid search.Source: §C.2 · pp. 35–36
  • DPS used gradient-norm-scaled likelihood steps after its residual-based scaling performed poorly with DDBM, while DAPS alternated reverse diffusion and Langevin sampling and DDS used conjugate-gradient data consistency.DAPS used 20 reverse steps and 50 Langevin steps per iteration; its Langevin sampling became unstable at very small t. DDS tuned λ and CG steps, and DPS’s gradient-norm scaling performed better on validation in all tested cases.Source: §C.2 · p. 36

C.3 Implementation details of GTP for CT reconstruction

  • Grad-GTP uses a time-varying likelihood that vanishes at the source and equals the measurement likelihood at the target, with gradient-norm-scaled steps and 1,000 sampling timesteps.The likelihood is g(x,t)=(1−t)·1/(2β^2)∥A x̂0−y∥^2; λ was grid-searched.Source: §C.3 · p. 37
  • Strict Prox-GTP uses the Grad-GTP likelihood and solves each proximal problem with five gradient-descent steps, gradient clipping, and 1,000 sampling timesteps.The inner-step learning rate was 1e-2, gradients were clamped to [−1,1], and λ was grid-searched.Source: §C.3 · p. 37
  • Default Prox-GTP computes a denoised estimate, applies an efficient proximal data-consistency update, and renoises the result; its update is solved by conjugate gradients.The CT implementation used 200 sampling timesteps and grid-searched λ and the number of CG iterations.Source: §C.3 · p. 37
  • Unpaired GTP trains a target-domain DDBM conditioned on reduced SAM features, using one interpolated feature channel to initialize the bridge and source images for conditioning.SAM features were instance-normalized, grouped into eight sets of 32 channels, and averaged to a 64 × 64 × 8 feature map; at testing, MR features provide the conditioning and bridge starting point.Source: §C.3 · pp. 37–38

C.4 Implementation details of baseline methods for PET reconstruction

  • PET baselines used a parallelproj non-TOF 2D scanner model with CT-based attenuation correction; OSEM was grid-tuned, while PET-DDS used a CT-conditioned DDBM and preconditioned gradient descent.PET-DDS introduced the negative Poisson log-likelihood through a data-consistency update, then renoised the result.Source: §C.4 · p. 38

C.5 Implementation details of Prox-GTP for PET reconstruction

  • PET Prox-GTP solves its negative Poisson-likelihood update using Bregman proximal gradient descent, with a mirror-descent step and elementwise Newton solves for the proximal subproblem.The method uses the KL divergence induced by negative entropy and applies pointwise preconditioning; the proximal subproblem separates into convex one-dimensional problems.Source: §C.5 · p. 39

D.1 Unpaired Grad-GTP and Prox-GTP · D.2 Prox-GTP (strict) · D.3 DDS-GTP and DAPS-GTP

Across the tested reconstruction settings, translation-based variants improved on diffusion baselines, unpaired GTP used side information to recover anatomy without paired training, and strict Prox-GTP performed similarly to Grad-GTP.

D.1 Unpaired Grad-GTP and Prox-GTP

  • On 30-degree LA-CT, unpaired Grad-GTP exceeded DPS by 5.15 dB, and unpaired Prox-GTP exceeded DDS by 5.13 dB.Unpaired GTP generally underperformed its paired counterpart, but surpassed diffusion methods without side information in challenging settings.Source: §D.1 · p. 40, Table 2 · p. 41
  • Unpaired Prox-GTP recovered kidney anatomy in 8-view SV-CT, while both unpaired GTP methods recovered bone structures in 90-degree LA-CT.These examples show recovery of anatomical features despite the absence of paired training data.Source: §D.1 · p. 41

D.2 Prox-GTP (strict)

  • Prox-GTP (strict) generally performed similarly to Grad-GTP, with small gains or losses showing no clear pattern.The methods discretize the same underlying dynamics, which the authors identify as a reason for their similar results.Source: §D.2 · p. 41

D.3 DDS-GTP and DAPS-GTP

  • DDS-GTP and DAPS-GTP replace the diffusion models in their baseline algorithms with translation models, using the same parameter-tuning ranges as DDS and DAPS.The specific parameter values are given elsewhere in the paper.Source: §D.3 · p. 42
  • DDS-GTP achieved the best PSNR among all methods in every SV-CT setting and two LA-CT settings.Table 5 reports SV-CT averages; the prose also reports the LA-CT result.Source: Table 5 · p. 43, §D.3 · p. 43
  • On 30-degree LA-CT, DAPS-GTP and DDS-GTP each gained more than 7 dB over their diffusion-based counterparts.For 32-view SV-CT, the gains were smaller: +0.10 dB for DAPS-GTP over DAPS and +0.15 dB for DDS-GTP over DDS.Source: §D.3 · p. 43

D.4 Runtime results

GTP runtimes generally match those of diffusion-based counterparts, with strict Prox-GTP as an exception; for PET, Prox-GTP and PET-DDS have essentially the same runtimes.

D.4 Runtime results

  • Across 8-view SV-CT and 90-degree LA-CT, GTP algorithms are generally well-matched in runtime with diffusion-based counterparts.Table 7 reports average runtime per test image.Source: §D.4 · p. 43, Table 7 · p. 44
  • Strict Prox-GTP incurs significant runtime cost from solving its proximal step with gradient descent and backpropagation through the denoiser.This is identified as the primary exception to the runtime match in Table 7.Source: §D.4 · pp. 43–44, Table 7 · p. 44
  • Across three PET count levels, PET-DDS and Prox-GTP have essentially the same runtimes because they require similar numbers of network and system forward-model evaluations.Table 8 reports average runtime per test image across the count levels.Source: Table 8 · p. 44, §D.4 · p. 44

D.5 Additional visualizations

Additional visualizations show GTP reconstructions, unconditional CT samples, and unguided MRI-to-CT translations. GTP improves anatomical recovery in 60-degree LA-CT, while the unguided models produce diverse CT images and anatomically consistent translations.

16-view SV-CT

  • For 16-view SV-CT, DDS has slightly higher PSNR than Prox-GTP, but Prox-GTP recovers small details and feature shapes, including the left femur, more accurately.The visual comparison highlights methods that use MRI side information and zooms in on small anatomical details.Source: §D.5 · p. 45, Figure 4 · p. 46

60-degree LA-CT

  • For 60-degree LA-CT, Grad-GTP and Prox-GTP recover anatomy more consistently than diffusion baselines, especially in the missing angular region.Every baseline either misses significant anatomical features or produces serious artifacts.Source: §D.5 · p. 45, Figure 5 · p. 47

Unconditional CT samples

  • Unconditional CT samples from the DDBM show diverse chest, abdominal, and pelvic anatomy with fine details, supporting its quality as a generative prior.The uncurated samples were generated using 200 sampling steps.Source: §D.5 · p. 45, Figure 6 · p. 47

Unguided MRI-to-CT translations

  • Unguided DDBM MRI-to-CT translations are anatomically consistent with the reference MR images and, in most cases, the real CT images.The visualizations show reference MRI, translated CT, and corresponding real CT images.Source: §D.5 · p. 45, Figure 7 · p. 48

Related papers

Loading 2608.28872v1…