PaperlayerView exact revision on arXiv

Source-linked AI summary

Generative Translation Priors: Bayesian Imaging with Cross-Modality Image Translation

Evan Bell, Jiaming Liu, Yifan Chen, Yu Sun

arXiv:2608.28872v1eess.IVcs.CVcs.LG

TL;DR

Ill-posed imaging reconstruction could benefit from complementary co-available modalities, but measurement-driven use of cross-modality generative models remains largely unexplored. GTP turns diffusion-based image translation into a Bayesian prior, guides sampling with target measurements, and analyzes the resulting bias. Across CT-with-MRI and PET-with-CT tasks, the proposed algorithms achieve high-quality reconstruction and benefits over conventional and diffusion-based inverse solvers.

  • Problem

    Cross-modality generative models typically target image synthesis without measurement information, leaving their use as priors for measurement-driven reconstruction largely unexplored.

  • Method

    GTP injects a time-varying measurement likelihood into a pre-trained diffusion translation process and provides gradient- and proximal-guided discretizations with ground-truth-free bias estimators.

  • Results

    GTP achieves high-quality reconstruction across limited-angle and sparse-view CT with MRI side information and low-dose PET with CT side information, outperforming conventional and diffusion-based inverse solvers.

  • Takeaways & Limitations

    GTP provides a principled framework for using generative translation models as expressive cross-modal image priors in challenging measurement-driven reconstruction tasks.

Abstract

from arXiv · show

The ability to leverage images from co-available modalities to inform target-domain reconstruction is highly desirable in imaging algorithms. In this work, we introduce Generative Translation Priors (GTP)--a Bayesian framework that transforms diffusion-based image-to-image translation models into cross-modality image priors for ill-posed imaging inverse problems. GTP incorporates target-domain measurements through likelihood guidance, steering the translation process toward the desired posterior distribution. The framework is grounded in a theoretical analysis of the resulting posterior dynamics, which reveals an intrinsic bias introduced by likelihood guidance. We further characterize this bias and derive a ground-truth-free formulation for its estimation, enabling it to serve as a practical metric for assessing posterior sampling quality. Building on this analysis, we derive two discretized GTP algorithms based on gradient and proximal likelihood guidance, respectively. We validate GTP on computed tomography reconstruction with magnetic resonance side information, and on positron emission tomography reconstruction with computed tomography side information. Experiments demonstrate that GTP effectively incorporates complementary cross-modality information and achieves high-fidelity reconstruction even under severely undersampled measurements.

1 Introduction

GTP uses diffusion-based cross-modality translation as an image prior and injects target measurements through likelihood guidance to sample a posterior for ill-posed reconstruction. Its analysis characterizes guidance-induced bias and supports two practical algorithms evaluated across CT and PET reconstruction tasks.

  • Cross-modality reconstruction can exploit complementary images, but requires expressive inter-modal modeling, broad modality-pair generality, and a principled framework.
  • GTP converts a pre-trained diffusion translation process into a Bayesian posterior sampler by injecting a time-varying measurement likelihood during sampling.The guidance steers translation trajectories toward the desired posterior while preserving learned inter-modal dependencies.
  • Theoretical analysis reveals an intrinsic, likelihood-controlled bias in the resulting posterior dynamics and provides a ground-truth-free criterion for evaluating guidance schemes.The criterion supports assessing and selecting likelihood guidance for posterior sampling accuracy.
  • Grad-GTP and Prox-GTP impose measurement likelihood through gradient- and proximal-based updates corresponding to forward and backward Euler discretizations.Both methods have corresponding ground-truth-free bias estimators, whose estimates correlate strongly with posterior sampling accuracy.
  • GTP is evaluated on sparse-view and limited-angle CT with MRI guidance and PET reconstruction with CT guidance, including Poisson shot noise in the PET experiment.The reported experiments demonstrate high-quality reconstruction across diverse inverse problems.

2 Background

The background frames reconstruction as Bayesian inference with learned diffusion priors, then reviews translation-based and cross-modality approaches. It positions GTP as a measurement-driven use of cross-modality generative models that addresses an underexplored gap.

  • 2.1 Inverse Problems & Bayesian Inference: Imaging inverse problems recover unknown images from incomplete, noisy measurements, motivating Bayesian inference through likelihoods, priors, and posterior sampling.Posterior sampling is particularly useful when reconstruction uncertainty matters in clinical diagnosis and scientific discovery.
  • 2.1 Inverse Problems & Bayesian Inference: Classical Bayesian reconstruction uses data-fidelity terms and regularizers, while proximal methods handle structured or non-differentiable objectives through backward-gradient-like updates.The proximal quadratic term keeps the output near the input while a parameter controls the test function’s influence.
  • 2.2 Diffusion Models as Learned Image Priors: Diffusion models learn reverse stochastic dynamics from Gaussian noise toward image distributions, using neural score approximations for the intermediate marginal densities.The reverse SDE contains inherited drift, time-dependent diffusion, Brownian motion, and a learned score function.
  • 2.2 Diffusion Models as Learned Image Priors: Posterior diffusion methods combine a learned prior score with a likelihood score, but the time-dependent likelihood is generally intractable and existing update rules lack a unified theoretical connection.Prior work either approximates the likelihood score under simplifying assumptions or incorporates measurements through designed diffusion updates.
  • 2.3 Image Translation via Generalized Diffusion: Generalized diffusion models transport source-domain images to target-domain distributions through an SDE whose drift determines the translation dynamics.Different drift choices recover different translation-model instantiations, including denoising diffusion bridge models.
  • 2.4 Related Cross-Modality Imaging Methods: Cross-modality methods include joint optimization, supervised image-to-image regression, and generative translation, but generative approaches typically omit measurement information.Using cross-modality generative models as priors for measurement-driven reconstruction remains largely unexplored, motivating GTP.

3 Generative Translation Priors

GTP adapts a translation diffusion prior toward posterior sampling by injecting a time-dependent likelihood, while its analysis identifies and estimates the resulting intrinsic bias. Two discretizations, Grad-GTP and Prox-GTP, provide practical gradient- and proximal-guided algorithms with bias-based accuracy guarantees.

  • Framework construction: GTP adapts a calibrated translation diffusion process toward the posterior by introducing a time-dependent likelihood potential during sampling.The likelihood evolution is constrained at the terminal time to recover the desired posterior, while earlier evolution may be chosen freely.
  • Intrinsic bias: The intrinsic bias requires only quantities available during sampling, enabling ground-truth-free evaluation and selection of likelihood evolution schemes.The analysis quantifies posterior sampling accuracy through accumulated bias, although direct computation can involve a high-dimensional Laplacian.
  • Intrinsic bias: The adapted posterior density evolution decomposes into a runnable Fokker–Planck SDE structure plus residual terms called intrinsic bias.The residuals mean the adapted process generally does not follow the desired posterior density path exactly.
  • Intrinsic bias: Theorem 1 bounds final KL error by intrinsic-bias accumulation along the sampling trajectory, so reducing bias tightens the posterior accuracy guarantee.The bound also accounts for the evolving normalizing constant through endpoint terms.
  • Discretized algorithms: Grad-GTP and Prox-GTP discretize the likelihood-adapted process using gradient-based and proximal likelihood updates, respectively.Grad-GTP uses Euler–Maruyama with current-iterate evaluations, whereas Prox-GTP applies the likelihood implicitly through a proximal operator.
  • Discretized algorithms: Prox-GTP is especially suitable when the forward model admits efficient proximal operators, and its bias can be evaluated from proximal iterates using a Moreau-envelope approximation.In the Langevin-SDE special case, Prox-GTP coincides with the proximal Langevin algorithm.

4 Numerical Validations of the Bias Theory

The numerical validation tests whether intrinsic bias tracks posterior accuracy without ground-truth references. Across likelihood strengths, it closely follows reference-based metrics, identifying both insufficient and excessive measurement weighting.

  • Experimental setup: The validation constructs a Gaussian image prior from 70,000 grayscale FFHQ images and uses a closed-form Gaussian posterior as the reference.Measurements use a 100×1024 normalized-row Gaussian matrix and Gaussian noise with σ = 0.01.
  • Validation metrics: Posterior accuracy is assessed with MMD, which measures recovered-to-true posterior discrepancy, and NLL, which evaluates representation of the ground-truth image.The experiment compares these reference-based metrics with intrinsic bias as likelihood strength varies.
  • Validation results: Intrinsic bias closely tracks MMD and NLL across likelihood strengths, reaching its lowest values in approximately the same intermediate regime.In that regime, the posterior mean better agrees with ground truth while posterior standard deviation retains meaningful spatial variation.
  • Validation results: At small likelihood strengths, insufficient measurement information leaves the recovered posterior near the prior, producing high intrinsic bias, MMD, and NLL.The posterior mean deviates from ground truth and its uncertainty retains the prior’s spatial structure.
  • Validation results: The intrinsic bias provides a reference-free accuracy criterion because MMD and NLL require ground-truth information unavailable in practical inverse problems.The figure caption reports that it detects both underweighting and overweighting of the measurement likelihood.

5 Experiments on Cross-Modality Imaging

GTP is evaluated for cross-modality CT and PET reconstruction under sparse, limited-angle, and low-dose measurements. Across these ill-posed settings, likelihood-guided translation priors improve reconstruction quality and preserve anatomical detail from reference modalities.

  • Evaluation setup: GTP is evaluated on sparse-view and limited-angle CT with MRI guidance and low-dose PET with CT guidance, including Poisson PET noise.The translation priors use DDBMs trained on paired cross-modal images independently of downstream inverse problems.
  • Sparse-View CT: At 4-view SV-CT, Prox-GTP achieves the highest PSNR and SSIM, with more than 2 dB PSNR improvement over DDS.The advantage decreases as more projections become available because CT measurements provide more information.
  • Visual CT comparison: GTP more faithfully preserves subject-specific kidney anatomy than unconditional baselines and produces more anatomical detail than JTV's smoother reconstructions.The comparison supports learned cross-modal dependencies over hand-crafted cross-modal regularization for these structures.
  • Limited-Angle CT: Across four LA-CT angular settings, GTP achieves the highest PSNR, while Prox-GTP exceeds DDS by 1.92 dB at 90° and by more than 7 dB at 30°.The benefit becomes more pronounced as angular coverage decreases, when contiguous structural information is unobserved.
  • Limited-Angle CT: In 90° LA-CT, GTP recovers poorly observed bone structures more faithfully than methods without side information, while JTV remains limited on fine structures in missing-angle regions.The reference MR image supplies complementary anatomical information where CT measurements leave structures unobserved.
  • PET reconstruction: In PET reconstruction, Prox-GTP leads at 5 × 10^5 and 10^6 counts, improving over PET-DDS by 1.10 dB and 2.24 dB in PSNR, respectively.At 10^5 counts, PET-DDS exceeds Prox-GTP by only 0.40 dB PSNR and 0.001 SSIM; Prox-GTP also better recovers a small tumor lesion at 5 × 10^5 counts.

6 Conclusion

The paper presents GTP as a framework for using generative translation models as image priors in cross-modal reconstruction. It combines likelihood-guided sampling, bias estimation, and two practical algorithms, with experiments showing benefits across challenging imaging tasks.

  • GTP uses generative translation models as expressive image priors for cross-modal image reconstruction by injecting a time-varying measurement likelihood during sampling.
  • Likelihood injection introduces an intrinsic sampling bias that can be estimated without ground-truth information as a reference-free criterion for posterior accuracy.
  • Grad-GTP and Prox-GTP implement different discretizations of the likelihood-adapted dynamics.
  • Experiments on sparse-view and limited-angle CT with MRI and low-dose PET with CT demonstrate benefits over conventional reconstruction methods and diffusion-based inverse solvers.

Supplementary Material for “Generative Translation Priors: Bayesian Imaging with Cross-Modality Image Translation”

The supplementary material develops the mathematical proof framework for the GTP dynamics. It derives density evolution and related SDE and bias expressions through algebraic transformations and Fokker–Planck analysis.

  • Notation and assumptions: The proofs assume sufficiently regular probability densities and a function g(x, t) continuously differentiable in x and t.
  • Density transformation: The derivation transforms p_t into e^g(x,t)q_t(x) to eliminate p_t(x) and expand the resulting derivatives.
  • SDE correspondence: The transformed evolution is rearranged into a Fokker–Planck equation corresponding to an SDE, then translated back to π_t by restoring normalization.
  • Bias derivation: The bias expression simplifies by removing Laplacian and squared-gradient terms under the stated derivation.

A.3 Proof of Theorem 1

Theorem 1 is analyzed through the time evolution of a KL divergence between the recovered and target densities. Integrating the resulting relation yields a nonnegative accumulated-bias measure.

  • The proof begins by characterizing the time evolution of KL(π̂_t ∥ π_t) for the biased and target processes.
  • The transport terms cancel when the two evolution expressions are combined, leaving the terms needed for the bias characterization.
  • Integrating over t ∈ [0,T], dropping the nonpositive Fisher-information term, and taking absolute values produces the theorem's result.
  • The absolute value makes the accumulated bias a nonnegative measure.

B.1 Numerical experiments

The experiments evaluate intrinsic bias as a ground-truth-free indicator of posterior quality and examine numerical issues in likelihood-guided GTP sampling.

  • Intrinsic bias, MMD, and NLL attain their minima at the same guidance strength λ for Grad-GTP posterior samples.The intrinsic bias upper bounds the KL divergence between recovered and true posteriors.
  • The default diffusion-coefficient schedule can create numerical challenges when the likelihood term g(x_t, t) is too large.Rescaling g may prevent the process from reaching the true posterior.
  • Adjusting σ_t changes the step size on ∇g and is theoretically justified when the drift is modified to preserve marginal distributions.In practice, the authors retain the unmodified DDBM solver.
  • Intrinsic-bias estimation uses the harmonic mean estimator to compute changes in the normalizing constant through ∂_t log Z_t.The estimator is straightforward with generated-sample batches but may have high variance.

C.1 Training GTPs

The implementation trains diffusion bridge translation models and applies likelihood-guided gradient or proximal updates for paired and unpaired cross-modality reconstruction.

  • Variance-preserving DDBMs translate source samples to target samples between t = 1 and t = 0, matching the GTP theoretical framework.The same architecture and training settings are used for unconditional diffusion and translation models.
  • Baselines: Baseline implementations include FBP, TV, JTV, DPS, DAPS, and DDS, with task-specific optimization and likelihood-guidance settings.JTV promotes coincident edges between paired images, while DAPS alternates reverse diffusion and Langevin sampling.
  • Grad-GTP: Grad-GTP uses a time-varying likelihood that is zero at t = 1 and equals the measurement likelihood at t = 0.Its denoised estimate is supplied by D_θ(x, t).
  • Prox-GTP: Prox-GTP enforces data consistency by applying a proximal operator to the denoised estimate.The default implementation avoids gradient descent and backpropagation through the denoiser.
  • Unpaired variants: Unpaired GTP variants train a DDBM using target-domain data and align modalities through features from a pretrained SAM encoder.These variants are designed for settings without paired source and target images.

D.1 Unpaired Grad-GTP and Prox-GTP

Unpaired GTP methods use side information effectively despite lacking paired training data, while translation-based variants substantially improve diffusion baselines on difficult reconstruction tasks.

  • In 30-degree LA-CT, unpaired Grad-GTP outperforms DPS by 5.15 dB and unpaired Prox-GTP outperforms DDS by 5.13 dB.Unpaired variants generally underperform their paired counterparts but help in challenging settings.
  • Unpaired Prox-GTP recovers kidney anatomy in 8-view SV-CT, while both unpaired methods recover bone structures in 90-degree LA-CT.These qualitative results show successful use of side information without paired training data.
  • Strict Prox-GTP generally performs similarly to Grad-GTP across tasks and settings, with no clear pattern in its small gains or losses.Both methods discretize the same underlying dynamics.
  • Replacing diffusion models with translation models improves DAPS and DDS across almost all settings, with gains exceeding 7 dB on 30-degree LA-CT.The gains are smaller on relatively well-posed 32-view SV-CT: +0.10 dB for DAPS-GTP and +0.15 dB for DDS-GTP.

D.4 Runtime results

GTP runtimes are generally comparable to diffusion-based counterparts, except for strict Prox-GTP, whose gradient-based proximal solve is substantially more expensive.

  • GTP algorithms generally have runtimes comparable to their diffusion-based counterparts on SV-CT and LA-CT.The strict Prox-GTP variant is the primary exception.
  • Strict Prox-GTP incurs substantial computational cost because its proximal step uses gradient descent with backpropagation through the denoiser.
  • PET-DDS and Prox-GTP have essentially identical runtimes across count levels.Both require similar numbers of network evaluations and system-forward-model evaluations.

D.5 Additional visualizations

Additional visualizations show that GTP improves anatomical reconstruction in undersampled CT settings, while the underlying diffusion and translation models generate diverse, detailed, and anatomically consistent images.

  • 16-view SV-CT: On 16-view SV-CT, Prox-GTP reconstructs small anatomical details and feature shapes more accurately than DDS despite slightly lower PSNR.The femur and other small features are better reconstructed when MRI side information is used.
  • 60-degree LA-CT: On 60-degree LA-CT, Grad-GTP and Prox-GTP outperform diffusion baselines, recovering anatomical features with fewer artifacts, especially in the missing angular region.Every baseline either misses significant anatomy or contains serious artifacts.
  • Unconditional CT prior: The unconditional CT diffusion model generates diverse chest, abdominal, and pelvic anatomies with fine details and small features.The model is implemented as a DDBM and uses 200 sampling steps.
  • MRI-to-CT translation: Without measurement guidance, the translation model produces CT images whose features are consistent with reference MR images and, in most cases, the corresponding real CT images.The visualizations compare reference MR images, translated CT images, and real CT images.
  • Cross-modality reconstruction: The visual results indicate that GTP benefits from cross-modality information when reconstructing anatomical details under limited-view CT measurements.The qualitative comparisons specifically highlight improvements in small anatomical regions and missing angular regions.
Loading 2608.28872v1…