Source-linked AI summary
Frozen CLIP Priors for Robust Self-Supervised Poisson Inverse Problems
Laura C. Diaz-Delgado, Emmanuel Martinez, Henry Arguello
TL;DR
Photon-limited inverse imaging lacks clean ground truth and requires robustness to Poisson noise, sampling operators, and distribution shifts. The paper addresses this with an ADMM-inspired unrolled solver using a lightweight decoder on frozen CLIP features, plus GR2R and Equivariant Imaging self-supervision; experiments show competitive shifted-domain reconstruction and self-supervised performance approaching supervised training.
Problem
Self-supervised inverse imaging must reconstruct under photon-limited Poisson measurements when clean ground truth is impractical and robustness under dataset and acquisition shifts is required.
Method
The method combines an ADMM-inspired unrolled PnP solver, closed-form data consistency, a lightweight decoder on frozen CLIP RN50 features, and GR2R plus Equivariant Imaging self-supervision.
Results
Experiments on Poisson CFA demosaicing and deblurring show competitive reconstruction quality, improved robustness under dataset shifts, and favorable efficiency versus iterative PnP baselines.
Takeaways & Limitations
Frozen CLIP representations provide an effective parameter-efficient prior for Poisson inverse problems, while the combined self-supervised scheme supports performance approaching supervised training.
Takeaways & Limitations
The approach relies on keeping pretrained CLIP representations frozen because task-specific fine-tuning can degrade their pretrained invariances.
Abstract
from arXiv · showhide
Self-supervised learning for imaging inverse problems is increasingly important in photon-limited settings, where acquiring clean ground truth is impractical and reconstruction must remain stable under dataset and acquisition shifts. This challenge is amplified under Poisson noise, whose signal-dependent statistics interact with sampling operators (e.g., CFA mosaicing). Meanwhile, foundation vision encoders trained at web scale offer distortion-invariant, content-related representations that generalize well across domains, suggesting a promising route to build priors that transfer beyond the training distribution without expensive fine-tuning. This paper proposes an ADMM-inspired unrolled plug-and-play solver for Poisson inverse problems that decouples a closed-form data-consistency update from a parameter-efficient prior. The prior is implemented as a lightweight decoder operating on frozen CLIP RN50 dense multi-scale features, adapting foundation representations with less trainable parameters. For self-supervision, the method integrates GR2R measurement-domain re-corruption with an Equivariant Imaging regularizer via virtual acquisitions. Experiments on Poisson CFA demosaicing and deblurring show competitive quality, improved robustness under shifts, and self-supervised performance approaching supervised training.
1 Introduction
The introduction frames self-supervised Poisson inverse imaging as an ill-posed reconstruction problem where robust priors are needed under limited ground truth and distribution shifts. It proposes frozen CLIP features, an ADMM-inspired unrolled solver, and specialized self-supervision, with competitive shifted-domain results.
- Self-supervised reconstruction is practically necessary when clean ground truth is expensive or impossible to collect in microscopy, medical imaging, and low-light photography.
- Learned pixel-space priors often target narrow training distributions, leaving robustness under operator-induced shifts as a persistent limitation.
- Frozen CLIP RN50 dense multi-scale features provide a transferable, distortion-invariant, content-related backbone that can support restoration through a lightweight decoder.
- The proposed solver combines frozen CLIP features and a lightweight decoder with an ADMM-inspired unrolled PnP architecture for parameter-efficient, operator-aware restoration.
- GR2R measurement-domain re-corruption is coupled with Equivariant Imaging regularization to address signal-dependent Poisson noise and stabilize learning under sampling operators.
- Poisson CFA demosaicing and deblurring experiments show competitive quality, improved robustness under dataset shifts, and favorable reconstruction efficiency versus iterative PnP baselines.
2 Image Formation
The image-formation section models photon-limited measurements with element-wise Poisson statistics and a known linear forward operator. It specializes this model to CFA subsampling for demosaicing and shared-channel spatial convolution for deblurring.
- The unknown is a vectorized clean RGB image x ∈ R3n, while y ∈ Rm denotes the observed measurements.
- A known linear forward operator A maps the clean image to measurements, and γ > 0 controls Poisson shot-noise severity.
- Poisson demosaicing (CFA sampling): In Poisson demosaicing, A models Bayer BGGR color-filter-array subsampling using diagonal binary masks for the red, green, and blue channels.
- Poisson demosaicing (CFA sampling): The demosaicing acquisition produces a single-channel mosaiced observation y ∈ Rn corrupted by Poisson noise.
- Poisson deblurring (spatial convolution): In Poisson deblurring, A applies the same spatial convolution operator to each color channel using a shared blur kernel.
- Poisson deblurring (spatial convolution): The deblurring measurements contain m = 3HW values and form a blurred noisy RGB image under the Poisson model.
3 Method
The method unrolls ADMM-inspired iterations that alternate operator-aware data consistency with a CLIP-based plug-and-play prior. It uses frozen CLIP features with a lightweight decoder and combines supervised or Poisson-specific self-supervised training.
- Optimization formulation: The reconstruction solves a regularized inverse problem using quadratic data fidelity and a prior term weighted by λ.The quadratic surrogate enables closed-form data-consistency updates.
- Network and training: The network truncates T ADMM iterations into a finite-depth architecture whose shared trainable parameters are the decoder weights.Initialization uses the physics-driven back-projection A⊤y, and clean targets support direct supervised reconstruction training when available.
- Unrolled optimization: ADMM unrolling separates each iteration into a data-consistency update, a prior update, and dual-variable ascent.The x-step enforces measurement fidelity, while the z-step incorporates the plug-and-play prior.
- Data consistency: For CFA demosaicing, the data-consistency update decouples element-wise and is computed through per-pixel division.This follows from the diagonal structure of A⊤A for the CFA operator.
- Data consistency: For convolutional deblurring, the data-consistency update exploits Fourier-domain diagonalization and is efficiently computed with FFTs.The resulting operations are pointwise in the Fourier domain.
- CLIP denoising prior: The plug-and-play prior approximates the z-subproblem with a denoiser composed of a frozen CLIP encoder and a trainable decoder.The encoder supplies distortion-invariant, content-related dense multi-scale features, while the decoder maps them to RGB estimates.
- Self-supervised training: Self-supervised training combines GR2R measurement-domain re-corruption with Equivariant Imaging regularization for Poisson noise and CFA sampling.GR2R addresses signal-dependent, non-Gaussian Poisson noise, while the equivariant term stabilizes training against trivial measurement-consistent solutions.
4 Simulations and Results
Experiments evaluate the proposed method on Poisson demosaicing and deblurring across datasets, noise regimes, qualitative examples, efficiency, and ablations. The method remains competitive under dataset shifts, while self-supervision approaches supervised performance and the frozen CLIP design improves efficiency and reconstruction quality.
- Experimental setup: The proposed method is evaluated against DPIR, GSPnP, RAM, and Transfer CLIP on Poisson inverse problems.Experiments cover Poisson demosaicing and deblurring on BSDS500 and DIV2K, including dataset-shift evaluation.
- Poisson demosaicing: 30.53 dB: Ours (Self) improves over RAM by +0.07 dB on BSDS500 at γ = 0.01 and remains close to Ours (Sup) at 30.75 dB.On DIV2K under the same regime, Ours (Self) obtains 30.30 dB and improves over GSPnP by +0.94 dB.
- Poisson demosaicing: 26.98 dB: Ours (Self) exceeds GSPnP by +0.48 dB on BSDS500 at γ = 0.05.Its SSIM also improves over RAM from 0.7178 to 0.7579 on BSDS500 and from 0.7343 to 0.7435 on DIV2K.
- Quantitative results: Across two datasets and two noise regimes, Ours consistently performs best, with Ours (Sup) leading and Ours (Self) closely following.The results span CFA mosaicing and blur, supporting transfer across operators and dataset distributions.
- Poisson deblurring: At severe-noise deblurring, Ours (Self) improves over GSPnP by +0.42 dB on BSDS500 and +0.49 dB on DIV2K.The corresponding SSIM gains are +0.031 and +0.034, while supervised margins are larger.
- Runtime and complexity: Ours uses 0.09 TFLOPs and 0.0242 s, yielding approximately 100× speedup over Transfer CLIP and approximately 20–40× over DPIR/GSPnP.The 10.99M reported parameters are trainable parameters only; the CLIP backbone remains fixed/shared.
5 Conclusion
The work introduces a foundation-driven unrolled plug-and-play solver for Poisson inverse imaging, combining operator-specific data consistency with a compact prior based on frozen CLIP features. Self-supervision couples GR2R re-corruption with Equivariant Imaging, while experiments show competitive quality and improved robustness under dataset shifts.
- The solver unrolls ADMM iterations with a closed-form forward-operator data-consistency update and a lightweight decoder prior over frozen CLIP RN50 multi-scale features.The decoder is shared across unrolled iterations, restricting learning to a compact task adapter.
- GR2R measurement-domain re-corruption and Equivariant Imaging regularization enable self-supervised learning without clean ground truth.The combination matches signal-dependent Poisson statistics and stabilizes training under sampling operators.
- Experiments on Poisson CFA demosaicing and deblurring over BSDS500 and DIV2K demonstrate competitive quality and improved robustness under dataset shifts.Self-supervised performance approaches supervised training according to the reported conclusion.
A Real Photon-Limited Data
A zero-shot protocol evaluates the method on real photon-limited SID data using one held-out patch and self-supervised training on the remaining patches from the same image. Channel-wise Poisson calibration and an affine color transform support reconstruction and evaluation in realistic sensor conditions.
- A single real low-light SID image supplies a 512 × 512 test patch, while its remaining patches optimize Lself without clean or paired supervision.This is a zero-shot self-supervised adaptation protocol.
- Poisson scaling is estimated independently for RGB channels, yielding γrgb = (0.018, 0.017, 0.026).The reconstruction is performed in linear RGB before affine transformation for sRGB visualization and metric evaluation.
- The experiment illustrates adaptation to real photon-limited sensor statistics without clean-image supervision.The caption describes the result as demonstrating adaptability to realistic acquisition conditions.
B Broader Scope and Future Directions
The framework separates acquisition physics, observation statistics, and image priors, allowing the reconstruction architecture to extend beyond Poisson noise and the evaluated inverse problems. Future adaptations mainly change the statistical self-supervision or data-consistency solver while retaining the frozen-CLIP prior.
- The framework represents acquisition physics with a known linear operator, derives self-supervision from the statistical model, and uses frozen CLIP features with a lightweight decoder as the prior.This separation motivates extensions to broader noise models and additional inverse problems.
- Broader noise models: For other noise distributions, the frozen encoder, decoder, and unrolled splitting structure can remain unchanged while the re-corruption rule and likelihood objective are adapted.GR2R supports Gaussian, Poisson, Gamma, Binomial, and some non-Gaussian additive noise models.
- Noise calibration: The current implementation assumes the Poisson scaling parameter is known or estimated before training.Unknown-noise objectives such as UNSURE and PG-UNSURE are identified as complementary alternatives.
- Additional inverse problems: The same principle can extend to linear inverse problems including inpainting, super-resolution, compressed sensing, and coded-aperture imaging.The forward operator determines data consistency, while the CLIP module remains a reusable prior.
- Additional inverse problems: For incomplete operators with non-trivial null spaces, Equivariant Imaging and complementary multi-operator measurements remain important self-supervised structure.These components can be combined with measurement-domain self-supervision and are orthogonal to the CLIP prior.
C Additional Results
Additional results provide quantitative comparisons for Poisson demosaicing and deblurring on BSDS500 at intermediate shot-noise levels, alongside extended visual comparisons across datasets and noise levels.
- Table 5 reports quantitative results for Poisson demosaicing and deblurring on BSDS500 at γ = 0.02 and γ = 0.03.These are intermediate Poisson noise levels.
- Extended visual comparisons cover both inverse problems across different shot-noise levels and the BSDS500 and DIV2K datasets.The visual results complement the quantitative comparisons.
C.1.0 Additional Poisson Noise Levels
Additional experiments at γ = 0.02 and γ = 0.03 show stable reconstruction behavior as photon-limited degradation increases. The self-supervised method remains competitive across demosaicing and deblurring, consistently outperforming evaluated baselines while staying close to supervised performance.
- At γ = 0.02 and γ = 0.03, experiments evaluate CFA demosaicing and deblurring on BSDS500.These intermediate noise levels complement the main experiments and examine behavior as photon-limited degradation increases.
- +0.27 dB at γ = 0.02 and +0.53 dB at γ = 0.03 are the demosaicing gains of Ours (Self) over the strongest competing baseline.
- +0.25 dB at γ = 0.02 and +0.37 dB at γ = 0.03 are the corresponding deblurring gains over the strongest competing baseline.
- The supervised variant exceeds Ours (Self) by only 0.14–0.22 dB for demosaicing and 0.16–0.17 dB for deblurring.
- Performance decreases smoothly from γ = 0.02 to γ = 0.03, while Ours (Self) remains competitive and consistently outperforms the evaluated baselines.The reported trends indicate stability across intermediate photon-limited regimes rather than dependence on a small set of selected noise levels.
C.2.0 Additional Qualitative Results
Additional qualitative results cover Poisson demosaicing and deblurring across BSDS500 and DIV2K, including dataset-shift settings and multiple shot-noise levels. The figures compare noisy measurements, competing reconstructions, the proposed self-supervised and supervised variants, and reference images using overlaid PSNR/SSIM.
- Poisson Demosaicing: Figures 5–7 present qualitative Poisson demosaicing results on BSDS500 at γ = 0.05 and DIV2K at γ = 0.01 and γ = 0.05.
- Poisson Deblurring: Figures 8–10 present qualitative Poisson deblurring results on BSDS500 at γ = 0.05 and DIV2K at γ = 0.01 and γ = 0.05.
- Comparison Setup: Each comparison includes the noisy measurement, DPIR, Transfer CLIP, GSPnP, RAM, Ours (Self), Ours (Sup), and a reference image.
- Comparison Setup: Per-image PSNR/SSIM values are overlaid in the qualitative comparisons for direct visual assessment.