Source-linked AI summary
SP$^3$: Spherical Priors for Plug-and-Play Restoration
Sean Man, Ron Raphaeli, Matan Kleiner, Or Ronai
TL;DR
Image restoration needs tractable priors that preserve natural-image structure. SP3 uses Spherical Encoders as projection-based generative priors with closed-form data consistency, achieving comparable perceptual quality to diffusion and flow methods while being 3–630× faster.
Problem
Explicit, tractable prior functions remain challenging in image restoration, motivating alternatives such as Plug-and-Play denoising operators.
Method
SP3 alternates Spherical Encoder manifold projections with a closed-form data-consistency step through Half-Quadratic Splitting.
Results
3–630× faster than previous work while matching the perceptual quality of diffusion- and flow-based methods across noisy restoration problems.
Takeaways & Limitations
SP3 provides fast, gradient-free, anytime restoration, producing perceptually strong reconstructions from a single step and refining them with further iterations.
Takeaways & Limitations
SP3 prioritizes perceptual realism over classical distortion metrics, cannot provide MMSE solutions, and is not a posterior sampler.
Abstract
from arXiv · showhide
In this paper, we introduce SP$^3$, a novel Plug-and-Play algorithm that accelerates maximum a posteriori image restoration by replacing denoisers with Spherical Encoders (SE) as generative priors. SP$^3$ approximates the intractable proximal prior step by utilizing the SE tightly structured latent space as a robust projection onto the natural image manifold. Alternating this projection with a closed-form data-consistency step, via Half-Quadratic Splitting, achieves stable convergence without requiring gradient computation during inference. This unique formulation unlocks "anytime" restoration capabilities, producing sharp, plausible images from the first iteration. Evaluations across a variety of image restoration tasks demonstrate that SP$^3$ achieves perceptual quality comparable to state-of-the-art zero-shot diffusion and flow methods while being $3$-$630\times$ faster.
1 Introduction
SP3 uses a Spherical Encoder as a generative prior in a Plug-and-Play restoration framework, alternating projection and data-consistency steps through Half-Quadratic Splitting. It provides gradient-free, anytime restoration, with reported speed improvements of 3–630× while achieving perceptual quality comparable to prior PnP and diffusion/flow methods.
- Motivation: MAP restoration combines data fidelity with a natural-image prior, but explicit tractable prior functions are challenging to specify.This difficulty motivates replacing explicit prior optimization with easy-to-compute alternatives such as denoising operators.
- Method: SP3 uses a Spherical Encoder that maps images onto a latent spherical manifold as the generative prior for Plug-and-Play restoration.The method follows the Plug-and-Play principle of enforcing prior information through a projection operator.
- Method: SP3 alternates prior projection and data-consistency steps through Half-Quadratic Splitting, using a gradient-free optimization trajectory over the latent space.Figure 1 describes iterative projection of degraded observations onto a tightly structured latent space while enforcing data consistency.
- Results: 3–630× speed improvement is reported for SP3 compared to previous diffusion/flow-based work.The paper characterizes SP3 as extremely fast relative to these methods.
- Results: Anytime restoration produces a single-step reconstruction that can be further refined to improve perceptual quality, while results remain on par with different PnP and diffusion methods.This property is attributed to the Spherical Encoder prior.
2 Background and related work
The section introduces Sphere Encoder as a spherical-latent generative framework and positions it as the prior used by SP^3 for image restoration. It contrasts SP^3’s fast, gradient-free process with prior latent-optimization and diffusion/flow approaches that require inference-time gradients or are slow.
- Sphere Encoder: Sphere Encoder maps images to a uniformly distributed spherical latent space, unlike VAEs’ Gaussian latent distributions, addressing shortcomings such as posterior holes.SE uses an autoencoder to encode and decode images while structuring its latent representation on a sphere.
- Sphere Encoder: SE constructs spherical latents by RMS-normalizing encoder outputs and spherifying noisy vectors, with σ controlling the injected Gaussian-noise magnitude.The latent vector is v = f(E(x)), and noisy spherification is S(v, σ) = f(v + σ · e).
- SP^3: SP^3 uses SE as an image-restoration prior because its unique spherical-latent features support the method’s success.The paper explicitly adopts the SE framework as a prior for image restoration.
- Related work: Latent-space MAP restoration methods often perform poorly because they require propagating gradients through the generative model.These methods optimize latent vectors so the generated, degraded image matches the observed measurement.
- Related work: SP^3 uses a fast, gradient-free iterative process with SE, contrasting with diffusion/flow restoration methods whose gradients and iterative nature yield high quality at slow inference speeds.Earlier restoration networks may also be constrained by degradation processes observed during training, whereas pretrained priors enable zero-shot restoration.
3 Method
SP3 formulates restoration as MAP estimation in the spherical encoder latent space and uses the complete encoder–decoder as a stable projection prior. Alternating this projection with data consistency in an HQS-based PnP scheme improves fidelity without requiring differentiation during inference.
- MAP formulation and baselines: SP3 uses both the spherical encoder and decoder, addressing decoder-only baselines that omit encoder information integral to the learned prior.Decoder-only S-GD and S-PGD propagate gradients only through the decoder.
- Spherical projection: The projection P maps degraded OOD images toward semantically similar natural images, produces probable manifold samples, and refines quality through repeated application.P(·) = D(S(E(·), σ)) uses latent random noise of strength σ.
- Spherical projection: Within the first few steps, LPIPS and KID significantly drop across all evaluated noise levels, indicating rapid mapping of OOD inputs toward the natural image distribution.The experiment uses 100 AFHQ-Cat images degraded by Gaussian blur and additive Gaussian noise.
- Spherical projection: Latent noise σ > 0 stabilizes projection quality, whereas deterministic projection σ = 0 slowly diverges and remains perceptually worse.Injected latent noise helps produce more probable, realistic samples from the image manifold.
- Plug-and-play restoration: SP3 alternates unconstrained manifold projection with rigorous data consistency, balancing perceptual realism and measurement fidelity without differentiation during the algorithm.The alternating update scheme is an HQS instance, but fixed-point convergence requires strong contractive or averaged assumptions for P(x).
4 Experiments
Experiments compare SP3 with neural-denoiser PnP, generative-denoiser PnP, and zero-shot diffusion/flow methods across six restoration tasks on AFHQ-Cat and CelebA. SP3 delivers comparable perceptual quality to leading competitors while emphasizing fast and anytime restoration.
- Experimental setup: SP3 is compared with neural-denoiser PnP, generative-denoiser PnP, and zero-shot diffusion/flow restoration methods.The experiments use AFHQ-Cat at 256px and CelebA at 128px, with results generally reported for 20 SP3 steps.
- Experimental setup: All methods are evaluated on six noisy restoration tasks, including denoising, deblurring, super-resolution, and multiple inpainting settings.The task-specific degradations and dataset-dependent parameters are defined in the experimental setup.
- Metrics: The evaluation measures distortion with LPIPS and perception with KID, while also reporting PSNR and per-image wall-clock runtime.LPIPS is the main distortion metric, KID is the perception index, and PSNR is an additional distortion metric favoring smooth reconstructions.
- Speed & anytime generation: 3-630× speedup is achieved by SP3 relative to competing methods while maintaining at least comparable perceptual quality.Figure 7 compares KID and running time, showing SP3 at 1, 3, 5, 10, and 20 steps.
- Comparisons: SP3 achieves results comparable to the best competing methods, typically DPS and OT-ODE, while producing sharp results with fine details.The comparison is reported on perception-distortion plots for AFHQ and CelebA across different degradations.
- Speed & anytime generation: SP3 provides comparable perceptual results after a single step, and after 20 steps it remains competitive while enabling anytime restoration.The paper attributes this behavior to SE projecting degraded images onto the clean-image manifold.
5 Conclusion and limitations
SP3 is a fast, gradient-free Plug-and-Play method that combines Spherical Encoder manifold projections with closed-form data consistency for perceptual image restoration. Its perceptual focus brings limitations in distortion-oriented estimation and posterior sampling, alongside societal dual-use risks.
- Conclusion: SP3 uses Spherical Encoders, robust manifold projections, and closed-form data consistency to solve MAP image restoration without gradients.The method provides anytime generation while matching state-of-the-art diffusion and flow methods at a fraction of the computational cost.
- Limitations: SP3 prioritizes perceptual realism by forcing intermediate states onto the clean image manifold, limiting classical distortion performance and preventing MMSE solutions.MMSE solutions inherently favor smoother, averaged outputs.
- Limitations: SP3 solves the MAP objective but is not a posterior sampler like DPS [25].
- Societal impacts: Restoring clean images from partial measurements can benefit smartphone photography and noisy imaging, but the method also carries risks of unauthorized surveillance and deepfake generation.
A Additional Comparisons
This section compares all methods qualitatively on AFHQ-Cat and CelebA, and reports distortion, perception, and single-image runtime measurements.
- A Additional Comparisons: Figures S1-S6 and S7-S12 show qualitative comparisons of all methods on AFHQ-Cat and CelebA, respectively.The figures provide visual comparisons for the two datasets.
- A Additional Comparisons: Tables S1-S4 report PSNR and LPIPS distortion metrics and KID perception metrics for all methods on AFHQ-Cat and CelebA, respectively.Tables S1-S2 cover AFHQ-Cat, while Tables S3-S4 cover CelebA.
- A Additional Comparisons: Table S5 reports running time in seconds for all methods on a single image.The runtime comparison uses single-image execution time as its measure.
A.1 Qualitative comparisons
The section presents qualitative image-restoration comparisons on AFHQ-Cat and CelebA across deblurring, denoising, inpainting, paintbrush-inpainting, random inpainting, and super-resolution.
- Qualitative comparisons cover AFHQ-Cat deblurring, denoising, and inpainting results.
- Additional AFHQ-Cat comparisons evaluate paintbrush-inpainting, random inpainting, and super-resolution.
- The same six restoration settings are qualitatively compared on CelebA.
A.2 Quantitative results
The quantitative results section reports evaluations on AFHQ cats and CelebA, with KID values scaled by ×103. It also reports per-image inference times measured on a single NVIDIA L40S GPU.
- Quantitative evaluations on AFHQ cats are presented in Tables S1 and S2, with KID scaled by ×103.
- Quantitative evaluations on CelebA are presented in Tables S3 and S4, with KID scaled by ×103.
- Per-image inference times are reported as wall-clock runtimes measured using a single NVIDIA L40S GPU for all methods.
B Implementation details · B.1 Comparisons between different priors · B.2 Licences
SP^3 uses degradation-specific initialization and is mainly controlled by λ, σ, and an adjustable number of forward steps, with λ and σ selected empirically. Comparisons are calibrated through generative KID validation, while the evaluated datasets have specified usage licences.
- B Implementation details: SP^3 is primarily governed by λ, the data prox regularization coefficient, and σ, the injected latent noise scale.σ measures relative noise strength against the prior’s maximum training noise level.
- B Implementation details: The number of forward steps is flexible rather than a strict constraint because SP^3 supports anytime restoration.
- B Implementation details: Initialization uses the adjoint operator for denoising and deblurring, a convolution-based median filter approximation for inpainting, and bicubic upsampling for super-resolution.
- B Implementation details: λ and σ configurations were determined empirically by grid search on a held-out validation set.Their optimal values across evaluated inverse problems are reported in Tables S6 and S7 for AFHQ Cat and CelebA.
- B.1 Comparisons between different priors: Generative backbones were validated for mostly similar KID results to support fair comparisons between Spherical Encoder and diffusion/flow-based methods.Table S8 reports KID ×103 for different priors on AFHQ-Cat and CelebA.
- B.1 Comparisons between different priors: Diffusion-based methods can achieve state-of-the-art results despite a diffusion-model prior having a high KID number.
- B.2 Licences: AFHQ is released under CC BY-NC 4.0, whereas CelebA uses a custom non-commercial research license.
C Theoretical Analysis and Convergence · C.1 Preliminaries and Notations
The theoretical analysis frames SP^3 as a non-convex alternating scheme whose convergence is motivated by Kurdyka–Łojasiewicz and fixed-point theory. It formalizes SP^3 over finite-dimensional image, latent, and measurement spaces using a generative prior, data-fidelity proximal operator, and averaged-operator framework.
- C Theoretical Analysis and Convergence: Under the Kurdyka–Łojasiewicz property, alternating minimization schemes converge to a critical point, extending classical Half-Quadratic Splitting analysis toward nonlinear deep generative priors.These guarantees require treating alternating updates as a non-convex optimization problem.
- C Theoretical Analysis and Convergence: SP^3’s fixed-point convergence analysis follows standard fixed-point and convex optimization theory, although its formal assumptions are restrictive for deep generative models.The framework motivates the empirical analysis in Section 3.
- C.1 Preliminaries and Notations: The formal setup defines finite-dimensional image, latent, and measurement spaces as X = R^n, V = R^m, and Y = R^k, with a linear forward operator A containing a non-trivial null-space.These spaces support the fixed-point formulation of the restoration algorithm.
- C.1 Preliminaries and Notations: SP^3 updates the image state through an operator T, establishing the fixed-point representation used in the convergence analysis.The passage introduces the operator without displaying its full expression.
- C.1 Preliminaries and Notations: The prior mapping is P(x) = D(S(E(x))), while the data-fidelity proximal operator is defined for the convex loss ℓ(x) = ∥Ax − y∥2.Together, these operators specify the prior and measurement-consistency components of the update.
- C.1 Preliminaries and Notations: Setting η = 1/(2λ) exactly recovers the update step defined in Equation (12).This parameterization connects the operator formulation to the algorithmic update in the main text.
- C.1 Preliminaries and Notations: Nonexpansive operators are 1-Lipschitz, α-averaged operators combine the identity with a nonexpansive map, and firmly nonexpansive operators are exactly 1/2-averaged.These definitions provide the operator regularity notions used in the convergence framework.
- C.1 Preliminaries and Notations: The composition of α1-averaged and α2-averaged operators is α3-averaged for some α3 ∈ (0, 1).This composition property supports analyzing the averagedness of the combined SP^3 update.
C.2 Proof of Convergence
Under an averaged prior and a nonempty fixed-point set, SP^3’s iteration converges to a fixed point. The proof establishes Fejér monotonicity and asymptotic regularity, while noting that nontrivial null-spaces generally prevent theoretical linear-rate bounds.
- Operator construction: Because prox_ηℓ is 1/2-averaged and P is α-averaged, their composition T is β-averaged for some β ∈ (0, 1).The convexity assumptions make prox_ηℓ firmly nonexpansive.
- Fixed-point convergence: Theorem 1 guarantees convergence of x_k+1 = T(x_k) to x_∞ ∈ Fix(T) when P is α-averaged and Fix(T) is nonempty.The update sequence converges to a fixed point under these assumptions.
- Convergence proof: The iteration is Fejér monotone and bounded, with non-increasing distance to any fixed point; telescoping residual bounds imply ∥x_k − x_k+1∥ → 0.Krasnosel’skii-Mann theory then yields strict convergence to a fixed point in finite-dimensional space.
- Convergence-rate limitation: Strong convexity would yield linear convergence, but restoration operators with non-trivial null-spaces generally make ℓ(x) non-strongly convex and rate bounds difficult to establish.This limitation motivates relying on empirical convergence behavior.