Source-linked AI summary
Plug-and-Play Methods for Integrating Physical and Learned Models in Computational Imaging
Ulugbek S. Kamilov, Charles A. Bouman, Gregery T. Buzzard, Brendt Wohlberg
TL;DR
Computational imaging needs methods that combine physical forward models with learned priors while retaining useful reconstruction structure. The paper reviews PnP methods and their variations, including online and multi-agent approaches, concluding that they deliver state-of-the-art performance and provide a practical framework for integrating learned models with imaging physics. It also identifies modularity and denoiser/data-fitting pairings as a practical consideration.
Problem
Deep learning can have an important drawback relative to regularized inversion in computational imaging, motivating frameworks that integrate learned models with physical models.
Method
The paper provides a unified review of PnP algorithms, covering data-fidelity treatment, online variants, modular denoiser pairings, and multi-agent consensus equilibrium.
Results
PnP methods have been used across diverse computational-imaging applications and provide state-of-the-art performance.
Takeaways & Limitations
PnP offers a practical approach for integrating learned models with imaging physics in inverse problems.
Takeaways & Limitations
The paper identifies a practical boundary around modular design: data-fitting updates and denoisers are developed separately, creating multiple possible pairings.
Abstract
from arXiv · showhide
Plug-and-Play Priors (PnP) is one of the most widely-used frameworks for solving computational imaging problems through the integration of physical models and learned models. PnP leverages high-fidelity physical sensor models and powerful machine learning methods for prior modeling of data to provide state-of-the-art reconstruction algorithms. PnP algorithms alternate between minimizing a data-fidelity term to promote data consistency and imposing a learned regularizer in the form of an image denoiser. Recent highly-successful applications of PnP algorithms include bio-microscopy, computerized tomography, magnetic resonance imaging, and joint ptycho-tomography. This article presents a unified and principled review of PnP by tracing its roots, describing its major variations, summarizing main results, and discussing applications in computational imaging. We also point the way towards further developments by discussing recent results on equilibrium equations that formulate the problem associated with PnP algorithms.
I. HISTORICAL BACKGROUND
Computational imaging reconstruction is formulated as balancing measurement consistency with prior plausibility. PnP emerged by replacing proximal regularization steps in iterative optimization with black-box denoisers while retaining physical data-fidelity updates.
- Problem formulation: Inverse imaging estimates an unknown image from noisy measurements using an objective that combines data fidelity and prior regularization.The data-fidelity term measures consistency with observed measurements, while the regularizer enforces prior knowledge on the image.
- Plug-and-Play methods: The same proximal-to-denoiser substitution yields PnP versions of multiple proximal algorithms, including ADMM and FISTA.The review specifically discusses PnP-ADMM and PnP-FISTA, based on proximal splitting and proximal-gradient methods.
- Analytic priors: Common analytic priors include sparsity-promoting transforms, total variation, and Markov random field functions.A representative sparse prior is h(x) = τ∥W x∥1, with τ > 0 and W a suitable transform.
- Proximal optimization: ADMM solves composite objectives by alternating minimization and using proximal operators instead of requiring explicit knowledge of function gradients.ADMM uses an augmented Lagrangian formulation and iterates its updates until convergence.
- Proximal optimization: The proximal operator can be interpreted as a MAP estimator for an AWGN denoising problem.This interpretation connects optimization-based regularization with image denoising under additive white Gaussian noise.
- Plug-and-Play methods: PnP-ADMM replaces ADMM’s proximal regularization step with a black-box denoiser such as BM3D, then runs the remaining ADMM updates.This substitution allows denoisers to serve as prior models without requiring an explicit regularizer.
II. PLUG-AND-PLAY INTEGRATION OF PHYSICAL AND LEARNED MODELS
PnP integrates physical measurement models with learned image priors by alternating data-fidelity updates and denoising operations. This modular design supports broad reconstruction applications, model reuse across sensor configurations, and equilibrium-based analysis.
- PnP integrates physical and learned information to improve super-resolution results at 2× and 4× upsampling rates using the same DnCNN denoiser.
- Replacing a proximal operator with a pre-trained denoiser lets PnP use learned image priors within iterative algorithms such as ADMM and FISTA.
- PnP combines regularized inversion's flexibility to changes in the data-fidelity term with deep CNNs' representation capabilities.
- The same learned CNN can be reused across different measurement operators by changing the data-fidelity update, without retraining.
- PnP-ADMM is empirically fast and efficient, while PnP-FISTA uses a gradient step instead of requiring a data-fidelity proximal map.
- Black-box denoisers generally do not correspond to optimization problems, motivating consensus-equilibrium formulations for characterizing converged PnP solutions.
B. Regularization by Denoising
RED integrates a denoiser into a gradient-based reconstruction update and is more appropriately understood, for practical denoisers, through fixed-point equilibrium conditions. Problem-specific artifact-removal priors can improve reconstruction substantially, but sacrifice the measurement-operator independence of traditional PnP.
- RED formulation: RED-SD iterates a gradient step combining the data-fidelity gradient with the denoiser residual τ(x − D(x)).The update uses step size γ and regularization parameter τ.
- RED equilibrium: Practical RED denoisers are more appropriately analyzed as defining a fixed-point problem than as solving the originally derived optimization problem.The resulting fixed-point condition is the RED equilibrium condition.
- Convergence: For convex g and a nonexpansive denoiser, RED-SD converges sublinearly to a set of points satisfying the RED equilibrium condition.The convergence statement is based on monotone operator theory.
- Applications: In 20% subsampled random-projection compressive sensing, PnP and RED are compared with TV and ISTA-Net+ using AWGN or artifact-removal priors.The artifact-removal operator is trained for artifacts specific to PnP iterations.
- Applications: Problem-specific artifact-removal training can significantly improve results relative to an AWGN denoiser, but makes the prior no longer fully separate from the forward model.The AR operator is trained by embedding it in a supervised deep-unfolding architecture.
C. Online Plug-and-Play Algorithms
Online, stochastic, and incremental PnP methods replace batch data-fidelity computations with single-measurement or minibatch approximations. This makes per-iteration cost independent of the batch size and supports scalable reconstruction, with convergence under unbiased bounded-variance gradient estimates and suitable convexity and denoiser conditions.
- Variants: Online methods can also approximate the batch proximal operator with a partial proximal operator or use block-coordinate image updates.Block selection may be random, reshuffled by epochs, or organized into minibatches.
- Motivation: Batch PnP becomes expensive when evaluating the data-fidelity gradient scales linearly with the number of measurements.This motivates online, stochastic, and incremental approximations based on small measurement subsets.
- Application: In 3D algae reconstruction, SIMBA with minibatches of p = 10 produced results indistinguishable from RED-SD using all b = 89 measurements per iteration.Both methods used the same measurement model and DnCNN AWGN denoiser, while SIMBA had only a fraction of the per-iteration complexity.
- Online updates: Online PnP algorithms use a single component gradient, making per-iteration complexity independent of the number of measurement blocks b.They extend PnP-FISTA and RED-SD through online updates.
- Minibatches: Minibatch variants process several measurement blocks in parallel, improving efficiency on multiprocessor hardware.The minibatch size is p, with selected indices drawn from the available blocks.
- Convergence: Under unbiased bounded-variance gradient estimates, convex component data-fidelity terms, and firmly nonexpansive denoisers, online PnP converges sublinearly to fixed points.Uniform random block selection automatically satisfies unbiasedness.
D. Deep Unfolding and Deep Equilibrium Models
Deep unfolding truncates PnP iterations and trains the denoiser jointly, while deep equilibrium models extend this idea to an arbitrary number of iterations through implicit differentiation. These task-specific artifact-removal priors can improve reconstruction but reduce generality, whereas DEQ trades stored intermediates for fixed-point computation.
- Deep unfolding: Deep unfolding converts a PnP algorithm into a fixed-t architecture by truncating iterations and training CNN denoiser weights end-to-end.The training uses paired ground-truth and measurement data with a loss comparing predicted and true solutions.
- Deep unfolding: A truncated PnP algorithm trained through supervised optimization can be directly used for imaging.The loss function quantifies discrepancy between true and predicted solutions.
- Deep equilibrium models: Deep equilibrium models extend deep unfolding to an arbitrary number of iterations by using implicit differentiation at the PnP fixed point.DEQ replaces the finite-iteration output with a fixed-point solution.
- Deep equilibrium models: DEQ reduces training memory by avoiding storage of intermediate variables, but computing the fixed point can increase training computational complexity.The memory and computation trade-off is explicit in the DEQ formulation.
- Comparison with PnP: Traditional PnP uses an AWGN denoiser, whereas DU and DEQ use artifact-removal operators trained for artifacts specific to PnP iterations.The distinction concerns how the learned prior is trained and coupled to reconstruction.
- Comparison with PnP: Problem-specific DU/DEQ training can improve results but couples the prior to measurement operator A, reducing generality and increasing training complexity.The cited example trains the AR prior for reconstruction from random projections rather than AWGN denoising.
E. Related Approaches
Related approaches to PnP include patch-based and CNN learned priors, RED-inspired methods, and priors trained from class-, scene-, or image-specific data. Performance can also be targeted by fine-tuning denoisers for specialized image distributions.
- Data sources: Learned priors may use images from the same class, the same scene, or the single observed image.
- Learned priors: Early non-CNN learned priors include EPLL, which uses a cost-function approach on image patches.
- Learned priors: CNN-prior research has examined empirical denoiser advantages, modified FISTA objectives, convergence and accuracy benefits, and RED-related motivations.
- Specialized priors: PnP-inspired methods can improve performance by fine-tuning denoisers for a specialized distribution of images.The paper cites denoisers trained to regularize a 4D reconstruction problem in space and time.
F. Trade-offs and Limitations
PnP methods separate data-fitting updates from learned denoisers, promoting modularity and flexible pairings but potentially sacrificing reconstruction quality. Their performance also depends on denoiser training distributions, network architecture, hyperparameter choices, and training–application mismatch.
- PnP separates data-fitting updates and denoisers, allowing independent development and many possible pairings.
- This modular generality can reduce reconstruction quality relative to an end-to-end trained system under ideal conditions.
- DU and DEQ methods retain separate data-update and denoising modules while training the denoiser end-to-end to enhance reconstruction quality.
- PnP methods require tuning multiple hyperparameters, including the denoiser’s training-image distribution and assumed noise level.
- When the noise level is unknown during training, approaches include denoisers trained for multiple noise levels or reconciled using MACE.
- Neural-network architecture affects reconstruction quality and reconstruction time, but its selection is a complex, problem-specific design task.
- Learned priors can mismatch the training and application data distributions, creating an additional scope concern for PnP reconstruction.
III. MULTI-AGENT CONSENSUS EQUILIBRIUM
MACE extends consensus-equilibrium methods beyond PnP-ADMM by combining multiple physics-based, data-driven, or application-specific agents. In computational imaging, it can fuse differently oriented 2D denoisers with a tomographic model while retaining flexibility in model and solver choices.
- MACE extends PnP-ADMM to more than two update terms and interprets their solution through equilibrium.
- MACE permits physics-based, data-driven, and application-specific models; with a forward model and denoiser prior, its solution matches PnP while offering broader model and algorithm choices.
- Each MACE agent maps an input image to an improved output, with denoisers, proximal maps, artifact-removal functions, and heuristics as examples.
- MACE stacks each agent’s input and output images, then uses an averaging operator to impose consensus among the agents.
- The equilibrium condition requires the agents’ input-output differences to sum to zero at the shared consensus point.
- MACE equations can be converted into a fixed-point problem and solved with Mann iterations, which are guaranteed to converge when T is nonexpansive and has a fixed point.
- In 4D reconstruction, multi-slice fusion using three differently oriented 2D denoisers produces the best quality, whereas omitting one creates orientation-aligned streaks.
- Multiple 2D denoisers offer more efficient processing and can use widely available 2D training images, unlike data-limited 3D/4D training.
A. Implementing PnP algorithms
PnP implementations are supported by open-source libraries and have been applied across diverse computational-imaging problems. Applications show that PnP can adapt learned priors to changed acquisition settings and reduce memory while maintaining reconstruction quality.
- Open-source PnP implementations include SPORCO, PnP-MACE, and SCICO, a Python toolkit built on JAX for CPU/GPU transition, compilation, and automatic differentiation.
- PnP has been applied to superresolution, blind deconvolution, tomographic imaging, MRI, and synthetic aperture radar.
- DeCoLearn is a self-supervised CNN for removing artifacts from MCNUFFT images acquired with 1600 radial lines.
- RARE adapts DeCoLearn to 800-line data without retraining by using it as an image prior within PnP.
- SIMBA also has a computational advantage from mini-batch gradients and achieves comparable imaging quality to RED-SD.
V. FUTURE DIRECTIONS
PnP has become a standard computational-imaging tool because it integrates learned models with imaging physics in an accessible framework. Future work centers on convergence, acceleration, quality–modularity trade-offs, agent coordination, and new applications.
- Future PnP research includes identifying convergent denoisers, accelerating convergence, and managing trade-offs between modularity and reconstruction quality.
- Future MACE research must address agent hyperparameter selection, balancing multiple agents, and combining agents operating in different spaces.
- Many new computational-imaging applications remain to be explored.
- PnP methods have become a standard tool since their introduction in 2013 and provide state-of-the-art performance across diverse applications.
- Their early contribution was a practical approach for integrating learned models with imaging physics to solve inverse imaging problems.
- Alternative approaches may achieve better reconstruction performance but require potentially time-consuming, data-dependent application-specific training.
- PnP and MACE are especially useful when the forward model is not fixed or labeled problem-specific training data is insufficient.