PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Modeling: From Handcrafted Priors to Foundation Image Models

A. Murat Tekalp

arXiv:2610.36646v1eess.IVcs.CV

TL;DR

Image modeling must characterize plausible images while supporting both inverse problems and generation. The paper connects denoisers and generative models through score-based formulations, and describes practical boundaries including known forward operators and computationally slow inference for inverse problems.

Abstract

from arXiv · show

This paper unifies six decades of image modeling, evolving from handcrafted mathematical tractable priors to solve inverse problems in imaging to data-driven deep foundation image models for both image generation and solving inverse problems, under a common Bayesian framework that formulates image modeling as an implicit or explicit probabilistic characterization of a data manifold. Every image restoration or generative task fundamentally seeks to answer the same question: What makes an image look like a real image? In the Bayesian framework, this question is encapsulated by the prior probability density p(x), or its more optimization-friendly counterpart, the log-prior log p(x). While early works in solving inverse problems, such as Tikhonov, Total Variation (TV), and Non-local priors relied on explicit handcrafted definitions of log prior to model local smoothness, modern techniques, such as Plug-and-Play (PnP) priors and Denoising Diffusion models, are rooted in algorithmic uncoupling and implicit manifold learning that apply to both inverse problems and conditional image generation. By grounding imaging inverse problems and conditional image generation under a single mathematical engine: the gradient of the log-prior, or the Stein Score Function, we bridge the gap between signal processing and machine learning communities, turning what seems like parallel developments into a single, evolution of image modeling from handcrafted analytical models to models learned from big data, demonstrating that the pursuit of the denoising score is the common thread connecting variational optimization (in MAP estimation) to posterior sampling (in modern foundation image models).

IMAGE MODELING FOR SOLVING INVERSE PROBLEMS VS. CONDITIONAL IMAGE GENERATION

Inverse imaging uses a Bayesian posterior to combine measurement evidence with an image prior, while point estimation and posterior sampling extract different solutions from that distribution. Conditional generation uses conditional scores and classifier-free guidance to steer samples toward a prompt while trading diversity for visual fidelity.

Imaging Inverse Problems

  • For inverse imaging, the observation is a noisy degraded measurement, and Bayes’ rule gives the posterior used to infer the clean image.The forward operator and observation noise characterize the measurement model.Source: Imaging Inverse Problems · p. 2

Point Estimation:

  • MAP point estimation selects the most likely clean image fitting the data and prior, but MMSE averaging of plausible solutions can lose fine, high-frequency textures.The objective combines a data-fidelity term with a prior term that describes what a clean image looks like.Source: Point Estimation · p. 3

Posterior sampling:

  • Posterior sampling draws from the posterior using score-based diffusion guidance, flow matching, or Langevin dynamics, favoring perceptual quality over distortion.The choice between point estimation and posterior sampling depends on the desired reconstruction characteristics.Source: Posterior sampling · p. 3

Conditional Image Generation: Text-to-Image

  • Classifier-free guidance jointly estimates conditional and unconditional scores, then uses a guidance scale to amplify conditioning without explicitly evaluating a classifier.This trades sample diversity for visual fidelity.Source: Conditional Image Generation:… · pp. 3–4

Distinctions between Imaging Inverse Problems and Conditional Image Generation

Imaging inverse problems use likelihoods defined by physical measurements and noise, while text-to-image generation relies on a learned semantic mapping. The paper connects them through conditional sampling that combines measurement and semantic context.

Distinctions between Imaging Inverse Problems and Conditional Image Generation

  • Inverse problems use physics-defined likelihoods that can be computed from the forward operator and noise statistics, whereas text-to-image likelihoods require learned semantic mapping.The text-to-image mapping is learned from image-text pairs using alignment networks such as CLIP.Source: Distinctions between Imaging Inverse… · p. 4

Text-Guided Image Restoration: Combining Inverse Problems and T2I

  • Text-guided restoration samples along a generative trajectory conditioned jointly on physical measurements and semantic context.The framework assumes these conditions are independent given the true image; diffusion posterior sampling estimates the clean image and passes the physical residual through the network via the chain rule.Source: Text-Guided Image Restoration:… · p. 4

TRADITIONAL IMAGE PRIORS: HAND-CRAFTED MODELS

  • Before deep learning, handcrafted log-priors explicitly modeled natural-image regularities to regularize imaging inverse problems.These priors specify an explicit probabilistic characterization of image structure.Source: TRADITIONAL IMAGE PRIORS: HAND-CRAFTED… · p. 5

Tikhonov Regularization: L2 Smoothness Prior and Homogeneous Gaussian Random Fields

  • Tikhonov regularization penalizes the squared L2 norm of image gradients, equivalent to a stationary constant-mean Gaussian prior with a linear score.Its global smoothness assumption can oversmooth sharp edges and produce ringing because it treats variations equally and cannot adapt to local statistical changes.Source: Tikhonov Regularization: L2 Smoothness… · p. 5

Non-stationary Mean Non-stationary Variance Gaussian Model

  • The NMNV Gaussian model, allows local mean and variance to vary spatially, supporting edge-directed restoration.The Kuan filter uses these local statistics to smooth low-variance regions more strongly while preserving edges and texture in high-variance regions.Source: Non-stationary Mean Non-stationary… · pp. 5–6

Markov Random Fields

  • Markov random fields model spatial context through neighborhood-dependent pixel states and Gibbs distributions, but handcrafted local clique potentials struggle with long-range dependencies.Geman and Geman demonstrated their use in image restoration and segmentation.Source: Markov Random Fields · p. 6

Local Smoothness: Total Variation (TV) and Generalizations

  • Total Variation penalizes the spatial L1 norm of gradient magnitudes, favoring sharp edges over many small variations but potentially producing staircase artifacts.Generalizations include L1-TV, vectorial TV, non-local TV, and Total Generalized Variation to address different noise, color, texture, and staircasing concerns.Source: Local Smoothness: Total Variation (TV)… · pp. 6–7

Sparse Representations and Dictionary Learning Frameworks

Sparse modeling represents images or patches through a small number of dictionary atoms and offers several optimization formulations for estimating the coefficients. When the dictionary is unknown, dictionary learning alternates sparse coding and dictionary updates.

Sparse Representations and Dictionary Learning Frameworks

  • Sparse representations model an image or patch as a linear combination of dictionary atoms, with most latent coefficients set to zero.Under i.i.d. Gaussian noise and Laplace-distributed coefficients, the MAP objective yields L1 regularization.Source: Sparse Representations and Dictionary… · pp. 7–8
  • The penalized form uses λ to balance reconstruction fidelity and sparsity, while the sparsity-constrained form fixes a predefined nonzero-coefficient budget; the three forms are weakly equivalent for p = 1, but not p = 0.The sparsity-constrained form is natively solved via greedy pursuit metrics like OMP.Source: Sparse Representations and Dictionary… · pp. 7–8

Dictionary Learning Strategies:

  • K-SVD [12] learns an unknown dictionary by alternating OMP-based sparse coding with SVD updates to dictionary atoms and their coefficients.The passage also names MOD, ODL, NMF, and task-driven dictionary learning as alternatives.Source: Dictionary Learning Strategies · p. 8

Low-Rank Models:

Low-rank modeling promotes structure through sparsity in the singular-value domain, while non-local priors exploit repeated patches and structures across an image. Their Bayesian formulation interprets non-local regularization as a score-driven adaptive diffusion process.

Low-Rank Models:

  • Low-rank matrix approximation relaxes the non-convex rank constraint with the nuclear norm, which minimizes the sum of singular values.The passage describes a low-rank matrix as sparse in the singular-value domain.Source: Low-Rank Models · p. 8

Non-Local Priors: Patch-Based Self-Similarity

  • Non-local priors exploit repeated image patches, extending similarity-based averaging beyond immediate neighbors.Non-Local Means weights pixels by patch similarity, while localized search windows reduce computational complexity and limit spurious matches.Source: Non-Local Priors: Patch-Based… · p. 9
  • A Bayesian non-local graph prior yields a score corresponding to the Non-Local Laplacian, with strong smoothing where matching patches exist and regularization deactivated at unmatched isolated structures.The localized search window also reduces complexity from O(N^2 · |P|) to O(N · |S| · |P|).Source: Non-Local Priors: Patch-Based… · p. 10
  • Non-local methods include BM3D, which groups similar patches for collaborative transform filtering, and WNNM, which penalizes smaller singular values more than dominant structural ones.Non-local TV enforces piecewise flatness across structurally similar, non-contiguous regions to address localized staircasing.Source: Non-Local Priors: Patch-Based… · pp. 10–11

DENOISER AS A PRIOR: BRIDGE TO LEARNING IMAGE PRIOR FROM BIG DATA

Denoisers can serve as implicit priors within split optimization schemes, separating data fidelity from regularization. Proximal and variable-splitting algorithms provide ways to handle non-smooth priors and linear transforms.

DENOISER AS A PRIOR: BRIDGE TO LEARNING IMAGE PRIOR FROM BIG DATA

  • The denoiser-as-prior approach replaces handcrafted analytical priors with denoisers, from BM3D to deep CNNs, within a split data-fidelity and regularization objective.This division separates a smooth data term from a typically non-differentiable regularizer.Source: DENOISER AS A PRIOR: BRIDGE TO… · p. 11

Proximal Algorithms

  • Proximal gradient descent alternates a gradient step on the smooth data term with a proximal step for the non-smooth regularizer.For an L1 regularizer, soft thresholding clips coefficients below the threshold to zero and shrinks larger ones; ISTA is this scheme applied to Lasso, and FISTA adds acceleration.Source: Proximal Algorithms · p. 12

Variable Splitting Algorithms

  • HQS and ADMM introduce a variable to separate a linear transform such as Lx from the non-smooth regularizer, enabling separate inversion and proximal updates.HQS may require its penalty parameter to approach infinity for an exact solution, whereas ADMM uses a multiplier and converges to the exact solution without that requirement.Source: Variable Splitting Algorithms · pp. 12–13

Plug-and-Play (PnP) Framework

Plug-and-Play (PnP) replaces a handcrafted prior’s proximal operation with an off-the-shelf denoiser inside an inverse-problem solver. RED sought an explicit regularizer for denoisers, but practical denoisers generally do not satisfy the conditions that make its proposed gradient valid.

Plug-and-Play (PnP) Framework

  • PnP replaces a handcrafted prior’s proximal operator with an off-the-shelf denoiser in the ADMM z-step, enabling ADMM and HQS to solve inverse problems with denoisers.BM3D was used in the original demonstration.Source: Plug-and-Play (PnP) Framework · pp. 13–14

The Deep Learning Bridge (PnP-CNN)

  • CNN denoisers trained at different noise levels were plugged into HQS or ADMM, outperforming traditional PnP methods while running significantly faster.The cited work used DnCNN.Source: The Deep Learning Bridge (PnP-CNN) · p. 14

Regularization by Denoising (RED)

  • RED’s proposed gradient ∇R(x) = x − f(x) requires a symmetric denoiser Jacobian and local homogeneity, conditions practical denoisers violate.Practical patch-based and neural denoisers have asymmetric Jacobians; consequently, x − f(x) need not be the gradient of a scalar objective, and variational-optimization convergence proofs do not apply.Source: Regularization by Denoising (RED) · p. 15
  • Because practical denoisers violate Jacobian symmetry, RED follows a non-conservative vector field rather than descending an energy landscape, motivating alternative mathematical interpretations.Despite this mismatch with explicit MAP regularization, RED algorithms remain powerful.Source: Regularization by Denoising (RED) · p. 15

TWEEDIE’S FORMULA AND MMSE DENOISER AS SCORE ESTIMATOR

Tweedie’s formula links an optimal MMSE denoiser to the score of a Gaussian-smoothed image distribution, connecting denoising-based inverse solvers with score-based generative modeling. The paper presents handcrafted priors and learned models as different ways to represent and use the image manifold, while distinguishing point estimation from posterior sampling.

TWEEDIE’S FORMULA AND MMSE DENOISER AS SCORE ESTIMATOR

  • Under Gaussian noise, Tweedie’s formula expresses the MMSE estimate E[x|y] through the noisy observation and the score of its smoothed data distribution.An optimal MMSE denoiser therefore estimates the score, allowing an inverse-problem update to use a denoiser pass rather than an explicit algebraic prior.Source: TWEEDIE’S FORMULA AND MMSE DENOISER AS… · pp. 15–16

Interpretation of PnP and RED as Score-Matching by Denoising (SMD)

  • Score-Matching by Denoising interprets x − f(x) under an MMSE denoiser as the data-distribution score, making RED a score-root-finding task rather than explicit cost minimization.Reehorst and Schniter proposed this interpretation.Source: Interpretation of PnP and RED as… · p. 16

Transition from Classical to Generative Image Models - Explicit vs. Implicit Priors

  • The paper describes image modeling’s shift from analytical regularizers to data-driven priors whose gradients are computed operationally by denoisers using Tweedie’s formula.The learned prior is expressed as R(x) = −log pσ(x).Source: Transition from Classical to… · p. 16
  • A shared MMSE denoiser estimating the Stein score can support either MAP optimization or posterior simulation, but these approaches need not yield the same solution.Both use the same mathematical engine to represent the image prior.Source: Transition from Classical to… · p. 16

Common Basis of Generative Image Models: Learning High-dimensional Probability Densities

  • Generative models learn high-dimensional image densities, but differ in architecture and training: VAEs and GANs sample in one step, whereas diffusion, score-based, and flow models follow trajectories.The passage describes VAEs as using an encoder-decoder bottleneck and ELBO, and diffusion as using a predefined forward process with a learned reverse process.Source: Common Basis of Generative Image… · p. 17
  • Diffusion and score-based SDE models train across noise levels to estimate a time-dependent score, then use Langevin dynamics to move from white noise toward an image.The described trajectory runs from t = T to t = 0.Source: Common Basis of Generative Image… · p. 17
  • Flow Matching and Rectified Flows learn deterministic velocity fields for straight noise-to-data trajectories, rather than directly modeling the score field.The passage characterizes their sampling as faster and more stable than diffusion sampling.Source: Common Basis of Generative Image… · p. 17

4. Latent Diffusion Models

  • Latent diffusion combines a VAE’s learned p(x|z) with a score-matched diffusion prior p(z), while GANs frame generation as a two-player game whose discriminator-derived log-odds gradient pushes samples toward the real-image manifold.The paper presents diffusion-based foundation models as a continuous evolution centered on learning and traversing the image-manifold score, not a disjointed leap.Source: §4 · p. 18

GENERATIVE IMAGE MODELS: VARIATIONAL AUTOENCODER

Variational autoencoders address the intractability of image likelihoods and latent posteriors by approximating the posterior with a tractable distribution. They optimize an evidence lower bound, using reparameterization so gradients can train the encoder through stochastic latent sampling.

GENERATIVE IMAGE MODELS: VARIATIONAL AUTOENCODER

  • For images, computing the evidence p(x) and posterior p(z|x) is generally intractable because the evidence requires integrating over continuous latent values.A VAE represents an image as a nonlinear function of lower-dimensional latent variables.Source: GENERATIVE IMAGE MODELS: VARIATIONAL… · p. 18

Variational Inference and the Evidence Lower Bound

  • Variational inference approximates p(z|x) with a tractable q(z), and maximizing the ELBO minimizes its KL divergence from the true posterior.The ELBO separates an expected log-likelihood term from a latent-distribution regularization term.Source: Variational Inference and the Evidence… · p. 19

The Reparameterization Trick

  • The reparameterization trick expresses stochastic z as a differentiable function of distribution parameters and standard-normal noise, allowing gradients to flow back through the encoder.The encoder outputs the latent distribution’s mean and variance parameters.Source: The Reparameterization Trick · p. 20

Training VAE

The paper outlines VAE training and generation, then contrasts VAEs’ explicit likelihood-based modeling with GANs’ implicit adversarial approach. VAEs learn through an ELBO and generate from a sampled latent; GAN generators and discriminators compete toward an equilibrium.

Training VAE

  • VAE training encodes each image as latent-distribution parameters, reparameterizes a noise sample into a latent code, decodes it, and optimizes the batch ELBO.The described workflow updates model weights with an optimizer.Source: Training VAE · p. 20

VAE as a Generative Model

  • For generation, a trained VAE discards its encoder, samples z from a standard normal prior, and passes z through its decoder to produce an image.The passage describes VAEs as balancing reconstruction and regularization to synthesize plausible samples.Source: VAE as a Generative Model · pp. 20–21

GENERATIVE ADVERSARIAL NETWORKS

  • GANs are implicit density estimators: they transform latent noise into data without providing a probability for a specific sample, unlike VAE likelihood models.GAN generation is framed as a competitive two-player game.Source: GENERATIVE ADVERSARIAL NETWORKS · p. 21

Architecture of GAN and the Adversarial Metaphor

  • A GAN generator maps latent noise to samples while its discriminator predicts whether an image is real; discriminator feedback guides the generator toward outputs it cannot distinguish from training images.The described equilibrium has discriminator accuracy of 50%, equivalent to random guessing.Source: Architecture of GAN and the… · p. 21

Mathematical Foundations: The Minimax Formulation

  • The GAN minimax objective has the discriminator maximize correct real-versus-fake classification while the generator minimizes the value function to confuse it.For a fixed generator, the optimal discriminator is a density-ratio estimator comparing real-data and generated-sample densities.Source: Mathematical Foundations: The Minimax… · p. 22

Global Optimality and the Jensen-Shannon Divergence

GAN training connects distribution matching to image generation, while the section traces architectural and objective changes addressing training difficulty. Autoregressive models instead factor image generation into sequential predictions, later extending that approach to latent tokens.

Global Optimality and the Jensen-Shannon Divergence

  • The GAN generator’s criterion is equivalent to minimizing the Jensen-Shannon divergence, reaching its global minimum when the generated and data distributions are identical.At that point, C(G) = −log(4) and D(x) = 0.5 everywhere.Source: Global Optimality and the… · p. 22

From Vanilla GAN to DCGAN

The Vanishing Gradient Problem:

  • Early GAN training can leave the generator with vanishing gradients; the non-saturating loss proposed in provides stronger early gradients while preserving the same global equilibrium.Source: The Vanishing Gradient Problem · p. 23

Mode Collapse and Lack of Diversity:

Beyond JSD: The Wasserstein GAN (WGAN)

AUTOREGRESSIVE IMAGE MODELS

  • Autoregressive models factor an image’s joint distribution into conditional probabilities in raster-scan order, capturing high-frequency detail at the cost of sequential computation.Source: AUTOREGRESSIVE IMAGE MODELS · p. 24

Architectures

  • PixelRNN and PixelCNN enforce causal pixel generation; PixelCNN supports parallel training, while Gated PixelCNN enables conditional generation.PixelCNN++ [36] refines efficiency and quality with a discretized logistic mixture likelihood.Source: Architectures · p. 24
  • VQ-VAE compresses images into discrete latent codes for autoregressive prediction; the original DALL-E combined VQ-VAE with a Transformer for text-conditioned image generation.Source: Architectures · p. 25

TRAJECTORY-BASED DEEP GENERATIVE MODELS: DIFFUSION, SCORES, AND FLOWS

Trajectory-based generative models transform a prior distribution into an image distribution along time-dependent paths. The section links discrete diffusion objectives to continuous score-based dynamics, with the Stein score providing the quantity needed to parameterize generation.

TRAJECTORY-BASED DEEP GENERATIVE MODELS: DIFFUSION, SCORES, AND FLOWS

  • Trajectory-based models, including DDPMs, score-based generative models, and flow matching, transform a known prior into a data distribution along time-dependent paths.Source: TRAJECTORY-BASED DEEP GENERATIVE… · p. 25

The Continuous-Time SDEs and Probability Flows

  • A forward SDE degrades clean data toward noise, and its reverse-time SDE uses the Stein score to move from the prior back toward the data manifold.The score points toward higher-probability regions at the current noise scale.Source: The Continuous-Time SDEs and… · p. 26

The Probability Flow ODE:

  • The reverse SDE and Probability Flow ODE share the same marginal densities, and both require the score function to parameterize their generative trajectories.Source: The Probability Flow ODE · p. 26

Score-Based Diffusion: Unifying DDPMs and SGMs

Flow Matching: Moving Beyond Stochastic Trajectories

Flow Matching replaces stochastic trajectories with deterministic velocity fields and can train those fields through conditional paths. Straightened paths reduce sampling evaluations, while the synthesis also connects flow matching to score-based diffusion and latent diffusion models.

Flow Matching: Moving Beyond Stochastic Trajectories

Conditional Flow Matching (CFM):

Optimal Transport and Rectified Flows:

The Unified Mathematical Synthesis

  • The Probability Flow ODE of score-based diffusion is identical to a Flow Matching ODE, framing the model progression as increasingly straight paths while preserving data-manifold expressive capacity.Source: The Unified Mathematical Synthesis · p. 29
  • Latent Diffusion reduces high-resolution computation by denoising compressed representations, and cross-attention injects prompts or other conditioning into the diffusion backbone.Source: The Unified Mathematical Synthesis · p. 30

FOUNDATION IMAGE MODELS: THE UNIVERSAL IMAGE PRIOR

Foundation Image Models (FIMs) are presented as universal image priors learned from massive multimodal data, extending image modeling toward zero-shot use across generation, understanding, and inverse problems. The section distinguishes them from narrower deep generative models and outlines approaches for applying FIM priors to inverse problems.

FOUNDATION IMAGE MODELS: THE UNIVERSAL IMAGE PRIOR

  • FIMs learn a universal natural-image-manifold representation by scaling to billions of parameters and training across massive cross-modal datasets.The paper frames this as a shift from narrow, single-task generation toward versatile, zero-shot open-domain understanding and synthesis.Source: FOUNDATION IMAGE MODELS: THE UNIVERSAL… · p. 31

What Distinguishes Foundation Image Models from Deep Generative Models

  • FIMs differ from deep generative models through scale, multimodal alignment, and zero-shot task generalization, with frozen priors usable in arbitrary iterative solvers.The section contrasts patch-based DiTs and global attention with convolutional U-Nets’ local inductive bias; FIM priors can address inverse problems without retraining the base network.Source: What Distinguishes Foundation Image… · pp. 31–32

Classification of Foundation Image Models

Solving Inverse Problems in Imaging Using FIM Priors

  • Inverse-problem methods use either task-specific supervised fine-tuning or inference-time optimization with a frozen FIM, including posterior sampling, latent optimization, and Plug-and-Play denoising.Fine-tuned adapters can handle unknown degradations but remain task-specific; zero-shot posterior sampling requires a known degradation operator and can be slow, while Plug-and-Play uses the FIM as a denoising block.Source: Solving Inverse Problems in Imaging… · p. 33, B. Inference-Time Zero-Shot,… · pp. 34–35

IMPLICIT NEURAL REPRESENTATION

Implicit Neural Representations encode images as continuous coordinate-to-color functions and provide differentiable gradients for spatial analysis and fitting observations. Their architectural bias favors smooth structure, while positional encoding or sinusoidal activations address their difficulty learning high-frequency detail; the discussion also notes open challenges and task-dependent evaluation.

IMPLICIT NEURAL REPRESENTATION

  • An Implicit Neural Representation (INR) models an image as a continuous, differentiable function mapping spatial coordinates to RGB values, queried at desired resolution.Its differentiability supports gradients with respect to both coordinates and network weights.Source: IMPLICIT NEURAL REPRESENTATION · p. 36

INR as an Architectural Prior:

  • Randomly initialized INRs favor smooth, low-frequency patterns, enabling denoising when training stops before the network learns the noise.This architectural prior is related to Deep Image Prior.Source: INR as an Architectural Prior · p. 36

Overcoming the ”Spectral Bias”:

  • Positional encoding or sinusoidal activations address MLP spectral bias, which makes networks learn high-frequency details more slowly than low-frequency components.Fourier features let the network represent both smooth gradients and sharp edges.Source: Overcoming the ”Spectral Bias” · p. 36

DISCUSSION AND CONCLUSIONS

  • Foundation Image Models offer zero-shot versatility, but hallucinations, training-data biases, and limited fine-grained control remain challenges, and evaluation must balance fidelity with photo-realism according to task.The discussion contrasts fidelity for recognition tasks with photo-realism for artistic photography.Source: DISCUSSION AND CONCLUSIONS · p. 37

Related papers

Loading 2610.36646v1…