PaperlayerView exact revision on arXiv

Source-linked AI summary

Modern Regularization Methods for Inverse Problems

Martin Benning, Martin Burger

arXiv:1801.09922v1math.NA

TL;DR

Ill-posed inverse problems require regularization to incorporate prior knowledge and make approximate inversion feasible. This paper surveys modern nonlinear methods, emphasizing variational approaches, their analysis, applications, and links to statistical inverse problems, multiscale decompositions, and learning theory.

  • Problem

    Ill-posed inverse problems require regularization because data and solutions generally lack continuous dependence, especially under measurement errors.

  • Method

    The paper provides a structured survey of modern nonlinear regularization methods, with fundamentals, convergence definitions, variational models, applications, and related developments.

  • Results

    Modern nonsmooth convex variational models improve reconstruction when appropriate prior information is available, while related iterative methods can reduce systematic errors and bias.

  • Takeaways & Limitations

    Variational regularization connects inverse problems with image processing, compressed sensing, statistical modeling, multiscale decompositions, and deep neural network architectures.

  • Takeaways & Limitations

    Quadratic-fidelity variational methods systematically bias recovery of singular vectors, although iterative regularization methods can overcome this reconstruction bias.

Abstract

from arXiv · show

Regularization methods are a key tool in the solution of inverse problems. They are used to introduce prior knowledge and make the approximation of ill-posed (pseudo-)inverses feasible. In the last two decades interest has shifted from linear towards nonlinear regularization methods even for linear inverse problems. The aim of this paper is to provide a reasonably comprehensive overview of this development towards modern nonlinear regularization methods, including their analysis, applications, and issues for future research. In particular we will discuss variational methods and techniques derived from those, since they have attracted particular interest in the last years and link to other fields like image processing and compressed sensing. We further point to developments related to statistical inverse problems, multiscale decompositions, and learning theory.

1 Introduction

Inverse problems are often ill-posed, so regularization introduces stable approximations and prior knowledge to make meaningful reconstruction possible. The paper surveys the shift from linear toward nonlinear methods, emphasizing variational techniques and related developments.

  • Motivation: Ill-posed inverse problems commonly lack continuous dependence of solutions on measured data, making regularization necessary when errors are present.Regularization replaces direct inversion with an approximation whose stability permits solving an approximate problem.
  • Foundations: A regularization method approximates the generalized inverse K† by a parameterized family Rα with improved stability properties.For linear methods, Rα consists of bounded linear operators converging pointwise to K† as α → 0.
  • Foundations: The parameter α must be chosen in relation to the noise level δ, interpreted as an error bound deterministically or a variance-like quantity stochastically.
  • Modern developments: Research has shifted toward nonlinear regularization methods, driven by variational models, sparsity, compressed sensing, Bayesian priors, and learning techniques.Applying learning paradigms is challenging because inverse problems usually lack ground-truth data and instead provide reconstructions dependent on regularization and noise.
  • Paper scope: The paper surveys modern nonlinear regularization methods, their analysis and applications, and fundamentals such as definitions and convergence requirements.It mainly assumes bounded linear operators on Banach spaces while indicating extensions to nonlinear operators and metric spaces.

2 A Little History of Regularization Methods

Regularization developed from early efforts to replace ill-posed inverse problems with well-posed approximations into a broad nonlinear field. Spectral linear methods dominated earlier theory, while variational, sparse, statistical, and iterative approaches expanded both analysis and applications.

  • Early foundations: Regularization methods approximate ill-posed problems by parameterized families of well-posed problems so meaningful solutions can be computed.Tikhonov regularization arises by combining least-squares data fitting with a regularization functional.
  • Early foundations: Early variational formulations generalized least-squares discrepancies and regularization functionals, although quantitative estimates and broader motivations were initially limited.
  • Linear methods: Linear regularization research in the 1970s and 1980s developed early stopping, truncated singular value decompositions, discretization, and projection methods using spectral analysis.
  • Nonlinear methods: Systematic nonlinear inverse-problem analysis began in the late 1980s with well-posedness, convergence, and convergence-rate results for nonlinear Tikhonov regularization.
  • Modern developments: Variational models, sparsity, wavelet shrinkage, compressed sensing, and Bayesian methods broadened regularization beyond classical linear approaches.
  • Applications: By around 2010, most medical-imaging methods had shifted from linear regularization to mainly variational methods, except fully sampled CT and statistically motivated EM.

3 Variational Modeling

Variational regularization balances data fidelity against a functional encoding solution preferences, enabling nonsmooth and structured reconstructions. Its modern variants include total variation, sparsity, Bregman distances, and infimal convolutions, each with characteristic benefits and limitations.

  • Core formulation: Variational regularization minimizes a weighted combination of data fidelity and a regularization functional, with α controlling their relative influence.The parameter is naturally small when the model should approach pure data fitting in the noiseless case.
  • Core formulation: Data fidelity can be chosen from noise statistics: Gaussian noise yields least squares, while Poisson likelihoods can improve reconstruction in high-noise photon-counting settings.
  • Total variation: Total variation regularization accommodates piecewise-constant or discontinuous solutions that are not represented well by Sobolev-space penalties.It is formulated in BV(Ω), whose distributional gradients are vectorial Radon measures.
  • Total variation: Total variation often produces piecewise-constant solutions, but can create staircasing in smoothly varying regions, motivating higher-order and other variants.
  • Total variation: For suitable derivative combinations, the regularizer becomes an equivalent norm after excluding a finite-dimensional nullspace.
  • Sparsity: Sparse variational models can guarantee that sufficiently many coefficients vanish, while related convex functionals support reconstruction of multiple peaks at unknown locations.
  • Infimal convolutions: Infimal convolution combines regularization approaches to define functionals intended to combine their advantages.
  • Bregman distances: Bregman distances compare discontinuity sets and their orientations, making them suited to structural priors that provide edge information.Contrast inversion can produce large Bregman distances; infimal convolution of Bregman distances is proposed as a possible remedy.

4 Fundamentals of Nonlinear Regularization

The section develops foundations for nonlinear regularization by identifying prior-selected limiting solutions, stability, convergence, and convergence rates in deterministic and generalized settings. It extends linear concepts to multivalued and multiparameter methods without requiring parameter convergence in the definition.

  • The framework begins with linear regularization in Hilbert spaces and seeks an analogue for nonlinear methods in Banach spaces.
  • Regularization approximates an unbounded generalized inverse through more stable operators, with parameter choice tied to the noise level.In linear methods, the operators are bounded and converge pointwise to the generalized inverse as α → 0.
  • Convergence rates are generally arbitrarily slow and therefore require restricted smoothness classes Mν, where ν measures smoothness or convergence order.
  • Nonlinear regularization generalizes minimum-norm solutions through prior-selected, potentially multivalued solutions and set-valued convergence concepts.Selection operators encode prior knowledge, while Kuratowski limsup captures stability through convergent subsequences whose limits solve the limiting problem.
  • The generalized framework allows vector-valued parameters and does not require the regularization parameter to converge as noise vanishes.This accommodates variational, iterative, multiparameter, and learning-based regularization schemes.
  • In multiparameter regularization, different parameter components may have different limiting behavior, including positive limits for relative parameters.

5 Variational Regularization Methods

The section formulates variational regularization as minimization of data fidelity plus a regularization functional, then establishes existence, stability, convergence, and Bregman-distance analysis under convexity and compactness assumptions.

  • Prior-selected solutions are obtained by minimizing the regularization functional over best approximate solutions, with parameter dependence determined by the model.For scalar J(u, α) = αJ1(u), the selection operator is independent of α; for an infimal convolution, it can depend only on a relative parameter.
  • Variational regularization selects minimizers of a data-fidelity term combined with a proper, lower semicontinuous, convex regularization functional.
  • Under Assumption 5.1, variational models have minimizers for every data value and parameter, and their solution sets are convex.
  • For convergent data sequences, selected variational solutions admit weak-star convergent subsequences whose limits solve the variational problem for the limiting data.
  • In the standard model J(u, α) = αJ(u), the condition δ α → 0 yields convergence of variational regularization methods.
  • Range and source conditions support error estimates in Bregman distances, and under Legendre data fidelity they are equivalent.The analysis also bounds the residual and symmetric Bregman distance.
  • With quadratic fidelity, positive α produces a systematic reconstruction bias for singular vectors, whereas iterative regularization can overcome this bias.

6 Iterative Regularization Methods

Iterative regularization applies robust iterations to ill-posed problems and uses early stopping to regularize noisy-data reconstructions. The section develops Bregman-based algorithms, convergence properties, bias correction, and numerical examples.

  • Iterative regularization paradigm: Iterative methods can exhibit semi-convergence: they initially approach the exact solution, then diverge on noisy data, motivating early stopping.The discrepancy principle selects a stopping index by comparing the residual with the noise level.
  • Bregman iteration: Bregman iteration replaces the regularization functional with a generalized Bregman distance and updates subgradients across iterations.The algorithm initializes α, noisy data, an initial iterate, and a subgradient, then repeatedly minimizes the resulting objective.
  • Bias and scales: Bregman iterations can correct systematic bias in variational reconstructions and introduce progressively finer-scale features during the iteration.For one-homogeneous J, the section explicitly connects bias correction with generalized singular vectors and describes an inverse scale-space behavior.
  • Bias and scales: In a compressed-sensing toy example, Bregman iteration significantly reduced average absolute bias while maintaining comparable standard deviation to the Morozov model.The experiment used a random 128×512 forward matrix, sparse vectors with nine nonzero entries, and one hundred noisy-data instances.
  • Linearized Bregman iteration: Algorithm 2 has analogous monotonicity and Fejér results, and if a finite iterate exactly satisfies Kuk∗ = f under the stated range condition, it belongs to S(f, α).The corresponding numerical illustration uses linearized Bregman iteration for a deconvolution problem.

7 Bias and Scales

The section connects regularization bias with feature scale and develops iterative, two-step, and spectral approaches for reducing bias or isolating scales. It also identifies conditions and computational boundaries for nonlinear spectral reconstruction.

  • Bias and scales: Variational regularization has larger bias on small-scale features, linking debiasing to multiscale analysis.The section frames bias and scale as closely related through eigenfunctions and eigenvalues.
  • 7.1 Inverse Scale space: Inverse scale space flow is the time-continuous limit of Bregman iteration and removes bias for one-homogeneous regularization under the stated singular-vector data condition.For input data vσ satisfying the specified condition, the reconstruction has no bias relative to variational regularization.
  • 7.1 Inverse Scale space: Exact decomposition of sums of singular vectors requires K-orthogonality and the (SUB0) condition.Under these assumptions, ordered singular-vector components are recovered over successive time intervals determined by their coefficients and singular values.
  • 7.2 Two-Step Debiasing: Two-step debiasing minimizes fidelity over solutions sharing the first-step support or subgradient, and its second step relates to a Bregman iteration at infinite regularization parameter.The subgradient formulation generalizes the refitting idea from the ℓ1 norm to arbitrary convex regularizations.
  • 7.2 Two-Step Debiasing: In TV denoising, Bregman iteration and two-step debiasing reduce contrast loss, while Bregman iteration appears to restore more small details.For larger α, smaller structures can be restored when they are represented in the subgradient but not the variational-model primal variable.
  • 7.3 Nonlinear Spectral Transform: Nonlinear spectral decomposition localizes singular vectors with different scales in different iterate-difference components, enabling spectral separation by band-pass filtering.The construction represents iterates through differences of subsequent iterates, with localization depending on α and the singular value.

8 Applications

The paper surveys modern regularization applications, emphasizing medical imaging examples where nonlinear and variational methods address undersampled or otherwise ill-posed reconstruction tasks.

  • Regularization methods have enabled applications including superresolution, PET, STED microscopy, and MR reconstruction.
  • 8.1 Velocity-Encoded Magnetic Resonance Imaging: Velocity-encoded MRI combines Fourier-like acquisition with regularization to reconstruct flow information from sampled measurements.Gradient-coil programming can encode velocity, while a variable transformation reduces the forward model to a sub-sampled Fourier transform.
  • 8.2 Dynamic MRI with Structural Prior: Dynamic MRI uses temporal smoothness and spatial total variation to exploit correlations between time steps under strong undersampling.The proposed functional combines these components and is reported to produce high-resolution reconstructions from extreme temporal undersampling on simulated data.
  • 8.3 Nonlinear Spectral Image Fusion: Nonlinear spectral TV decomposition fuses multiscale features from aligned and segmented images, but challenging cases may require manual preprocessing.The paper illustrates this with a fusion of a banknote, Gauß, and Newton images.

9 Advanced Issues

The paper identifies nonconvex data fidelities and machine-learning-related methods as advanced directions extending iterative variational regularization.

  • Advanced issues include extending iterative variational methods to nonconvex problems and modern machine-learning approaches.

9.1 Nonconvex Optimization

The paper develops proximal-gradient and Bregman-based approaches for nonconvex optimization, with convergence results under convexity, smoothness, and KL assumptions.

  • The framework replaces conventional convex data fidelities with a differentiable, potentially nonconvex energy functional.
  • 9.1.1 Proximal Gradient Method: Bregman proximal gradient linearizes the nonconvex part and damps updates through a Bregman distance generated by a Legendre functional.
  • Convexity of the relevant functional yields sufficient energy decrease, while stronger assumptions provide gradient bounds for the iterates.The analysis uses assumptions including Lipschitz continuity, strong convexity, coercivity or bounded level sets, and suitable step sizes.
  • In finite dimensions, KL assumptions yield global convergence of iterates to critical points under the stated existence and regularity conditions.For the proximal-gradient scheme, E + J must be a KL function with at least one critical point; related split schemes have analogous requirements.
  • The required strong convexity of J* imposes a restrictive smoothness condition on J, motivating a functional splitting strategy.

9.2 Nonlinear Inverse Problems

The paper extends the nonconvex optimization framework to nonlinear inverse problems and demonstrates nonlinear regularization on blind deconvolution and velocity-encoded MRI.

  • Nonlinear inverse problems fit the nonconvex framework by using nonlinear forward operators with convex or nonconvex data fidelities.The corresponding convergence theory applies, while regularization convergence requires additional analysis.
  • Linearization of the forward operator supports proximal-gradient and iteratively regularized Gauss–Newton strategies for nonlinear variational regularization.
  • Ill-conditioning still requires early stopping even when finite-dimensional convergence to a critical point follows from the theory.
  • 9.2.1 Nonlinear Landweber Regularization: Nonlinear Landweber regularization estimates both an image and an unknown convolution kernel in blind deconvolution.The example uses nonnegative, mean-preserving kernels and stops by the discrepancy principle; the reconstruction remains remarkable despite slight kernel-induced artifacts.
  • For velocity-encoded MRI, nonlinear Landweber reconstructs velocity directly from the nonlinear forward problem using a scaled H1 regularizer.The paper compares this reconstruction with fully sampled, zero-filled, and TGV-based results from sub-sampled data.

9.3 Learning

The section connects learned parameters and neural-network architectures with iterative regularization, while identifying conditions that preserve desirable convergence behavior. It shows how variable-metric updates can yield ReLU networks and motivate structured parameter learning.

  • Parameter learning: Parameter choice is central to regularization, with Morozov’s discrepancy principle used to determine when iterative regularization should stop.For iterative methods, the stopping iteration serves as the regularization parameter.
  • Parameter learning: Learned parameters can be estimated from training pairs by minimizing losses between reconstructions and ground-truth signals while incorporating prior knowledge through a regularization functional.The regularization operator maps noisy data and parameters to reconstructions, and the optimization may be simultaneous or sequential.
  • Parameter learning: The parameter-learning formulation is generic enough to include supervised machine learning with early stopping and current state-of-the-art parameter-learning approaches for inverse problems.When the reconstruction arises from an optimization problem, the formulation is also a bilevel optimization problem.
  • Iterative regularization and deep neural networks: Deep neural-network architectures are closely related or equivalent to linearized Bregman iteration with variable-metric data fidelity, providing insight into learning more stable architectures.The section focuses on the connection between modern deep networks and iterative regularization methods.
  • Iterative regularization and deep neural networks: Choosing the pointwise nonnegative characteristic functional produces the standard ReLU neural-network architecture for the primal update.The resulting parameters A_k and b_k retain forms derived from the variable-metric inverse-problem formulation.
  • Iterative regularization and deep neural networks: Conditions on the variable metrics can guarantee convexity and support monotonicity properties, including Fejér monotonicity of the iterates under a stated discrepancy inequality.Convexity requires suitable positive-semidefiniteness conditions involving Q_k and I − K*Q_kK.

10 Conclusions & Outlook

The conclusion presents modern convex variational and iterative regularization as versatile tools for inverse problems with suitable prior information, while outlining open theoretical and practical challenges. These include stochastic uncertainty quantification, spectral understanding, joint reconstructions, and inverse-problem-specific learning theory.

  • Conclusions: Nonsmooth convex variational models improve reconstruction when appropriate prior information is available, and iterative methods based on them can reduce systematic errors and bias.The same developments also provide insights into scale properties, spectral and multiscale decompositions, and links to deep neural networks.
  • Outlook: Stochastic models and uncertainty quantification remain major challenges, especially for non-Gaussian priors in Banach spaces and iterative regularization in stochastic settings.The survey describes the Gaussian case as comparatively well understood while identifying posterior convergence and infinite-dimensional statistical inference as open topics.
  • Outlook: Eigenvalue problems and spectral decompositions offer geometric understanding of inverse problems and regularization, although their practical reach remains unclear.They partly close the gap to singular value decomposition as a standard tool for linear regularization.
  • Outlook: Methods that alternately reconstruct multiple unknowns can effectively compute Nash equilibria and work well in engineering practice, but lack a systematic convergence theory.Motion-corrected reconstruction is given as an example involving alternating image and motion estimation.
  • Outlook: High-dimensional joint reconstruction problems remain open in modelling and analysis, including biomedical image-motion reconstruction and strongly undersampled dynamic or spectral problems.These applications are identified as current and future development areas.
  • Outlook: Machine learning is expected to become important for inverse-problem regularization, but learning theory must address ill-posedness and the difficulty of obtaining meaningful training data.Current learning architectures do not capture all inverse-problem-specific aspects of ill-posedness.
Loading 1801.09922v1…