Source-linked AI summary
Learning spatially varying regularisation parameters of low regularity for image reconstruction
Kostas Papafitsoros, Luca Calatroni, Andreas Kofler
TL;DR
Variational image reconstruction needs regularisation weights whose spatial structure and regularity are not fully understood, especially when weights are learned adaptively. This chapter reviews theory across regularity classes and examines hybrid methods that learn weights with deep networks while retaining model-based reconstruction. The examples show low-regularity learned weights adapting to image structure and, in denoising, to specific noise instances, while MRI results do not show the same adaptation.
Problem
The chapter addresses how the regularity and structural properties of spatially varying regularisation weights affect variational image reconstruction.
Method
The chapter synthesises theoretical results across regularity classes and reviews neural-network approaches that learn spatially varying weights for model-based variational reconstruction.
Results
Numerical examples find highly detailed, low-regularity weights that adapt to image structure and specific noise instances in denoising, whereas MRI weights do not show the same noise- or-mask adaptation.
Takeaways & Limitations
Learned regularisation maps can remain interpretable through classical model-based reconstruction while providing highly adaptive spatial weighting.
Takeaways & Limitations
Existence of solutions for the dyadic-weight problem remains open and may require compactness conditions on admissible weights.
Abstract
from arXiv · showhide
In this chapter, we review and discuss the regularity properties of spatially adaptive regularisation weight functions used in variational image reconstruction. Incorporating such weights into classical model-based regularisers, such as Total Variation (TV) and Total Generalised Variation (TGV), allows the regularisation strength to vary across the image and adapt to local image content. When appropriately estimated, these weights can thus significantly improve edge and detail preservation in the reconstructions. We review the existing theoretical literature on this topic for different regularity classes, including constant, continuous, and piecewise constant functions. Our discussion is motivated by recent work on hybrid image reconstruction methods that combine model-based regularisation with deep neural networks to learn highly adaptive regularisation weights. In particular, we discuss how the structural properties of these weights influence the reconstruction from both theoretical and practical perspectives. Through representative examples in image denoising and magnetic resonance imaging (MRI) reconstruction, we demonstrate that the learned weights are often of low regularity and can adapt not only to the image structure but also to the specific noise realisation. We conclude by highlighting several directions for future research on this topic.
1 Introduction
Inverse imaging reconstructs grayscale images from corrupt, noisy measurements using variational regularisation, whose spatially varying parameters can adapt regularisation strength across the image to preserve detail.
- Inverse imaging: Inverse imaging seeks to recover an unknown image from corrupt and noisy measurements generated by a forward operator and random noise.The forward operator may be simple, as in denoising, or more involved and potentially non-invertible, making recovery ill-posed.
- Variational regularisation: Variational regularisation combines a data fidelity term with a regulariser to produce reconstructions that are consistent with the data and mathematically well-posed.The regulariser supplies additional regularity and strongly influences reconstruction quality and stability.
- Regularisation parameters: The regularisation parameter balances fidelity and regularisation strength, with different values changing the structure of the reconstructed solution.Traditionally, this parameter is treated as a constant function.
- Spatial adaptation: A non-constant parameter map Λ : Ω →(R+)ℓ makes the regularisation strength spatially varying across the image.The map can also be spatio-temporally varying in time-dependent problems.
- Spatial adaptation: Spatially varying parameters aim to produce sharper structures and better detail preservation by weighting regularisation differently throughout the spatial domain.The chapter examines their functional regularity, associated variational solutions, and links to deep-neural-network-based parameter selection.
2 Model-based regularisation functionals
Classical regularisers trade off smoothness, edge preservation, and staircasing, while spatially varying weights introduce richer solution structures and adaptation opportunities alongside additional existence questions.
- Tikhonov regularisation: Tikhonov regularisation produces smooth reconstructions that destroy edges because minimisers lie in H1(Ω) for d=2.The embedding C(Ω) ⊂ H1(Ω) rules out jump discontinuities in minimisers.
- Total Variation: TV preserves discontinuities, and its L2-denoising solutions do not create jump discontinuities absent from the data, up to an H^(d−1)-null set.This structural result assumes f∈BV(Ω) ∩ L∞(Ω).
- Total Generalised Variation: TGV extends TV to preserve edges while reducing staircasing, with its two parameters controlling regularisation strength and the interplay of its two terms.When λ0/λ1 is sufficiently high, TGV essentially behaves like TV.
- Continuous weights: Continuous weights can permit new jumps where the weight gradient jumps, while in one dimension the sign of DΛ′({x}) determines whether a jump or a flat neighbourhood appears.For DΛ′({x})>0, the jump size equals DΛ′({x}) under the stated support condition; for DΛ′({x})<0, the solution is locally constant.
- Discontinuous weights: Piecewise constant dyadic weights can yield structured adaptive models, but general existence of solutions remains open and the condition λ∈[c,1/c] is crucial for the cited existence result.The cited construction also has a minimum square length beyond which further partition refinement is not beneficial.
3 Discretisation and Bayesian interpretation
The chapter discretises weighted TV and TGV models and derives their Bayesian interpretation through spatially varying Gibbs priors. Under Gaussian noise, MAP estimation yields the corresponding variational reconstruction energies, while fixed weights determine local regularisation strength.
- Discretisation: The discretisation uses finite-difference operators for horizontal and vertical gradients and a corresponding symmetric gradient for vector fields.The image is represented as u∈R^N, with D=(D_h,D_v) and discrete symmetric-gradient components.
- Bayesian interpretation: Under an i.i.d. Gaussian noise model, MAP estimation combines the quadratic data-fidelity term with the negative log-prior.For fixed noise variance, the likelihood contributes 1/(2σ^2)||Au−f||_2^2, up to a constant independent of u.
- Bayesian interpretation: Weighted TV is modelled by local exponential potentials on gradient magnitudes with spatially varying rates, forming a non-stationary Markov random field rather than independent gradient variables.Neighbouring gradients share pixel values, so the local quantities are Gibbs potentials in the resulting image prior.
- Bayesian interpretation: Large TV weights concentrate local gradients near zero and impose stronger regularisation, whereas small weights permit larger gradients and favour edges and fine structures.The weights therefore encode spatially varying prior strength in the reconstruction model.
- Bayesian interpretation: Weighted TGV retains an auxiliary vector field and introduces analogous first- and second-order local Gibbs potentials for its two spatially varying weights.The resulting augmented prior on (u,w) has a negative log whose image-dependent part is the weighted TGV energy.
- Bayesian interpretation: For fixed weights, the normalisation term is independent of the unknown variables and can be omitted in MAP estimation, leaving the weighted TGV energy before minimisation over the auxiliary field.A hierarchical extension can instead treat the parameter maps as unknowns and assign hyperpriors, optionally with spatial regularity assumptions.
4 Practical computation of regularisation weights across regularity classes
The chapter contrasts bilevel and neural-network approaches for learning spatially varying regularisation weights, emphasizing computational cost, interpretability, and weight regularity. Experiments show that unrolled networks produce detailed weights whose noise adaptation differs between denoising and MRI reconstruction.
- Bilevel optimisation: Bilevel optimisation learns regularisation weights through nested upper- and lower-level problems, but repeated reconstructions create severe computational bottlenecks.These methods are therefore used mainly for denoising and simple 2D tasks.
- Neural network-unrolled approach: An unrolled network maps the data to spatially varying regularisation parameters, then performs a fixed number of model-based reconstruction iterations.The reconstruction remains interpretable as a variational solution with a classical regulariser.
- Neural network-unrolled approach: End-to-end supervised training requires data-ground-truth pairs rather than ground truths paired with optimal regularisation parameters.The network parameters are optimized so the resulting reconstruction approaches the ground truth.
- Regularity classes: Low-regularity weights use abrupt transitions and can adapt more closely to image details than high-regularity weights with smooth transitions.The chapter’s discrete examples compare bilevel H1-regularized weights with unrolled weights lacking imposed additional regularity.
- Numerical experiments: In denoising, independently inferred maps are highly adapted to each noise realization: cross-applying them produces significantly inferior reconstructions.The maps remain visually detailed and yield similar reconstructions when each is applied to its own noisy image.
- Numerical experiments: In MRI, inferred maps are more regular and cross-application yields very similar reconstructions, indicating limited adaptation to specific noise instances and undersampling masks.One proposed explanation is that the maps are learned from A* f_i, which dampens fine-scale noise details.
5 Conclusions
The chapter identifies unresolved theoretical questions about low-regularity learned weights, despite their practical advantages and apparently artifact-free numerical reconstructions. Future work should clarify their effects on discontinuities, approximation, existence, and noise adaptation across inverse problems.
- Low-regularity weights appear advantageous in practice, but their effect on reconstruction structure remains insufficiently understood.The authors specifically question whether less regular weights may create more discontinuities or spurious artifacts, although numerical examples do not appear to show them.
- In one-dimensional denoising, differentiable weights can make minimisers constant where the weight derivative is sufficiently large.The chapter proposes investigating whether analogous behavior occurs at abrupt jumps in nondifferentiable weights.
- Highly oscillatory weights may be unnecessary for best approximation in weighted TV, creating a tension with the chapter’s numerical experiments.The proposed explanation is a difference between the previously studied problem (P) and the continuously refined-weight problem (P̂).
- Whether continuously refining the weights in (P̂) is beneficial depends on whether, and under which conditions, that problem has a solution.If solutions do not exist, the authors suggest studying relaxed formulations over an augmented space containing suitable limits of piecewise constant functions.
- The differing noise adaptation observed in denoising and MRI requires rigorous explanation, potentially involving forward-operator ill-posedness and adjoint removal of fine-scale noise.The chapter leaves this relationship as an open direction rather than establishing the proposed mechanism.