Source-linked AI summary
Physics-Based Generative Adversarial Models for Image Restoration and Beyond
Jinshan Pan, Jiangxin Dong, Yang Liu, Jiawei Zhang, Jimmy Ren, Jinhui Tang, Yu-Wing Tai, Ming-Hsuan Yang
TL;DR
Image restoration is ill-posed, and feed-forward or unconstrained GANs may fail to preserve correct structures or satisfy the imaging physics. The paper proposes an end-to-end physics-constrained GAN that enforces consistency with observed inputs and performs favorably across restoration tasks and baselines.
Problem
Feed-forward models and unconstrained methods may produce incorrect structures because they estimate the mapping without ensuring consistency with the physics model.
Method
The method jointly trains a GAN with a physics-model constraint so restored outputs remain consistent with observed images across low-level vision tasks.
Results
The proposed method performs favorably against state-of-the-art algorithms on each task and generates clearer images than an unconstrained GAN baseline.
Takeaways & Limitations
Using the known physics model helps avoid trivial solutions while improving restoration quality, robustness to noise, and convergence relative to unconstrained GANs.
Takeaways & Limitations
The formulation assumes a corresponding image-degradation physics model is available to enforce consistency between recovered and observed images.
Abstract
from arXiv · showhide
We present an algorithm to directly solve numerous image restoration problems (e.g., image deblurring, image dehazing, image deraining, etc.). These problems are highly ill-posed, and the common assumptions for existing methods are usually based on heuristic image priors. In this paper, we find that these problems can be solved by generative models with adversarial learning. However, the basic formulation of generative adversarial networks (GANs) does not generate realistic images, and some structures of the estimated images are usually not preserved well. Motivated by an interesting observation that the estimated results should be consistent with the observed inputs under the physics models, we propose a physics model constrained learning algorithm so that it can guide the estimation of the specific task in the conventional GAN framework. The proposed algorithm is trained in an end-to-end fashion and can be applied to a variety of image restoration and related low-level vision problems. Extensive experiments demonstrate that our method performs favorably against the state-of-the-art algorithms.
1 INTRODUCTION
Image restoration is ill-posed because estimating a clear image from an observed input requires constraints beyond a learned inverse mapping. The paper proposes enforcing the physics model within a GAN so outputs remain consistent with inputs while producing realistic restoration results.
- Problem: Restoration tasks estimate a clear image from an observed input under an image formation model.Examples include deblurring, super-resolution, dehazing, and deraining.
- Problem: Because estimating the unknown image is ill-posed, additional constraints are needed to regularize the solution.MAP methods use likelihood and image-prior terms for this purpose.
- Problem: Feed-forward inverse mappings can fail because the solution space is large and random initialization does not adequately estimate the mapping.The resulting restoration may not be sufficiently accurate.
- Motivation: Adversarial losses encourage realistic outputs but do not ensure that generated contents remain consistent with the inputs.Prior GAN-based restoration methods therefore may produce incorrect structures and details.
- Contribution: The method is motivated by enforcing consistency between restored results and observed inputs under the degradation process.The figure caption states that this constraint can produce better results than using no constraint and applies across several tasks.
- Contribution: The proposed method constrains a GAN with the physics model and jointly trains it end to end for restoration and low-level vision tasks.The physics constraint requires the estimated result G(y) to remain consistent with the observed image y.
2 RELATED WORK
Related work applies GANs and end-to-end neural networks to low-level vision, including restoration and blind deblurring. However, existing approaches may not ensure that generated results satisfy the image formation process.
- Generative adversarial networks: GANs have been used for vision tasks, including image deblurring, dehazing, and related low-level vision problems.The paper positions its method within this expanding GAN literature.
- Image deblurring: Blind deblurring methods either estimate blur kernels before conventional restoration or directly restore clear images with end-to-end networks.The latter avoids explicit kernel estimation but may still violate the image formation process.
3 IMAGE RESTORATION WITH GAN
GANs provide an adversarial prior for image restoration, but unconstrained models do not guarantee consistency with the degradation model. The paper frames restoration as combining data fidelity under the physics model with adversarial regularization.
- GAN formulation: GAN training alternates a generator that produces samples and a discriminator that distinguishes generated images from real data.The generator seeks to fool the discriminator while the discriminator performs binary classification.
- GAN formulation: When the observed image y conditions the generator, the adversarial loss encourages G(y) to have a distribution closer to clear images.The loss value is larger when generated and clear-image distributions are more similar.
- Restoration objective: The MAP formulation combines a data term enforcing consistency with the input degradation model and a regularizer modeling image properties.The data term and image prior correspond to likelihood and prior information.
- Restoration objective: The regularizer can act like a discriminator that favors clear images, allowing adversarial loss to regularize the restoration solution space.This interpretation connects adversarial learning with the image prior in restoration.
- Motivation: Unconstrained GANs may generate outputs inconsistent with the image degradation model because they do not account for the data term.The paper motivates adding a physics-based constraint to address this issue.
4 PROPOSED ALGORITHM
The proposed algorithm combines a generative network, two discriminators, and an image degradation model to enforce both realistic restoration and consistency with observed inputs. It is trained end-to-end and can accommodate task-specific physics models and existing network architectures.
- The algorithm uses a generator, two discriminators, and an image degradation model to constrain restoration outputs.Dg evaluates closeness to ground-truth image distributions, while Dh evaluates consistency between regenerated and observed images.
- The generator first produces an intermediate restored image, which is passed through the degradation model before consistency is evaluated.For deblurring, the generator maps blurred input yi to an intermediate deblurred image G(yi), followed by the image degradation model.
- Task-specific physics models support dehazing, super-resolution, and other vision applications.The paper gives explicit dehazing and super-resolution models and states that corresponding physics models can be used for other tasks.
- The networks are jointly trained end-to-end, but training uses a known blur kernel and is therefore not fully blind.The learned generator can be applied directly to input images at test time, whereas the training-stage physics constraint uses known blur information.
- Existing generative and discriminative network architectures can be used once the physics model is specified for a task.The implementation uses a CycleGAN-like generator and PatchGAN discriminators, while Table 1 lists the network configurations.
- The objective combines adversarial, pixel-wise, stability, and physics-consistency terms with weighted parameters.The least square GAN loss is used for the adversarial term, while λ, γ, and µ weight components of the objective.
5 EXPERIMENTAL RESULTS
Experiments evaluate the physics-constrained GAN across deblurring, dehazing, super-resolution, and deraining. Across these tasks, the method performs favorably against state-of-the-art methods and produces clearer, more structurally faithful results.
- Experimental scope: The proposed algorithm is evaluated on image deblurring, dehazing, super-resolution, and deraining tasks.The experiments include synthetic and real-image settings across these restoration problems.
- Image deblurring: On text-image deblurring, the proposed method produces clearer images with recognizable characters than conventional, CNN-based, and CycleGAN methods.The comparison uses retrained competing methods and evaluates the text-image dataset with PSNR and SSIM.
- Image deblurring: On face-image deblurring, the proposed algorithm performs favorably against statistical-prior and deep-neural-network methods and effectively deblurs heavily blurred faces.Figure 4 reports fewer artifacts, while Table 3 compares PSNR and SSIM against state-of-the-art methods.
- Image deblurring: On natural-image deblurring, the method performs well on synthetic tests, dynamic scenes, and real blurred images, producing clearer outputs with sharper characters.The method is trained on uniform blur but also performs well when dynamic-scene blur is approximated locally as uniform blur.
- Image dehazing: For dehazing, the image degradation model constraint helps preserve structures and produce clearer images with fine details, while the method performs favorably on an outdoor dataset and real hazy images.The method avoids estimating the transmission map and atmospheric light and produces more vivid restored-image colors on real hazy images.
- Super-resolution and deraining: For super-resolution and deraining, the proposed method applies the physics-constrained framework to new degradation models and removes rainy streaks while generating clearer images.The super-resolution evaluation uses Set5 and compares against SRCNN, ESPCN, VDSR, SRGAN, and EDSR.
6 ANALYSIS AND DISCUSSION
The analysis shows that physics-model constraints improve restoration quality, convergence, and robustness relative to unconstrained GAN or inverse-mapping approaches, while performance remains limited by complex mixed degradations.
- Effect of the image degradation model constraint: Without the image degradation model constraint, BaseGAN produces unclear images whose restored structures differ substantially from the input.The proposed constrained method instead generates much clearer images on the synthetic hazy-image evaluation.
- Loss-function ablation: Removing any of the proposed loss functions reduces effectiveness, whereas using them enables clearer restored images.This comparison is reported for the ablation in Table 8 and Figure 11(c)-(f).
- Relation with GAN-based methods: The method avoids the trivial solutions associated with CycleGAN and performs favorably against state-of-the-art algorithms on each task.Because the physics model is known, the method does not need to learn the additional generator F used by CycleGAN.
- Robustness and convergence: The method performs well under noise levels ranging from 0% to 10% and converges well within 200 epochs in terms of PSNR.These findings are reported on clear text images with added random noise and on a text-image deblurring dataset, respectively.
- Robustness and convergence: Using the physics model makes training more stable and improves convergence compared with a GAN using one discriminator and one generator, despite increased training complexity from multiple discriminators.The sensitivity study also reports good performance for λ in [40, 100], with slightly better results at λ=50, and insensitivity to the number of ResBlocks.
- Limitations: The method is less effective when multiple degradation factors, such as rain and haze, occur because the physics model does not describe the complex formation process well.An example shows failure to remove rain or snow from the input image.
7 CONCLUDING REMARKS
The paper concludes that enforcing the physical formation process within an end-to-end GAN provides a broadly applicable approach for image restoration and low-level vision, with favorable performance against state-of-the-art methods.
- Conclusion: The proposed end-to-end physics-constrained GAN applies across image restoration and low-level vision tasks and performs favorably against state-of-the-art methods.Its applicability follows from deriving the physics model from the image formation process of each low-level problem.