PaperlayerView exact revision on arXiv

Source-linked AI summary

PUGAN: Physical Model-Guided Underwater Image Enhancement Using GAN with Dual-Discriminators

Runmin Cong, Wenyu Yang, Wei Zhang, Chongyi Li, Chun-Le Guo, Qingming Huang, Sam Kwong

arXiv:2306.08918v2eess.IVcs.CV

TL;DR

Underwater images are degraded by water-induced absorption and scattering, while existing GAN and physical-model approaches offer complementary strengths. PUGAN combines learned physical-model inversion, degradation-guided CNN enhancement, and style-content dual discrimination, with experiments reporting superiority across benchmark evaluations.

  • Problem

    Underwater absorption and scattering degrade image quality, while GAN-based and physical-model methods provide complementary advantages that motivate their combination.

  • Method

    PUGAN learns physical-model parameters to generate a color-enhanced image, uses it to guide a DQ-equipped TSIE-subnet, and applies Dual-D style-content adversarial constraints.

  • Results

    PUGAN's extensive experiments on different benchmarks demonstrate the superiority of the method and the effectiveness of each module.

  • Takeaways & Limitations

    The framework combines physical inversion, regional differential feature learning, and style-content synergy within one GAN-based enhancement model.

Abstract

from arXiv · show

Due to the light absorption and scattering induced by the water medium, underwater images usually suffer from some degradation problems, such as low contrast, color distortion, and blurring details, which aggravate the difficulty of downstream underwater understanding tasks. Therefore, how to obtain clear and visually pleasant images has become a common concern of people, and the task of underwater image enhancement (UIE) has also emerged as the times require. Among existing UIE methods, Generative Adversarial Networks (GANs) based methods perform well in visual aesthetics, while the physical model-based methods have better scene adaptability. Inheriting the advantages of the above two types of models, we propose a physical model-guided GAN model for UIE in this paper, referred to as PUGAN. The entire network is under the GAN architecture. On the one hand, we design a Parameters Estimation subnetwork (Par-subnet) to learn the parameters for physical model inversion, and use the generated color enhancement image as auxiliary information for the Two-Stream Interaction Enhancement sub-network (TSIE-subnet). Meanwhile, we design a Degradation Quantization (DQ) module in TSIE-subnet to quantize scene degradation, thereby achieving reinforcing enhancement of key regions. On the other hand, we design the Dual-Discriminators for the style-content adversarial constraint, promoting the authenticity and visual aesthetics of the results. Extensive experiments on three benchmark datasets demonstrate that our PUGAN outperforms state-of-the-art methods in both qualitative and quantitative metrics.

I. INTRODUCTION

Underwater image enhancement is difficult because absorption, scattering, and complex imaging conditions cause color distortion, blur, and other degradation. PUGAN combines physical modeling with GAN-based learning, using learned physical parameters, degradation-aware enhancement, and dual adversarial constraints.

  • Underwater absorption and scattering produce color distortion, blurring, and difficult imaging conditions that hinder downstream ocean-related tasks.
  • Traditional non-physical methods rely on handcrafted pixel adjustments and can cause over- or under-enhancement, while physical models improve interpretability and address color distortion.
  • GANs provide strong learning ability but limited generalization, motivating a framework that combines GAN learning with physical underwater imaging models.
  • PUGAN uses a Phy-G containing the Par-subnet for physical-parameter estimation and the TSIE-subnet for CNN-based enhancement guided by a color-enhanced image.
  • The DQ module quantifies scene distortion to guide targeted encoder-feature reinforcement, while Dual-D imposes style-content adversarial constraints on enhanced images.
  • PUGAN is reported to outperform state-of-the-art methods qualitatively and quantitatively on three benchmark datasets.

B. Learning-based UIE method

Learning-based UIE methods improve enhancement but often omit the physical imaging model, while prior physical-model methods can depend on limited or fragile priors. PUGAN learns physical parameters, uses physical inversion to guide CNN enhancement, and applies style-content adversarial constraints.

  • Deep learning has improved underwater enhancement, but GAN-based methods and other learning-based approaches often do not account for the physical model.
  • Recent methods embed physical models into deep networks to emphasize degraded regions or separate color correction and dehazing sub-tasks.
  • PUGAN physically inverts a learned color-enhanced image and feeds it as guidance into a secondary CNN enhancement stage with degradation quantization.
  • The Phy-G estimates transmission and attenuation parameters, while TSIE uses the color-enhanced image and transmission map to produce the final enhanced image.
  • Dual-D constrains overall style through one discriminator and structural-content authenticity through another discriminator incorporating the depth map.
  • The framework overview combines Phy-G and Dual-D under a GAN architecture with global similarity, perceptual, style adversarial, and content adversarial losses.

A. Physical Model-Guided Generator

PUGAN’s Physical Model-Guided Generator combines a learned physical-model inversion stage with CNN-based enhancement. Its Par-subnet estimates attenuation, depth, and transmission to produce a color-enhanced image that guides subsequent processing.

  • The Phy-G generator is a two-stage UIE network comprising the Par-subnet and TSIE-subnet.
  • Physical model: The physical imaging model relates observed image I, restored image J, background light A, and transmission map t; t describes unscattered light reaching the camera.
  • Physical model: The model describes attenuation as exponential with scene depth, so depth reflects scene attenuation.
  • Physical model inversion: Model inversion uses estimated transmission and attenuation to generate a color-enhanced image, with one term correcting color and another removing background-light influence.
  • Physical model inversion: The estimated transmission map and original image are used together to restore the color-enhanced underwater image through model inversion.
  • Par-subnet: The Par-subnet estimates attenuation coefficient β and depth d, which are used to compute the transmission map t.

2) Two-Stream Interaction Enhancement Subnetwork (TSIE-subnet):

TSIE-subnet uses the original and color-enhanced underwater images in a two-stream encoder, then applies degradation-guided feature fusion and reinforcement before one-stream decoding. Its DQ module combines color-enhancement differences and transmission-map cues to target severely degraded regions.

  • Two-stream architecture: The color-enhanced image provides an interpretable physical-model inversion result that guides subsequent CNN-based re-enhancement.The first stage estimates physical-model parameters and generates the color-enhanced image; the second stage uses it as auxiliary information.
  • Degradation quantization: The DQ module adaptively locates and quantifies severely degraded regions using the transmission map and differences between color-enhanced and original features.The transmission map identifies degradation-prone regions, while feature differences indicate how much information should be supplemented.
  • Two-stream architecture: The two-stream architecture feeds the original and color-enhanced images into separate encoders, whose features are fused before decoding.The encoders extract multilevel features, while the decoder is symmetric to the encoder and produces the final enhanced image.
  • Degradation quantization: A thresholded feature-difference signal and transmission-derived degradation measure determine regional weights for encoder-feature reinforcement.The threshold α is set to 0.7, and larger degradation values receive stronger enhancement during decoding.
  • Feature decoding: The updated features are decoded through convolution-residual blocks with upsampling to generate the final enhanced underwater image.The decoder uses 3 × 3 convolutional layers, upsampling, ReLU activation, and residual operations.

B. Dual-Discriminators (Dual-D)

The Dual-Discriminators structure uses adversarial loss to constrain generated underwater images toward realistic and clear results.

  • Adversarial discrimination: The discriminator evaluates whether the generated enhanced underwater image is real or fake, driving further generator optimization when it fails to appear real.This process is constrained by an adversarial loss.

D LGAN(G, D)

PUGAN’s Dual-D structure jointly constrains image style and content through two discriminators, using patch-level judgments to emphasize local texture and regional enhancement.

  • Style-content synergy: Dual-D jointly constrains the generated image’s overall style authenticity and content-sensitive regional enhancement.Discriminator1 evaluates overall style, while Discriminator2 addresses differences in perceptual importance across image regions.
  • Patch-level discrimination: Patch-based discrimination is chosen because whole-image scoring may overlook content-dependent enhancement differences across regions.The method adopts Markovian PatchGAN rather than focusing only on the global image.
  • Patch-level discrimination: Markovian PatchGAN classifies each N × N patch as real or fake, helping capture local texture and style with fewer parameters and faster execution.For a 256×256×c input, four convolution layers produce a 16×16×1 output whose average is the discriminator response.
  • Style-content adversarial loss: The style-content adversarial loss uses separate terms for the two discriminators and compares generated and ground-truth images, including their estimated depth maps.D1 and D2 denote the style and content discriminators, while dE and dY are estimated depth maps from generated and real enhancement images.

C. Training Strategy and Loss Function

PUGAN is trained in two stages: Par-subnet pretraining is followed by joint training of the fixed Par-subnet, TSIE-subnet, and Dual-D. The final objective combines global similarity, perceptual, and style-content adversarial losses.

  • Training strategy: The first stage pre-trains Par-subnet offline, after which its parameters are fixed during second-stage PUGAN training.TSIE-subnet and Dual-D are trained alternately in the second stage.
  • Training strategy: Par-subnet pretraining uses 200 synthetic images spanning five open-ocean and five coastal water types with depth and attenuation-coefficient labels.Each model stage is trained separately for 60 epochs with batch size 4 and learning rate 1e−4.
  • Training strategy: Second-stage training uses 800 paired UIEB images and 1250 synthetic underwater images.These data are used after Par-subnet pretraining is completed.
  • Loss function: The final loss combines global similarity, perceptual, and adversarial objectives to improve visual pleasantness while maintaining image authenticity.The adversarial terms separately represent style and content constraints, with scaling factors controlling the contributions of the loss components.

A. Implementation Details

PUGAN is evaluated on three real underwater-image benchmarks using reference and non-reference metrics, alongside traditional and deep learning-based methods. It ranks first on PSNR and MSE across all three testing datasets, while traditional methods lead several no-reference metrics.

  • Datasets: PUGAN is independently evaluated on UIEB, UFO-120, and EUVP, each divided into training and testing datasets.The UIEB split uses 800 image pairs for training and 90 pairs for testing.
  • Evaluation metrics: Quantitative evaluation uses PSNR and MSE, while non-reference metrics assess colorfulness, sharpness, contrast, and fog density.Higher non-reference scores indicate better image quality in the stated evaluation convention.
  • Compared methods: The comparison includes six non-learning-based methods and eight deep learning-based methods.The listed baselines include GDCP, ACDE, HLRP, MLLE, UNTV, SPDF, FUnIE-GAN, WaterNet, UWCNN, JI-Net, ACPAB, TOPAL, and Ucolor.
  • Reference metrics: PUGAN ranks first in PSNR and MSE on Test-UIEB, Test-UFO, and Test-EUVP.Compared with the second-best PSNR competitor, its percentage gains are 5.1%, 2.1%, and 2.2%, respectively; its minimum MSE gains are 30.8%, 15.8%, and 2.9%.
  • Reference metrics: Traditional methods generally trail deep learning-based methods on PSNR and MSE because of the limited representational power of hand-crafted features.The statement is based on average results reported for Test-UIEB, Test-UFO, and Test-EUVP.
  • No-reference metrics: ACDE leads UIQM, GDCP leads FDUM, and HLRP leads UICQE and CCF across the three datasets.Deep learning methods have no overall advantage on these no-reference metrics; PUGAN ranks second for UICQE on Test-UIEB.

2) Qualitative Evaluation:

Qualitative comparisons show PUGAN producing results closer to ground truth across color, brightness, definition, complex scenes, and low-contrast or low-light conditions. Ablations further support the roles of physical-model guidance, DQ, two-stream interaction, and both discriminators.

  • Qualitative comparison: PUGAN is reported as closer to ground truth in color, brightness, and definition than the comparison methods.The qualitative comparisons are presented in Fig. 6.
  • Color distortion correction: PUGAN corrects greenish and bluish distortions more naturally than most comparison methods, which can retain or introduce color casts.The examples include greenish scenes and cases where GDCP, HLRP, or MLLE produce green, red, or nearly colorless results.
  • Complex scenes: In a cluttered fish scene with varied colors and depths, PUGAN is less wrong-colored, blurry, or hazy than the cited comparison methods.GDCP, HLRP, and ACPAB distort fish colors, while SPDF, WaterNet, and Ucolor produce blurry or hazy results.
  • Low contrast and Low light: PUGAN better handles low contrast and low light, whereas several comparison methods show insufficient enhancement or introduce additional distortion.The passage specifically links forward scattering with low contrast and describes the need to enlarge contrast while removing blur.
  • Par-subnet ablation: Deepening the depth estimator improves the color-enhanced image MSE from 1.74 to 0.93, but better depth quality does not necessarily improve the final enhancement.The paper attributes this difference to weak depth-to-final-output correlation and the second-stage network’s fault tolerance.
  • TSIE-subnet ablation: The full two-stream TSIE architecture outperforms single-stream variants, and adding J′ as guidance further improves performance.Single-stream variants are described as inferior quantitatively, with one producing blur and another dispersion visually.
  • DQ module ablation: Replacing DQ with simple concatenation reduces performance, while retaining only one DQ branch remains below the full model and can introduce artifacts or reddish results.The DQ module combines feature differences and transmission characteristics to reinforce seriously degraded regions.
  • Dual-D ablation: Removing either discriminator negatively affects evaluation metrics and visual quality, supporting the Dual-Discriminators design.The ablation labels identify the variants as removing Discriminator1 or Discriminator2.

D. Discussions

PUGAN achieves the best reported performance among compared methods but has a large model size, and it struggles with severe turbidity and uneven lighting.

  • PUGAN achieves the best performance among the compared algorithms, with a model size of 660 MB and testing time of 0.14s per image.Ucolor is second in overall performance and has a 616 MB model size.
  • The model is computationally heavy because its two-stage design combines parameter estimation with correction enhancement and uses resource-intensive UNet components.The paper specifically identifies skip-connections and linear layers as major resource consumers.
  • The authors suggest lightweight backbones, removal of redundant parameters, edge refinement, and brightness correction as future improvements.Fully connected layers are identified as especially relevant targets for parameter reduction.
  • PUGAN does not work well when degradation is severe, including turbid water and uneven lighting.These conditions can make edges and details difficult to observe and can produce uneven enhancement.
  • Turbid scenes may produce unreasonable color changes, while uneven lighting may cause over-enhancement in relatively bright regions.The authors attribute the lighting issue partly to the lack of a specifically designed lighting factor.

V. CONCLUSION

The paper proposes PUGAN, which combines physical inversion with CNN-based enhancement and uses a DQ-equipped interaction subnet plus dual discriminators. Experiments on different benchmarks support the method’s superiority and the effectiveness of its modules.

  • PUGAN combines a physical model and a CNN-based model within a GAN framework for underwater image enhancement.Its Phy-G uses physical inversion to generate a color-enhanced image before CNN-based enhancement.
  • The Par-subnet generates a color-enhanced image through physical inversion, while the TSIE-subnet uses regional and differential feature learning to produce the final result.The TSIE-subnet is equipped with a DQ module.
  • The Dual-D structure uses a style-content synergy mechanism to judge generator reconstruction results.The design uses two independent discriminators to support visual aesthetics and content authenticity.
  • Experiments on different benchmarks demonstrate PUGAN’s superiority and the effectiveness of each module.
Loading 2306.08918v2…