PaperlayerView exact revision on arXiv

Source-linked AI summary

Adaptive deep learning framework for robust unsupervised underwater image enhancement

Alzayat Saleh, Marcus Sheaves, Dean Jerry, Mostafa Rahimi Azghadi

arXiv:2212.08983v3cs.CV

TL;DR

Underwater enhancement is difficult to train with supervision because high-quality paired images are limited and underwater degradation complicates image processing. UDNet combines probabilistic uncertainty modeling with adaptive reference maps in an unsupervised framework. The authors report that it outperforms ten state-of-the-art methods across seven metrics and eight datasets, while noting remaining challenges from backscatter and environmental variability.

  • Problem

    Paired, high-quality underwater training data are limited, while distortion, low contrast, and wavelength-dependent absorption make enhancement challenging.

    Source: §1 · pp. 1–2

  • Method

    UDNet is an unsupervised framework that combines probabilistic uncertainty modeling, SGMCSS-generated reference maps, and PAdaIN for adaptive enhancement.

    Source: §1 · pp. 2–3

  • Results

    UDNet outperforms ten state-of-the-art underwater enhancement methods across seven metrics and eight datasets.

    Source: §6 · p. 23

  • Takeaways & Limitations

    The authors report competitive quantitative and qualitative performance across eight public datasets without relying on ground truth data.

    Source: §5 · p. 23

    Backscatter, occasional unrealistic enhancements from statistical modeling, and environmental variability can limit clarity and generalization across scenarios.

    Source: §5 · p. 23

Abstract

from arXiv · show

One of the main challenges in deep learning-based underwater image enhancement is the limited availability of high-quality training data. Underwater images are difficult to capture and are often of poor quality due to the distortion and loss of colour and contrast in water. This makes it difficult to train supervised deep learning models on large and diverse datasets, which can limit the model's performance. In this paper, we explore an alternative approach to supervised underwater image enhancement. Specifically, we propose a novel unsupervised underwater image enhancement framework that employs a conditional variational autoencoder (cVAE) to train a deep learning model with probabilistic adaptive instance normalization (PAdaIN) and statistically guided multi-colour space stretch that produces realistic underwater images. The resulting framework is composed of a U-Net as a feature extractor and a PAdaIN to encode the uncertainty, which we call UDnet. To improve the visual quality of the images generated by UDnet, we use a statistically guided multi-colour space stretch module that ensures visual consistency with the input image and provides an alternative to training using a ground truth image. The proposed model does not need manual human annotation and can learn with a limited amount of data and achieves state-of-the-art results on underwater images. We evaluated our proposed framework on eight publicly-available datasets. The results show that our proposed framework yields competitive performance compared to other state-of-the-art approaches in quantitative as well as qualitative metrics. Code available at https://github.com/alzayats/UDnet .

1 INTRODUCTION

UDNet addresses the difficulty of enhancing underwater images when paired, high-quality training data are scarce. It combines uncertainty modeling and adaptive reference generation, with reported state-of-the-art performance across multiple datasets.

1 INTRODUCTION

  • Wavelength-dependent absorption and scattering cause colour casts and other degradation, making underwater images difficult to process and analyze.Blue light travels farther underwater than other wavelengths, contributing to the blue appearance of underwater objects.Source: §1 · p. 1, Figure 1 · p. 2
  • Supervised enhancement depends on paired underwater and ground-truth images that are challenging and costly to acquire, while existing methods may generalize poorly.Traditional methods also struggle with diverse underwater conditions, and supervised models may overfit specific domains.Source: §1 · pp. 1–2
  • UDNet is an unsupervised framework that learns without ground-truth or manually annotated images, using probabilistic uncertainty modeling.Its SGMCSS module generates diverse reference maps, and PAdaIN adapts to varying image characteristics.Source: §1 · pp. 2–3
  • Unlike pixel-level uncertainty approaches such as (Fu et al., 2022b), UDNet models distributional uncertainty through adaptive references and probabilistic feature normalization.The authors state that this design generalizes across varied underwater environments without paired training data.Source: §1 · p. 2
  • UDNet outperforms or matches existing supervised and unsupervised methods, with strong generalizability across multiple underwater datasets.The introduction reports state-of-the-art results in both qualitative and quantitative evaluations.Source: §1 · pp. 2–3

2 RELATED WORK

Related work spans physical-prior, model-free, deep-learning, and probabilistic approaches to underwater enhancement. The paper positions UDNet as extending uncertainty-based methods with adaptive reference selection and probabilistic feature normalization.

2.1 Prior-Based Methods

  • Prior-based methods reverse estimated optical parameters using physical image-formation models, but their assumptions may fail in complex underwater environments.Examples include red-channel, underwater dark-channel, and light-attenuation priors.Source: §2.1 · p. 3

2.2 Model-Free Methods

  • Model-free methods improve visibility through intensity redistribution or feature enhancement, but can struggle with diverse underwater lighting, scattering, and turbidity.Examples include CLAHE, white balance, and Retinex-based methods.Source: §2.2 · p. 3

2.3 Deep Learning-Based Methods

  • Deep-learning methods learn enhancement representations from training data, but commonly rely on paired datasets and may remain limited to specific training domains.The review identifies CNN- and GAN-based approaches among these methods.Source: §2.3 · pp. 3–4

2.4 Probabilistic-Based Methods

  • UDNet extends probabilistic underwater enhancement beyond pixel-level uncertainty by randomly selecting references and dynamically aligning latent feature distributions.The authors contrast this with (Fu et al., 2022b), which focuses on pixel-wise uncertainty, and report improved generalization across diverse datasets.Source: §2.4 · p. 4

3 METHOD

UDNet is organized as three abstract building blocks: reference-map generation, feature extraction, and probabilistic adaptive instance normalization.

3 METHOD

  • A statistically guided multi-colour space stretch module generates the reference maps used by UDnet.Source: §3 · p. 4
  • A conditional variational autoencoder forms the feature-extractor block.Source: §3 · p. 4
  • Probabilistic adaptive instance normalization is the third abstract building block in the framework.Source: §3 · p. 4

3.1 Uncertainty Distribution

UDNet treats enhancement as uncertain because different images can require different corrections and a clean target may be unavailable. It uses randomly selected enhanced references and probabilistic feature modeling to generate outputs that account for this uncertainty.

3.1 Uncertainty Distribution

  • The enhancement target is ambiguous: images may need different contrast, saturation, or gamma adjustments, while the true clean image is often unavailable.The authors contrast this with deterministic methods that must compromise among possible enhancement results.Source: §3.1 · p. 4
  • During training, UDNet randomly selects among enhanced reference maps rather than relying on a fixed set of labels.The references are generated through contrast, saturation, or gamma correction; the implicit variable z represents enhancement uncertainty.Source: §3.1 · pp. 4–5
  • UDNet models the clean-image distribution conditioned on the input and a maximum-probability uncertainty sample drawn from p(z|x).The model learns these distributions and uses them to generate enhanced images.Source: §3.1 · p. 5
  • The SGMCSS module creates statistically guided reference maps that feed a U-Net-like cVAE, whose PAdaIN block encodes feature uncertainty for image generation.The cVAE is trained with reconstruction and KL-divergence losses, then samples its learned latent prior during testing.Source: Figure 2 · p. 5, §3.1 · p. 6

3.2 Comparison with Uncertainty-Inspired Underwater Image Enhancement

The section compares UDNet with the uncertainty-inspired method of Fu et al. (2022b), noting that both use uncertainty but differ in how they model, leverage, and learn from it.

3.2 Comparison with Uncertainty-Inspired Underwater Image Enhancement

  • The section sets up a comparison of UDNet with Fu et al. (2022b)’s uncertainty-inspired underwater image enhancement method.The comparison focuses on how the two methods model, leverage, and learn from uncertainty.Source: §3.2 · p. 6
  • Both methods utilize uncertainty to enhance underwater images.Source: §3.2 · p. 6
  • Their approaches to modeling, leveraging, and learning from uncertainty are markedly different.Source: §3.2 · p. 6

Similarities

UDNet and Fu et al. (2022b)’s method both use uncertainty in underwater image enhancement, motivated by variability across underwater conditions.

Similarities

  • Both UDNet and Fu et al. (2022b)’s method incorporate uncertainty modeling for underwater image enhancement.Source: Similarities · p. 6
  • The shared rationale is that enhancement is uncertain under varied lighting, water turbidity, and diverse underwater environments.Source: Similarities · p. 6
  • Both approaches recognize these environmental factors as sources of inherent uncertainty in enhancement.Source: Similarities · p. 6

Differences

UDNet models distributional uncertainty through adaptive reference selection and probabilistic feature normalization, whereas Fu et al. (2022b) model pixel-level uncertainty. Their reference-map generation and learning approaches also differ.

Differences

  • UDNet models distributional uncertainty through adaptive reference selection and probabilistic feature normalization, while Fu et al. (2022b) model pixel-level uncertainty with learned uncertainty maps.Fu et al.’s maps refine enhanced images and quantify uncertainty per pixel.Source: Differences · p. 6
  • UDNet generates reference maps with SGMCSS, adjusting input contrast and saturation and applying gamma correction; Fu et al. use references from existing enhancement algorithms.Source: Differences · p. 6
  • UDNet learns unsupervised from randomly selected, statistically guided references without paired or manually annotated data, whereas Fu et al. require manually created references for supervised learning.Source: Differences · p. 6

3.3 Reference Maps Generation

The framework addresses scarce reference maps by generating uncertain pseudo-labels without human intervention, then improving their colour and contrast with a statistically guided multi-colour-space module.

3.3 Reference Maps Generation

  • Limited reference maps for degraded inputs motivate generating training references from UIEBD images and their algorithm-produced reference maps.UIEBD contains real-world underwater images and references generated by 12 state-of-the-art enhancement algorithms.Source: §3.3 · p. 6

3.3.1 Autogenration of three reference maps from the input image

  • Three enhanced references are generated from each input, with one randomly selected to introduce uncertainty; adding more references did not increase model accuracy.The selection uses the same intuition as volunteer selection in UIEBD but without human intervention.Source: §3.3.1 · pp. 6–7
  • The three reference-generation adjustments are contrast, saturation, and gamma correction, chosen to simulate underwater changes in contrast, saturation, brightness, and colour.For contrast adjustment α is constant across pixels; for saturation adjustment it varies by pixel.Source: §3.3.1 · p. 7

3.3.2 Statistically Guided Multi-Colour Space Stretch for Colour Correction

  • SGMCSS improves pseudo-label visual quality by processing images in RGB, HSI, and Lab spaces, while preserving hue, intensity/lightness, or colour-opponent channels as described for each space.RGB channels are stretched individually; HSI stretches S and I while preserving H, and Lab stretches a and b while maintaining L.Source: §3.3.2 · pp. 7–8
  • SGMCSS combines average- and maximum-statistic colour balancing with residual enhancement, then merges the colour-space outputs to correct balance and enhance contrast.Its non-parametric colour correction is designed for known RGB means and standard deviations and can accommodate new pixel-value distributions.Source: §3.3.2 · pp. 7–8

3.4 Feature Extraction

UDnet uses a two-branch U-Net feature extractor and probabilistic distributions to model enhancement uncertainty. Its training objective combines reference-based enhancement loss with KL alignment of posterior and prior distributions.

3.4 Feature Extraction

  • The feature extractor feeds image representations to PAdaIN, which transforms their enhancement statistics to create the enhanced image.The posterior distributions are modeled as normal distributions whose mean and variance depend on the input image and reference map.Source: §3.4 · pp. 8–9
  • A two-branch U-Net builds posterior distributions from the raw image and reference map during training, while the test branch estimates a prior from the raw image alone.The mean and standard-deviation distributions represent uncertainty in possible enhancements.Source: §3.4 · p. 8
  • The probabilistic distributions let UDnet model input uncertainty and generate enhanced images consistent with the input and reference map, without manual annotation or large amounts of data.During training, random samples from the learned distributions are used to predict an enhanced image, and the enhancement loss penalizes deviation from the reference map.Source: §3.4 · p. 9, §3.4.1 · p. 9
  • At test time, latent codes are determined only by the input image, and repeated PAdaIN and output-block evaluations generate variants for maximum-probability selection.The model need not be retrained to generate these variants.Source: §3.4 · p. 9

3.4.1 Loss Function

  • The total training loss combines enhancement loss with KL divergences between posterior and prior distributions, supporting enhanced-image mapping while incorporating uncertainty.The enhancement loss uses mean square error and weighted VGG16 perceptual loss; β weights the total objective’s KL terms.Source: §3.4.1 · pp. 9–10

3.5 Probabilistic Adaptive Instance Normalization (PAdaIN)

PAdaIN uses learned uncertainty distributions to adapt instance normalization for underwater enhancement, generating multiple possible appearances without known content and style images. Its adaptive uncertainty modeling differs from static uncertainty maps and is intended to support generalization across diverse datasets.

3.5 Probabilistic Adaptive Instance Normalization (PAdaIN)

  • PAdaIN uses uncertainty distributions learned during training to encode feature uncertainty and generate multiple enhanced versions reflecting ambiguity in underwater enhancement.The paper describes PAdaIN as a modified AdaIN algorithm for underwater image enhancement.Source: §3.5 · p. 10
  • Unlike Fu et al. (2022b), UDnet randomly selects reference images during training rather than using static uncertainty maps, with the stated aim of improving robustness across diverse underwater environments.The paper also says PAdaIN accounts for latent-space variability to generalize across datasets with diverse characteristics.Source: §3.5 · p. 10
  • PAdaIN uses random samples from posterior mean and standard-deviation distributions as AdaIN parameters, allowing enhancement without known content and style images.The posterior distributions are learned using a cVAE.Source: §3.5 · p. 10

4 EXPERIMENTS

The experiments evaluate UDnet on public underwater-image datasets using quantitative, qualitative, visual-perception, and ablation tests. They also examine model-component effects, including PAdaIN’s contribution to enhancement quality.

4.1 Datasets

  • UIEBD was the only training dataset, used without ground-truth labels; the other seven of the eight public datasets were reserved for evaluation.EUVP, UFO, and UIEBD include paired and unpaired images, while the remaining datasets contain only unpaired images.Source: §4.1 · p. 11

4.2 Evaluation Metrics

  • Evaluation uses PSNR, SSIM, MAD, and GMSD on paired test sets and UIQM, MUSIQ, and NIQE on unpaired sets, with no-reference scores treated as indicative rather than definitive.Higher PSNR, SSIM, UIQM, and MUSIQ and lower MAD, GMSD, and NIQE indicate better scores under the stated metric definitions.Source: §4.2 · pp. 11–12

4.3 Implementation Details

  • UDnet was trained for around 500 epochs with batch size 10 and Adam, using MSE, VGG perceptual, and KL-divergence losses; inference averaged 0.03 seconds per image.Training used 256 × 256 crops and a learning rate of 1 × 10−4; the inference-time passage notes that speed depends on image size, complexity, and hardware.Source: §4.3 · p. 12

4.5 Quantitative Comparisons

  • Against 10 prior methods, UDnet leads all four full-reference metrics on EUVP, performs strongly on UFO, and exceeds supervised models on EUVP and UFO despite unsupervised training.The paper also reports the highest PSNR and SSIM on EUVP and the lowest MAD and GMSD on EUVP and UFO; it notes slightly lower performance than fully supervised methods on UIEBD.Source: §4.4 · p. 12, §4.5 · pp. 16, 17, 19, Table 2
  • UDnet has the highest NIQE on FISHTRAC, FishID, RUIE, and SUIM, second-best UIQM on three datasets, and leads selected metrics across UIEBD, EUVP, UCCS, and UIQS.The authors caution that no-reference metrics may not accurately represent image quality and treat their scores as references.Source: Table 3 · p. 14, §4.5 · p. 19, Table 4

4.6 Qualitative Comparisons

  • Visual comparisons show UDnet brightening and increasing contrast in low-light images and balancing green-tinted colours where several compared methods fail.The paper also reports that competing outputs can show over-enhancement, poor contrast, saturation, or over- or under-enhanced backgrounds, which it says UDnet avoids.Source: §4.6 · p. 20

4.8 Ablation Study

  • On UIEBD, the full model outperformed ablated variants in PSNR and SSIM; qualitative tests identify reference-stage colour adjustment and VGG loss as beneficial, while applying colour adjustment to all inputs or adding reference maps degraded outputs.The further PAdaIN comparison is reported in Table 6, with the text stating that each component plays a crucial role but giving no specific values here.Source: Table 5 · p. 21, §4.8 · pp. 21–22, Table 6 · p. 22

4.7 Visual Perception Improvement

  • After enhancement, matched feature points between two RUIE frames increased from 74 to 594, alongside a marked increase in SIFT keypoints.The study used SIFT for keypoint detection and RANSAC for matching.Source: §4.7 · pp. 20–21

4.8 Ablation Study

  • 22.23 dB PSNR and 0.812 SSIM were achieved by full UDnet, the best ablation result; removing PAdaIN lowered performance, while the baseline scored lowest.The full model includes SGMCSS, cVAE, and PAdaIN; the study attributes PAdaIN’s contribution to encoding feature uncertainties.Source: §4.8 · p. 22

5 DISCUSSION

UDNet combines unsupervised probabilistic uncertainty modeling with SGMCSS and PAdaIN to enhance varied underwater images. It performs competitively across eight public datasets, but backscatter and statistical assumptions remain challenges.

5 DISCUSSION

  • Probabilistic uncertainty modeling lets UDNet adaptively enhance underwater images with varying characteristics without paired raw and enhanced training images.The approach is framed as an alternative to supervised methods that rely on large paired datasets.Source: §5 · p. 23
  • UDNet performs competitively across eight public datasets, outperforming or matching state-of-the-art methods quantitatively and qualitatively.Its unsupervised learning does not require ground-truth data, and SGMCSS and PAdaIN address variations in contrast, color balance, and illumination.Source: §5 · p. 23
  • Backscatter, especially at greater distances, limits visual clarity; statistical models can also produce unrealistic enhancements, and environmental variability may hinder optimal generalization.The stated environments range from oceans to lakes.Source: §5 · p. 23
  • Future work will explore alternative CNN architectures, multi-resolution approaches, and improved reference-map generation to capture finer details and reduce reliance on statistical assumptions.Source: §5 · p. 23
  • The authors identify potential uses in environmental monitoring, marine biology, and underwater archaeology, including clarifying images for ecosystem, species, and artifact studies.Source: §5 · p. 23
  • UDNet may also serve as a reference-map-generation framework in medical imaging or satellite enhancement, where ground truth is challenging to obtain.Source: §5 · p. 23

6 CONCLUSION

The conclusion presents UDNet as an unsupervised framework for underwater image enhancement using probabilistic uncertainty modeling and an encoder-decoder architecture. It reports broad benchmark performance and generalization, particularly on unpaired datasets.

6 CONCLUSION

  • UDNet outperforms ten state-of-the-art underwater image enhancement methods across seven metrics and eight datasets.The conclusion also describes its generalization ability, particularly for unpaired datasets.Source: §6 · p. 23
  • UDNet combines probabilistic uncertainty modeling with an encoder-decoder architecture, including SGMCSS and PAdaIN, to enhance images without manually labeled data.Source: §6 · p. 23
  • The authors position UDNet as a practical tool for diverse underwater applications, citing its versatility and effectiveness across datasets.Source: §6 · p. 23
Loading 2212.08983v3…