PaperlayerView exact revision on arXiv

Source-linked AI summary

Diving Deeper into Underwater Image Enhancement: A Survey

Saeed Anwar, Chongyi Li

arXiv:1907.07863v1cs.CVcs.LGeess.IV

TL;DR

Underwater enhancement research lacks a comprehensive account of deep methods and robust evidence across datasets, while real imagery remains difficult because degradation and benchmark limitations persist. This paper surveys deep learning methods, models, datasets, and metrics, then compares algorithms qualitatively and quantitatively. It concludes that progress is constrained by synthetic training data, limited datasets, and inadequate evaluation metrics, motivating more realistic data and underwater-specific metrics.

  • Problem

    Deep underwater enhancement lacks a comprehensive survey and sufficiently explored qualitative and quantitative comparisons across diverse datasets.

  • Method

    The paper reviews deep enhancement algorithms, image formation models, datasets, and evaluation metrics, then conducts systematic qualitative and quantitative comparisons.

  • Results

    The evaluation identifies limitations in current deep algorithms and biases or shortcomings in existing benchmark datasets and evaluation metrics.

  • Takeaways & Limitations

    Future progress requires more realistic training datasets and evaluation metrics that incorporate underwater image properties.

  • Takeaways & Limitations

    Current methods often fail to generalize from synthetically generated underwater scenes to real-world images, while unsupervised learning remains an open problem.

Abstract

from arXiv · show

The powerful representation capacity of deep learning has made it inevitable for the underwater image enhancement community to employ its potential. The exploration of deep underwater image enhancement networks is increasing over time, and hence; a comprehensive survey is the need of the hour. In this paper, our main aim is two-fold, 1): to provide a comprehensive and in-depth survey of the deep learning-based underwater image enhancement, which covers various perspectives ranging from algorithms to open issues, and 2): to conduct a qualitative and quantitative comparison of the deep algorithms on diverse datasets to serve as a benchmark, which has been barely explored before. To be specific, we first introduce the underwater image formation models, which are the base of training data synthesis and design of deep networks, and also helpful for understanding the process of underwater image degradation. Then, we review deep underwater image enhancement algorithms, and a glimpse of some of the aspects of the current networks is presented including network architecture, network parameters, training data, loss function, and training configurations. We also summarize the evaluation metrics and underwater image datasets. Following that, a systematically experimental comparison is carried out to analyze the robustness and effectiveness of deep algorithms. Meanwhile, we point out the shortcomings of current benchmark datasets and evaluation metrics. Finally, we discuss several unsolved open issues and suggest possible research directions. We hope that all efforts done in this paper might serve as a comprehensive reference for future research and call for the development of deep learning-based underwater image enhancement.

1 Introduction

Underwater image enhancement matters because degraded imagery limits engineering and research uses, while deep learning has not yet delivered the expected performance. The paper surveys current methods, benchmarks them, and identifies limitations and research directions.

  • Underwater images support tasks such as archaeology and surveillance, but absorption, scattering, artificial lighting, and backscatter degrade their quality.
  • Conventional and data-driven methods, including CNNs and GANs, are being explored for underwater image enhancement.
  • Deep learning has improved many vision tasks, yet its appealing performance has not been observed consistently in underwater image enhancement.
  • The paper surveys algorithms across architectures, parameters, training data, losses, and configurations, and compares them qualitatively and quantitatively on diverse datasets.
  • The paper also reviews evaluation metrics and datasets, identifies benchmark limitations and metric bias, and suggests future research directions.

2 Background

Underwater image formation models explain degradation and support training-data synthesis and network design. The survey contrasts atmospheric, simplified, and revised models, emphasizing that physical accuracy increases with complexity.

  • Atmospheric scattering models describe underwater degradation by modeling attenuated scene light and ambient atmospheric light reaching the sensor.
  • The atmospheric model assumes nonselective scattering and wavelength-independent attenuation, limiting its realism for underwater imaging.
  • Real underwater imaging differs substantially from atmospheric scattering because water has wavelength-selective attenuation and more complicated optical properties.
  • The simplified underwater formation model accounts for wavelength-dependent transmission, which contributes to contrast degradation and color casts.
  • The revised model incorporates neglected effects involving veiling light, water absorption, and distinct attenuation of direct and scattering signals.
  • The atmospheric model is suitable mainly for shallow water with low backscatter, while the more accurate revised model has received little attention because of its complexity.

3 Deep Underwater Image Enhancement Algorithms

Deep underwater enhancement algorithms are broadly divided into CNN-based and GAN-based approaches, but the survey argues that architectural differences provide a more informative categorization.

  • CNN-based algorithms aim to remain faithful to the original underwater image, whereas GAN-based algorithms target improved perceptual quality.
  • The CNN-versus-GAN division is described as naive, so the survey categorizes networks according to architectural differences.
  • Figure 1 presents the categorization, followed by method details organized into categories based on essential aspects.

3.1 Encoder-Decoder models

Encoder-decoder architectures are used across several underwater enhancement models, with variants differing in training data, losses, and domain-robustness mechanisms.

  • 3.1 Encoder-Decoder models: Encoder-decoder architectures are used to advance underwater image enhancement research.
  • 3.1.1 P2P: The P2P model uses a symmetric three-layer convolutional encoder and three-layer deconvolutional decoder with ReLU activations except at the output.
  • 3.1.1 P2P: P2P training uses real-world images with milk-based simulated degradation, 10,000 training images, 2,000 testing images, 66×66 patches, ℓ2 loss, and SGD.
  • 3.1.2 UIE-DAL: UIE-DAL combines a UNET encoder-decoder with a nuisance classifier intended to learn features agnostic to underwater type.
  • 3.1.2 UIE-DAL: UIE-DAL combines ℓ2, nuisance, and adversarial losses and trains first without, then with, the nuisance classifier.
  • 3.1.3 UGAN: UGAN uses a fully convolutional WGAN-GP discriminator, a transmission-map reconstruction output, and a global skip connection for residual learning.
  • 3.1.3 UGAN: UGAN is trained on distorted and undistorted ImageNet subsets for 100 epochs with Adam, learning rate 10^-4, 256×256×3 inputs, and combined ℓ1 and Wasserstein-1 loss.

3.2 Modular designs

Modular designs repeatedly apply shared block structures to learn features in deep underwater enhancement networks. Examples include UWCNN’s densely connected CNN blocks and DenseGAN’s multiscale dense generator and discriminator.

  • UWCNN: UWCNN is an end-to-end CNN trained on synthetic underwater datasets to address low contrast and distorted color.Its architecture contains three densely connected building blocks, each comprising three densely connected convolutional layers.
  • UWCNN: UWCNN uses ten synthetic datasets spanning open-ocean and coastal water types from clear to highly turbid conditions.The datasets are synthesized with different attenuation coefficients from RGB-D NYU-v2 images.
  • DenseGAN: DenseGAN combines dense connections, residual learning, and multiscale processing for underwater image enhancement.Its generator begins with convolutional and normalization layers, uses two MSDB blocks, and ends with deconvolution and TanH layers.
  • DenseGAN: Each MSDB uses two differently dilated branches whose features are repeatedly concatenated and fused with a 1×1 convolution.The discriminator follows a PatchGAN-like design with five spectrally normalized layers.
  • DenseGAN: DenseGAN’s implementation uses 256×256×3 patches, batch size 32, ADAM optimization, and a learning rate of 10^-3.The first generator filters are 7×7 and 3×3 with 64 and 128 feature maps, respectively.

3.3 Multi-branch designs

Multi-branch networks process different representations or feature types through parallel subnetworks before combining their outputs. The surveyed examples include UIE-Net, gated fusion CNNs, and FGAN.

  • Design principle: Multi-branch designs learn different features from one input or exploit distinct inputs through separate branches.This architectural strategy is used to separate complementary enhancement functions.
  • UIE-Net: UIE-Net uses a sharing network to feed parallel color-correction and haze-removal subnetworks.The branches output a color-corrected image and transmission map, respectively, using matching four-layer structures with different output channels.
  • UIE-Net: UIE-Net captures contextual information with branch filters of 1×1, 3×3, 5×5, and 7×7.Training uses 32×32 patches synthesized from 200 clear images, with 2×10^5 image patches.
  • UIE-Net: UIE-Net uses ℓ2 loss, input smoothing, and guided filtering on the transmission map to remove possible artifacts.The survey identifies it as one of the pioneering deep-learning approaches in this direction.
  • DUIENet: DUIENet is trained on a real-world dataset of 950 underwater images, including 890 with reference images selected through volunteer majority voting.Three preprocessed inputs are transformed and fused using learned confidence maps.
  • FGAN: FGAN processes multiple inputs through separate branches and sums their features before generator loss computation.Its discriminator uses five spectrally normalized convolutional layers and follows PatchGAN.

3.4 Depth-guided networks

Depth-guided networks explicitly estimate transmission or depth information to address scattering-related degradation. The survey covers URCNN, UIR-Net, and WaterGAN-based restoration pipelines.

  • Design principle: Transmission or depth maps are used because they relate to scattering-induced degradation and can guide underwater image restoration.Predicting these maps is presented as a natural design choice for enhancement networks.
  • URCNN: URCNN predicts a transmission map with a VGG-modified convolutional network and refines the input image using that output.A global skip connection enforces residual learning.
  • URCNN: URCNN trains on 1,800 synthetic images and tests on 200 images generated from NYU data with random attenuation and background light.The network has 25 layers with 64 feature maps per layer and uses ℓ2 loss.
  • UIR-Net: UIR-Net separately estimates background light and transmission through BL-Net and TM-Net.The two subnetworks use Euclidean and scale-invariant minimum-square-error losses, respectively, with 12,000 synthetic images generated using 29 ambient lights.
  • WaterGAN: WaterGAN uses RGB-D images to simulate underwater images for color correction before a restoration network estimates depth and corrects color.Its GAN generator creates synthetic images while the discriminator distinguishes them from real images.
  • WaterGAN: WaterGAN forms 15,000 underwater images from multiple marine and Kinect datasets, allocating 12,000 for training and 3,000 for testing.These data support the subsequent depth-estimation and color-correction restoration pipeline.

3.5 Dual Generator GANs

Dual-generator GANs use multiple generators to predict improved underwater images, either sharing features or cascading one generator’s output into another. The surveyed methods include weakly supervised color transfer, multiscale CycleGAN restoration, and stacked conditional GANs.

  • Design principle: Dual-generator GANs commonly use two generators with one or two discriminators to share features or cascade predictions.The category is organized around multi-generator designs for underwater enhancement.
  • UWGAN: UWGAN is a weakly supervised color-transfer method that relaxes the need for paired underwater images.Its cycle structure learns mappings between underwater and high-quality air-image domains.
  • UWGAN: UWGAN combines adversarial, cycle-consistency, and SSIM losses while training on 3,800 underwater and 3,800 high-quality air images.The images are resized to 256×256 and evaluated with 70×70 PatchGAN discriminators.
  • MCycleGAN: MCycleGAN adds multiscale SSIM loss to CycleGAN to transfer turbid underwater images toward recovered-style images.Transmission maps from the dark channel prior provide depth information through three binary filters.
  • MCycleGAN: MCycleGAN combines adversarial, cycle-consistent, and multiscale SSIM losses with a nine-ResNet-block generator and 70×70 PatchGAN discriminator.Its training images are 256×256.
  • UIE-sGAN: UIE-sGAN stacks haze-detection and color-correction subnetworks, each containing a generator and discriminator.Training uses adversarial losses for both networks and a consistency loss.

3.6 Network Specifics

The survey examines network loss, depth, parameters, and input-size choices in underwater image enhancement models, while noting that exact parameter comparisons are limited by unavailable implementations.

  • Network Loss: Loss functions commonly minimize per-pixel errors with ℓ1 or ℓ2, sometimes combined with SSIM or gradient losses.
  • Network Depth and Parameters: Underwater enhancement networks remain shallow, with fewer than 45 layers; WaterGAN is the deepest listed network at 42 layers.
  • Input Patch Size: Most algorithms operate on full-size images, potentially incorporating wavelength dissipation across red, green, and blue channels.
  • Input Patch Size: Some methods reduce images to a predefined size and require upsampling as post-processing, including MCycleGAN, DenseGAN, and UWGAN.
  • Network Depth and Parameters: Exact parameter counts cannot be compared fairly because most network models are not publicly available.

4 Experimental Settings

The survey evaluates underwater enhancement using real-world datasets, automatic metrics, and qualitative comparisons across degradation types. Results show method performance varies with data and degradation, while severe backscatter, noise, artifacts, and color casts remain unresolved.

  • Real-world underwater image datasets: Synthetic datasets are limited by inaccurate formation models, hard assumptions, insufficient images, and scene-specific coverage.
  • Real-world underwater image datasets: UIEBD contains 950 real-world images, including 890 with references selected from 12 enhanced results and 60 challenging images without satisfactory references.
  • Real-world underwater image datasets: Existing real-world datasets often have monotonous content, limited degradation types, and no ground truth because paired degraded and true scenes are impractical to acquire.
  • Evaluation metrics: Automatic evaluation uses PSNR, MSE, SSIM, PCQI, UCIQE, and UIQM, while human evaluation is used because real ground-truth data are scarce.
  • Evaluation metrics: UIQM is reference-free and combines colorfulness, sharpness, and contrast measures to evaluate underwater image quality.
  • Benchmark results: DUIENet performs best on UIEBD, whereas UWCNN performs worst; however, superiority is difficult to establish because architecture, data, depth, patch size, channels, and loss differ.
  • Qualitative comparisons: CNN outputs are generally more faithful to original colors and achieve higher PSNR than GAN outputs, which target perceptual quality.
  • Qualitative comparisons: Bluish tones remain difficult to remove: DUIENet and UWCNN perform best, while UWGAN and DenseGAN introduce artificial colors associated with unpaired training data.

5 Future and Emerging Directions

Future work centers on closing the gap between synthetic training and real underwater scenes, while developing better datasets, objectives, metrics, and learning paradigms. The survey also highlights domain knowledge and difficult high-backscatter imagery as important unresolved challenges.

  • Underwater enhancement still trails other image-enhancement tasks, leaving substantial room for advancement.
  • Objective functions and evaluation metrics: Current objective functions and evaluation metrics often ignore underwater physical properties, and visual comparisons can disagree with quantitative rankings.The survey therefore calls for specialized objectives and metrics for underwater enhancement.
  • Prior knowledge: Human domain or prior knowledge may improve enhancement because experts assess image quality using more than visual content alone.The survey identifies augmenting current techniques with prior knowledge as a possible research direction.
  • High-backscatter imagery remains especially difficult: current methods may introduce artifacts and color casts, while ULFID images defeat all methods.The survey notes that low backscatter is easier to remove than high backscatter.
  • Unsupervised learning: Unsupervised, zero-shot, and few-shot learning are proposed for limited-data settings, but developing unsupervised learning remains an open research problem.Few-shot learning uses the limited available datasets so networks learn from only a few images.
  • Real vs. Synthetic: Synthetic training data may not match real underwater scenes, limiting the generalization of models trained on artificially produced datasets.The survey suggests improving synthetic-data generation, including transferring underwater style to simulated scenes with GANs.

6 Conclusion

The survey covers deep learning methods, datasets, evaluation metrics, and visual and quantitative comparisons for underwater image enhancement. It identifies limitations in current architectures, synthetic training data, metrics, and datasets, and proposes future directions.

  • The survey comprehensively reviews CNN- and GAN-based underwater image enhancement methods, datasets, evaluation metrics, and their limitations.
  • Systematic comparisons on benchmark datasets combine quantitative metrics and visual results to assess algorithm performance, difficulty, and robustness.
  • Current methods largely modify existing architectures such as encoder-decoder networks and CycleGAN, without architectures or loss functions specifically designed for underwater enhancement.
  • Synthetic training data limit model generalization, while current research lacks large training datasets and purpose-built underwater evaluation metrics.
  • The survey suggests incorporating underwater image properties into evaluation metrics and using GANs to create more realistic datasets.
Loading 1907.07863v1…