PaperlayerView exact revision on arXiv

Source-linked AI summary

UIEC^2-Net: CNN-based Underwater Image Enhancement Using Two Color Space

Yudong Wang, Jichang Guo, Huan Gao, Huihui Yue

arXiv:2103.07138v2cs.CV

TL;DR

Underwater enhancement must address image degradation while RGB-only CNNs inadequately represent luminance and saturation, and prior methods face limitations across scenes. UIEC^2-Net integrates RGB and HSV processing with a learned curve layer and pixel-level attention in an end-to-end CNN; experiments on synthetic and real-world images report good subjective and objective performance.

  • Problem

    Underwater images suffer from low contrast, brightness problems, color deviations, blur, and uneven illumination, while many CNN methods use RGB alone and are insensitive to luminance and saturation.

  • Method

    UIEC^2-Net combines an RGB pixel-level block, an HSV global-adjust block with learned piece-wise linear curves, and a pixel-level attention map block in an end-to-end CNN.

  • Results

    Experiments on synthetic and real-world underwater images report good performance in subjective comparisons and objective metrics, including improved color-cast restoration and retained image details.

  • Takeaways & Limitations

    Integrating RGB and HSV color spaces provides the paper's basis for enhancing underwater color, luminance, saturation, and detailed information within one CNN.

Abstract

from arXiv · show

Underwater image enhancement has attracted much attention due to the rise of marine resource development in recent years. Benefit from the powerful representation capabilities of Convolution Neural Networks(CNNs), multiple underwater image enhancement algorithms based on CNNs have been proposed in the last few years. However, almost all of these algorithms employ RGB color space setting, which is insensitive to image properties such as luminance and saturation. To address this problem, we proposed Underwater Image Enhancement Convolution Neural Network using 2 Color Space (UICE^2-Net) that efficiently and effectively integrate both RGB Color Space and HSV Color Space in one single CNN. To our best knowledge, this method is the first to use HSV color space for underwater image enhancement based on deep learning. UIEC^2-Net is an end-to-end trainable network, consisting of three blocks as follow: a RGB pixel-level block implements fundamental operations such as denoising and removing color cast, a HSV global-adjust block for globally adjusting underwater image luminance, color and saturation by adopting a novel neural curve layer, and an attention map block for combining the advantages of RGB and HSV block output images by distributing weight to each pixel. Experimental results on synthetic and real-world underwater images show the good performance of our proposed method in both subjective comparisons and objective metrics. The code are available at https://github.com/BIGWangYuDong/UWEnhancement.

I. INTRODUCTION

Underwater images often suffer from low contrast, brightness, color deviations, blur, and uneven illumination, while existing RGB-based and physical-model methods have important limitations. UIEC^2-Net addresses these issues by integrating RGB and HSV processing with pixel-level attention in an end-to-end CNN.

  • Motivation: Underwater image degradation includes low contrast and brightness, color deviations, blurry details, and uneven bright speck.These defects reduce the usefulness of underwater image information for engineering and research tasks.
  • Limitations of Existing Methods: Traditional enhancement methods and simplified physical models can perform poorly across underwater image types and may only suit specific scenes.Physical-model approaches rely on assumptions about multiple parameters that may not accurately represent underwater formation conditions.
  • Proposed Method: UIEC^2-Net combines an RGB pixel-level block, an HSV global-adjust block, and an attention map block in one end-to-end CNN.The RGB block handles denoising and color-cast removal; the HSV block adjusts saturation and brightness; attention combines their outputs pixel by pixel.
  • Proposed Method: The HSV global-adjust block learns piece-wise linear curves to adjust underwater image properties, especially saturation and brightness.This provides differentiable HSV-space transformations within the neural network.
  • Contribution: The method introduces deep-learning-based underwater enhancement using both RGB and HSV color spaces, which the authors identify as a first.The contribution targets image properties such as luminance, saturation, and color cast that RGB-only processing does not directly represent.

II. RELATE WORK

Prior underwater enhancement research includes deep-learning CNN and GAN approaches, but realistic training data remain difficult to synthesize. Multiple-color-space enhancement has mainly used conventional algorithms, motivating a deep-learning method that combines RGB and HSV processing.

  • Deep-Learning Methods: Deep-learning underwater enhancement methods include encoder-decoder, modular, multi-branch, map-prediction, and multiple-generator architectures.These methods are broadly organized into CNN-based and GAN-based categories.
  • Datasets: Realistic underwater training pairs are difficult to synthesize because underwater image formation depends on factors such as turbidity and lighting conditions.This distinguishes underwater enhancement datasets from several other low-level vision tasks with abundant synthetic degradations.
  • Multiple Color Spaces: Conventional underwater enhancement methods have used multiple color spaces, including RGB with HSI or HSV, for contrast and visual-quality improvement.Examples include histogram stretching, CLAHE, contrast correction, and recursive adaptive histogram modification.
  • Research Gap: The paper presents RGB and HSV integration in deep-learning-based underwater enhancement, extending multiple-color-space processing beyond conventional algorithms.The authors describe this as the first application of these two color spaces in deep-learning-based underwater enhancement.

A. Network Architecture

UIEC^2-Net combines RGB pixel processing, HSV global adjustment, and pixel-level attention in an end-to-end network for underwater enhancement.

  • Network Architecture: UIEC^2-Net uses three blocks: RGB pixel-level processing, HSV global adjustment, and attention-based fusion.The RGB block denoises and removes color cast; the HSV block refines luminance and saturation; attention combines their outputs.
  • RGB pixel-level block: The RGB pixel-level block is a fully convolutional network without downsampling that outputs an H × W × 3 image.It uses eight 3×3 convolutional layers with stride 1, batch normalization, nonlinear activations, and a final sigmoid.
  • Limitations: The authors identify U-Net and residual-network backbones as possible future improvements to the RGB pixel-level block.This is presented as a limitation of the current architecture rather than an evaluated modification.
  • HSV global-adjust block: The HSV global-adjust block converts the RGB output to HSV and predicts knot points for piecewise linear curves that globally refine image properties.The predicted curves scale pixel values, with examples shown in Figure 3.
  • HSV global-adjust block: Four HSV curves adjust value, saturation, and hue to refine luminance, saturation, and color before conversion back to RGB.The block produces an H × W × 3 HSV result and uses differentiable HSV-to-RGB conversion.
  • Attention map block: The attention map block predicts six pixel-level weights and combines the RGB and HSV-derived outputs into the final enhanced image.Three channels weight the RGB block output and three weight the HSV block output.

B. Loss Function

Training uses four loss components spanning RGB and HSV spaces, with L1 and SSIM terms applied to intermediate and final outputs.

  • Loss Function: UIEC^2-Net is trained end-to-end with four loss components, including HSV-specific and RGB-space terms.The RGB terms are L1, SSIM, and perceptual losses; the HSV term measures HSV-space differences.
  • HSV loss: The HSV loss operates on hue, saturation, and value channels in conical HSV color space.HSV channels are defined over H ∈ [0, 2π), S ∈ [0, 1], and V ∈ [0, 1].
  • L1 loss: L1 loss measures RGB-pixel differences between predictions and ground truth in both the RGB block and the complete network.The loss is therefore applied to an intermediate RGB output and the final enhanced image.
  • SSIM loss: SSIM loss imposes structure and texture similarity using grayscale images converted from RGB.Each pixel’s SSIM is computed over an 11×11 surrounding image patch, and the loss is applied twice during training.
  • Perceptual loss: The perceptual loss compares VGG-19 feature representations of the network output and ground-truth image.Layer 4_3 is selected to capture both color and semantic information.
  • Loss weighting: Loss weights are set to ωℓ1 = 1, ωSSIM = 1, ωhsv = 1, and ωperc = 0.5.The schedule changes λpixel and λwhole from 0.5/0.5 during the first 20 epochs to 0.1/0.9 later.

C. Color Space Transformations

The network uses differentiable RGB-to-HSV and HSV-to-RGB transformations so color-space processing remains compatible with end-to-end backpropagation.

  • Color Space Transformations: UIEC^2-Net relies on differentiable RGB→HSV and HSV→RGB conversions for end-to-end learning by backpropagation.The paper identifies these conversions as necessary components of the two-color-space pipeline.
  • RGB→HSV: RGB→HSV conversion uses minimum and maximum operations and conditional expressions that can be processed within PyTorch.The resulting conversion supports backpropagation despite the conditional statements.
  • HSV→RGB: HSV→RGB conversion replaces complex formulas with piecewise linear functions defined by linear segments and knot points.The piecewise construction provides gradients between knot points and is illustrated in Figure 4.
  • HSV→RGB: The HSV-to-RGB formulas operate on each pixel’s hue, saturation, and value components.The cited variables h_j, s_j, and v_j denote those components for pixel j.

IV. EXPERIMENTAL RESULTS

The evaluation compares UIEC^2-Net with traditional and deep-learning underwater enhancement methods on synthetic and real-world images.

  • Experimental Results: The study performs qualitative and quantitative comparisons on both synthetic and real-world underwater images.The comparison includes traditional methods and recent deep-learning-based enhancement methods.
  • Compared Methods: Baselines include HE, WB, UDCP, ULAP, UWGAN, UGAN, UWCNN, and DUIENet.The listed methods span histogram-based, physical-model, GAN-based, and CNN-based approaches.

A. Implementation Details

The network is trained on paired synthetic and real-world underwater images, with fixed resizing and random cropping applied during preprocessing.

  • A. Implementation Details: Training uses 800 real-world image pairs from UIEBD and 1,200 synthetic pairs generated from NYU-v2 RGB-D images.The combined training set contains 2,000 images.
  • A. Implementation Details: Input images are resized to 350×350 and randomly cropped to 320 × 320 during training.

B. Experiment on Synthetic Datasets

Synthetic-image experiments compare UIEC^2-Net with traditional, physical, CNN, and GAN-based methods using visual comparisons and full-reference metrics. The authors report strong generalization across underwater image categories and best synthetic full-reference performance.

  • B. Experiment on Synthetic Datasets: Synthetic testing evaluates enhancement results against HE, WB, UDCP, ULAP, UWGAN, UGAN, UWCNN, DUIENet, and reference images.Figure 5 presents the visual comparison layout for the synthetic testing set.
  • B. Experiment on Synthetic Datasets: UIEC^2-Net shows good generalization across varied synthetic underwater categories and restores color casts better than comparison methods.Other methods perform better in only one or two categories or struggle with color casts and image details.
  • B. Experiment on Synthetic Datasets: Synthetic results are assessed with MSE, PSNR, and SSIM as full-reference image-quality metrics.Lower MSE and higher PSNR indicate closer image content to the ground truth, while higher SSIM indicates greater structural and texture similarity.
  • B. Experiment on Synthetic Datasets: UIEC^2-Net achieves the best full-reference image-quality performance on synthetic underwater images.The authors report higher SSIM than the second-best method and higher PSNR with correspondingly lower MSE than the compared methods.

C. Experiment on Real Datasets

Real-world experiments cover several underwater appearance categories and combine subjective comparisons with full-reference and no-reference quality evaluation. UIEC^2-Net is reported to handle haze, color casts, brightness, saturation, and details effectively.

  • C. Experiment on Real Datasets: The real-world test set covers greenish, bluish, yellowish, shallow-water, and limited-illumination underwater images.Corresponding reference images are provided by UIEBD.
  • C. Experiment on Real Datasets: Subjective comparisons arrange raw images, eight enhancement baselines, UIEC^2-Net, and reference images from left to right.The comparisons include shallow-water, bluish, greenish, low-illuminated, and yellowish images.
  • C. Experiment on Real Datasets: UIEC^2-Net effectively removes haze and color casts, adjusts brightness and saturation, and produces less noise and better details than corresponding reference images in some examples.
  • C. Experiment on Real Datasets: Real-world full-reference evaluation uses MSE, PSNR, and SSIM, with UIEC^2-Net achieving the best reported performance.The authors associate this result with handling image details.
  • C. Experiment on Real Datasets: No-reference evaluation uses UCIQE and UIQM, with UIQM computed from weighted colorfulness, sharpness, and contrast measures.The stated weights are c1 = 0.0282, c2 = 0.2953, and c3 = 3.5753.

D. Ablation Study

The ablation study examines the HSV global-adjust and attention map blocks against RGB-only processing. The authors report a quality-metric trade-off alongside subjective improvements and describe how attention mitigates HSV-related artifacts.

  • D. Ablation Study: The ablation compares UIEC^2-Net with RGB-only processing and variants without the attention map block.
  • D. Ablation Study: Adding the HSV and attention blocks decreases UCIQE and UIQM, but the authors consider the sacrifice necessary for overall network performance.
  • D. Ablation Study: The HSV block improves background color-cast processing and saturation, while it can also reduce contrast and saturation through over-processing.
  • D. Ablation Study: HSV processing may distort colors in reddish regions because the H-channel is sensitive.The attention map block combines raw, RGB, and HSV features at pixel level to avoid some HSV problems.
  • D. Ablation Study: The attention map block can also avoid noise blocks in the enhanced output.

V. CONCLUSION

UIEC^2-Net combines RGB- and HSV-based processing in a CNN for underwater image enhancement, effectively reducing color cast while preserving image detail across synthetic and real-world images.

  • UIEC^2-Net combines RGB and HSV color spaces in a CNN-based underwater image enhancement method.
  • Its RGB pixel-level block removes color cast and denoises, while the HSV global-adjust block refines luminance and saturation.
  • The method improves background-color restoration and retains detailed information in underwater images.
  • Experiments on synthetic and real-world underwater images, using qualitative and quantitative assessment, demonstrated the method’s effectiveness.
Loading 2103.07138v2…