PaperlayerView exact revision on arXiv

Source-linked AI summary

Wavelength-based Attributed Deep Neural Network for Underwater Image Restoration

Prasen Kumar Sharma, Ira Bisht, Arijit Sur

arXiv:2106.07910v3eess.IVcs.CV

TL;DR

Underwater restoration must address wavelength-dependent attenuation and the resulting channel asymmetry, which conventional equal-context processing does not handle well. Deep WaveNet uses wavelength-guided channel-specific receptive fields and attention-based skip refinement for simultaneous enhancement and super-resolution, and reports stronger benchmark and high-level-task performance, while failure cases show that enhancement can still alter pose-estimation outcomes.

  • Problem

    Wavelength-dependent attenuation causes underwater color distortion, while existing UIR networks often use equal receptive fields across color channels.

  • Method

    Deep WaveNet simultaneously enhances and super-resolves underwater images using attenuation-guided channel-specific receptive fields and block attention-based skip refinement.

  • Results

    Deep WaveNet outperforms best-published UIR works and enhanced images improve performance across high-level tasks, including semantic segmentation and diver 2D pose estimation.

  • Takeaways & Limitations

    Wavelength-specific contextual design and attentive feature refinement provide a paper-supported approach for underwater enhancement and super-resolution across benchmark datasets.

  • Takeaways & Limitations

    Some enhanced images reduce high-level vision performance, including false-positive or missed detections in diver 2D pose estimation.

Abstract

from arXiv · show

Background: Underwater images, in general, suffer from low contrast and high color distortions due to the non-uniform attenuation of the light as it propagates through the water. In addition, the degree of attenuation varies with the wavelength resulting in the asymmetric traversing of colors. Despite the prolific works for underwater image restoration (UIR) using deep learning, the above asymmetricity has not been addressed in the respective network engineering. Contributions: As the first novelty, this paper shows that attributing the right receptive field size (context) based on the traversing range of the color channel may lead to a substantial performance gain for the task of UIR. Further, it is important to suppress the irrelevant multi-contextual features and increase the representational power of the model. Therefore, as a second novelty, we have incorporated an attentive skip mechanism to adaptively refine the learned multi-contextual features. The proposed framework, called Deep WaveNet, is optimized using the traditional pixel-wise and feature-based cost functions. An extensive set of experiments have been carried out to show the efficacy of the proposed scheme over existing best-published literature on benchmark datasets. More importantly, we have demonstrated a comprehensive validation of enhanced images across various high-level vision tasks, e.g., underwater image semantic segmentation, and diver's 2D pose estimation. A sample video to exhibit our real-world performance is available at \url{https://tinyurl.com/yzcrup9n}. Also, we have open-sourced our framework at \url{https://github.com/pksvision/Deep-WaveNet-UnderwaterImage-Restoration}.

1 INTRODUCTION

Underwater restoration is difficult because wavelength-dependent attenuation creates color distortions, while conventional methods often use equal receptive fields across channels. Deep WaveNet addresses this by assigning channel-specific context and refining features, with enhanced images also evaluated for high-level vision tasks.

  • Motivation: Blue light traverses farthest because of its shortest wavelength, motivating a larger receptive field for the blue channel.The proposed rationale links blue-channel dominance with wavelength-specific context assignment.
  • Motivation: Wavelength-dependent attenuation and marine snow create color casts, scattering, and challenging underwater restoration conditions.The attenuation varies with wavelength, while marine snow increases light scattering.
  • Motivation: Existing UIR schemes commonly process color channels with equal receptive field sizes, despite differing underwater attenuation ranges.The paper questions whether homogeneous contextual sizes are suitable for underwater images.
  • Approach: Deep WaveNet uses wavelength-driven multi-contextual design and attentive residual learning to filter irrelevant features.The paper presents these as novel components for underwater image restoration.
  • Evaluation: The framework evaluates enhanced images on high-level tasks, including underwater semantic segmentation and diver 2D pose estimation.Figure 2 presents sample results across high-level vision tasks.

2 RELATED RESEARCH: AN OVERVIEW

Prior underwater restoration research includes non-physical, physical-model, and data-driven approaches, but existing methods may retain artifacts because they do not tailor receptive fields to channel-specific attenuation. Deep WaveNet combines wavelength-guided contextual processing with attention-based skip refinement and evaluates both enhancement and super-resolution.

  • Existing literature: Prior UIR methods span underwater enhancement and single-image super-resolution, including non-physical, physical-model, and data-driven approaches.The review organizes the literature along restoration task and modeling strategy dimensions.
  • Major observations: Existing methods may produce color distortion because deep CNNs are directly applied without supervising receptive-field context across channels.The paper attributes this concern to differing attenuation ranges in underwater imagery.
  • Proposed approach: Deep WaveNet simultaneously enhances and increases the spatial resolution of degraded underwater images using attenuation-guided channel-specific receptive fields.The model assigns larger context to blue, smaller context to green, and further reduced context to red.
  • Proposed approach: A block attention-based skip-refinement mechanism adaptively regulates channel-specific information flow across the network.Intermediate stages aggregate multi-contextual features while suppressing irrelevant color-localized skip information.
  • Evaluation: The study compares the framework with nearly 20 published methods across more than 12 image-quality metrics and includes ablation experiments.The evaluation covers underwater enhancement and super-resolution.

3.1 Convolutional Block Attention Module (CBAM)

CBAM refines intermediate features through channel and spatial attention, and Deep WaveNet applies it after color-localized skip connections rather than immediately after convolution layers. The refined features are reported to benefit enhanced underwater image construction.

  • CBAM operation: CBAM computes channel and spatial attention maps for an intermediate feature map and multiplies them with input features for adaptive refinement.This attention operation produces refined intermediate features.
  • Deep WaveNet integration: Deep WaveNet applies CBAM after color-localized skip connections in stages 2 and 4 instead of using it only after convolution layers.This placement supports adaptive refinement of skip features.
  • CBAM operation: The module uses pooling, a 7×7 convolution, a sigmoid function, and a one-hidden-layer multilayer perceptron to compute attention components.The cited formulation identifies these operations and components.
  • Effect: The refined features are reported to benefit construction of visually pleasant enhanced underwater images.The paper links this observation to results discussed later in the study.

3.2 Network Architecture

Deep WaveNet uses wavelength-driven, channel-specific receptive fields and attentive refinement to restore underwater images, with an optional super-resolution extension.

  • Network goal: Deep WaveNet learns a unified model for simultaneous enhancement and super-resolution of degraded underwater images.The network denotes degraded and enhanced images as D and E, respectively.
  • Stages 1–2: Stage 1 generates channel-specific features using wavelength-driven contextual sizes.The degraded image is processed channel-wise to obtain multi-contextual features.
  • Stages 1–2: Stage 2 generates color-specific distortion residuals and adaptively refines them with CBAM modules.The refinement suppresses noisy color-specific features while preserving details needed for global color correction.
  • Stages 3–4: Stage 3 converts attentive multi-contextual residuals into a global color-correction residual map, and Stage 4 reconstructs the enhanced image.Stage 4 uses a deconvolution layer, an attentive residual block, and a final deconvolution layer.
  • Super-resolution sub-stage: The super-resolution sub-stage uses convolution and pixel-shuffle operations after Stage 4 to increase spatial resolution.The design supports 2×, 3×, and 4× scale factors; output channels are set to 3s^2 for scale factor s.

3.3 Model Learning

Deep WaveNet combines pixel-wise, perceptual, and structural losses, adjusting the objective according to whether enhancement alone or joint enhancement and super-resolution is trained.

  • Loss functions: The model first incorporates Mean Squared Error ℓ2 loss against the original clean, higher-resolution underwater image.The authors note that ℓ2 minimization may produce blurry artifacts.
  • Loss functions: Perceptual loss compares VGG16 relu2_2 features of the predicted and ground-truth images to retain high-frequency details.VGG16 is pretrained on ImageNet.
  • Loss functions: SSIM loss minimizes structural differences between the enhanced image E and original image O.SSIM is added alongside L2 and perceptual losses.
  • Objective: The objective weights λP and λS are empirically set to 0.02 and 0.5, respectively.These weights apply to the perceptual and SSIM terms in the combined objective.
  • Task-specific training: For enhancement alone, L2 and perceptual losses are sufficient, whereas joint enhancement and super-resolution additionally uses SSIM.The SSIM weight λS is set to zero for image-enhancement-only training.

4 EXPERIMENTAL SETUP

Experiments evaluate Deep WaveNet for underwater enhancement and single-image super-resolution across EUVP, UIEB, and UFO-120 benchmarks against established methods using reference and reference-less image-quality metrics.

  • Datasets: EUVP enhancement uses 11,435 paired training images and 515 paired test images, each sized 256 × 256.
  • Datasets: UIEB contains 890 paired images, split into 800 training and 90 test images, with 5-fold cross-validation for fair comparison.
  • Datasets: UFO-120 provides 1,500 paired training images and 120 test images for simultaneous enhancement and 2×, 3×, and 4× super-resolution.
  • Competing methods: Enhancement comparisons include established physical, non-physical, and learning-based methods, while super-resolution comparisons include SRCNN, SRResNet, SRGAN, SRDRM, SRDRM-GAN, and Deep SESR.
  • Metrics: Evaluation uses MSE, PSNR, SSIM, UIQM, NIQE, PCQI, UISM, VIF, average entropy, and average gradient metrics.

5 RESULTS

Deep WaveNet outperforms published methods across underwater enhancement and super-resolution evaluations, with gains in image-quality metrics and qualitative fidelity. Enhanced outputs are also assessed for segmentation and pose-estimation applications.

  • High-level vision tasks: Enhanced images from multiple restoration methods are qualitatively evaluated for semantic segmentation and diver 2D pose estimation.
  • Underwater image enhancement: Deep WaveNet achieved the best result on 7 of 12 adopted image-quality metrics for underwater enhancement.
  • Underwater image enhancement: On UIEB, the method significantly outperformed existing works on all reported metrics, including a 13% PSNR improvement over WaterNet.
  • Underwater image enhancement: On the Challenge set, Deep WaveNet improved UIQM and NIQE over published methods, while retaining water’s bluish appearance better than Ucolor.
  • Underwater super-resolution: For underwater SISR, Deep WaveNet improved SSIM and UIQM despite slight PSNR underperformance, including an approximately 12% SSIM gain over Deep SESR at 4×.
  • Underwater super-resolution: Qualitative SISR results describe Deep WaveNet outputs as artifact-free and perceptually similar to ground truth, unlike methods retaining color distortions or noise traces.

6 EFFECT ON HIGH-LEVEL VISION TASKS

Deep WaveNet enhanced underwater images for semantic segmentation and diver 2D pose estimation, but some enhancement cases reduced high-level vision performance.

  • 6.1 Underwater Image Semantic Segmentation: Deep WaveNet enhanced images supported underwater semantic segmentation evaluated across five classes, including divers, wrecks, robots, fish, and reefs.The evaluation used SUIM, which assigns every pixel to one of five underwater scene categories.
  • 6.2 Underwater Divers 2D Pose Estimation: Enhanced images from Deep WaveNet produced more accurate diver pose key points than existing methods and degraded images.The comparison used OpenPose-based diver 2D pose estimation.
  • 6.3 Failure Case: Some samples showed enhancement declining high-level vision performance, with degraded, enhanced, and clean images compared using detected human-body poses.Figure 9 presents these failure cases across underwater images.
  • 6.3 Failure Case: OpenPose sometimes replaced a detected fish false positive with a random-background false positive or missed a leg detected before enhancement.The authors associate these cases with altered color or texture distributions and OpenPose’s outdoor-image expectations.

7 ABLATION STUDY

Ablations show that perceptual and SSIM-based losses improve restoration, while wavelength-driven contexts and CBAM jointly improve the model over homogeneous-context baselines.

  • 7.1 Effect of Cost Functions: Perceptual loss significantly improved enhancement over L2, while SSIM-based loss improved both SSIM and PSNR for underwater super-resolution.L2 alone produced minimal super-resolution performance and was associated with blurry denoised images.
  • 7.2 Effect of Wavelength-driven Contextual Sizes and CBAM: Deep WaveNet-1 used homogeneous 3 × 3 contexts without CBAM, Deep WaveNet-2 added wavelength-driven contexts, and Deep WaveNet-3 added CBAM to homogeneous contexts.These baselines isolate contextual-size and attention effects.
  • 7.2 Effect of Wavelength-driven Contextual Sizes and CBAM: Deep WaveNet combined wavelength-driven multi-contextual formulation with CBAM and achieved notable improvement over Deep WaveNet-1.The authors report the homogeneous-context, no-CBAM model as comparatively weakest among the described baselines.
  • 7.2.1 Run-time and Memory Usages of Different Baselines: The baseline study measured computational overhead using run-time and memory usage on an Nvidia P100 16 GB GPU.Training used 256 × 256 images with batch size 5, while testing used batch size 1.

8 CONCLUSIONS

The paper proposes Deep WaveNet for underwater enhancement and super-resolution using wavelength-specific receptive fields and attention-based feature refinement.

  • 8 Conclusions: Deep WaveNet assigns channel-specific receptive field sizes according to wavelength to learn diverse local and global underwater-image features.The model targets simultaneous underwater image enhancement and super-resolution.
  • 8 Conclusions: A block attention mechanism adaptively refines learned features and is reported to improve the proposed scheme’s performance.The architecture is extensible to spatial super-resolution of enhanced underwater images.
  • 8 Conclusions: The proposed scheme is reported to outperform existing best-published works across benchmark datasets, supported by ablation experiments.The paper also identifies underwater video enhancement and super-resolution as future work.
Loading 2106.07910v3…