PaperlayerView exact revision on arXiv

Source-linked AI summary

SGUIE-Net: Semantic Attention Guided Underwater Image Enhancement with Multi-Scale Perception

Qi Qi, Kunqian Li, Haiyong Zheng, Xiang Gao, Guojia Hou, Kun Sun

arXiv:2201.02832v1eess.IVcs.CV

TL;DR

Underwater enhancement lacks realistic paired references, making diverse degradation difficult to learn. SGUIE-Net uses semantic guidance and multi-scale perception, with experiments on four real-world benchmarks verifying its effectiveness and color restoration accuracy.

  • Problem

    Underwater enhancement lacks realistic reference images corresponding to degraded inputs, limiting effective data-driven learning for diverse degradation types.

  • Method

    SGUIE-Net uses semantic associations and a multi-scale, dual-branch architecture to guide enhancement across semantically related regions and degradation types.

  • Results

    Experiments on four real-world underwater benchmarks verified SGUIE-Net's effectiveness and color restoration accuracy, while ablations supported its principal modules.

  • Takeaways & Limitations

    Semantic guidance connects uncommon degradation types with learned distributions while supporting global enhancement and recovery of clear local details.

Abstract

from arXiv · show

Due to the wavelength-dependent light attenuation, refraction and scattering, underwater images usually suffer from color distortion and blurred details. However, due to the limited number of paired underwater images with undistorted images as reference, training deep enhancement models for diverse degradation types is quite difficult. To boost the performance of data-driven approaches, it is essential to establish more effective learning mechanisms that mine richer supervised information from limited training sample resources. In this paper, we propose a novel underwater image enhancement network, called SGUIE-Net, in which we introduce semantic information as high-level guidance across different images that share common semantic regions. Accordingly, we propose semantic region-wise enhancement module to perceive the degradation of different semantic regions from multiple scales and feed it back to the global attention features extracted from its original scale. This strategy helps to achieve robust and visually pleasant enhancements to different semantic objects, which should thanks to the guidance of semantic information for differentiated enhancement. More importantly, for those degradation types that are not common in the training sample distribution, the guidance connects them with the already well-learned types according to their semantic relevance. Extensive experiments on the publicly available datasets and our proposed dataset demonstrated the impressive performance of SGUIE-Net. The code and proposed dataset are available at: https://trentqq.github.io/SGUIE-Net.html

I. Introduction · II. Related Works · A. Deep Enhancement Models Trained with Paired Samples

Underwater enhancement is challenged by dynamically changing conditions, light attenuation, low contrast, and blurred details, while scarce paired data limits deep models’ coverage of diverse degradations. The paper addresses this gap with SGUIE-Net, which uses semantic associations and multi-scale perception to guide robust, scene-consistent enhancement.

  • I. Introduction: Underwater robots support ocean exploration, but changing local conditions, floating sand, plankton, and illumination differences complicate underwater imaging.These conditions contribute to challenging underwater vision environments.
  • I. Introduction: Light attenuation produces low contrast and blurred details, motivating extensive research into improving underwater image visual quality.The degradation directly challenges vision-based underwater operations.
  • I. Introduction: In-situ acquisition of degraded and clear underwater images is nearly impossible, limiting paired samples for models covering diverse degradation types.Synthetic underwater datasets were proposed as a compromise.
  • I. Introduction: Semantic maps provide region boundaries and high-level associations between images, supporting reliable supervision, consistent enhancement of related regions, and improved training generalization.The paper identifies semantic maps as auxiliary constraints for mining supervisory information and robust enhancement.
  • I. Introduction: SGUIE-Net introduces semantic attention guidance and multi-scale perception to build enhancement guidance for uncommon degradations that are semantically related to well-learned types.The network incorporates high-level semantic information into its learned features.
  • A. Deep Enhancement Models Trained with Paired Samples: CycleGAN-based synthesis, FGAN, DenseGAN, MLFcGAN, and FUnIE-GAN use generated paired underwater images, while UIEB provides 890 manually selected real-world paired images.UIEB was constructed as a real-world underwater image enhancement benchmark.
  • A. Deep Enhancement Models Trained with Paired Samples: Using UIEB, Water-Net fuses inputs through three predicted confidence maps, whereas UICoE-Net introduces correlation feature matching units to communicate mutual correlation.These approaches seek to exploit limited high-quality underwater training images through learning strategies and network structures.
  • A. Deep Enhancement Models Trained with Paired Samples: Paired-sample shortages constrain existing models, and high-level scene-consistent constraints remain insufficient despite efforts to share features among images.The passage attributes same-scene nonconsistency partly to ambiguity in training labels.

B. Semantic Guidance in Computer Vision · III. Proposed Method · A. Overall Network with Multi-Scale Perception

Semantic guidance has been used across high-, mid-, and low-level computer-vision tasks, while SGUIE-Net applies semantic information as a high-level, cross-image constraint for underwater enhancement. Its two-branch design combines original-scale attention-aware enhancement with multi-scale semantic perception to address uneven degradation and preserve detail texture.

  • B. Semantic Guidance in Computer Vision: Semantic guidance has been widely explored in visual navigation, image generation, image-image matching, image-sentence matching, and Re-IDentification.In these tasks, semantic information commonly constrains associated results and provides guiding or error-correcting signals.
  • B. Semantic Guidance in Computer Vision: Semantic information is increasingly used in low-level tasks including image enhancement, restoration, and manipulation.The cited examples establish semantic guidance as an emerging component of low-level image processing.
  • A. Overall Network with Multi-Scale Perception: SGUIE-Net contains two enhancement branches for multi-scale perception: an original-scale main branch and a semantic-level bottom branch.The main branch uses cascaded attention-aware enhancement, residual groups, and semantic guided feature extraction and fusion; the bottom branch learns multi-scale semantic association features.
  • B. Semantic Guidance in Computer Vision: Semantic context can guide gray-level tonemapping, emphasize semantically relevant colors, and perform defocus magnification in image enhancement.The passage describes semantic concepts being linked to image characteristics before applying these guided operations.
  • A. Overall Network with Multi-Scale Perception: SGUIE-Net uses semantic information as a high-level, cross-image constraint to mine limited supervisory information more effectively.Input images are partitioned into semantic-object sub-regions using manual masks or pre-trained underwater segmentation models such as SUIM-Net.
  • A. Overall Network with Multi-Scale Perception: The original-scale main branch provides differentiated enhancement for uneven degradation while maintaining detail texture.Its cascaded attention-aware enhancement module addresses spatial-detail loss associated with encoder-decoder structures.

B. Cascaded Attention-aware Enhancement Module

The cascaded attention-aware enhancement module (CAM) is the main-branch structure designed to preserve detailed texture and high-level dependencies while learning multiscale image features at the original resolution.

  • B. Cascaded Attention-aware Enhancement Module: CAM operates at the original image scale to preserve detailed texture appearance and reduce the spatial-detail losses of encoder-decoder enhancement methods.The module is introduced because repeated downsampling can lose pixel-wise spatial details and dependencies in high-level features.
  • B. Cascaded Attention-aware Enhancement Module: CAM uses three cascaded residual groups with skip connections to construct the main branch.Each residual group stacks four feature attention blocks, while long skip connections address training difficulties from network depth.
  • B. Cascaded Attention-aware Enhancement Module: Each feature attention block cascades channel attention and pixel attention to learn higher-level features.The architecture combines channel- and pixel-attention modules within the feature attention block.

D. Semantic Guidance Extraction and Fusion Module

The semantic guidance fusion module combines global attention features with multi-scale semantic-region attention to compensate for CAM’s missing multi-scale information. It crops corresponding global regions, fuses them with semantic features, and returns semantically guided global features to CAM for enhancement.

  • Semantic Guidance Extraction and Fusion Module: SGF fuses the raw image’s global attention feature with semantic-region residual features and a semantic region mask.The three inputs are fg ∈ R^C×H×W, R ∈ R^C×h×w, and m ∈ R^1×h×w.
  • Semantic Guidance Extraction and Fusion Module: Residual semantic-region features pass through double Conv-BN-ReLU units and a sigmoid function to produce multi-scale semantic region attention features.The resulting feature is denoted X_k for the k-th semantic region.
  • Semantic Guidance Extraction and Fusion Module: The module uses enlarged semantic-region rectangles for context awareness but removes surrounding redundancy during fusion, retaining attention-aware regional features.The semantic regions input to SRM is the largest outer rectangle around the actual semantic object.
  • Semantic Guidance Extraction and Fusion Module: Corresponding global feature regions are cropped using semantic-region locations and combined element-wise with semantic attention features.This produces the fused regional feature of the main branch.
  • Semantic Guidance Extraction and Fusion Module: The resulting global features injected with semantic guidance are sent to CAM for residual perception and final enhancement.This completes the feedback of semantic guidance into the main enhancement branch.

E. Loss Function … B. Experiment Settings

The paper uses pixel-wise ℓ2 supervision to preserve sharp edges and details, and evaluates SGUIE-Net through dataset construction, comparisons, and ablation studies. Experiments employ a specified PyTorch training setup and introduce SUIM-E to provide enhancement references for semantic guidance.

  • E. Loss Function: The ℓ2 loss minimizes pixel-wise error to better preserve sharp edges and details in enhanced images.The loss is defined over the enhancement and ground-truth images.
  • E. Loss Function: In the loss formulation, E and EGT denote the enhancement of input image I and its ground truth, while i and j index pixel locations.
  • IV. Experiments: The experiments introduce the SUIM-E dataset, compare SGUIE-Net with state-of-the-art methods on six datasets, and conduct ablation studies for each component.
  • IV. Experiments: The study includes experimental settings, SUIM-E construction, state-of-the-art comparisons, and ablations to assess SGUIE-Net’s components.
  • A. Implementation Details: SGUIE-Net is implemented in PyTorch with Kaiming initialization, Adam optimization, an initial learning rate of 1e-4, linear decay to zero, and batch size 1.Training uses 256×256 inputs with random cropping and flipping on an NVIDIA RTX 3090 GPU.
  • B. Experiment Settings: SUIM-E supplements SUIM underwater images with hand-picked corresponding enhancement references because existing underwater datasets lack both references and semantic segmentation maps.The dataset was constructed to more fully verify the semantic-guided network.

1) Datasets: · 2) Compared Methods:

The evaluation uses UIEB and additional underwater datasets, with semantic maps predicted by pre-trained SUIM-Net. SGUIE-Net is compared against six recent methods spanning physical-model, physical-model-free, and deep-learning approaches, using visual and quantitative assessments.

  • 1) Datasets:: UIEB contains 890 underwater images with manually selected enhancement references, including 800 used for training/validation and 90 for testing.Another 60 challenging reference-free images are also used for testing.
  • 1) Datasets:: Pre-trained SUIM-Net predicts semantic segmentation maps for both training and test images.The method is additionally evaluated for generalizability and color-correction accuracy on RUIE.
  • 2) Compared Methods:: SGUIE-Net is compared with six methods from the previous three years: ULAP, CBF, HUE, Water-Net, Ucolor, and UICoE-Net.The comparison includes one physical model-based method, two physical model-free methods, and three deep-learning-based methods.
  • 2) Compared Methods:: Visual comparisons on the SUIM-E test set include raw underwater images, six competing methods, SGUIE-Net, and reference images.The displayed competitors are ULAP, CBF, HUE, Water-Net, Ucolor, and UICoE-Net.
  • 2) Compared Methods:: Quantitative comparison on the SUIM-E test set reports MSE(×103), PSNR, SSIM, UIQM, and UCIQE.Traditional methods and paired-reference deep-learning methods are separated with lines, and the top three scores are marked in red, green, and blue.
  • 2) Compared Methods:: The compared methods comprise ULAP as a physical model-based method, CBF and HUE as physical model-free methods, and Water-Net, Ucolor, and UICoE-Net as deep-learning methods.These method categories are explicitly used to organize the comparison.
  • 2) Compared Methods:: Because CBF source code was unavailable, its results use code reproduced by another researcher, while other methods use codes released by their authors.This establishes the implementation conditions for the comparison.

C. Evaluation of Visual Enhancement Performance

Across SUIM-E, UIEB, RUIE, and EUVP, SGUIE-Net provides stable, natural enhancement under color distortion, low light, backscatter, and other challenging conditions. It achieves the best reported full-reference results on UIEB and the highest visual perceptual quality on the evaluated challenging datasets, even with incomplete semantic guidance.

  • Qualitative comparisons: SGUIE-Net stably corrects severe color deviations without introducing extra color distortion and produces more natural results through semantic region-wise enhancement.The module fine-tunes enhancement according to scene context and semantic regions.
  • Quantitative comparisons: On UIEB, SGUIE-Net achieves the best results among tested methods on all three full-reference metrics.This agrees with the qualitative comparisons, where competing traditional methods fail or introduce oversaturation and artifacts on difficult images.
  • Semantic-guidance robustness: Despite using predicted, unfine-tuned semantic segmentation on UIEB, SGUIE-Net achieves the best score among all compared methods and the highest visual perceptual quality score across the reported challenging datasets.The predicted semantic information may be incomplete, limiting performance, but the method remains comparatively stable.
  • Challenging cases: On UIEB challenging images, SGUIE-Net handles low lighting and color deviation while smoothly improving contrast, whereas competing methods fail in extreme or poorly learned scenarios.ULAP fails, HUE produces excessive saturation, CBF retains low contrast, and Water-Net and UICoE-Net fail in extreme cases.
  • Generalization and robustness: Across RUIE and EUVP, SGUIE-Net generalizes robustly by avoiding extra colors and improving contrast, brightness, and detailed textures more effectively than the compared methods.Water-Net, UICoE-Net, and Ucolor remove bluish or greenish distortion but remain unsatisfactory in these other visual properties.

D. Evaluation of Color Restoration Performance

The color-restoration evaluation tests robustness and accuracy on SQUID and Color-Checker7 using comparison models trained on SUIM-E. Qualitative and quantitative analyses assess performance across challenging depths, imaging devices, and color-patch differences.

  • Evaluation setup: Comparisons on SQUID and Color-Checker7 use models trained on the SUIM-E dataset for testing.These datasets are used to analyze color-restoration robustness and accuracy.
  • SQUID evaluation: SQUID images captured at depths from 3–6 meters to 20–30 meters expose diverse challenges for the comparison methods.The evaluation includes underwater images photographed across substantially different water depths.
  • SQUID evaluation: Water-Net and UICoE-Net remove haze better but fail to recover color, whereas Ucolor slightly improves color yet fails to deblur and recover details.The comparison highlights different weaknesses in haze removal, color recovery, deblurring, and detail restoration.
  • Quantitative color evaluation: Color restoration is quantitatively evaluated with average reproduction angular error (AE), denoted as ψ, between gray-scale patches and corresponding pure gray colors in RGB space.The authors use Berman et al.’s evaluation code to assess deep learning-based methods.
  • Color-Checker7 evaluation: Color-Checker7 contains seven underwater images from a shallow swimming pool captured with different cameras, with a color checker photographed in each image.The dataset evaluates robustness to different imaging devices and color-restoration accuracy.
  • Color-Checker7 evaluation: Color-Checker7 color restoration is measured with CIEDE2000 by comparing enhancement-result patches against corresponding ground-truth Macbeth Color Checker patches.The measurement follows Ancuti et al.’s procedure.

E. Ablation Study

The ablation study shows that the complete SGUIE-Net outperforms variants lacking CAM, SRM, or FAB, while semantic multi-scale guidance and attention components improve quantitative and visual enhancement quality.

  • Overall ablation results: The full SGUIE-Net achieves the best SSIM, PSNR, and MSE performance on both SUIM-E and UIEB compared with all ablation models.This supports the effectiveness of the CAM, SRM, and FAB modules.
  • Semantic region-wise enhancement: Nearly 4% higher SSIM, 13% higher PSNR, and 39% lower MSE are obtained with SRM than without SRM on SUIM-E.SRM captures multi-scale perception and semantic relations across images to provide high-level enhancement guidance.
  • Semantic region-wise enhancement: Proper semantic guidance improves color correction and produces visually pleasing enhancements relative to models without SRM, with random region splits, or without the U-Net block.Figure 15 compares these variants and the complete model.
  • Feature attention block: The w/o-PA and w/o-CA variants outperform w/o-FAB, but both remain below the full model on SSIM, PSNR, and MSE.The PSNR score of w/o-PA is about 0.18 higher than w/o-CA.
  • Feature attention block: Removing channel attention can introduce artificial colors because pixel-only attention lacks weighted perception of different feature components.The FAB module combines cascaded channel attention and pixel attention.

V. Conclusion

SGUIE-Net combines semantic attention guidance with multi-scale perception for underwater image enhancement. Its semantic and dual-branch architectures address regionally uneven degradation while improving global enhancement and local detail recovery.

  • Conclusion: SGUIE-Net introduces semantic information as high-level guidance through a semantic region-wise enhancement module.The module bridges uncommon degradation types and the learned distribution of underwater degradation.
  • Conclusion: The attention mechanism senses uneven degradation across semantic regions and feeds this information back to global attention features.This feedback uses embedded semantic guided feature extraction and fusion.
  • Conclusion: A complementary dual-branch, multi-scale feature perception architecture enables global enhancement while recovering clear local details.The architecture is designed to obtain good global enhancement and restore local detail clarity.
  • Conclusion: The paper reports extensive experiments on four real-world underwater benchmark datasets.The supplied conclusion passage states the evaluation scope but does not provide quantitative results.

(Supplementary Material)

The supplementary material details the newly constructed SUIM-E dataset and presents additional visual comparisons across several underwater image benchmarks.

  • It provides further details about the newly constructed SUIM-E dataset.
  • It includes visual comparisons on the UIEB challenging set, RUIE, EUVP, and SQUID benchmarks.

I. More Details about the Proposed SUIM-E Dataset · II. More Visual Comparisons with Perceptual Scores

The paper introduces SUIM-E as a real-world underwater enhancement dataset with references and semantic annotations, and provides broader visual comparisons with perceptual scores across four datasets.

  • I. More Details about the Proposed SUIM-E Dataset: SUIM-E supplements the SUIM dataset with corresponding enhancement references for raw underwater images.The references were generated using 12 underwater image enhancement methods and a commercial application.
  • I. More Details about the Proposed SUIM-E Dataset: The candidate-reference generation uses CE, Fusion, GCHE, HistogramPiror, HUE, IBLA, Retinex, TwoStep, UCM, ULAP, DCP, and dive+ 1.These methods provide the candidate enhancement references used to supplement SUIM.
  • I. More Details about the Proposed SUIM-E Dataset: SUIM-E contains 1635 real-world underwater images with corresponding high-quality reference images.The dataset also includes pixel annotations for eight object categories.
  • I. More Details about the Proposed SUIM-E Dataset: The eight annotated object categories include fish, reefs, aquatic plants, wrecks/ruins, human divers, robots, and sea-floor.Fish are identified as vertebrates and reefs as invertebrates in the supplied passage.
  • I. More Details about the Proposed SUIM-E Dataset: SUIM-E is presented as the first real-world underwater dataset containing both corresponding enhancement references and semantic segmentation maps.This combines enhancement supervision with semantic segmentation information.
  • II. More Visual Comparisons with Perceptual Scores: Additional visual comparisons are provided on the UIEB challenging set, RUIE, EUVP, and SQUID datasets.The comparisons appear in Figures 4, 5, 6, and 7, respectively.
  • II. More Visual Comparisons with Perceptual Scores: Each enhancement result is labeled with a perceptual score, offering an intuitive comparison across the evaluated datasets.The passage states that these scores further demonstrate the superiority of the proposed method.
Loading 2201.02832v1…