PaperlayerView exact revision on arXiv

Source-linked AI summary

Underwater Image Restoration Through a Prior Guided Hybrid Sense Approach and Extensive Benchmark Analysis

Xiaojiao Guo, Xuhang Chen, Shuqiang Wang, Chi-Man Pun

arXiv:2501.02701v1cs.CV

TL;DR

Underwater images are degraded by color distortion and reduced clarity from light-water interactions. The paper introduces a multiscale prior-guided hybrid restoration framework and a real-world benchmark, reporting overall superiority to 37 existing methods despite not leading in every individual case.

  • Problem

    Underwater images suffer color casts, blur, and reduced clarity, while existing UIR benchmarks lack standardized datasets and evaluation metrics.

  • Method

    The paper combines CNN-based Detail Restorers, a Transformer-based Feature Contextualizer, multiscale processing, and a Color Balance Prior in GuidedHybSensUIR.

  • Results

    The framework outperforms 37 existing state-of-the-art methods overall across benchmark datasets and metrics, though it does not achieve the best result in every individual case.

  • Takeaways & Limitations

    The paper provides a prior-guided multiscale restoration framework and a standardized real-world benchmark for comparing underwater image restoration methods.

Abstract

from arXiv · show

Underwater imaging grapples with challenges from light-water interactions, leading to color distortions and reduced clarity. In response to these challenges, we propose a novel Color Balance Prior \textbf{Guided} \textbf{Hyb}rid \textbf{Sens}e \textbf{U}nderwater \textbf{I}mage \textbf{R}estoration framework (\textbf{GuidedHybSensUIR}). This framework operates on multiple scales, employing the proposed \textbf{Detail Restorer} module to restore low-level detailed features at finer scales and utilizing the proposed \textbf{Feature Contextualizer} module to capture long-range contextual relations of high-level general features at a broader scale. The hybridization of these different scales of sensing results effectively addresses color casts and restores blurry details. In order to effectively point out the evolutionary direction for the model, we propose a novel \textbf{Color Balance Prior} as a strong guide in the feature contextualization step and as a weak guide in the final decoding phase. We construct a comprehensive benchmark using paired training data from three real-world underwater datasets and evaluate on six test sets, including three paired and three unpaired, sourced from four real-world underwater datasets. Subsequently, we tested 14 traditional and retrained 23 deep learning existing underwater image restoration methods on this benchmark, obtaining metric results for each approach. This effort aims to furnish a valuable benchmarking dataset for standard basis for comparison. The extensive experiment results demonstrate that our method outperforms 37 other state-of-the-art methods overall on various benchmark datasets and metrics, despite not achieving the best results in certain individual cases. The code and dataset are available at \href{https://github.com/CXH-Research/GuidedHybSensUIR}{https://github.com/CXH-Research/GuidedHybSensUIR}.

I. INTRODUCTION

Underwater images suffer from color casts, blur, and reduced clarity caused by light absorption, scattering, and suspended particles. The paper proposes a multiscale hybrid restoration framework with a Color Balance Prior and establishes a broad benchmark for comparison.

  • Motivation: Underwater imaging exhibits blue or green color casts and blurriness because water differentially absorbs wavelengths while scattering and suspended particles reduce clarity.Red light is absorbed most quickly, followed by green and blue, and these degradations hinder marine segmentation, localization, and recognition.
  • Motivation: UIR must restore blurred details as well as correct color, contrast, and saturation to approach in-air image quality.
  • Approach: CNN-based local processing and Transformer-based long-range modeling are combined in a U-shaped Hybrid Sense UIR network for fine- and coarse-scale restoration.The hybrid design addresses fine-scale blurry details and coarser-scale color performance using complementary sensing capabilities.
  • Approach: The Color Balance Prior guides color restoration and converts the Hybrid Sense UIR system into the Guided Hybrid Sense UIR Framework.It is defined from RGB channel averages and is intended to direct restoration toward balanced color distributions.
  • Benchmark: The benchmark aggregates 5,600 training images, 490 paired testing samples, and 460 non-reference test samples from four real-world datasets, covering 37 existing UIR methods.The benchmark addresses inconsistent training data, test data, and evaluation metrics across prior studies.
  • Results: Extensive experiments report that the proposed framework outperforms state-of-the-art methods overall on underwater image restoration.

1) Traditional Approaches:

Prior underwater restoration methods include model-based, model-agnostic, and deep learning approaches, but real-world complexity, synthetic-to-real gaps, and evaluation limitations remain. The paper emphasizes diverse real-world datasets for robust development and assessment.

  • 1) Traditional Approaches:: Model-based methods estimate physical variables with handcrafted priors, but often struggle with complex real-world underwater scenes.Examples include transmission, illumination, dark-channel, attenuation, statistical, blurriness, haze-line, and illumination-sparsity priors.
  • 1) Traditional Approaches:: Model-agnostic methods use contrast adjustment, histogram equalization, color correction, fusion, CLAHE, UCM, and Retinex refinements without physical imaging models.
  • 2) Deep Learning-based Approaches:: Early deep learning methods predicted physical imaging elements, while later paired-data approaches learned directly from underwater images and clear references.CNN, GAN, Transformer, semi-supervised, and unsupervised methods represent major directions in the field.
  • 2) Deep Learning-based Approaches:: Deep learning methods incorporating priors use transmission, physical knowledge, illumination, and color histograms to address color casts, visibility, and global degradation.
  • 2) Deep Learning-based Approaches:: Quantitative and qualitative analyses indicate that existing deep learning methods still have room for improvement.
  • B. Underwater Image Datasets: Synthetic datasets can create a gap from real underwater images, so the benchmark excludes them to maintain relevance to real-world applications.
  • B. Underwater Image Datasets: Real-world datasets include paired and non-reference collections spanning varied environments, illumination, depth of field, blur, and color casts.Such diversity is described as essential for developing and evaluating methods that generalize to real-world scenarios.

III. METHODOLOGY

GuidedHybSensUIR uses a U-shaped architecture to combine local detail restoration with global color and contextual enhancement across multiple scales. Its Detail Restorer, Feature Contextualizer, and Scale Harmonizer modules use quaternion-based and attention-based processing to reconstruct underwater images.

  • Overall architecture: The U-shaped architecture combines CNN-based local restoration with Transformer-based global enhancement and decoder-side multi-scale fusion.The encoder uses Detail Restorer modules for local details, while the bottleneck Feature Contextualizer models long-range relationships and the decoder merges features across scales.
  • Overall architecture: The Color Balance Prior guides global color correction strongly in the Feature Contextualizer and weakly during final decoding.The prior is embedded in the Feature Contextualizer and skip-connected to the decoder output.
  • 1) Detail Restorer: The Detail Restorer uses six sequential quaternion convolutional units integrating Residual Context Blocks and Nonlinear Activation-Free Blocks.Quaternion convolutions combine the RCB and NAFB branches in a constrained fusion process.
  • 1) Detail Restorer: Quaternion feature fusion constrains interactions between RCB and NAFB outputs, promoting more stable cooperation than unconstrained branch fusion.The quaternion representation sets the real component to zero and combines branch outputs through Hamilton-product-based convolution.
  • 1) Detail Restorer: The NAFB selectively passes or suppresses information through gated element-wise operations, while the RCB captures channel- and spatial-contextual information.The RCB uses contextual feature maps and residual connections; the NAFB uses convolutional processing followed by a simple gate.

2) Feature Contextualizer:

The Feature Contextualizer operates at a coarser scale to model global context and color dependencies through stacked Multi-Attention Quaternion blocks. Its ACT, KFT, and SAT branches combine prior-guided cross-attention with image-feature self-attention.

  • Feature Contextualizer: The Feature Contextualizer uses four residual Multi-Attention Quaternion blocks followed by Spatial Pyramid Pooling at four pooling granularities.Each MAQ block contains ACT, KFT, and SAT transformers operating in parallel.
  • Attention mechanisms: The ACT uses prior features as Query and image features as Key and Value to emphasize image channels similar to the Color Balance Prior.Its inter-channel cross-attention adjusts image contextual information toward the prior-guided color representation.
  • Feature Contextualizer: The contextualizer focuses on inter-channel rather than inter-patch relationships, reducing the computational burden associated with quadratic Vision Transformer attention.Earlier Detail Restorer modules handle local details, allowing the contextualizer to operate on broader contextual relationships.
  • Attention mechanisms: The KFT uses image features as Query and prior features as Key and Value to retain prior information consistent with the image.This design suppresses inappropriate prior information because the color balance prior may be inaccurate.
  • Attention mechanisms: The SAT applies self-attention within image features to capture their inter-channel dependencies and internal relationships.Unlike ACT and KFT, SAT does not use cross-attention with the color balance prior.

3) Scale Harmonizer:

The Scale Harmonizer integrates features from the bottleneck, lower decoder layers, and same-scale Detail Restorer skip connections. Feature Calibrators then modulate the fused representation with conditioned scaling and shifting.

  • Scale Harmonizer: Scale Harmonizers concatenate incoming decoder features with skip-connected Detail Restorer features at the corresponding scale before harmonization.Incoming features may originate from the Feature Contextualizer or from an upsampled lower decoding layer.
  • Feature Calibrator: The Feature Calibrator modulates input features using learnable scale and shift parameters conditioned on three convolutional feature granularities.The conditioned parameters are generated from averaged multi-granularity features extracted from the input feature map.
  • Feature Calibrator: Three groups of Feature Calibrators harmonize fused channels by sequentially scaling features and adding learned biases.This process adjusts feature amplitudes and biases to optimize the fused representation.

B. Color Balance Prior

The Color Balance Prior models balanced RGB intensity as a guide for correcting wavelength-induced underwater color casts. It is integrated strongly for global contextual correction and weakly near final decoding to preserve learned features while maintaining color balance.

  • Color-balance motivation: Underwater wavelength-dependent attenuation motivates balancing red, green, and blue channel intensities toward an in-air-like distribution.The formulation focuses on estimating and adjusting illumination because geometry is treated as constant and reflectance is illumination-invariant.
  • Prior derivation: The derivation links each channel’s average intensity ai to its illuminant Ii, yielding equal channel averages when ideal in-air illumination has IR = IG = IB.The derivation assumes constant illumination, independent geometry and reflectance, and uniformly distributed scene colors.
  • Prior construction: The prior is formed from the average of the red, green, and blue channels, assigning the same value to all three channels at each pixel.This simple construction follows the even-intensity condition associated with the Gray-World Assumption.
  • Guidance role: The prior is intended to steer restoration toward enhanced and consistent color results rather than being used directly as a raw value.The model extracts feature embeddings from the prior because direct values may not capture contextual relationships and intricate color-correction patterns.
  • Guidance role: The Color Balance Prior guides global color correction strongly in the Feature Contextualizer and is merged once near final decoding as a subtle color reminder.This placement balances long-range feature interaction with the prior’s approximate nature.

C. Loss Function

The restoration objective combines pixel fidelity, structural similarity, and perceptual similarity losses. Their weighted combination is designed to preserve image content while aligning restored outputs with reference images and human visual perception.

  • Composite objective: The composite loss is L = w1 · Lf + w2 · Ls + w3 · Lp, combining fidelity, structural, and perceptual objectives.The empirical weights are w1 = 1, w2 = 0.3, and w3 = 0.7.
  • Pixel fidelity: Lf uses SmoothL1Loss to measure pixel-level discrepancies between the restored image R and target reference T while balancing large-error sensitivity and outlier robustness.The threshold β controls the quadratic-to-linear transition and is set to β = 1.0.
  • Structural similarity: Ls maintains similarity in luminance, contrast, and structure between restored and target images.Its formulation includes the Structural Similarity Index Measure.
  • Perceptual similarity: Lp uses LPIPS to compare restored and target images through features extracted from a pretrained AlexNet model on ImageNet.LPIPS supplies the perceptual component of the composite objective.

A. Benchmark

The paper constructs a multi-dataset benchmark to address inconsistent comparisons caused by differing training data, test data, and metrics. Balanced data selection and paired and unpaired evaluation sets are intended to support fairer assessment of UIR methods.

  • Benchmark motivation: Existing UIR methods lack a standardized benchmark because studies differ in training datasets, testing datasets, and evaluation metrics.
  • Training data: The benchmark uses training data from UIEB, EUVP, and LSUI, combining diverse real-world underwater images from three datasets.UIEB contributes higher-resolution paired data, while EUVP and LSUI provide larger paired subsets.
  • Training data: The training set is balanced by selecting 2,000 paired images each from EUVP and LSUI and duplicating UIEB training data to reduce dataset dominance.
  • Test data: Evaluation includes unpaired test sets from UIEB, EUVP, and RUIE to assess generalization across untrained domains and varying image difficulties.The unpaired sets contain 60 UIEB images and 200 images each from EUVP and RUIE.
  • Benchmark purpose: Balancing multiple high-quality sources is intended to promote fair assessments and meaningful comparisons without bias toward a single dataset.

B. Evaluation Metrics

The evaluation combines reference-based metrics for reconstruction, structure, and perception with non-reference underwater-quality metrics. The paper also reports visual comparisons and cautions that UCIQE and UIQM can diverge from perceived restoration quality.

  • Reference-based metrics: PSNR, SSIM, and LPIPS evaluate pixel accuracy, structural similarity, and perceptual similarity against reference images.
  • Visual comparisons: Visual comparisons report stronger color correction, contrast, texture preservation, and detail restoration than traditional and deep learning-based alternatives.The reported advantages include performance in both well-lit and poorly lit regions.
  • Non-reference metrics: UCIQE and UIQM evaluate underwater image quality without requiring reference images.UCIQE considers chroma and saturation, while UIQM includes colorfulness, sharpness, and contrast.
  • Metric limitations: UCIQE and UIQM may not accurately reflect perceptual quality because their simplified HVS features and linear models neglect nonlinear human perception.
  • Metric limitations: The benchmark retains UCIQE and UIQM for generalization assessment while acknowledging their limitations.

C. Implementation Details

The paper evaluates GuidedHybSensUIR against conventional and deep learning restoration methods on paired and unpaired real-world underwater test sets. Results show strong full-reference and non-reference performance, favorable efficiency, and qualitative effectiveness despite imperfect UCIQE ranking.

  • Benchmark and evaluation: 37 methods were compared, including 14 conventional and 23 deep learning approaches retrained on the same benchmark dataset.The comparison covered consistent training settings for deep learning methods.
  • Benchmark and evaluation: Paired evaluation used UIEB, EUVP, and LSUI with PSNR, SSIM, LPIPS, UCIQE, and UIQM.The paired test sets contained 90 UIEB, 200 EUVP, and 200 LSUI images.
  • Paired results: The method achieved the highest scores across most full-reference metrics, while ranking second to Semi-UIR in UIEB PSNR by 0.04%.It exceeded Semi-UIR by 3.9% in SSIM and 16.7% in LPIPS on UIEB, and outperformed it across all reported metrics on EUVP and LSUI.
  • Metric results: GuidedHybSensUIR achieved the highest UCIQE on UIEB and the second-highest scores on EUVP and LSUI among deep learning methods.UCIQE and UIQM were treated as indicators of colorfulness, sharpness, and contrast despite known limitations.
  • Efficiency: The method was not the most efficient, but its computational cost was lower than that of most other deep learning methods.This indicates a balance between restoration performance and resource use.
  • Unpaired results: On unpaired UIEB, EUVP, and RUIE sets, the method did not attain the highest average UCIQE score but showed strong generalization across diverse domains.A visual comparison reported superior restored-image quality despite a third-place average UCIQE ranking among deep learning approaches.
  • Overall finding: The reported quantitative results collectively highlight the effectiveness of GuidedHybSensUIR.This conclusion follows the paired and unpaired benchmark analyses.

2) Qualitative Comparison:

Qualitative comparisons show that GuidedHybSensUIR improves color, contrast, texture, and fine-grained detail across challenging underwater scenes. Ablations further attribute these gains to complementary contextual, detail-restoration, multiscale, prior-guided, and skip-connected components.

  • Qualitative comparison: Traditional methods often fail to remove underwater haze or introduce new color distortions in complex ocean environments.The qualitative comparison contrasts these failures with stronger deep learning restoration results.
  • Qualitative comparison: GuidedHybSensUIR produces more saturated and contrasted colors while restoring texture and fine-grained color details in highlighted and dark regions.The comparison reports improvements in both color and texture details.
  • Qualitative comparison: Its qualitative generalization is reported as superior to other high-UCIQE methods across unpaired UIEB, EUVP, and RUIE images.The comparison emphasizes robustness across images from varied sources and characteristics.
  • Ablation studies: ACT and KFT significantly improve ablation performance by modeling long-range dependencies and global color relationships.SAT also had the largest individual impact among Feature Contextualizer components.
  • Ablation studies: Combining Detail Restorer and Feature Contextualizer improves all metrics over Feature Contextualizer alone, with substantial PSNR and SSIM gains.The result emphasizes Detail Restorer’s role in recovering fine-grained details.
  • Ablation studies: Scale Harmonizer modules further improve results by integrating multiscale features.This supports the role of cross-scale feature fusion in the architecture.
  • Ablation studies: The Color Balance Prior and its skip connection significantly improve performance over the baseline, guiding accurate color restoration.A separate input-to-output skip connection also consistently improves results by preserving information and facilitating multiscale fusion.
  • Ablation studies: The full model achieves the best overall performance when all components are enabled, combining global and local enhancements with prior guidance and skip connections.The ablation attributes the result to a synergistic architecture.
Loading 2501.02701v1…