PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Adaptation for Underwater Image Enhancement

Zhengyong Wang, Liquan Shen, Mei Yu, Kun Wang, Yufei Lin, Mai Xu

arXiv:2108.09650v1cs.CV

TL;DR

Synthetic-data-trained underwater enhancement models face synthetic-to-real and within-real distribution gaps. TUDA addresses them with dual alignment, rank-based quality assessment, and easy-hard adaptation, while reported evaluations show strong perceptual-quality and quality-assessment performance.

  • Problem

    Synthetic-trained models face an inter-domain gap with real underwater images, while varied real degradations create an intra-domain gap that can produce artifacts and over-enhancement.

  • Method

    TUDA uses dual image- and feature-level alignment first, then rank-based quality assessment to divide real data into easy and hard samples for pseudo-label-based adaptation.

  • Results

    TUDA achieves the best reported RUIQA and perceptual scores among compared methods, while RUIQA correlates with MOS at about 0.900 and gains 0.5 to 0.65 over UCIQE and UIQM.

  • Takeaways & Limitations

    The results support jointly adapting across synthetic-real and easy-hard real-image distributions for more convincing enhancement across diverse underwater scenes.

Abstract

from arXiv · show

Recently, learning-based algorithms have shown impressive performance in underwater image enhancement. Most of them resort to training on synthetic data and achieve outstanding performance. However, these methods ignore the significant domain gap between the synthetic and real data (i.e., interdomain gap), and thus the models trained on synthetic data often fail to generalize well to real underwater scenarios. Furthermore, the complex and changeable underwater environment also causes a great distribution gap among the real data itself (i.e., intra-domain gap). However, almost no research focuses on this problem and thus their techniques often produce visually unpleasing artifacts and color distortions on various real images. Motivated by these observations, we propose a novel Two-phase Underwater Domain Adaptation network (TUDA) to simultaneously minimize the inter-domain and intra-domain gap. Concretely, a new dual-alignment network is designed in the first phase, including a translation part for enhancing realism of input images, followed by an enhancement part. With performing image-level and feature-level adaptation in two parts by jointly adversarial learning, the network can better build invariance across domains and thus bridge the inter-domain gap. In the second phase, we perform an easy-hard classification of real data according to the assessed quality of enhanced images, where a rank-based underwater quality assessment method is embedded. By leveraging implicit quality information learned from rankings, this method can more accurately assess the perceptual quality of enhanced images. Using pseudo labels from the easy part, an easy-hard adaptation technique is then conducted to effectively decrease the intra-domain gap between easy and hard samples.

I. INTRODUCTION

Underwater enhancement must address degradation, the synthetic-to-real inter-domain gap, and substantial variation among real images. TUDA responds with two adaptation phases combining dual alignment, quality ranking, and easy-hard adaptation.

  • Challenges: Underwater images commonly suffer blurriness, color casts, low contrast, and bluish or greenish tones caused by light absorption and suspended particles.Enhanced visibility can also support downstream detection and recognition tasks.
  • Challenges: Synthetic-to-real domain shift causes models trained on synthetic data to lose performance on real images with different distortion distributions.This inter-domain gap is illustrated as a major challenge for generalization to real underwater scenarios.
  • Challenges: Learning-based methods benefit from data and computation, but distortion-free real underwater reference images are impractical to collect, encouraging synthetic-data training.Synthetic data provides corresponding reference images and can simulate multiple underwater types and degradation levels.
  • Challenges: Illumination, water bodies, depth, seasons, and weather create diverse real-image degradations, producing an intra-domain gap that can yield artifacts, noise, and over-enhancement.Models may perform well on some real images yet poorly on others when this variation is ignored.
  • Proposed approach: TUDA jointly addresses both gaps through inter-domain dual alignment and intra-domain easy-hard adaptation.Its first phase combines synthetic-to-real translation, enhancement, image-level adaptation, and feature-level adaptation; its second phase uses rank-based quality assessment and pseudo labels.

B. Domain Adaptation

Domain adaptation reduces distribution differences across domains and can operate at the image or feature level. TUDA applies this framework to underwater enhancement through dual alignment followed by quality-based easy-hard adaptation.

  • Domain adaptation: Domain adaptation aims to reduce distribution gaps between domains and may be performed at the image level or feature level.It has been applied in areas including image dehazing, semantic segmentation, and depth prediction.
  • Domain adaptation: Prior intra-domain adaptation work decomposed a target domain into subdomains using entropy statistics before aligning them.The cited semantic-segmentation approach used self-supervision and entropy-map alignment for its subdomains.
  • TUDA: TUDA uses a first-phase dual-alignment network with an image translation part and an image enhancement part for synthetic-to-real adaptation.The translation component learns a robust synthetic-to-real transformation, while the enhancement component uses translated and real images.
  • TUDA: In the second phase, rank-based quality assessment separates real images into easy and hard samples before easy/hard adaptation closes their distribution gap.The method uses a trustworthy easy set with generated pseudo labels and unpaired hard samples.

A. 1st phase: Inter-domain Adaptation

The inter-domain adaptation phase uses a dual-alignment network that translates synthetic images toward real-image realism before enhancement, while jointly aligning image- and feature-level representations.

  • Network design: The dual-alignment network contains an image translation module followed by an image enhancement module.Synthetic samples and corresponding labels are translated into realistic images, then paired translated data trains the enhancement network.
  • Image-level alignment: Image-level alignment uses WGAN-GP adversarial learning to make translated synthetic images resemble unpaired real images.The discriminator distinguishes translated images from real images, with λimg = 10 used as the penalty parameter.
  • Image-level alignment: Color-tone matching divides synthetic and real images into blue, green, and blue-green subsets to accelerate synthetic-to-real translation.The subsets are determined from the average blue-channel value in the CIELab color space.
  • Image-level alignment: A semantic content loss uses pretrained VGG-19 features to preserve content during synthetic-to-real translation.The loss aggregates features from conv1-1 through conv5-1 with experimentally specified layer weights.
  • Feature-level alignment: Feature-level adversarial alignment is added to the enhancement part, using shared encoder weights for real and translated input pipelines.The feature-level penalty parameter λfeat is set to 10 in the experiments.
  • Joint optimization: Image- and feature-level alignments are jointly optimized end-to-end through an overall inter-domain loss with four trade-off weights.The weights λ1, λ2, λ3, and λ4 are set to 1, 100, 10, and 0.0005, respectively.

B. 2nd phase: Intra-domain Adaptation

The intra-domain adaptation phase separates real underwater images into easy and hard samples according to the assessed quality of their enhanced results, then adapts between these groups.

  • Easy-hard separation: Real images are divided into easy and hard samples because varied degradation distributions create an intra-domain gap.Easy samples resemble the training distribution, whereas hard samples produce poorer enhancement results.
  • Quality-based separation: RUIQA is used to assess enhanced-image quality because existing underwater quality metrics may be inaccurate in some cases.The rank-based assessment method is intended to support more effective separation of real underwater images.

1) Rank-based Underwater Image Quality Assessment (RUIQA):

RUIQA learns perceptual quality from underwater image rankings and uses predicted quality scores to classify enhanced real images into easy and hard samples for adaptation.

  • Motivation: Existing deep IQA models perform unsatisfactorily on underwater images with diverse distortions because their pretraining captures ground-image information.This motivates learning ranking information from underwater images instead.
  • RUIQA pipeline: RUIQA trains a network on underwater ranking data, then fine-tunes it to predict enhanced-image perceptual quality.Its three stages are ranking-data generation, Siamese ResNet-50 training, and fine-tuning for quality prediction.
  • Ranking dataset: The ranking dataset is built from 800 selected underwater images enhanced by eight methods and scored on a continuous quality scale from 1 to 5.Pairwise ranking labels are derived from the resulting mean opinion scores.
  • Ranker training: A Siamese ResNet-50 learns ranking information using pairwise images and ranking-order labels, after which one branch initializes the quality-assessment backbone.The training uses a margin-ranking loss.
  • Quality prediction: RUIQA predicts quality from multi-scale features extracted from four backbone layers and mapped into perceptual quality vectors.The fine-tuning network replaces the final global-average-pooling and fully connected layers with four feature-processing blocks.
  • Easy-hard classification: Higher predicted quality identifies easy samples, while a selected easy-sample ratio determines the threshold and assigns remaining images to the hard group.Easy enhanced results are reported as having higher perceptual quality and being closer to human perception; easy outputs also provide pseudo labels for adaptation.

2) Easy/Hard Adaptation:

Easy/hard adaptation translates easy samples toward hard-image characteristics while preserving content, then enhances them and aligns easy and hard feature distributions.

  • 2) Easy/Hard Adaptation:: The intra-domain translator makes easy samples indistinguishable from hard images through adversarial training.The translated easy image is constrained to remain similar in content to the original easy image.
  • 2) Easy/Hard Adaptation:: Content preservation uses semantic content loss based on weighted VGG-19 feature differences.The loss uses layers conv1-1 through conv5-1 with weights 1/32, 1/16, 1/8, 1/4, and 1.0.
  • 2) Easy/Hard Adaptation:: The translated easy image is passed to the intra-domain enhancement network to obtain the enhanced output.
  • 2) Easy/Hard Adaptation:: The intra-domain enhancement network is trained with content and perceptual losses, whose trade-off weights are c=0.8 and d=0.2.
  • 2) Easy/Hard Adaptation:: Feature-level adaptation aligns feature-map distributions between translated easy samples and hard samples using an intra-domain discriminator.The discriminator compares the feature maps of xet and xh during end-to-end training.

C. Architecture Details

The architecture uses translator modules without down-sampling and enhancement modules built from dense blocks under a U-Net structure; testing retains only enhancement and quality-assessment components.

  • C. Architecture Details: The translator omits down-sampling to avoid losing valuable image information.
  • C. Architecture Details: PatchGAN discriminators perform local real-versus-fake discrimination for both image and feature representations.
  • C. Architecture Details: The enhancement module stacks dense blocks under a U-Net structure.
  • C. Architecture Details: The testing pipeline uses only the enhancement modules and the proposed rank-based image-quality assessment method.

IV. EXPERIMENTS

Experiments evaluate TUDA using synthetic and real underwater data, with visual comparisons, a two-stage testing pipeline, and specified optimization settings.

  • IV. EXPERIMENTS: The study compares TUDA with existing methods on four publicly available real underwater benchmarks and includes ablation and complexity analyses.
  • IV. EXPERIMENTS: The synthetic data comprise 9 water types, with 7200 training images and 1800 testing images.
  • IV. EXPERIMENTS: The real database contains 3900 images, divided into 2900 training images and 1000 testing images.
  • IV. EXPERIMENTS: Testing first applies inter-domain enhancement, then uses rank-based quality assessment to route low-scoring cases to intra-domain enhancement.
  • IV. EXPERIMENTS: Training uses Adam optimization with learning rates of 1 × 10^-4 and 2 × 10^-4, batch size 4, and 200 epochs followed by linear decay over 100 epochs.

B. Experiment Settings

Experiments assess TUDA on multiple real underwater benchmarks using no-reference metrics, user studies, color error, quantitative comparisons, and visual inspection. Results report strong perceptual and visual performance, while highlighting limitations of existing metrics and competing methods.

  • B. Experiment Settings: Real-image performance is measured with UCIQE, UIQM, and RUIQA, where higher scores indicate better human visual perception.
  • C. Comparisons with State-of-the-Art Methods: TUDA achieves the best performance in RUIQA and perceptual scores, while producing relatively low average color-restoration error.
  • C. Comparisons with State-of-the-Art Methods: Methods trained on real data or synthetic data remain limited when they ignore intra-domain or inter-domain gaps, respectively.
  • C. Comparisons with State-of-the-Art Methods: UCIQE can rank HE-Prior highly despite its lowest perceptual score, because the metric may emphasize local color features over whole-image quality.
  • C. Comparisons with State-of-the-Art Methods: Visual comparisons show TUDA reducing color distortion and blurriness while recovering details and producing visually pleasing results across challenging datasets.

D. Ablation Studies and Analysis

The experiments evaluate RUIQA, proposed components, λ, and computational cost through performance and ablation analyses.

  • The study evaluates RUIQA, component contributions, λ selection, and model efficiency through ablation and comparative experiments.

1) Effectiveness of the Proposed RUIQA method:

RUIQA is evaluated against established underwater quality metrics and through ablation studies, while TUDA’s inter-domain adaptation and efficiency are also examined.

  • 1) Effectiveness of the Proposed RUIQA method:: RUIQA achieves the best rank and linear correlations with MOS, reaching 0.900 and gaining 0.5 to 0.65 over UCIQE and UIQM.Larger values indicate better perceptual-quality correlation.
  • 1) Effectiveness of the Proposed RUIQA method:: RUIQA outperforms direct quality prediction and ImageNet-pretrained initialization in the reported ablation study.
  • 1) Effectiveness of the Proposed RUIQA method:: The visual comparisons assess UCIQE, UIQM, and RUIQA as measures of subjective quality.
  • 1) Effectiveness of the Proposed RUIQA method:: The visual comparisons include easy and hard samples, with the full model producing the most visually pleasing results particularly on hard samples.

3) Effectiveness of the intra-domain adaptation phase:

The intra-domain adaptation phase uses RUIQA-based quality assessment to divide real images into easy and hard samples and improve enhancement of difficult cases.

  • 3) Effectiveness of the intra-domain adaptation phase:: Adding intra-domain adaptation improves average RUIQA and perceptual scores by up to 2.186 and 0.205, respectively, over inter-domain adaptation alone.The gains indicate improved perceptual quality on the evaluated real-image set.
  • 3) Effectiveness of the intra-domain adaptation phase:: The intra-domain module addresses noises and over-enhancement artifacts remaining in some hard samples after inter-domain adaptation.The reported examples characterize the module as robust for extremely hard real underwater images.
  • 3) Effectiveness of the intra-domain adaptation phase:: RUIQA scores enhanced real images, whose ranked MOS values separate the data into easy and hard samples for intra-domain training and testing.The ratio λ controls this separation and is studied experimentally.
  • E. Model Complexity Analysis: The efficiency comparison measures parameters, FLOPs, and running time on UIEB’s 950 images of size 256x256x3.The comparison uses a specified CPU, GPU, and memory configuration.
  • E. Model Complexity Analysis: TUDA is reported to combine good performance with efficiency, while other methods exhibit different trade-offs among runtime, FLOPs, parameters, and generalization.
  • V. CONCLUSION: The conclusion reports that TUDA jointly addresses inter-domain and intra-domain gaps through dual alignment, RUIQA, and easy/hard adaptation.
Loading 2108.09650v1…