Source-linked AI summary
WaterGAN: Unsupervised Generative Network to Enable Real-time Color Correction of Monocular Underwater Images
Jie Li, Katherine A. Skinner, Ryan M. Eustice, Matthew Johnson-Roberson
TL;DR
Underwater restoration is difficult because wavelength-dependent water-column effects require parameters and paired training data that are hard to obtain. The paper proposes WaterGAN to generate aligned synthetic underwater imagery from in-air RGB-D data and uses it to train monocular color correction, which produces accurate and viewpoint-consistent results while implicitly estimating coarse depth.
Problem
Underwater image restoration must address wavelength-dependent attenuation and scattering, but large paired datasets with depth and true-color ground truth are difficult to collect.
Method
WaterGAN adversarially uses in-air RGB-D images and unlabeled site-specific underwater images to generate synthetic training pairs for a monocular color-correction network.
Results
The pipeline produced accurate, viewpoint-consistent color correction, inverted the generator model on synthetic data, and estimated normalized real-image depth with 0.11m RMSE against stereo reconstruction.
Takeaways & Limitations
The approach provides a practical route to training monocular underwater restoration without real underwater ground-truth color and depth pairs.
Takeaways & Limitations
The centered-vignetting assumption is invalid for the MHL dataset, leaving residual vignetting in restored images.
Abstract
from arXiv · showhide
This paper reports on WaterGAN, a generative adversarial network (GAN) for generating realistic underwater images from in-air image and depth pairings in an unsupervised pipeline used for color correction of monocular underwater images. Cameras onboard autonomous and remotely operated vehicles can capture high resolution images to map the seafloor, however, underwater image formation is subject to the complex process of light propagation through the water column. The raw images retrieved are characteristically different than images taken in air due to effects such as absorption and scattering, which cause attenuation of light at different rates for different wavelengths. While this physical process is well described theoretically, the model depends on many parameters intrinsic to the water column as well as the objects in the scene. These factors make recovery of these parameters difficult without simplifying assumptions or field calibration, hence, restoration of underwater images is a non-trivial problem. Deep learning has demonstrated great success in modeling complex nonlinear systems but requires a large amount of training data, which is difficult to compile in deep sea environments. Using WaterGAN, we generate a large training dataset of paired imagery, both raw underwater and true color in-air, as well as depth data. This data serves as input to a novel end-to-end network for color correction of monocular underwater images. Due to the depth-dependent water column effects inherent to underwater environments, we show that our end-to-end network implicitly learns a coarse depth estimate of the underwater scene from monocular underwater images. Our proposed pipeline is validated with testing on real data collected from both a pure water tank and from underwater surveys in field testing. Source code is made publicly available with sample datasets and pretrained models.
I. INTRODUCTION
Underwater imaging supports seafloor mapping but suffers wavelength-dependent attenuation and scattering, while paired underwater training data with depth and true-color ground truth is difficult to obtain. WaterGAN addresses this gap by generating aligned synthetic underwater imagery for training monocular color correction.
- Underwater attenuation and scattering shift scene colors toward blue or green and introduce haze that reduces effective resolution.
- Deep learning for underwater restoration is constrained by the difficulty of collecting large datasets with depth and true-color ground truth.
- WaterGAN uses real unlabeled underwater images to learn a realistic representation of water-column properties at a survey site.
- Given in-air images and depth maps, WaterGAN generates synthetic underwater images aligned with the original RGB-D data.
- The aligned dataset trains a color-correction network that maps raw monocular underwater images to restored images appearing as if captured in air.
II. BACKGROUND
Prior underwater restoration methods either explicitly estimate physical parameters or rely on image priors and preprocessing, but these approaches face site variation, dense-depth requirements, or limited modeling of range-dependent effects. Deep-learning approaches instead learn the complex image-formation process from simulated or environmental data.
- Physics-based restoration methods require prior experiments or detailed knowledge because attenuation parameters vary with survey-site water composition and quality.
- Gray-world and histogram-equalization methods can improve appearance but lack range-dependent information, causing viewpoint-dependent color changes.
- Some consistency-restoration methods enforce scene-wide color consistency but require dense depth maps.
- Deep-learning methods have been used to estimate parameters of the complex underwater physical model, including with simulated data for dehazing.
- WaterGAN incorporates real field data to learn a realistic representation of environmental conditions for raw underwater images from a specific survey site.
III. TECHNICAL APPROACH
The proposed pipeline combines WaterGAN with a color-correction network to restore monocular underwater images in real time. WaterGAN adversarially adapts in-air RGB-D images to the characteristics of unlabeled underwater imagery from a local survey area.
- WaterGAN is trained adversarially using in-air RGB-D images and unlabeled raw underwater images from a specific survey site.
- The generator renders underwater images from in-air RGB-D inputs while assuming water-column effects are mostly uniform within a local area.
- The generated images train a color-correction network that compensates for location-specific water-column effects in real time.
A. Generating Realistic Underwater Images
WaterGAN generates realistic underwater images by combining in-air RGB-D inputs with learned, physically motivated models of attenuation, scattering, and camera effects. Its adversarial training uses unlabeled underwater images to match field image-formation properties while preserving scene structure.
- WaterGAN trains a generator and discriminator adversarially to make synthetic underwater images resemble real unlabeled underwater images.The discriminator classifies real and synthetic samples, while the generator is optimized to produce images classified as real.
- The generator models underwater image formation through attenuation, backscattering, and a camera model.These stages are designed to alter relative color and intensity without changing the underlying RGB-D scene structure.
- G-I: Attenuation: Attenuation applies wavelength-dependent exponential decay based on in-air irradiance, camera-to-scene range, and an estimated positive coefficient.The coefficient varies with water composition and quality across survey sites.
- G-II: Scattering: Backscattering adds a depth-dependent, wavelength-sensitive mask to the attenuated image to represent underwater haze without distorting scene structure.The mask is generated from depth and noise through convolutional layers and added to the output of the attenuation stage.
- G-III: Camera Model: The camera model accounts for vignetting and applies a constrained linear sensor response before producing the final synthetic image.The vignetting mask is generated at the image dimensions, and full-resolution samples can be produced after upsampling relevant components.
B. Underwater Image Restoration Network
The restoration pipeline uses two fully convolutional stages trained on WaterGAN-rendered underwater images and their in-air RGB-D sources. It first estimates coarse relative depth from an underwater image, then uses that estimate with the image to restore color.
- The first network reconstructs a coarse relative depth map from a downsampled synthetic underwater image.Its input is 56 × 56 × 3 and its output is a 56×56×1 relative depth map, later upsampled for restoration.
- The second network restores color using the underwater image together with its estimated relative depth map.The color correction module processes RGB-D input and uses extra downsampling and upsampling stages to increase output resolution.
- Both modules are based on fully convolutional encoder-decoder networks modeled after SegNet.The restoration design adds skipping layers to preserve texture and high-frequency information useful for downstream applications.
IV. EXPERIMENTAL SETUP
The experiments combine indoor in-air RGB-D datasets with controlled tank and field underwater data. The restoration architecture is evaluated using synthetic-training and real-testing image pathways.
- The in-air training pool contains 15000 RGB-D images compiled from four indoor Kinect datasets.The datasets are B3DO, UW RGB-D Object, NYU Depth, and Microsoft 7-scenes.
- The network architecture trains on synthetic underwater images and uses real underwater images during testing to produce restored color outputs.The first stage estimates relative depth, and the second stage combines the underwater image and depth estimate for restoration.
- Controlled imaging tests use an artificial rock platform and a diving color board to provide reference ground truth in a pure-water tank.The submerged platform supports collection of the MHL dataset.
A. Artificial Testbed
The artificial testbed uses a submerged man-made rock platform with an attached color board in a pure-water tank to support controlled evaluation.
- The MHL survey submerged a 4 ft × 7 ft man-made rock platform in a pure-water test tank and attached a color board for reference.The survey compiled over 7000 underwater images.
B. Field Tests
The field tests include underwater imagery from Port Royal, Jamaica, and a coral reef system near Lizard Island, Australia, collected with a handheld diver rig.
- Field datasets: Port Royal data comprise 6,500 images from a single dive at a submerged city containing natural and man-made structures.The seafloor depth reached approximately 1.5 m.
- Field datasets: Lizard Island data come from a coral reef system and were collected using the same handheld diver rig.
C. Network Training
WaterGAN is trained separately for each survey site to generate synthetic underwater imagery, after which the color correction and depth networks are trained on generated paired data.
- WaterGAN training: WaterGAN models each survey site using real underwater samples for its discriminator and equal numbers of in-air RGB-D pairs for its generator.Training used a Titan X (Pascal), batch size 64, and learning rate 0.0002.
- Synthetic data generation: The experiments generated 15,000 rendered underwater images for each model, matching the size of the compiled RGB-D dataset.Once trained, the model can generate an arbitrary amount of synthetic data.
- Downstream network training: The color correction and depth networks were trained from scratch using 12,000 generated training images and 3,000 validation images.Both networks were trained on a Titan X (Pascal) GPU.
V. RESULTS AND DISCUSSION
WaterGAN produces more viewpoint-consistent color correction than the compared methods and supports learned color and relative-depth recovery, while its centered-vignetting assumption limits restoration on MHL data.
- Qualitative evaluation: The proposed method shows more consistent color across varying views, with reduced vignetting and attenuation compared with the other methods.Histogram equalization appears visually appealing but lacks knowledge of range-dependent effects.
- Color accuracy: The method has the lowest color-correction error for blue, red, and magenta, while histogram equalization performs best for cyan, yellow, and green.The proposed method still outperforms the remaining methods for cyan and yellow.
- Color consistency: The proposed method shows the lowest mean variance across each color channel for scene points viewed from multiple images.This variance measures color consistency across viewpoints.
- Synthetic validation: The trained network is able to invert the model encoded by the generator on synthetic validation data.Validation uses RMSE for both color and depth.
- Depth estimation: The depth network recovers accurate relative depth rather than necessarily absolute depth because monocular depth estimation has inherent scale ambiguity.Normalized estimated depth versus normalized stereo depth has RMSE 0.11m on MHL data.
- Architecture analysis: Skipping layers reduces stable-loss training from around 30 to 15 epochs for core training and from 10 to 5 epochs for full-model training.The comparison also indicates that skipping layers preserves high-frequency input information.
- Limitations: The centered-vignetting assumption is invalid for MHL, leaving residual vignetting in restored images despite partial correction.The authors suggest adding a parameter for the vignetting-pattern center and expanding the generator to cover more complex effects and conditions.
- Resources: Source code, sample datasets, and pretrained models are publicly available.
VI. CONCLUSIONS
WaterGAN models underwater images from in-air RGB-D data and supports monocular color correction evaluated on controlled and field data. The method produces accurate, viewpoint-consistent outputs, while its separately trained components remain a stated area for improvement.
- WaterGAN models underwater images from RGB-D in-air data using a generator structure that incorporates underwater image formation.
- The color correction network is trained on RGB-D pairs and corresponding generated images to restore monocular underwater images.
- The method was evaluated on controlled and field data, with qualitative and quantitative evidence of accuracy and consistency across viewpoints.
- Figure 5 compares raw images and corrected results from four baselines against the proposed method across MHL, Lizard Island, and Port Royal datasets.
- Figure 6 examines color-correction results with and without skipping layers through a zoomed-in image comparison.
- WaterGAN and the color correction network are trained separately, while joint training is identified as a future direction.