Source-linked AI summary
Simultaneous Enhancement and Super-Resolution of Underwater Imagery for Improved Visual Perception
Md Jahidul Islam, Peigen Luo, Junaed Sattar
TL;DR
Underwater distortions make conventional super-resolution outputs uninformative, motivating simultaneous enhancement and super-resolution for robust robot vision. Deep SESR jointly learns enhancement, saliency, and upscaling, achieving strong cross-dataset performance and fast single-board inference.
Problem
Underwater distortions degrade hue, sharpness, contrast, and color, while conventional super-resolution can amplify noise and optical distortions into uninformative regions of interest.
Method
Deep SESR is an end-to-end generative model that jointly learns saliency prediction and 2×–4× enhancement and super-resolution on shared hierarchical features.
Results
Deep SESR achieves state-of-the-art underwater enhancement and super-resolution, better natural-image generalization, competitive terrestrial results, and fast single-board inference.
Takeaways & Limitations
With 10 MB memory and 129 ms per frame on an Nvidia AGX Xavier, Deep SESR is suitable for near real-time visually guided underwater robots.
Takeaways & Limitations
Existing underwater SISR models do not account for underwater distortions and therefore rely on a secondary enhancement network, limiting end-to-end SESR applicability.
Abstract
from arXiv · showhide
In this paper, we introduce and tackle the simultaneous enhancement and super-resolution (SESR) problem for underwater robot vision and provide an efficient solution for near real-time applications. We present Deep SESR, a residual-in-residual network-based generative model that can learn to restore perceptual image qualities at 2x, 3x, or 4x higher spatial resolution. We supervise its training by formulating a multi-modal objective function that addresses the chrominance-specific underwater color degradation, lack of image sharpness, and loss in high-level feature representation. It is also supervised to learn salient foreground regions in the image, which in turn guides the network to learn global contrast enhancement. We design an end-to-end training pipeline to jointly learn the saliency prediction and SESR on a shared hierarchical feature space for fast inference. Moreover, we present UFO-120, the first dataset to facilitate large-scale SESR learning; it contains over 1500 training samples and a benchmark test set of 120 samples. By thorough experimental evaluation on the UFO-120 and other standard datasets, we demonstrate that Deep SESR outperforms the existing solutions for underwater image enhancement and super-resolution. We also validate its generalization performance on several test cases that include underwater images with diverse spectral and spatial degradation levels, and also terrestrial images with unseen natural objects. Lastly, we analyze its computational feasibility for single-board deployments and demonstrate its operational benefits for visually-guided underwater robots. The model and dataset information will be available at: https://github.com/xahidbuffon/Deep-SESR.
1. Introduction
The paper frames simultaneous enhancement and super-resolution as a unified need for robust, computationally constrained visual perception, especially underwater. Deep SESR addresses this need with an end-to-end model, a multi-modal objective, and the UFO-120 dataset, with reported gains across underwater and terrestrial imagery.
- Motivation: Single-image super-resolution can amplify noise and optical distortions, making perceptual restoration essential for robust visual perception in noisy environments.The motivation explicitly includes underwater imagery and visually guided robotic perception.
- Motivation: A unified enhancement and super-resolution approach had not yet been explored in depth for computationally constrained real-time applications.
- Motivation: Underwater attenuation, refraction, and backscatter produce range- and wavelength-dependent distortions, including hue distortion, blurriness, low contrast, and color degradation.
- Contributions: Deep SESR is an end-to-end unified model using dense residual-in-residual subnetworks and a multi-modal objective for perceptual restoration and saliency prediction.The objective addresses chrominance-specific color degradation, sharpness, contrast, and high-level feature representation.
- Contributions: The UFO-120 dataset provides over 1500 annotated training samples and an additional 120 test samples for large-scale SESR training.
- Results: Deep SESR reportedly outperforms state-of-the-art methods on underwater tasks, generalizes better to unseen natural images, and remains competitive on terrestrial datasets without additional training or tuning.The paper also analyzes computational aspects and robotic deployment benefits.
2. Background
Background work addresses underwater enhancement, super-resolution, and saliency prediction largely as separate or task-specific problems. The paper identifies limitations in generalization, end-to-end integration, and attention modeling for visually guided underwater robots.
- Underwater Enhancement: Physics-based underwater enhancement methods use filters, illumination assumptions, or revised formation models to correct optical distortions and recover pixel intensities.
- Underwater Enhancement: Learning-based enhancement models approximate mappings from distorted to enhanced images and achieve state-of-the-art image-to-image translation performance.
- Underwater Enhancement: GAN-based underwater enhancement models can use paired or unpaired training but are prone to training instability and require careful hyper-parameter and loss-function choices.
- Super-Resolution: Existing underwater SISR models do not account for underwater distortions and therefore rely on a secondary enhancement network.
- Saliency Prediction: Saliency prediction identifies interesting foreground regions using visual or task-driven cues, including luminance, color, texture, depth, motion, or query relevance.
- Saliency Prediction: Underwater saliency research mainly supports object detection, leaving general attention modeling for faster visual search and scene understanding insufficiently addressed.
3. Problem Formulation
The paper formulates SESR as a generative mapping from distorted low-resolution inputs to enhanced high-resolution images, with shared-feature saliency prediction. UFO-120 supplies paired, annotated data across multiple degradation orders and upscaling factors for supervised training.
- 3.1. Learning SESR: SESR generates perceptually enhanced high-resolution images from low-resolution and possibly distorted input measurements.
- 3.1. Learning SESR: Deep SESR jointly learns a mapping to saliency maps, enhanced input-resolution images, and final high-resolution outputs on a shared feature space.
- 3.1. Learning SESR: UFO-120 pairs 640×480 high-resolution ground truths with distorted inputs at 320×240, 214×160, and 160×120 resolutions.
- 3.2. Data Preparation: The UFO-120 Dataset: The dataset combines existing enhancement and super-resolution sources with human-labeled saliency maps to form paired training data.
- 3.2. Data Preparation: The UFO-120 Dataset: UFO-120 contributes over 1500 training samples and 120 test samples collected across multiple oceanic locations and water types.
- 3.2. Data Preparation: The UFO-120 Dataset: The dataset uses style transfer, Gaussian blurring, and bicubic down-sampling to generate distorted low-resolution samples.
- 3.2. Data Preparation: The UFO-120 Dataset: Set-U applies Gaussian blurring before bicubic down-sampling, while Set-O applies the reverse order.
- 3.2. Data Preparation: The UFO-120 Dataset: Set-F interchanges the blurring and down-sampling order with probability 0.5, contributing to nine combinations across 2×, 3×, and 4× scaling.
4. Deep SESR Model
Deep SESR combines residual dense feature extraction, saliency prediction, and separate enhancement and upsampling branches in an end-to-end architecture. Its seven-part objective targets saliency, contrast, color, content, and sharpness to restore underwater imagery.
- Network architecture: Deep SESR ties residual dense blocks, FENet, and AAN into one end-to-end architecture for combined SESR learning.FENet extracts hierarchical features while AAN predicts saliency in the shared feature space.
- Feature extraction: Parallel 3 × 3 and 5 × 5 branches, followed by a second residual stage, learn locally dense features while maintaining fast feature extraction.The first stage uses eight RDB blocks per branch; the later stage uses four 3×3 RDB blocks and generates 32 feature maps.
- Output generation: Separate output layers generate an enhanced LR image and a 2×, 3×, or 4× SESR image from progressively upsampled features.Enhancement is supervised at a shallow output, while deconvolution layers produce the final higher-resolution result.
- Loss formulation: The training objective combines seven loss components to supervise the mapping from distorted input to saliency, enhancement, and SESR outputs.The components address saliency prediction, contrast, color, high-level content, and sharpness, with scaling factors empirically tuned as hyperparameters.
- Perceptual supervision: Contrast loss uses saliency-derived foreground pixels and CMI to shift distorted images toward higher foreground-to-background intensity contrast.The authors report that this coupling lets AAN influence enhancement and improves training stability by avoiding premature convergence and over-fitting.
- Perceptual supervision: Content loss preserves high-level VGG-19 feature representation, while sharpness loss measures SESR blurriness recovery through local image gradients.The sharpness formulation is motivated by significant gradient-norm disparities between distorted images and high-resolution ground truth.
5. Experimental Results
Deep SESR qualitatively restores underwater color, sharpness, and contrast while jointly predicting salient foreground regions. Across enhancement and super-resolution evaluations, it achieves strong perceptual and quantitative performance against existing models.
- Enhancement: Deep SESR rectifies greenish hue, restores true pixel colors, and recovers global sharpness while focusing saliency predictions on foreground regions.The saliency output guides contrast improvement, and ablations assess contributions from the learning objective's loss terms.
- Enhancement: Removing LLR_Content impairs color rendition and finer texture detail, while removing LLR_Contrast produces considerably lower-contrast images.These ablations support distinct roles for the content and contrast-related losses.
- Enhancement: Deep SESR produces sharper enhanced images than the compared underwater enhancement models while achieving competitive or better PSNR and SSIM and generally better UIQM.The comparison includes physics-based and learning-based SOTA models.
- Enhancement: Ablation without LLR_P causes a 9.47% drop in UIQM values, supporting the contribution of the saliency-related objective to enhancement quality.The paper relates this objective to improving UICM.
- Super-Resolution: Deep SESR outperforms the compared super-resolution models by considerable margins on UIQM and generates sharper, higher-quality HR images from distorted and undistorted LR inputs.The evaluation covers 2×, 3×, and 4× scaling on UFO-120 and USR-248, with competitive PSNR and SSIM on USR-248.
- Super-Resolution: Deep SESR avoids noise from unnecessary over-correction and shows similar performance trends across Set-U, Set-F, and Set-O down-sampling conditions.Relative quantitative scores are reported in Table 3.
6. Generalization Performance
Deep SESR generalizes beyond its supervised underwater data to unseen underwater and terrestrial images. It retains coherent color, texture, sharpness, contrast, and saliency outputs while remaining substantially faster than a physics-based comparison.
- Unseen Underwater Images: On unseen underwater images, Deep SESR produces pixel intensities perceptually similar to a physics-based approximation while also generating HR images and saliency maps.The model is reported to offer more than 10 times faster run-time in this comparison.
- Terrestrial Images: On terrestrial images with unseen objects such as grass, faces, and clothing, Deep SESR produces perceptually coherent color and texture enhancement.Further tuning and training on terrestrial datasets could improve performance.
7. Operational Feasibility & Design Choices
Deep SESR is designed for near real-time robotic deployment through compact execution and configurable branches. Its reported single-board implementation supports enhancement, saliency prediction, and optional region-of-interest super-resolution.
- Operational Feasibility: On an Nvidia AGX Xavier, Deep SESR requires 10 MB of memory and runs at 129 ms per frame, or 7.75 FPS.These measurements characterize the default single-board deployment.
- Design Choices: Decoupling enhancement and saliency prediction from the frozen model reaches 10.02 FPS, a 22% speed increase when HR perception is unnecessary.An automatically selected RoI can receive SESR processing with an additional 25 ms.
- Design Choices: Replacing FENet-2d with single-branch FENet-1d provides a 12.5% speed-up with only 1.8%/1.5%/1.8% lower PSNR/SSIM/UIQM scores on UFO-120.The paper describes the resulting qualitative images as indistinguishable and the trade-off as admissible in practical applications.
- Operational Feasibility: Use-case-specific design choices preserve robust SESR performance while supporting near real-time robotic deployments.The paper points to supplementary material for further operational demonstration.
8. Conclusion
The paper introduces Deep SESR as an efficient learning-based solution for simultaneous enhancement, super-resolution, and saliency prediction in underwater imagery. It contributes the UFO-120 dataset and reports strong performance, generalization, and fast inference for visually guided underwater robots.
- Deep SESR performs 2×−4× simultaneous enhancement and super-resolution with saliency prediction on a shared feature space.The paper also presents its network architecture, loss functions, and end-to-end training pipeline.
- The UFO-120 dataset contributes over 1500 annotated samples for large-scale SESR training.
- Deep SESR provides state-of-the-art underwater enhancement and super-resolution, better generalization on natural images, and competitive terrestrial results.
- Deep SESR achieves fast inference on single-board platforms, supporting near real-time use by visually guided underwater robots.The paper identifies computational efficiency and application-specific design choices as factors supporting this use.
Appendix I: Dataset Information
The appendix lists datasets used for underwater enhancement, underwater super-resolution, and terrestrial super-resolution experiments.
- The UFO-120 dataset is listed as a resource for underwater imagery.
- The USR-248 dataset is listed among the underwater datasets.
- The EUVP and UImNet datasets are listed as additional underwater datasets.
- Set5, Set14, Sun80, and other terrestrial datasets are listed for terrestrial super-resolution.
Appendix II: Credits for Media Resources
The appendix credits the image and video resources used for the paper’s media materials, including underwater scenes, marine life, wrecks, and underwater robots.
- The credits include underwater photographs and marine-life imagery from Flickr, SeaPics, Atlas of Living Australia, PondExperts, and WallpaperUP.
- The credits include underwater videos depicting coral reefs, fish, sharks, ocean scenes, and other aquatic environments.
- The credits include videos of shipwrecks, wreck diving, caves, and submerged structures.
- The credits include underwater robot and remotely operated vehicle footage, including mapping and navigation examples.
- Additional credited resources cover coral reefs, shark footage, underwater wrecks, and underwater robotic platforms.
- The remaining credits document further wreck-diving scenes and underwater robots used in aquatic environments.
- The appendix also credits additional underwater wreck and robot videos, including BlueROV2 and other underwater platforms.
- Further credited material features underwater robots and sea-trial or diving footage from several named creators and organizations.