PaperlayerView exact revision on arXiv

Source-linked AI summary

Semantic Segmentation of Underwater Imagery: Dataset and Benchmark

Md Jahidul Islam, Chelsey Edge, Yuyang Xiao, Peigen Luo, Muntaqim Mehtaz, Christopher Morse, Sadman Sakib Enan, Junaed Sattar

arXiv:2004.01241v3cs.CVeess.IV

TL;DR

Underwater semantic segmentation lacks broadly applicable datasets and mature methods because underwater imagery differs from terrestrial imagery. The paper introduces SUIM, benchmarks segmentation models, and proposes SUIM-Net, which combines competitive performance with fast, memory-efficient inference for underwater robotic vision.

  • Problem

    Underwater semantic segmentation lacks general-purpose large-scale annotated datasets, while terrestrial models are not directly applicable to underwater imagery.

  • Method

    The paper constructs the SUIM dataset, evaluates state-of-the-art segmentation models, and develops fully convolutional encoder-decoder variants of SUIM-Net.

  • Results

    SUIM-Net achieves F and mIOU scores within 5% of the respective top scores, while SUIM-NetRSB runs at 28.65 FPS and uses over 10 times less memory than several state-of-the-art models.

  • Takeaways & Limitations

    The dataset and model support semantic segmentation, object localization, saliency prediction, visual servoing, and detailed underwater scene understanding.

Abstract

from arXiv · show

In this paper, we present the first large-scale dataset for semantic Segmentation of Underwater IMagery (SUIM). It contains over 1500 images with pixel annotations for eight object categories: fish (vertebrates), reefs (invertebrates), aquatic plants, wrecks/ruins, human divers, robots, and sea-floor. The images have been rigorously collected during oceanic explorations and human-robot collaborative experiments, and annotated by human participants. We also present a benchmark evaluation of state-of-the-art semantic segmentation approaches based on standard performance metrics. In addition, we present SUIM-Net, a fully-convolutional encoder-decoder model that balances the trade-off between performance and computational efficiency. It offers competitive performance while ensuring fast end-to-end inference, which is essential for its use in the autonomy pipeline of visually-guided underwater robots. In particular, we demonstrate its usability benefits for visual servoing, saliency prediction, and detailed scene understanding. With a variety of use cases, the proposed model and benchmark dataset open up promising opportunities for future research in underwater robot vision.

I. INTRODUCTION

Underwater semantic segmentation remains less developed than terrestrial segmentation because underwater imagery differs substantially and lacks general-purpose annotated datasets. The paper addresses this gap with the SUIM dataset, benchmark evaluations, and the efficient SUIM-Net model.

  • Underwater imagery differs from terrestrial data in object categories, background patterns, and optical distortion artifacts, limiting direct transfer of terrestrial models.
  • General-purpose underwater semantic segmentation lacks large-scale annotated datasets for training and benchmark evaluation.
  • The SUIM dataset provides 1525 natural underwater images with ground-truth semantic labels and a 110-image test set.
  • The dataset covers fish, reefs, aquatic plants, wrecks or ruins, human divers, robots or instruments, and sea-floor or rocks.
  • SUIM-Net is an encoder-decoder model designed to achieve competitive segmentation performance with much faster runtime than state-of-the-art models.

II. THE SUIM DATASET

SUIM is a multi-category underwater dataset with pixel-level annotations, varied image sources and resolutions, and benchmark test data. Its labels encode eight semantic classes using compact RGB color codes.

  • SUIM labels eight categories: waterbody background, human divers, aquatic plants, wrecks or ruins, robots, reefs, fish, and sea-floor or rocks.
  • The dataset uses 3-bit binary RGB colors to represent its eight object categories in image space.
  • The dataset contains 1525 RGB images for training and validation plus 110 test images for benchmark evaluation.
  • Samples were collected during oceanic explorations and human-robot cooperative experiments across locations and water types.
  • Seven human participants produced pixel annotations using guidelines for potentially confusing object classes.

III. USE CASES AND RELATED WORK

Underwater segmentation research builds on fully convolutional deep models but remains limited by domain differences, scarce datasets, and narrow prior category coverage. Existing class-agnostic methods do not generalize to multi-object semantic segmentation.

  • Fully convolutional networks use CNN backbones to extract hierarchical features for semantic segmentation.
  • Terrestrial segmentation solutions are not directly applicable underwater because object categories and image statistics differ, with underwater distortion artifacts also present.
  • Class-agnostic foreground methods group regions using local salient features but cannot generalize to multi-object semantic segmentation.
  • Prior underwater CNN segmentation studies reported results for only sea-grass, sand, and rock, while many state-of-the-art models remained unevaluated.

B. Visual Attention Modeling and Servoing

Underwater robots need pixel-level attention and localization for exploration, tracking, and visual servoing. Semantic segmentation adds object identity and tighter boundaries beyond class-agnostic saliency maps and bounding-box detectors.

  • B. Visual Attention Modeling and Servoing: Semantic segmentation provides tighter object boundaries and pixel-level detection accuracy than bounding boxes, supporting more robust tracking.
  • B. Visual Attention Modeling and Servoing: Semantic attention can support tracking known objects, discovering objects of interest, and maintaining human or robot companions within an AUV’s field of view.
  • B. Visual Attention Modeling and Servoing: The figure compares segmentation-derived maps with DDD object detection and Deep SESR and ASNet class-agnostic saliency prediction.
  • B. Visual Attention Modeling and Servoing: Semantic saliency maps encode the spatial distribution and interactions of objects, unlike class-agnostic maps that identify only foreground regions.

IV. THE SUIM-NET MODEL

SUIM-Net is a fully convolutional encoder-decoder model that combines skip connections and residual learning to balance real-time inference with segmentation performance. Its variants use residual skip blocks or a deeper pre-trained VGG-16 encoder.

  • SUIM-Net uses a fully convolutional encoder-decoder architecture with skip connections between mirrored composite layers.The architecture includes a base model with an optional residual skip block (RSB).
  • Each residual skip block contains three convolutional layers, each followed by batch normalization and ReLU.
  • The SUIM-NetRSB variant combines skip connections with residual learning to target real-time inference and reasonable segmentation performance.
  • The SUIM-NetVGG variant prioritizes improved performance using 12 encoding layers of a pre-trained VGG-16 network.Its detailed architecture is referenced in Figure 5c.
  • An RSB can connect either an intermediate convolutional layer or the block input for local residual learning.The two options are selected with skip=0 and skip=1, respectively.

B. Training Pipeline and Implementation Details

The training pipeline uses supervised cross-entropy optimization for mapping underwater images to semantic labels, with quantitative results reported as mean ± variance. Comparisons identify the best and next three scores for each metric.

  • Training learns a mapping from natural underwater images X to semantic labels Y using supervised standard cross-entropy loss.The implementation uses TensorFlow, a Linux host with an Nvidia GTX 1080, and the Adam optimizer.
  • Table II reports quantitative performance scores as mean ± variance.
  • The best score and the next three scores for each comparison are colored red and blue, respectively.

V. BENCHMARK EVALUATION

The benchmark compares multiple state-of-the-art segmentation models under semantic-segmentation and saliency-prediction configurations. The setups differ in output structure and target-mask construction.

  • The benchmark evaluates FCN8, SegNet, UNet, PSPNetMobileNet, and DeepLabV3 with multiple backbone or input variants.Variants include CNN, VGG-16, ResNet-50, grayscale, RGB, and MobileNet configurations.
  • The evaluation compares state-of-the-art models under two training configurations.These configurations are semantic segmentation and single-channel saliency prediction.
  • Semantic segmentation: Semantic segmentation predicts five output channels for HD, WR, RO, RI, and FV, while the remaining categories are treated as background.Separate predicted masks are combined into RGB masks for visualization.
  • Saliency prediction: Saliency prediction uses a single output channel in which HD, RO, FV, and WR pixels are assigned intensity 1.0 and all others 0.0.The output is thresholded and visualized as binary images.

A. Evaluation Criteria

The evaluation uses region similarity and contour accuracy metrics to assess predicted pixel labels and object-boundary localization. These are quantified with the Dice/F score and mean IoU.

  • Region similarity measures the correctness of predicted pixel labels relative to ground truth using the Dice coefficient, also called the F score.
  • The F score is calculated from precision P and recall R as F = 2×P×R/(P+R).
  • Contour accuracy measures object-boundary localization using mean IoU, where IoU is the area of overlap divided by the area of union.

B. Quantitative and Qualitative Analysis

The benchmark compares segmentation and saliency models across standard metrics, while SUIM-Net variants balance competitive accuracy with substantially faster and more efficient inference. Qualitative results further examine fine-grained localization and saliency prediction for underwater objects.

  • Quantitative comparison: UNetRGB, FCN8V GG, and DeepLabV 3 generally achieve the highest F and mIOU scores across semantic segmentation and saliency prediction.SegNetResNet and PSPNetMobileNet remain competitive but show less consistent performance across object categories.
  • Quantitative and qualitative comparison: SUIM-Net variants provide consistent, competitive region similarity and object localization, with F and mIOU scores within 5% of the respective top scores.Their generated masks are qualitatively comparable to ground-truth labels, although leading models produce finer object contours.
  • Computational efficiency: 22.46 FPS allows SUIM-NetV GG to retain faster inference than FCN8V GG while providing better object localization performance in general.The variant adds learning capacity relative to SUIM-NetRSB, trading some efficiency for improved localization.
  • Use-case evaluation: SUIM-NetV GG masks provide more precise pixel-level localization of divers and robots and fewer missed detections, especially in occluded or low-contrast regions.The additional semantic information also supports improved saliency prediction compared with class-agnostic models that focus on high-contrast foreground regions.

VI. CONCLUSION

The paper addresses limitations in underwater robot vision by introducing a large annotated dataset, benchmarking existing methods, and proposing an efficient segmentation model. SUIM-Net combines competitive segmentation with faster runtime for near-real-time underwater robotic tasks.

  • Dataset and benchmark: SUIM provides 1525 annotated images and benchmark test data with pixel annotations for eight underwater object categories.The categories include fish, reefs, plants, wrecks/ruins, humans, robots, sea-floor/sand, and waterbody background.
  • Model and evaluation: The paper benchmarks state-of-the-art semantic segmentation approaches and presents SUIM-Net as a fully convolutional encoder-decoder model.The model is designed to balance segmentation performance and computational efficiency.
  • Conclusion: SUIM-Net offers considerably faster runtime than state-of-the-art approaches while maintaining competitive semantic segmentation performance.The stated balance supports near-real-time use by visually guided underwater robots in attention modeling and servoing tasks.

APPENDIX I: DATA AUGMENTATION PARAMETERS

The appendix specifies the data-augmentation configuration and lists implementation repositories used for the evaluated models.

  • Data augmentation: Data augmentation uses a 0.2 rotation range, 0.05 width and height shifts, 0.05 shear and zoom ranges, and enabled horizontal flipping.Other augmentation parameters remain at their library defaults.
  • Implementation resources: The implementation references repositories for FCN, bilinear upsampling, SegNet, PSPNet, DeepLabv3, and Deep SESR variants.The listed repositories provide the model implementations used in the study.
Loading 2004.01241v3…