PaperlayerView exact revision on arXiv

Source-linked AI summary

MODS -- A USV-oriented object detection and obstacle segmentation benchmark

Borja Bovcon, Jon Muhovič, Duško Vranac, Dean Mozetič, Janez Perš, Matej Kristan

arXiv:2105.02359v2cs.CVcs.LG

TL;DR

Robust obstacle detection is needed for safe USV navigation, but conventional segmentation metrics do not capture navigation-relevant errors. MODS benchmarks object detection and obstacle segmentation; MaskRCNN leads overall detection with recognition, FCOS leads detection-only in the danger zone, and WaSR achieves the best dynamic-obstacle F1.

  • Problem

    Conventional per-pixel segmentation scores do not capture navigation-relevant obstacle errors, although robust obstacle detection is needed for safe USV navigation.

    Source: §I · p. 1, B. Obstacle detection by segmentation · p. 6

  • Method

    MODS combines an annotated IMU-synchronized stereo dataset with object-detection and obstacle-segmentation protocols, evaluating static obstacle-water edges separately from dynamic-obstacle true and false positives.

    Source: §I · p. 2, B. Obstacle detection by segmentation · p. 6

  • Results

    MaskRCNN yields the best overall detection with recognition, FCOS leads detection-only in the danger zone, and maritime-specific WaSR achieves the best dynamic-obstacle F1 score.

    Source: §VII · p. 13

  • Takeaways & Limitations

    Small objects are typically missed by evaluated object detectors, resulting in low recall on the maritime dataset.

    Source: §VII · p. 13

Abstract

from arXiv · show

Small-sized unmanned surface vehicles (USV) are coastal water devices with a broad range of applications such as environmental control and surveillance. A crucial capability for autonomous operation is obstacle detection for timely reaction and collision avoidance, which has been recently explored in the context of camera-based visual scene interpretation. Owing to curated datasets, substantial advances in scene interpretation have been made in a related field of unmanned ground vehicles. However, the current maritime datasets do not adequately capture the complexity of real-world USV scenes and the evaluation protocols are not standardised, which makes cross-paper comparison of different methods difficult and hinders the progress. To address these issues, we introduce a new obstacle detection benchmark MODS, which considers two major perception tasks: maritime object detection and the more general maritime obstacle segmentation. We present a new diverse maritime evaluation dataset containing approximately 81k stereo images synchronized with an on-board IMU, with over 60k objects annotated. We propose a new obstacle segmentation performance evaluation protocol that reflects the detection accuracy in a way meaningful for practical USV navigation. Nineteen recent state-of-the-art object detection and obstacle segmentation methods are evaluated using the proposed protocol, creating a benchmark to facilitate development of the field. The proposed dataset, as well as evaluation routines, are made publicly available at vicos.si/resources.

I. INTRODUCTION

The introduction motivates camera-based USV obstacle perception, identifies the lack of a comprehensive benchmark for comparing methods, and presents MODS as a unified dataset and evaluation resource.

I. INTRODUCTION

  • USV perception addresses maritime object detection of dynamic obstacles when shoreline maps are available, and obstacle segmentation of navigable versus non-navigable pixels when maps are absent or unreliable.The segmentation labels distinguish obstacle, water, and sky; object detection assumes static obstacles such as piers and shore can be avoided using GPS and maps.Source: §I · pp. 1–2, Figure 1 · p. 1
  • Different private datasets and non-standardized evaluation measures make objective comparisons difficult, while a comprehensive USV-specific benchmark with a large, diverse evaluation dataset and protocols is missing.Source: §I · p. 2
  • MODS contributes an IMU-synchronized stereo RGB dataset with over 80k frames and over 60k annotated dynamic obstacles, plus curated training sets and an obstacle-segmentation protocol.The benchmark evaluates popular state-of-the-art maritime object-detection and obstacle-segmentation approaches, and makes the datasets and evaluation routines publicly available.Source: §I · p. 2

II. RELATED WORK · A. Maritime detection and segmentation · B. Maritime visual perception datasets

Prior maritime detection and segmentation methods face limitations from visual ambiguity, moving-camera conditions, and small or static obstacles. Existing datasets vary in scene diversity, vantage point, annotation, and task coverage; this work introduces a much larger, diverse USV benchmark and standardized evaluation resources.

A. Maritime detection and segmentation

  • Saliency-based methods,,,, fail when objects resemble water or conditions include fog and glitter, while classical background subtraction, produces false positives as USVs rock.Deep-learning foreground extraction addresses the static-camera limitation but still struggles with visual ambiguity.Source: A. Maritime detection and segmentation · p. 2
  • Stereo plane-fitting detects close obstacles but misses distant or minimally protruding ones and fails in calm water; adding an IMU addresses the calm-water limitation.General-purpose detection CNNs,,,, and an extended Faster R-CNN, classify ships but miss static obstacles and dynamic objects absent from training.Source: A. Maritime detection and segmentation · p. 2
  • Hand-crafted segmentation,, uses sensor fusion to improve water-edge estimation and reduce glitter false positives, but simple visual features fail in diverse water conditions and mirroring.CNN segmentation, outperforms hand-crafted methods but remains weak on mirroring and small obstacles;,, and explore network changes and visual-inertial fusion, as deep learning becomes dominant.Source: A. Maritime detection and segmentation · pp. 2–3

B. Maritime visual perception datasets

  • Kristan et al. introduced a maritime obstacle-detection dataset with 12 diverse USV-captured sequences, later extended by Bovcon et al. to 28 stereo-camera sequences synchronized with an IMU.Both datasets were recorded in the same port and annotate horizon, water-edge, and large and small dynamic obstacles; recordings span different weather conditions.Source: B. Maritime visual perception datasets · p. 3
  • The surveillance dataset has 51 RGB and 30 NIR sequences, but mostly fixed on-shore or high-vantage views and static scenes; later added coarse instance labels.Vessel-classification patches lack locations for detector evaluation, while the ∼2k-image dataset omits static and dynamic obstacles.Source: B. Maritime visual perception datasets · p. 3
  • The 191-image segmentation dataset has limited diversity, weak train-test separation, and only water/non-water labels; the ∼1300-image dataset labels water, sky, and obstacles across varied conditions.Most datasets target detector evaluation, with only a few designed to train segmentation methods.Source: B. Maritime visual perception datasets · p. 3
  • The benchmark contains more than 80k stereo images and over 60k annotated objects, versus 31k images in the second-largest publicly available dataset, and provides evaluation protocols, training datasets, and a public toolkit.The authors describe it as an order of magnitude larger and the most challenging benchmark because recent methods perform significantly worse on it than on related benchmarks.Source: B. Maritime visual perception datasets · p. 3

III. DATASET ACQUISITION HARDWARE

The benchmark dataset was captured aboard a real small-sized USV equipped with navigation sensors and a forward-facing stereo camera system. The section details the stereo hardware, its mounting and capture constraints, and its calibration with the IMU.

III. DATASET ACQUISITION HARDWARE

  • The real small-sized USV uses a steerable thrust propeller, reaches 2.5m/s, and carries LIDAR, compass, GPS, an IMU, two side cameras, and a main stereo camera system.Figure 2 shows the USV on a mission, its acquisition hardware, and the dataset acquisition areas along the Slovenian coast.Source: §III · p. 3, Figure 2 · p. 4
  • The main stereo system uses two Vrmagic VRmS-14/C-COB CCD sensors with a 0.3m baseline, 3.5mm focal-length lenses, f/1.4 maximum aperture, and 132.1◦FOV.The cameras are part of a Vrmagic VRmMFC system.Source: §III · p. 4
  • Mounted approximately 0.7m above the water and aligned with the IMU, the stereo system captures 1,278 × 958-pixel images at 10 frames per second over USB-2.0.Automatic aperture adjustment responds to lighting conditions; the IMU is calibrated with the stereo system using the IMU-camera calibration toolbox.Source: §III · p. 4

IV. DATASET CONSTRUCTION · A. Evaluation dataset acquisition · B. Evaluation dataset annotation protocol

The authors prioritized a carefully acquired and annotated evaluation dataset, while assembling training data from existing maritime datasets to reduce evaluation bias. The evaluation set was designed to represent varied coastal conditions and realistic obstacle encounters, with annotations focused on navigation-relevant obstacle boundaries.

IV. DATASET CONSTRUCTION

  • Most resources went to acquiring and annotating the evaluation set, while existing maritime datasets were consolidated for training to reduce CNN overfitting to evaluation-set characteristics.This division was intended to reduce evaluation bias while supporting methods that may need maritime fine-tuning.Source: §IV · p. 4

A. Evaluation dataset acquisition

  • Eight voyages over seven months in 2018–2019 covered two geographically disjoint Slovenian coastal areas, with varied times, weather, and realistic close-range obstacle encounters.An expert piloted the USV through scenarios involving approaching or passing obstacles.Source: A. Evaluation dataset acquisition · p. 4
  • Examples in Figure 3 show MODS covering varied obstacle appearances and types, including boats, buoys, swimmers, and swans.Source: Figure 3 · p. 4
  • About 48 hours of synchronized sensor footage yielded 94 selected sequences containing 80,828 images, including diverse obstacle interactions and visually challenging conditions.Captured sensors included stereo cameras, an IMU, compass, and GPS; challenges included sun-glitter, sea-foam, shellfish farms, and floating debris.Source: A. Evaluation dataset acquisition · p. 4

B. Evaluation dataset annotation protocol

  • Every tenth frame was annotated; dynamic obstacles received tight boxes and vessel, person, or other labels, while static obstacles were marked along the water-obstacle edge.The edge was selected because errors there can cause collisions, unlike inaccuracies in an obstacle's upper region; the static annotation followed the MODD protocol,.Source: B. Evaluation dataset annotation… · p. 4
  • Annotation proceeded in two stages: crowdsourcing first, followed by expert adjustments to ensure high-quality results.Source: B. Evaluation dataset annotation… · p. 4

1) Stage 1: Crowd-sourced annotations.:

Crowd workers annotated dynamic-obstacle bounding boxes and obstacle–water edge polygons through a web-based workflow. The authors used instructions, timing constraints, validation, and screening to support annotation quality and consistency.

1) Stage 1: Crowd-sourced annotations.:

  • Annotators labeled dynamic obstacles with bounding boxes and obstacle–water edges with polygons using a modified VIA web interface.Clickworker recruited and organized annotators, and the annotation server ran on an Amazon AWS instance.Source: 1) Stage 1: Crowd-sourced annotations · pp. 4–5
  • Each 10-image task followed illustrated instructions and quizzes, while a 60-second minimum per image and a 3-day batch deadline were used to support quality and focused completion.Annotators received a fixed password after passing instructions and could skip the quiz on later recruitment for the same task.Source: 1) Stage 1: Crowd-sourced annotations · p. 5
  • The team validated tools and instructions on 100 batches assigned to 100 recruits, then adjusted batch fees and added clarification instructions based on quality and timing.Source: 1) Stage 1: Crowd-sourced annotations · p. 5
  • After screening 600 prospective annotators, the team selected 101 sufficiently skilled workers for final annotation across 16 jobs, reassigning timed-out batches until completion.Each of the three screening jobs contained 200 batches assigned to 200 annotators; final jobs assigned 50 batches to the skilled annotators.Source: 1) Stage 1: Crowd-sourced annotations · p. 5

2) Stage 2: Annotation refinement.: … A. Object detection by bounding boxes

The section refines maritime annotations, assembles training data for detection and segmentation, and specifies an object-detection evaluation protocol adapted to difficult waterborne scenes.

2) Stage 2: Annotation refinement.:

  • Expert review, correction, and consistency screening produced 63,579 dynamic-object annotations and 10,706 obstacle–water edge annotations, present in 99.3% of frames.Researchers added vessel, person, and other labels to dynamic obstacles and propagated labels across frames.Source: 2) Stage 2: Annotation refinement · p. 5
  • For difficult or incomplete annotations, images receive a global exhaustiveness label, and ignore regions exclude poorly boxed objects from evaluation, following and extending LVIS.The global label supports a federated evaluation protocol, while ignore regions address cases such as diagonally placed mooring ropes.Source: 2) Stage 2: Annotation refinement · p. 5

C. Training datasets

  • 24,090 images and 145,334 annotated objects form the detection training set, merged from MODD1, MODD2, and SMD and re-annotated.Images without obstacles were excluded unless they contained glitter; ambiguous images were removed, and remaining boxes were corrected and missing annotations added.Source: C. Training datasets · p. 6, Figure 8 · p. 6
  • Deep segmentation training uses MaSTr1325, with 1,325 images labeled per pixel as water, obstacle, or sky, and similar augmentation routines.Augmentation combines geometric transformations with color transfer from seven target-domain images to the reference training images.Source: C. Training datasets · p. 6

A. Object detection by bounding boxes

  • Object detection follows the COCO/LVIS protocol,, counting a detection as a true positive when its bounding-box IoU exceeds 0.3 and reporting F1 as the primary measure.Precision and recall are calculated across all dataset images.Source: A. Object detection by bounding boxes · p. 6
  • Detections above the annotated water edge are excluded unless overlapping ground truth, and false positives are disregarded in images marked as not exhaustively annotated.This focuses evaluation on dynamic obstacles and prevents missing detections above the water edge from counting as false negatives.Source: A. Object detection by bounding boxes · p. 6

B. Obstacle detection by segmentation

The proposed protocol targets navigation-relevant errors that ordinary per-pixel scores can miss, evaluating static-obstacle water edges separately from dynamic-obstacle detections. It accounts for object shape, fused components, and tolerable over-segmentation when scoring dynamic obstacles.

B. Obstacle detection by segmentation

  • Static obstacles are evaluated by water-edge localization accuracy µA and detection robustness µR, with edge pixels correct when within Θw = 20px of ground truth.This targets boundary errors relevant to navigation rather than relying on mean per-pixel Jaccard scores, which can understate the impact of water misclassified in the USV path.Source: B. Obstacle detection by segmentation · p. 6
  • For dynamic obstacles, a binary obstacle mask yields a true positive when a ground-truth box contains enough obstacle pixels, accommodating fused components and moderate over-segmentation.For objects larger than 250 pixels, the object-specific threshold uses DEXTR-estimated object pixels within the box; objects of 250 pixels or fewer use a fixed threshold of Θ = 0.5.Source: B. Obstacle detection by segmentation · pp. 6–7
  • False positives are assessed after adjusting water-edge regions to ignore slight over-segmentation and recomputing connected components, with unassigned components counted as false positives.For remaining components, effective false positives are estimated from component width beyond assigned true-positive widths, scaled by αsc = 1.1; false positives are counted only on fully annotated images.Source: B. Obstacle detection by segmentation · p. 7
  • The protocol reports precision, recall, and F1, plus average true-positive and false-positive detections per 100 images.Source: B. Obstacle detection by segmentation · p. 7

C. Danger zone · VI. THE BENCHMARK · A. Maritime object detection evaluation

The section evaluates maritime perception with a proximity-aware danger zone and describes the MODS benchmark’s object-detection evaluation setup.

C. Danger zone

  • The danger zone is a radial area around the USV whose farthest point is reachable within ten seconds at an average speed of 1.5m/s.Source: C. Danger zone · p. 7
  • Camera-IMU geometry estimates the danger zone per image, and detection performance is reported for both the whole image and that zone.Source: C. Danger zone · p. 7

VI. THE BENCHMARK

  • MODS evaluations cover state-of-the-art object-detection and obstacle-segmentation methods, using an Intel Core i7-7700 CPU and Nvidia RTX2080 Ti GPU with 11GB GRAM.Source: §VI · p. 7

A. Maritime object detection evaluation

1) Results:

MODS object detector results show low scores in standard and class-agnostic evaluation, while detection of large obstacles in the USV danger zone is reliable. Detecting and classifying small, distant objects remains difficult.

Results

  • Setup1 uses standard detection evaluation in the obstacle-appearing image region; Setup2 ignores class labels, and Setup3 further restricts evaluation to the USV danger zone.The latter setups assess dynamic obstacle detection for path planning and collision avoidance, with Setup3 focusing on immediate collision threats.Source: 1) Results · p. 8
  • Mean F1 scores were 0.08 in Setup1, 0.12 in Setup2, and around 0.95 in Setup3, versus approximately 0.65 on COCO for state-of-the-art detectors.MODS scenes challenge standard detectors because small, distant objects are difficult to distinguish from waves, foam, and reflections and to classify correctly.Source: Table II · p. 8, 1) Results · p. 8
  • MaskRCNN usually leads in Setup1 and Setup2, especially for small objects at about twice the score of FCOS, while FCOS leads at all scales in Setup3.YOLOv4 outperforms MaskRCNN for large objects in Setup1; detector scores vary more in Setup1 and Setup2, and SSD is mostly worst but competitive in Setup3.Source: 1) Results · p. 8
  • All evaluated detectors reliably detect large obstacles within the USV danger zone, but detecting and classifying small objects at large distances remains an open problem.The paper calls for further research to improve detection methods and adapt their architectures to the maritime domain.Source: 1) Results · p. 8

B. Obstacle segmentation evaluation

The section evaluates 16 obstacle-segmentation methods under a shared three-class training setup, with F1 scores broken down by obstacle size and methods characterized by parameter count and segmentation time.

B. Obstacle segmentation evaluation

  • Sixteen methods were evaluated: two recent handcrafted maritime approaches and fourteen deep-learning methods spanning scene segmentation, panoptic segmentation, multimodal fusion, lightweight models, and maritime-specific networks.The maritime-specific networks were IntCatchAI and WaSR; the handcrafted methods were DeFP and ISSM.Source: B. Obstacle segmentation evaluation · pp. 8–9
  • Figure 12 reports obstacle F1 scores by COCO size category and distinguishes results across Setup1, Setup2, and Setup3.The caption defines small, medium, and large areas using thresholds of < 322, 322–962, and > 962, respectively.Source: Figure 12 · p. 9
  • All networks output sky, water, and obstacle labels per pixel; deep networks were trained for 10 epochs with early stopping using RMSProp and a 10^-5 initial learning rate.Backbones used published pretrained values; modified output layers were Xavier-initialized, and the learning rate used polynomial decay of 0.9 with momentum 0.9.Source: B. Obstacle segmentation evaluation · p. 9
  • Table III reports each tested method’s trainable parameter count and average milliseconds needed to segment one image.The supplied passage gives the table’s scope but not its cell values.Source: Table III · p. 9

1) Results:

The results identify WaSR as the strongest overall segmentation method and the safest option in the danger zone, while showing trade-offs across water-edge localization, obstacle types and sizes, augmentation, and processing speed.

1) Results:

  • AGC helps shallow networks, but deeper networks generally perform better with AG or AGC2; AGC2 gives DeepLabpan and RefineNet the best overall F1, while AG is best for most methods in the danger zone.Because the danger zone is crucial for safe navigation, the subsequent experiments use AG augmentation.Source: 1) Results · pp. 9–10, Table IV · p. 10
  • BiSeNet and RefineNet have the best water-edge robustness at 97.6%, followed by DeepLabpan at 97.5%, FCDeNet103 at 97.4%, WaSR at 97.1%, and DeepLab3+ at 97.0%.Segmentation methods overshoot falsely estimated edges 90.5% of the time; BiSeNet, DeepLabpan, RefineNet, and FCDeNet103 have the best overall localization accuracy, though FCDeNet103 and RefineNet more often undershoot.Source: 1) Results · p. 10, Table V · p. 11, Figure 14 · p. 11
  • WaSR achieves the highest overall F1, followed by DeepLabpan, RefineNet, BiSeNet, and DeepLab3+, which trail by 0.2%, 0.4%, 1.1%, and 5.5%, respectively.WaSR averages 2.6 false alarms per 100 images while detecting many true positives, and qualitative examples show robustness to sea foam, reflections, and visual ambiguities.Source: 1) Results · p. 11, Table V · p. 11, Figure 14 · p. 11
  • WaSR provides the best danger-zone compromise, outperforming MaskRCNN in precision and recall and thus achieving the strongest F1 among tested methods.MaskRCNN also has high true-positive and low false-positive rates; common buoys and boats often appear in the danger zone, where object detectors can detect them.Source: 1) Results · p. 11, Table VI · p. 12
  • DeepLab3+ most often detects persons and vessels but produces twice as many false positives as other methods, while RefineNet slightly leads vessel detection and performs best on other obstacle types.DeepLabpan performs best overall for person detection; the remaining four methods—DeepLabpan, BiSeNet, RefineNet, and WaSR—are approximately equally successful on vessels.Source: 1) Results · pp. 11–12, Table VII · p. 12
  • Obstacles larger than 10,000 pixels are detected at an average rate above 98% overall and in the danger zone, while tiny obstacles are often missed and account for 450 false alarms on average.RefineNet has the best true-positive rate across multiple size classes, whereas WaSR has the lowest false-discovery rate in every class; about 32% of tiny-obstacle false alarms occur in the danger zone.Source: 1) Results · pp. 12–13
  • Accurate segmentation methods average 38 FPS and are considered real-time for typical 10 Hz USV cameras, except DeepLabpan; IntCatchAI is fastest but has low detection accuracy.The hand-crafted ISSM and DeFP methods run at 34.4 ms and 9.0 ms per frame on a desktop CPU, respectively, but their obstacle detection performance is significantly lower.Source: 1) Results · p. 13

VII. CONCLUSION

MODS introduces a challenging USV benchmark for maritime object detection and obstacle segmentation, and its evaluations expose small-object and water-edge weaknesses in existing methods. The findings point to a need for maritime-specific, efficient perception methods.

VII. CONCLUSION

  • MODS provides 80,828 carefully annotated stereo images synchronized by IMU, training sets, and evaluation protocols for maritime object detection and segmentation-based obstacle detection.The benchmark evaluates several state-of-the-art methods using the proposed protocols.Source: §VII · p. 13
  • Object detectors scored approximately six times lower F1 than on COCO; MaskRCNN led detection with recognition, while FCOS led danger-zone detection-only and ran nearly twice as fast.FCOS reached approximately 14 fps on a high-end GPU, near the on-board cameras’ 10 Hz frame rate. The dataset is more challenging than evaluations constrained to large maritime objects.Source: §VII · p. 13
  • WaSR achieved the best dynamic-obstacle F1, while RefineNet and BiSeNet detected water-obstacle edges robustly but sometimes overshot the water edge.Geometric augmentation mattered more than color transfer; WaSR’s false-positive rate was halved in the danger zone versus BiSeNet. Large objects were detected above 98%, but performance fell for small objects such as persons and buoys; edge detection exceeded 97%, and segmentation methods ran at approximately 14 fps on a high-end GPU.Source: §VII · p. 13
  • The results motivate maritime-specific methods that better detect small objects, estimate water-obstacle edges, reduce false positives on tiny objects, and run on embedded low-power GPUs.The paper suggests using image sequences to improve detection, for example through tracking or architectures operating directly on consecutive frames.Source: §VII · p. 13

Related papers

Loading 2105.02359v2…