Source-linked AI summary
A Dataset And Benchmark Of Underwater Object Detection For Robot Picking
Chongwei Liu, Haojie Li, Shuchang Wang, Ming Zhu, Dong Wang, Xin Fan, Zhihui Wang
TL;DR
Existing underwater object-detection datasets lack unified test annotations and contain similar images or incomplete labels, complicating fair algorithm comparison. This paper introduces DUO through dataset collection, deduplication, and re-annotation, alongside an efficiency-and-accuracy benchmark for academic and industrial reference.
Problem
Existing datasets often lack test-set annotations and contain similar images or incomplete labels, preventing a unified benchmark for comparing underwater object-detection algorithms.
Method
The paper collects and re-annotates available underwater datasets into DUO, removes overly similar images, and benchmarks SOTA detectors using accuracy and efficiency indicators.
Results
DUO contains 7,782 underwater images with four annotated classes, while its benchmark evaluates detector accuracy and efficiency in a robot-embedded setting.
Takeaways & Limitations
DUO and its benchmark provide a reference for academic research and industrial applications and support more comprehensive evaluation of underwater detectors.
Takeaways & Limitations
High-accuracy and high-efficiency underwater detection remains an open direction, with substantial room for performance improvement on DUO.
Abstract
from arXiv · showhide
Underwater object detection for robot picking has attracted a lot of interest. However, it is still an unsolved problem due to several challenges. We take steps towards making it more realistic by addressing the following challenges. Firstly, the currently available datasets basically lack the test set annotations, causing researchers must compare their method with other SOTAs on a self-divided test set (from the training set). Training other methods lead to an increase in workload and different researchers divide different datasets, resulting there is no unified benchmark to compare the performance of different algorithms. Secondly, these datasets also have other shortcomings, e.g., too many similar images or incomplete labels. Towards these challenges we introduce a dataset, Detecting Underwater Objects (DUO), and a corresponding benchmark, based on the collection and re-annotation of all relevant datasets. DUO contains a collection of diverse underwater images with more rational annotations. The corresponding benchmark provides indicators of both efficiency and accuracy of SOTAs (under the MMDtection framework) for academic research and industrial applications, where JETSON AGX XAVIER is used to assess detector speed to simulate the robot-embedded environment.
1. INTRODUCTION
Underwater robot picking relies on object detection, but existing datasets lack accessible test annotations, contain redundant or incomplete data, and do not support a unified comparison. The paper introduces DUO and a benchmark designed to address these gaps.
- Underwater object detection locates sea creatures for robots that automatically capture them in open-sea farms.
- Existing URPC datasets lack accessible test-set annotations, forcing researchers to create their own splits and retrain competing methods.This increases workload and produces inconsistent comparisons across studies.
- URPC datasets contain many similar or duplicate images, and URPC2017 retains only 15% after similar-image removal, encouraging overfitting.
- DUO collects and re-annotates available underwater datasets, producing 7,782 images with four classes and more accurate annotations.The classes are holothurian, echinus, scallop, and starfish.
- The accompanying MMDetection-based benchmark reports both detector accuracy and efficiency for academic and industrial reference.Efficiency is assessed on JETSON AGX XAVIER to simulate an embedded underwater-robot environment.
2. BACKGROUND
The background traces underwater detection for robot picking to URPC datasets and describes the collected sources underlying DUO. These datasets vary in size, resolution, capture conditions, and class coverage.
- URPC2017 initiated underwater object detection for open-sea farming within the Underwater Robot Picking Contest.The contest included target recognition, fixed-point grasping, and autonomous grasping tracks.
- The DUO source datasets include URPC2017, URPC2018, URPC2019, URPC2020ZJ, URPC2020DL, and UDD.
- URPC2017 contains 17,655 training and 985 testing images from six videos, whose frames are highly similar because they were sampled every 10 frames.
- URPC2018 and URPC2019 lack test-set annotations, while URPC2018 also includes images from an artificial underwater environment.
- URPC2020ZJ and URPC2020DL contain 5,543 and 6,575 training images, respectively, and both lack test-set annotations.
3. PROPOSED DATASET
DUO is constructed by deduplicating and re-annotating available underwater datasets, yielding diverse scenes, more accurate labels, and statistics dominated by small, long-tailed object instances.
- 3.1. Image Deduplicating: PHash removes duplicate or overly similar images while preserving varied underwater scenarios.The hash remains approximately stable when image content is not significantly modified, enabling within-scenario deduplication.
- 3.1. Image Deduplicating: 7,782 images remain after deduplication, including 6,671 training images and 1,111 testing images.The new dataset has a 95% retention rate and retains various underwater scenes.
- 3.2. Re-annotation: A GFL detector followed by manual annotation addresses missing, incorrect, unavailable, and incomplete class labels.The process first trains on originally labeled images, predicts all 7,782 images, and uses predictions as groundtruth candidates for re-annotation.
- 3.3. Dataset Statistics: 74,515 objects span holothurian, echinus, scallop, and starfish, with echinus comprising 67.3% and producing an obvious long-tail distribution.The class counts are 7,887, 50,156, 1,924, and 14,548, respectively.
- 3.3. Dataset Statistics: Most objects occupy 0.3% to 1.5% of image area, while most images contain 5 to 15 instances averaging 9.57.DUO therefore combines massive small instances with a long-tail distribution.
4. BENCHMARK
The benchmark evaluates detector accuracy and efficiency under standardized MMDetection settings, including embedded-device speed. Results expose accuracy–efficiency trade-offs and persistent challenges for small-object detection.
- 4.1. Evaluation Metrics: The benchmark reports COCO mAP metrics, class-specific AP, parameter count, FLOPs, and FPS for accuracy and efficiency evaluation.AP is measured across IoU=0.50:0.05:0.95, with separate size and class metrics.
- 4.2. Implementation Details: SOTA detectors are evaluated as multi- and one-stage methods with ResNet18, ResNet50, and ResNet101 backbones, then deployed on AGX for efficiency assessment.The benchmark uses MMDetection and standardized training configurations, including ImageNet-pretrained backbones and fixed image resizing.
- 4.3. Benchmark Analysis: ATSS and GFL can achieve both high accuracy and high efficiency, unlike the general trade-off observed between Cascade R-CNN and RetinaNet.Cascade R-CNN generally has high accuracy and low efficiency, whereas RetinaNet generally has low accuracy and high efficiency.
- 4.3. Benchmark Analysis: The accuracy difference between multi-stage and one-stage methods in AP is not obvious, while APS is consistently the lowest size-specific AP.APSc is substantially behind the other class AP values because scallop has the fewest instances.
- 4.3. Benchmark Analysis: 7.4 FPS is the maximum AGX speed, making the evaluated efficiency difficult to deploy on underwater robots.Large parameter counts and FLOPs are associated with low FPS on AGX.
5. CONCLUSION
The paper introduces DUO and a corresponding benchmark to fill gaps in underwater object-detection research. Their diverse scenes, improved annotations, and joint accuracy–efficiency evaluation support academic and industrial reference use.
- 5. CONCLUSION: DUO and its benchmark address community gaps through diverse underwater scenes, more reasonable annotations, and comprehensive detector evaluation.The benchmark includes both efficiency and accuracy indicators and is intended as a reference for academic research and industrial applications.