PaperlayerView exact revision on arXiv

Source-linked AI summary

FathomNet: A global image database for enabling artificial intelligence in the ocean

Kakani Katija, Eric Orenstein, Brian Schlining, Lonny Lundsten, Kevin Barnard, Giovanna Sainz, Oceane Boulais, Megan Cromwell, Erin Butler, Benjamin Woodward, Katy Croff Bell

arXiv:2109.14646v4cs.CVcs.LGeess.IV

TL;DR

Large-scale visual monitoring of marine life is difficult, and machine learning in the ocean is limited by inconsistent data and a shortage of large labeled datasets. FathomNet aggregates standardized, expertly curated marine imagery in a public database. Models trained on its data reduced annotator effort by 81% in one use case, while the authors note that no single general-use ocean image classifier or detector is foreseeable.

  • Problem

    Marine machine learning has had limited success amid a lack of data standardization and insufficient large, labeled image datasets, constraining visual monitoring at the scales needed for ocean stewardship.

    Source: Introduction · pp. 2–3

  • Method

    FathomNet is a publicly accessible database that standardizes and aggregates expertly curated labeled marine imagery from distributed sources.

    Source: Results · p. 3

  • Results

    81% reduction in human annotator effort in one use case, using FathomNet-trained models on visual data from other underwater platforms and institutions.

    Source: Discussion · p. 5

  • Takeaways & Limitations

    FathomNet-based workflows can assist, but not replace, expert taxonomists and ecologists in annotation.

    Source: Discussion · p. 5

    The authors do not expect a single general-use ocean image classifier or detector soon, citing too little expert-annotated data, undiscovered organisms, and challenges from distribution shifts.

    Source: Discussion · p. 5

Abstract

from arXiv · show

The ocean is experiencing unprecedented rapid change, and visually monitoring marine biota at the spatiotemporal scales needed for responsible stewardship is a formidable task. As baselines are sought by the research community, the volume and rate of this required data collection rapidly outpaces our abilities to process and analyze them. Recent advances in machine learning enables fast, sophisticated analysis of visual data, but have had limited success in the ocean due to lack of data standardization, insufficient formatting, and demand for large, labeled datasets. To address this need, we built FathomNet, an open-source image database that standardizes and aggregates expertly curated labeled data. FathomNet has been seeded with existing iconic and non-iconic imagery of marine animals, underwater equipment, debris, and other concepts, and allows for future contributions from distributed data sources. We demonstrate how FathomNet data can be used to train and deploy models on other institutional video to reduce annotation effort, and enable automated tracking of underwater concepts when integrated with robotic vehicles. As FathomNet continues to grow and incorporate more labeled data from the community, we can accelerate the processing of visual data to achieve a healthy and sustainable global ocean.

Introduction

Monitoring marine life at the ocean-wide spatiotemporal scales needed for stewardship is difficult, while below-surface biological sampling has lagged behind other ocean observations. Imaging can provide fine-scale biological observations, but its expert-intensive analysis and the shortage of accessible, diverse labeled marine datasets constrain automation.

Introduction

  • Marine observation must scale across a vast, rapidly changing ocean, but traditional sampling is resource-intensive and below-surface biological sampling has lagged.Distributed platforms and open data structures have advanced other forms of ocean observation, while large-scale biological sampling below surface waters remains constrained.Source: Introduction · p. 2
  • Imaging can identify animals to species, reveal community structure and spatial relationships, and capture fine-scale behavior, but complex imagery requires resource-intensive expert classification.Acoustics capture population- and group-scale dynamics but make individual identification difficult; eDNA offers broad community views, with source attribution and population-size relationships still under study.Source: Introduction · p. 2
  • Marine imaging supports biological sampling, underwater vehicle navigation, and public engagement, making visual data valuable for understanding and communicating about the ocean.Remotely controlled and autonomous platforms make imaging deployable across environments.Source: Introduction · p. 2
  • Annotation tools exist, but limited expert availability and the costs of annotating and storing footage leave automated marine-image annotation in demand.Processing complex scenes and organisms requiring expert classifications cannot scale without investment, capacity building, and advances in automation.Source: Introduction · p. 2
  • Deep-learning annotation can reach finer taxonomic levels, but requires publicly available, large-scale labeled image datasets.Random forests have classified plankton images at greater than 90% accuracy, while unsupervised learning has limited application for sufficiently granular marine-image annotation.Source: Introduction · p. 2
  • Marine labeled datasets largely focus on plankton, while macroscopic organisms lack an equivalent, readily accessible repository spanning diverse ocean communities and spatial scales.WHOI-Plankton includes 3M images across 109 concepts, and EcoTaxa includes 150M images from 10 imaging systems; plankton datasets contain regions of interest and exclude larger animals, while CoralNet is restricted to coral imagery.Source: Introduction · pp. 2–3

Results

FathomNet aggregates standardized, community-contributed marine imagery, and its use cases show both annotation assistance on other institutions’ video and sustained animal tracking with robotic vehicles. The NOAA example also shows that cross-region model output is not reliable on its own.

FathomNet database

  • FathomNet is a distributed, publicly available labeled-image dataset built on FAIR principles and Darwin CORE formats, with a website and REST API backed by a relational database.The database integrates community web services, including WoRMS.Source: Results · p. 3
  • The database aims for 1,000 independent observations per accepted Animalia species, exceeding 200 million observations, and can expand beyond biota to equipment, geological features, and debris.Its concept tree can incorporate additional taxonomic hierarchies and aggregate existing underwater datasets.Source: Results · p. 3

FathomNet dataset statistics

  • As of July 2022, FathomNet contained 84,454 images, 175,873 localizations, and 2,244 concepts from 204 collections.The database’s localizations typically occupy less of the frame and appear in more diverse poses than those in ImageNet, COCO, and iNat2017; annotation coverage varies by environment and taxonomic rank.Source: Results · p. 3

Benthic animal detection

  • On NOAA Musician Seamount images, the MBARI model produced many background false positives and negatives under distribution shift, yet substantially eased human annotation.The authors report that it is not reliable on its own; camera angle, unseen taxa, organism density, and coral morphology confounded detection.Source: Results · p. 4

Midwater activity detection

  • 43% IOU and 63% event-level recall; the activity-recognition routine flagged animals or objects in 19% of Deep Discoverer footage for review.The routine used filtered detections to generate video segments for human operators to review.Source: Results · p. 4

Machine learning-integrated tracking

  • 100% of one-second bins received 3D position information during the longest continuous ML-Tracking trial, which followed a siphonophore for 5.27 hours.The evaluation included nearly 50 hours of stereo and high-definition footage.Source: Results · p. 5

Discussion

FathomNet’s expert-annotated data supported cross-institution model use that reduced annotator effort in one case, while current data and distribution shifts limit general-purpose deployment. Its longer-term value depends on community contributions and use.

Discussion

  • MBARI’s FathomNet seed annotations cost approximately $165k, compared with estimated annotation costs of approximately $2,464k for ImageNet 2012 and $85k for COCO 2014.FathomNet’s initial labels rely on domain experts, whereas ImageNet and COCO use crowd-workers; MBARI’s source VARS data also represents approximately $144M in ship time.Source: Discussion · p. 5
  • FathomNet’s intended goals require substantial community engagement, high-quality contributions across diverse groups, and broad database use.The authors frame workflows as assisting, not replacing, expert taxonomists and ecologists.Source: Discussion · p. 5
  • The FathomNet Model Zoo shares community-contributed trained models for use on marine visual data, while future baseline datasets and annual challenges are envisioned as ways to develop algorithms.The envisioned challenges are contingent on FathomNet growing to include additional concepts and imagery.Source: Discussion · p. 5
  • 81% less human annotation effort was needed in one use case when MBARI-trained models generated proposals and activity-detected segments on other institutions’ video.The work also demonstrated high-level taxonomic models on other institutional data, including NOAA’s ROV Deep Discoverer.Source: Discussion · p. 5
  • A single general-use ocean classifier is not expected soon because expert annotations are insufficient, many organisms remain undiscovered, and natural distribution shifts challenge models.Differences in imaging systems, deployment mode, and environment challenged transfer between the broadly similar MBARI and NOAA datasets; models may still aid new-dataset annotation or strategic retraining.Source: Discussion · p. 5
  • A public, growing image resource could support education, conservation, and broader participation in ocean visual-data challenges.The authors describe possible community campaigns and game-based participation as ways to aggregate shared taxonomic knowledge and contribute science.Source: Discussion · p. 6

Methods

The methods describe how FathomNet aggregates imagery and annotations from multiple repositories, how its dataset characteristics are assessed, and how annotation costs are estimated. They also detail annotation coverage, image variability, data-use terms, and community resources.

Methods

  • FathomNet was seeded with curated imagery and metadata from MBARI, NGS, and NOAA, drawing on more than 30 years of underwater visual data while leaving further source data for future augmentation.MBARI’s VARS contributes imagery from eight imaging systems and four ROVs; its database contains approximately 27,400 video hours and 536,000 frame grabs, and FathomNet includes approximately 75% of VARS localizations at publication.Source: Methods · p. 6
  • NGS’s Deep Sea Camera System and NOAA’s OER video collections supplied additional data, with expert annotations and localizations contributing images and annotations to FathomNet.The initial NOAA contribution included data from two CAPSTONE cruises; the NGS contribution included 2,963 images and 3,256 bounding-box annotations.Source: Methods · pp. 6–7
  • FathomNet statistics benchmark the dataset against ImageNet, COCO, and iNat2017, while describing concepts, instances, taxonomic breakdowns, and annotation distributions.Aggregate statistics were computed from the Rest API in October 2021; taxonomic labels were back-propagated from annotator labels using the knowledgebase.Source: Methods · p. 7
  • Annotation coverage was assessed as average recall on 50 randomly selected images per taxonomic level for two benthic and midwater organisms, with an expert adding missing labels until all biological objects were localized.The false-detection rate of existing annotations was less than 0.1% for each concept.Source: Methods · p. 7
  • Pose variability was compared by averaging equally sized random image samples for medusae and echinoidiae in FathomNet and their closest ImageNet synsets, jellyfish and starfish.The method treats blurrier, more uniformly gray average images as indicating greater pose variability.Source: Methods · p. 8
  • FathomNet’s data-use policy licenses submitted annotation data under CC BY-ND 4.0 and permits hosted-image use for machine-learning training and development across commercial, academic, nonprofit, and government purposes.For other image uses, users must contact the original copyright holder, whose information is tracked in the database.Source: Methods · p. 8
  • The FathomNet ecosystem supports sharing trained models through the Model Zoo and lets community members validate submissions and augment existing labels through the web portal.The authors also provide code, blogs, and a YouTube channel; expert validation is especially helpful for images lacking full-coverage annotations.Source: Methods · p. 8
  • Cost estimates put annotation of MBARI’s initial 66,039-image contribution at approximately $165,100, while estimating COCO and ImageNet annotation at approximately $2.1M and $6.1M, respectively.These estimates use an AMT target hourly rate of $3.25 for comparison; the FathomNet figures reflect expert annotators paid $80 per hour and vary by habitat and image complexity.Source: Methods · pp. 8–9

Author contributions statement

The authors describe contributions spanning conception, database and pipeline development, data contributions, analyses, and manuscript preparation.

Author contributions statement

  • KK, KCB, and BW conceived FathomNet, while BS built the database, pipelines, API, and website back-end.KB developed the Python API and supported data ingestion from VARS.Source: Author contributions statement · p. 12
  • EO generated database statistics and worked on representative use cases with BW and KK; MC facilitated data contributions and ran NOAA-OER statistics.BW developed Tator-to-FathomNet pipelines incorporating NOAA and NGS data.Source: Author contributions statement · p. 12
  • LL and GS generated most VARS labels in FathomNet, and KK wrote the manuscript with significant contributions from EO.All authors reviewed the manuscript.Source: Author contributions statement · p. 12

Additional information

The paper identifies public repositories for its code, data, and referenced models and reports no competing interests.

Additional information

  • The manuscript’s code and data are available through the FathomNet Code Repository and database.The paper lists fathomnet.org and the FathomNet GitHub repository.Source: Additional information · p. 12
  • The benthic and midwater use-case models are listed in the FathomNet Model Zoo or at the cited model references.Source: Additional information · p. 12
  • The authors declare no competing interests.Supplementary information is also noted as available with the paper, including video.Source: Additional information · p. 12

Tables and Figures

The table and figures document FathomNet’s data-entry structure, portal and software architecture, dataset characteristics, and model-use examples. They also contrast annotation demands and image properties across sources.

Tables and Figures

  • Table 1 divides collection and image data-entry fields into required, recommended, and suggested categories.Collection categories follow the Darwin CORE Archive data standard.Source: Table 1 · p. 13
  • Figure 1 contrasts traditional single-concept annotations with localized multi-concept labels across midwater and benthic examples.The left-column images are largely iconic; the right-column images show non-iconic views with multiple concepts.Source: Figure 1 · p. 14
  • Figure 2 shows portal features for concept and filtered searches, image details, and tools to augment or correct annotations and localizations.Filtered searches can use geographic location or concept-tree terms.Source: Figure 2 · p. 15
  • Figure 3 presents a software architecture comprising a JavaScript REST API, SQL Server database, NATS.io messaging bus, and external web services.Examples of external services include WoRMS, VARS, and MarineRegions.Source: Figure 3 · p. 15
  • Figure 4 compares instance counts, concept counts, their averages, and relative instance sizes across FathomNet, COCO, ImageNet, and iNat2017.Most objects in FathomNet and iNat are small compared with those in ImageNet and COCO.Source: Figure 4 · p. 16
  • Figure 5 contrasts average image patches and annotation coverage: FathomNet suggests more pose and size diversity than ImageNet, while benthic images typically lack more annotations than midwater images.Coverage was checked by a human expert on 50 randomly selected images at each taxonomic level.Source: Figure 5 · p. 17
  • Figure 6 contrasts Monterey Bay validation with Musician Seamount application, where background false positives and negatives sometimes overwhelm correct annotations.The validation matrix’s background false negatives are mostly attributed to incomplete original annotations.Source: Figure 6 · p. 18
  • Figure 7 compares object-detection activity recognition with expert annotations, showing a low-contrast detection and a miss caused by unfocused, blurry optics.The low-contrast example is contrast-enhanced.Source: Figure 7 · p. 19
Loading 2109.14646v4…