Source-linked AI summary
Leveraging existing sparse point annotations for benthic imagery dense segmentation
Cesar Borja, Breck A. McCollum, Jarret E. Byrnes, Kenneth Sebens, Ana C. Murillo
TL;DR
Existing sparse benthic point annotations are difficult to trust for dense segmentation because some points propagate into harmful pseudo-masks. The paper filters unsuitable points before using SAM2 propagation and refines overlaps, producing higher-quality supervision and notable downstream gains.
Problem
Given vast repositories of sparse annotations, it remains unclear which points can be trusted for propagation and which resulting mask regions should be retained.
Method
The method propagates sparse point-labels with SAM2, then prunes unsuitable points and trims propagated masks to mitigate semantic leaks and class overlaps.
Results
The strategy produced the largest gains on noisier datasets, removing 34% of points on the benthic dataset and 25% on Coralscapes.
Takeaways & Limitations
Automatically discarding unsuitable points can improve pseudo-ground-truth quality and downstream fine-grained benthic segmentation without additional expensive human annotation.
Takeaways & Limitations
Appearance-based filtering is unreliable for classes with few point-labels and struggles to separate near-identical classes.
Abstract
from arXiv · showhide
The health of marine ecosystems is a critical indicator of global environmental change, yet the physical constraints of underwater observation and the intrinsic challenges of processing marine imagery severely limit the scalability of systematic monitoring. While recent visual foundation models such as the Segment Anything Model (SAM) series show great promise, they still struggle with the fine-grained recognition required in these complex scenarios and still require expert supervision. Our work addresses this gap by bridging state-of-the-art foundation models with existing sparse supervision. Because historical benthic surveys are typically annotated with only a few sparse expert points per image, we utilize these legacy point-labels as visual prompts for SAM2. Our primary contribution is a novel mechanism to automatically identify which of these points are suitable, and which are actively harmful, when used for propagation. By filtering out unreliable points, we extract high-quality pseudo-ground-truth masks capable of training more accurate, fine-grained semantic segmentation models. We demonstrate the effectiveness of our approach on public benthic data and introduce a new, challenging benchmark featuring real-world sparse expert annotations, paving the way for scalable ecological analysis.
1 Introduction
Benthic photographic surveys support monitoring of climate-driven ecological change but require labor-intensive expert analysis, while foundation models remain weak on fine-grained and rare classes. This work filters harmful sparse point labels before propagation and trims resulting masks, introducing a benchmark for realistic random annotations.
- Motivation: Ocean warming and species range shifts increase the need for scalable monitoring, while fixed-location benthic photographs provide repeatable records of species composition.Ecologists use these photographic snapshots to count, measure, and identify species without physically demanding and expensive fieldwork.
- Problem: Benthic image analysis remains time-consuming and dependent on skilled species-identification expertise despite software-assisted annotation workflows.The paper motivates further automation because manual analysis imposes a substantial burden on ecology researchers.
- Problem: Visual foundation models such as the SAM family still struggle with fine-grained and rare benthic classes, requiring additional expert supervision for accurate, specific models.These limitations motivate leveraging existing sparse expert annotations rather than relying only on foundation-model predictions.
- Problem: Randomly or uniformly sampled points can lie on class edges or ambiguous textures, producing pseudo-masks that bleed across object boundaries even when the original labels are correct.The framework addresses which existing points are trustworthy for propagation and which mask regions should be retained.
- Contributions: The framework combines pruning to discard unsuitable points before expansion with trimming to remove spillover or overlap afterward, improving dense masks from sparse labels.The authors also introduce a benchmark derived from long-term monitoring with randomly placed expert points and validate the method on new and public datasets.
2 Related Work
Prior work propagates sparse benthic point annotations into dense masks using superpixels, image features, or foundation models, while related research addresses label-noise filtering, pseudo-label refinement, and prototype-based classification. This work builds on these directions by using SAM2 point prompts and reliability checks to address propagation errors.
- Sparse point-label propagation for benthic imagery: CoralNet established sparse expert points as the dominant annotation regime for benthic surveys, motivating dense-mask propagation methods such as CoralSeg.The supplied passage identifies sparse point annotation as the prevailing regime and names CoralSeg as a downstream propagation approach.
- Sparse point-label propagation for benthic imagery: PLAS uses superpixel label purity, D+NN uses denoised DINOv2 nearest-neighbor matching, and SSeg combines SAM2 with superpixels while actively guiding new point placement.When ecological surveys already provide the labeled points, SSeg’s active point-selection paradigm cannot be applied without acquiring new annotations.
- Instance selection and label-noise filtering: The pruning method follows confident learning by identifying unsuitable points directly from held-out reference points, calibrating its threshold without training or model tuning.This distinguishes the approach from AUM, which detects mislabeled examples through training-time margin statistics.
- Pseudo-label reliability and refinement: Related pseudo-label methods use confidence thresholding or separate low-confidence regions, while mask-refinement methods such as SAMRefiner regrow cleaner masks with a foundation model.The approach likewise includes a refinement stage that trims obtained masks when necessary.
- Prototype-based segmentation: The trimming step repurposes per-image nearest-prototype classification as a reliability filter on foundation-model masks rather than as the primary segmenter.This connects the method to prototype-based segmentation approaches including ProtoSeg and PANet.
- Foundation model for image segmentation: SAM2 expands point labels into dense masks using each point as a visual prompt, but transformer-feature artifacts can cause patch-level drift, motivating complementary DINOv3-based patch reliability checks.The cited artifacts include high-norm outlier tokens and position-driven grid patterns that mislead similarity-based propagation.
3 Benthic datasets: a new long-term monitoring benchmark
The study uses Coralscapes and UCSD Mosaics as established densely annotated benthic datasets, then introduces a long-term monitoring benchmark with sparse expert point labels. The benchmark reflects real ecological imagery challenges, including difficult class distinctions and large-scale temporal survey data.
- Existing benthic datasets: The study selects Coralscapes, a 39-class densely annotated coral-reef dataset, and UCSD Mosaics, comprising 16 large 10K × 10K coral-reef mosaics.Both datasets provide high-quality semantic segmentation annotations across diverse classes.
- New long-term monitoring benchmark: The authors release a benchmark from a long-term monitoring project whose images span years and whose point annotations were not guided by segmentation.This annotation process makes label propagation more challenging than in datasets designed explicitly for segmentation.
- New long-term monitoring benchmark: The benchmark covers 24 foreground classes, with 313 training images, 78 test images, and about 180 expert point labels per image.Experts placed 200 random points per photo using CoralNet; points on mobile fauna were excluded.
- Monitoring motivation: The photos support a large monitoring effort tracking sessile invertebrate communities through annual summer surveys across multiple nearby sites, motivating automated segmentation for processing extensive data.The broader project provides a substantial source for documenting ecological change over time.
- Dataset challenges: The dataset contains near-identical colonial tunicates, morphologically similar encrusting forms, and a heterogeneous mixed-turf category that challenge generic and appearance-based models.These class characteristics make the benchmark difficult for existing methods.
4 Segmentation-aware sparse point label propagation
The method converts sparse point annotations into dense semantic segmentations with SAM2, then improves propagation quality by pruning harmful points and trimming locally inconsistent mask regions using DINOv3 features and per-image class prototypes.
- Propagation: Each sparse labeled point is supplied to SAM2 as a visual prompt, producing an independent dense mask whose outputs are combined into the image segmentation.The pipeline propagates every fixed point-label before subsequent quality-control stages.
- Merging: The final trimmed masks are merged into dense labels by ground-truth voting after propagation, pruning, and trimming.The pipeline overview describes merging the surviving masks into final dense labels.
- Trimming: Surviving masks are trimmed patch by patch when a region is more similar to another class than to its assigned class, preserving reliable mask portions.The trimming score compares each patch’s similarity to its own prototype against its maximum similarity to other class prototypes.
- Pruning: A pruning score identifies propagated masks whose covered patches disagree with the point’s class prototype, and unsuitable point-labels are removed with their masks.DINOv3 encodes 16 × 16-pixel patches into 1024-dimensional embeddings, while prototypes are computed per image and class.
- Pruning: The pruning threshold is the median score of verified-unsuitable propagated points, calibrated using a 75% expansion set and 25% validation witness set.A point is pruned when its score is at least τprune; if no verified-unsuitable evidence exists, the method prunes nothing.
- Pruning: Pruning proceeds worst-first while preserving at least one lowest-scoring point-label for every class, so the number of removed points varies by image.Scores are standardized per image before applying the class guard.
5 Experiments
Experiments evaluate SAM2-propagated pseudo-masks by training SegFormer models across three benthic-image datasets and comparing reliability-filtering strategies. The proposed pruning and trimming approach improves segmentation, with pruning contributing more than trimming while both remain complementary.
- Datasets and evaluation: The evaluation trains SegFormer models on SAM2-propagated pseudo-masks across the benthic dataset, Coralscapes, and UCSD Mosaics.The datasets contain 24, 39, and 34 classes, respectively; the benthic dataset averages 180 expert point-labels per image.
- Filtering strategies: The comparison isolates unreliable-supervision handling because every strategy receives the same point-labels and identical supervision.The results therefore rank filtering strategies in a weakly supervised sparse-point regime rather than measure performance comparable to densely supervised training.
- Filtering strategies: On the two noisier datasets, Ours separates clearly from keep all, whereas UCSD Mosaics methods cluster around 44 mIoU.The authors attribute UCSD Mosaics’ behavior to high-quality images, sharp boundaries, and sampled points rarely falling on ambiguous regions.
- Filtering strategies: 22.6 and 23.5 versus 27.5: random pruning underperforms keep all on the benthic dataset, showing that selecting which points to remove matters more than pruning alone.Random baselines discard as many points as Ours but without a reliability estimate.
- Ablation: 31.0 mPA after pruning versus 29.8 after trimming and 27.5 without either: both stages improve results, but pruning contributes more.The ablation indicates that pruning bad points before propagation and trimming bad regions afterwards are complementary.
- Qualitative analysis: Qualitative examples show pruning and trimming removing unreliable expansions and mislabeled regions, sometimes recovering pixels mislabeled by keep all.These examples are presented for the benthic dataset and Coralscapes.
6 Conclusion
The framework bridges sparse legacy annotations and dense semantic segmentation by filtering harmful propagation points and resolving class overlaps. It improves pseudo-ground-truth masks, while limitations remain for classes with few labels and near-identical appearances.
- Contributions: The method bridges sparse legacy annotations with dense semantic segmentation by addressing harmful label noise from randomly sampled survey points.The noise arises from boundary ambiguity when SAM2 is applied directly to survey points.
- Contributions: Pre-processing pruning removes unsuitable propagation points, while post-processing trimming resolves class overlaps and improves generated pseudo-ground-truth masks.These contributions were validated on established public benchmarks.
- Resources: The work introduces a challenging dataset with real-world sparse expert annotations alongside the validated method.The dataset complements experiments on established public benchmarks.
- Limitations: Appearance-based signals become unreliable for classes with few point-labels, and near-identical classes remain difficult to separate using appearance alone.The framework relies on per-image class prototypes, which are unreliable when a class has few point-labels.
- Impact: Despite these limitations, the framework offers a practical tool for using existing ecological image repositories without additional expensive human annotation effort.The authors also release their code, models, and new benchmark.