Source-linked AI summary
Progressively Refining SAM for Accurate Phytoplankton Segmentation
Jialin Li, Jinkun Jiang, Yuezun Li, Junyu Dong
TL;DR
Phytoplankton segmentation remains underexplored amid difficult visual structures and scarce pixel-level annotations. PhySeg adapts SAM2 with progressive multi-scale refinement and introduces manually annotated masks; it achieves the best reported mIoU, BF1, BF2, and Dice scores among compared methods.
Problem
Phytoplankton’s varied morphology, slender structures, low contrast, and ambiguous boundaries challenge segmentation, while suitable pixel-level annotations remain scarce.
Source: §1 · p. 1Method
PhySeg adapts SAM2 with a multi-step refiner that fuses multi-scale features to progressively correct coarse logits, alongside a manually annotated phytoplankton dataset.
Source: §2 · pp. 1–2, §3 · p. 2Results
PhySeg achieves the best mIoU, BF1, BF2, and Dice scores, reaching 91.87%, 68.32%, 84.02%, and 95.68%, respectively.
Source: §4 · p. 3Takeaways & Limitations
Abstract
from arXiv · showhide
Accurate phytoplankton segmentation is essential for ecological monitoring, yet it remains largely underexplored due to the subtle visual characteristics of phytoplankton and the scarcity of pixel-level annotations. To address these challenges, we propose PhySeg, a SAM-based segmentation framework equipped with a lightweight refinement module that progressively restores fine-grained details through multi-scale feature fusion. In addition, we construct a phytoplankton segmentation dataset by manually annotating instance-level masks for an existing dataset, providing a valuable resource for future research. Experimental results demonstrate that PhySeg improves segmentation accuracy, validating the effectiveness of the proposed module. Code is available at https://github.com/lijialin404/PhySeg.
1. INTRODUCTION
Phytoplankton morphology matters for understanding marine ecosystems, but pixel-level segmentation remains underexplored amid challenging visual structures and scarce annotations. PhySeg addresses these gaps with a SAM2-based refiner and a manually annotated dataset.
1. INTRODUCTION
- Pixel-level masks are needed to characterize individual organisms’ morphology more precisely than category labels, boxes, or trajectories allow.The paper links this detail to more thorough ecological analysis.Source: §1 · p. 1
- SAM is not designed specifically for phytoplankton, whose varied morphology, slender structures, low contrast, and ambiguous boundaries challenge fine-detail recovery.Source: §1 · p. 1
- High-quality segmentation annotations are scarce in existing phytoplankton datasets, limiting the data available for model adaptation.Surveys also identify limited annotated data as a persistent challenge.Source: §1 · p. 1
- PhySeg adapts SAM2 with a multi-scale refiner and extends PMID2019 with 11,990 manually annotated instance masks across 9,824 images.The experiments establish a baseline for phytoplankton segmentation.Source: §1 · p. 1
2. METHOD
PhySeg adapts SAM2 for phytoplankton images with a lightweight refiner that iteratively corrects coarse logits. Its feature schedule moves from structural cues toward finer local details.
2. METHOD
- PhySeg takes an image and box prompt, refines SAM2’s coarse logits with multi-scale features, and applies LoRA to adapt the image encoder.Source: §2 · p. 1
- Progressive refinement replaces one-shot upsampling with multiple correction steps, each using Feature Fusion and a Residual Head to predict residual logits.The residuals are added to coarse logits step by step.Source: §2 · pp. 1–2
- The feature schedule adds F8 at Step 0, combines F4 with F8 at Step 1, then uses F4 at Step 2 to shift from structural correction to local detail.F16, F8, and F4 have resolutions of 1/16, 1/8, and 1/4 of the input image, respectively.Source: §2 · p. 2
- Updated logits and fused features form a hidden state carried between steps, while shallow encoder feature S0 supplies local spatial detail at every step.An additional Step 0 fusion module aggregates input context to construct H0.Source: §2 · p. 2
- Feature Fusion aligns inputs with ROI, uses three dilated convolution branches for local information and global average pooling for shared context, then predicts residual logits.The Residual Head uses 3 × 3 and 1 × 1 convolutions.Source: §2 · p. 2
- The refined prediction is trained with the sum of focal loss and Dice loss.Source: §2 · p. 2
3. PHYTOPLANKTON SEGMENTATION DATASET
PhySegD adds manually annotated ground-truth masks to PMID2019, addressing the limited availability of pixel-level phytoplankton data. Its annotations require careful review because cell boundaries can be difficult to distinguish.
3. PHYTOPLANKTON SEGMENTATION DATASET
- PhySegD is constructed by manually annotating ground-truth masks for the PMID2019 dataset.Source: §3 · p. 2
- Bright-field halos can resemble semi-transparent cell boundaries, and similar-texture impurities can obscure edges, making annotation labor-intensive.The paper reports that expert review is needed in multiple rounds.Source: §3 · pp. 2–3
- PhySegD contains 9,824 images and 11,990 instances across 15 categories, split into 6,880 training, 1,968 validation, and 976 test images.The corresponding instance counts are 8,378, 2,395, and 1,217.Source: §3 · p. 3, Table 1 · p. 3
4. EXPERIMENTS
Experiments compare PhySeg with segmentation and refinement methods on PhySegD using region and boundary metrics; results favor PhySeg on key measures, with ablation trends consistent with its design.
Experimental Setup
- Evaluation uses PhySegD, mIoU and Dice for region accuracy, and BF1, BF2, and BF4 for boundary quality; training adapts SAM2 with LoRA before training only the refiner.Each training stage lasts 15 epochs.Source: §4 · p. 3
Comparison Results
- On the test set, PhySeg achieves the best mIoU (91.87%), BF1 (68.32%), BF2 (84.02%), and Dice (95.68%).Table 2 compares methods using box or mask prompts.Source: §4 · p. 3, Table 2 · p. 4
- In visual examples, PhySeg better preserves slender structures and corrects inaccurate regions in initial masks.Fig. 4 shows the original image, ground truth, SAM2, SAM2+LoRA, HQ-SAM2, and PhySeg.Source: §4 · p. 4, Figure 4 · p. 4
- Against HQ-SAM2, PhySeg improves mIoU, Dice, BF1, and BF2 but decreases BF4 by 0.12 percentage points.The paper characterizes the results as balancing boundary refinement and region correction.Source: §4 · p. 4
Ablation Study
- Removing S0, Step 0, or Steps 0 and 1 consistently degrades multiple metrics, although overlapping features and SAM2’s strong baseline can limit individual gains.Table 3 evaluates feature inputs and progressive refinement steps on the validation set.Source: §4 · p. 4, Table 3 · p. 4
5. CONCLUSION
The conclusion presents PhySeg as a SAM2-based phytoplankton segmentation method with a progressive refiner and reports a manually annotated PMID2019 dataset. It states that experiments demonstrate the method’s effectiveness.
Conclusion
- PhySeg builds on SAM2 and uses a refiner that progressively fuses multi-scale features to recover details and correct inaccurate regions.The paper describes this as improving the delineation of fine phytoplankton structures.Source: §5 · p. 4
- The authors manually annotate ground-truth masks for the existing PMID2019 dataset to construct a phytoplankton segmentation dataset.Source: §5 · p. 4
- The reported experiments demonstrate the effectiveness of the proposed method.Source: §5 · p. 4