Source-linked AI summary
Beyond Weak Labels: Prompt-Guided Local Refinement for Weakly Supervised Water Segmentation in High-Resolution Multispectral Imagery
Muhammad Farhan Humayun, Mohammad Imangholiloo, Afifah Shah, Tomi Westerlund, Jukka Heikkonen
TL;DR
High-resolution water segmentation lacks reliable, affordable pixel-level labels, while hydrographic vectors provide scalable but imperfect supervision. The paper uses a two-stage framework that learns initial masks from rasterized vector pseudo-labels and refines local regions with structured prompts for SAM 2. On a manually corrected validation set, refinement improves both SegFormer-B0 and U-Net, with sharper shorelines and reduced boundary spillover.
Problem
Reliable pixel-level ground truth is difficult and costly, while hydrographic vectors contain artifacts that limit their use as exact supervision for high-resolution water segmentation.
Method
A two-stage framework learns initial masks from pseudo-label supervision, then converts predictions into structured spatial prompts for localized SAM 2 refinement.
Results
SegFormer-B0 IoU improves from 0.9509 to 0.9535 and U-Net IoU from 0.9408 to 0.9486 on a manually corrected validation set.
Takeaways & Limitations
Prompt-guided refinement acts as a boundary-aware local correction module, improving shoreline alignment and difficult local structures in pseudo-label-based segmentation.
Abstract
from arXiv · showhide
High-resolution water mapping supports environmental monitoring and related applications, but accurate pixel-level labels are difficult and costly to produce. Official hydrographic vectors provide scalable weak supervision, but they contain artifacts like boundary noise, temporal mismatch, and omissions of small water structures. We propose a two-stage framework for weakly supervised water segmentation in high resolution multispectral imagery. Stage 1 learns initial masks from rasterized vector pseudo-labels, and Stage 2 converts these masks into structured component-wise prompts for localized refinement. On a manually corrected validation set, refinement improves SegFormer-B0 from 0.9509 to 0.9535 IoU and U-Net from 0.9408 to 0.9486 IoU, with corresponding F1 gains from 0.9749 to 0.9762 and 0.9695 to 0.9736. It leads to sharper shorelines, reduced boundary spillover, and better thin-structure delineation. The results indicate that prompt-guided refinement can improve pseudo-label-based water segmentation by targeting local errors that are poorly captured by global training supervision.
1. INTRODUCTION
High-resolution multispectral water mapping needs reliable pixel-level delineation, but manual labels are costly and hydrographic vectors contain supervision artifacts. The paper addresses this with two-stage prompt-guided local refinement targeting boundary and small-component errors.
- High-resolution imagery captures shorelines, narrow channels, and small water bodies but introduces irregular boundaries, fragmented structures, and local visual ambiguities.
- Hydrographic vectors provide inexpensive, scalable pseudo-labels but include boundary noise, temporal mismatch, simplification, and omissions.
- Global pixel-wise scores can hide shoreline, thin-channel, small-water-body, and island errors because many tiles contain large easy regions.
- The proposed framework first learns initial masks from pseudo-label supervision, then performs prompt-guided local refinement with SAM 2.
- Structured component-wise prompts direct refinement toward boundary adjustment and recovery of small water or land components.
2. MATERIALS AND METHODS
The study uses official hydrographic vectors as weak supervision for multispectral water segmentation, then refines predicted components with structured prompts and SAM 2. Evaluation compares the resulting pipeline with manually corrected strong labels.
- Data and weak supervision: The study combines RGB and NIR orthophotos with rasterized Finnish hydrographic vectors to create four-channel inputs and weak pseudo-label masks.The study area is Joensuu, Finland, and includes lakes, ponds, rivers, and narrow streams.
- Data and weak supervision: 810 strong-label validation tiles were used for evaluation, approximately following a 90:10 split.
- Stage 1: baseline segmentation: SegFormer-B0 predicts pixel-wise non-water and water logits from four-channel tiles, with probabilities obtained through softmax.The model uses pretrained weights and is adapted for binary water segmentation.
- Stage 1: baseline segmentation: Training uses pseudo-label masks and pixel-wise cross-entropy, while evaluation uses manually corrected strong labels.
- Stage 2: prompt generation: Connected water components from Stage 1 are converted into structured prompts containing bounding boxes, positive and negative points, refinement regions, and refinement modes.Points are sampled from component interiors, high-confidence foreground, nearby non-water, boundaries, or hole-like regions according to local geometry and confidence cues.
- Stage 2: local refinement: SAM 2 generates candidate masks for each component, which are filtered geometrically and merged only within restricted local refinement regions.This preserves the original Stage-1 prediction outside allowed regions while targeting boundaries, small structures, and locally ambiguous areas.
3. RESULTS AND DISCUSSION
Prompt-guided Stage 2 refinement improves weakly supervised water segmentation primarily through local boundary correction, reducing spillover and improving difficult shoreline, narrow-channel, and fragmented-structure delineation for both SegFormer-B0 and U-Net.
- Quantitative results: 12.8% fewer false positives improved SegFormer-B0 precision from 0.9669 to 0.9710, IoU from 0.9509 to 0.9535, and F1-score from 0.9749 to 0.9762.Recall slightly decreased from 0.9829 to 0.9815, indicating that refinement mainly suppressed false water detections.
- Quantitative results: U-Net refinement improved IoU from 0.9408 to 0.9486 and F1-score from 0.9695 to 0.9736 while reducing both false positives and false negatives.The gains also included recall improvement from 0.9666 to 0.9727 and accuracy improvement from 0.9744 to 0.9778.
- Interpretation: The aggregate gains are modest but meaningful because large easy regions can hide local errors in global metrics.Stage 2 is best interpreted as a boundary-aware local refinement module rather than a semantic replacement for Stage 1.
- Visual results: Figure 2 presents zoomed-in local error maps before and after SegFormer-B0 refinement, while Figure 3 compares Stage 1 and Stage 2 outputs for SegFormer-B0 and U-Net.Figure 3 places SegFormer-B0 examples in the top three rows and U-Net examples in the bottom three rows.
- Visual results: Visual comparisons show cleaner water-land transitions, reduced shoreline spillover, and improved local mask quality for both Stage-1 backbones.SegFormer-B0 examples improved from 0.872 to 0.948 and from 0.902 to 0.941 tile-level IoU, while one difficult U-Net case improved from 0.000 to 0.964.
4. CONCLUSION
The study presents a two-stage weakly supervised framework that uses rasterized hydrographic pseudo-labels for initial masks and structured spatial prompts for localized SAM 2 refinement. Refinement improves both SegFormer-B0 and U-Net, mainly through reduced false positives, sharper shorelines, and better handling of narrow or fragmented water structures.
- Conclusion: The framework combines Stage 1 pseudo-label training with Stage 2 structured-prompt SAM 2 refinement evaluated on manually corrected strong labels.Stage 1 uses SegFormer-B0 and U-Net, while Stage 2 converts their predictions into localized spatial prompts.
- Conclusion: SegFormer-B0 IoU increased from 0.9509 to 0.9535 and U-Net IoU increased from 0.9408 to 0.9486 after refinement.Corresponding F1 gains were reported for both baselines.
- Conclusion: Prompt-guided refinement is effective as a boundary-aware local correction module for pseudo-label-based water segmentation under limited annotations.The supported scope particularly includes fragmented or narrow water bodies with shoreline spillover.