Source-linked AI summary
Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Grouping
Chunming He, Kai Li, Yachao Zhang, Guoxia Xu, Longxiang Tang, Yulun Zhang, Zhenhua Guo, Xiu Li
TL;DR
WSCOS must segment visually concealed objects despite foreground–background similarity and sparse supervision. The paper combines SAM-generated pseudo labels with uncertainty-aware refinement and multi-scale feature grouping, reporting state-of-the-art performance across WSCOS tasks. Its scope includes settings where SAM itself can produce unreliable masks or point annotations are randomly selected.
Problem
WSCOS is difficult because concealed objects resemble their backgrounds and sparse annotations provide limited supervision for learning.
Method
The method uses sparse annotations as SAM prompts for pseudo-label generation, refines masks with augmentation fusion and entropy-based weighting or selection, and adds Multi-scale Feature Grouping.
Results
Experiments across multiple WSCOS tasks show that the method achieves state-of-the-art performance and outperforms baselines and existing methods.
Takeaways & Limitations
The framework provides a combined strategy for more reliable weak supervision and more coherent segmentation in single- and multiple-object concealed scenes.
Takeaways & Limitations
SAM can fail on concealed objects, while the reported point-supervision setup randomly selects foreground and background points from ground-truth masks.
Abstract
from arXiv · showhide
Weakly-Supervised Concealed Object Segmentation (WSCOS) aims to segment objects well blended with surrounding environments using sparsely-annotated data for model training. It remains a challenging task since (1) it is hard to distinguish concealed objects from the background due to the intrinsic similarity and (2) the sparsely-annotated training data only provide weak supervision for model learning. In this paper, we propose a new WSCOS method to address these two challenges. To tackle the intrinsic similarity challenge, we design a multi-scale feature grouping module that first groups features at different granularities and then aggregates these grouping results. By grouping similar features together, it encourages segmentation coherence, helping obtain complete segmentation results for both single and multiple-object images. For the weak supervision challenge, we utilize the recently-proposed vision foundation model, Segment Anything Model (SAM), and use the provided sparse annotations as prompts to generate segmentation masks, which are used to train the model. To alleviate the impact of low-quality segmentation masks, we further propose a series of strategies, including multi-augmentation result ensemble, entropy-based pixel-level weighting, and entropy-based image-level selection. These strategies help provide more reliable supervision to train the segmentation model. We verify the effectiveness of our method on various WSCOS tasks, and experiments demonstrate that our method achieves state-of-the-art performance on these tasks.
1 Introduction
WSCOS seeks concealed-object segmentation from sparse annotations, where foreground–background similarity and weak supervision limit reliable learning. The proposed method combines SAM-based pseudo labeling with uncertainty-aware refinement and multi-scale feature grouping, achieving state-of-the-art performance across WSCOS tasks.
- Intrinsic foreground–background similarity makes concealed objects difficult to distinguish and can produce incomplete or fragmented segmentation.
- WSCOS reduces annotation costs by learning from a few foreground or background points or scribbles instead of pixel-wise labels.
- SAM generates dense masks from sparse annotations used as prompts, turning weak annotations into pseudo labels for training a segmentation model.
- Multi-augmentation fusion, entropy-based pixel weighting, and entropy-based image selection improve the reliability of SAM-generated supervision.
- Multi-scale Feature Grouping aggregates cues across granularities to promote coherence and support complete segmentation of single and multiple objects.
- The method achieves state-of-the-art performance across various WSCOS tasks.
2 Related Works
Prior work covers fully supervised concealed-object applications and limited weakly supervised segmentation, while SAM offers strong general segmentation but struggles with poorly visible objects. WS-SAM adapts SAM to WSCOS and refines its pseudo labels for more stable training.
- SAM provides zero-shot generalization and high-quality segmentation, but studies report difficulties with poorly visible objects in concealed-object tasks.
- WS-SAM uses sparse annotations as SAM prompts to generate dense pseudo labels and introduces refinement based on augmentation consistency and prediction reliability.
- Research on weakly supervised concealed-object segmentation remains limited, and sparse-only supervision restricts the segmenter's discrimination capacity.
3 Methodology
The method combines SAM-generated pseudo labels with refinement strategies for weak supervision and multi-scale feature grouping for concealed-object segmentation. It jointly trains with sparse annotations and pseudo labels while using entropy to emphasize reliable supervision.
- Pseudo Labeling with SAM: SAM uses sparse annotations as prompts to generate dense masks that serve as pseudo labels for training.The approach addresses the limited supervision available from points or scribbles.
- Pseudo Label Refinement: Multi-augmentation fusion combines masks from transformed images after inverse transformation to the original image, producing a more reliable fused mask.Different augmented views can produce complementary masks, recovering regions missed by individual predictions.
- Pseudo Label Refinement: Entropy-based pixel weighting assigns higher weights to pixels that are confidently and consistently predicted across augmentations.Low entropy indicates lower prediction uncertainty in the fused mask.
- Pseudo Label Refinement: Entropy-based image selection filters images using absolute and relative uncertainty, with thresholds τa=0.1 and τr=0.5.Relative uncertainty is designed to accommodate small-object scenarios, while selection excludes highly unreliable images.
- Multi-scale Feature Grouping: MFG groups features into learnable prototypes at different granularities, iteratively updates prototypes, and aggregates grouping features to capture global and detailed information.The grouping process uses projected queries, keys, and values, GRU-based prototype updates, and T=3 iterations.
- Weakly-Supervised Training: The model is trained jointly with sparse annotations and SAM-generated pseudo labels using partial cross-entropy, cross-entropy, and intersection-over-union losses.The partial cross-entropy term uses sparse annotations, while the other two losses use the pseudo label.
4 Experiments
Experiments evaluate the method across three WSCOS tasks, supervision types, ablations, and further analyses. Results consistently favor the proposed WS-SAM and MFG designs, including stronger performance on multi-object images.
- Experimental Setup: Evaluations cover camouflaged object detection, polyp image segmentation, and transparent object detection using point annotations, with additional COD experiments using scribbles.The study uses four common segmentation metrics and compares against baselines and existing methods.
- Comparative Evaluation: The method achieves the best performance across COD datasets and remains superior to direct SAM fine-tuning with point or scribble supervision.Integrating SCOD into WS-SAM also remains below the proposed method.
- Comparative Evaluation: The method significantly surpasses SCOD on polyp image segmentation and surpasses all baselines and existing methods on transparent object detection.These results are reported for point supervision.
- Ablation Study: Adding multi-augmentation fusion, pixel-level weighting, and image-level selection improves WS-SAM over the single-mask baseline.The ablation isolates the contributions of the three pseudo-label refinement strategies.
- Ablation Study: MFG outperforms removing feature grouping, replacing grouping with slot attention, removing multiscale processing, or replacing the weighted gate with a fixed coefficient.This supports the proposed MFG design choices.
- Further Analysis: 10.7% is the performance gap between MFG and SCOD on 186 COD10K multi-object images, compared with 5.8% across all test images.MFG also improves a fully supervised baseline to a level comparative with state-of-the-art methods.
5 Conclusions
The paper concludes that WS-SAM provides refined pseudo-label supervision while MFG improves feature coherence for concealed-object segmentation. Across multiple WSCOS tasks, the combined method outperforms baselines and existing methods.
- 5 Conclusions: WS-SAM generates masks with SAM and refines them through augmentation fusion, pixel-level uncertainty weighting, and image-level uncertainty filtration.The refined masks train the segmentation model with more reliable supervision.
- 5 Conclusions: MFG groups extracted clues to improve feature coherence, alleviating incomplete segmentation and supporting better multiple-object segmentation.Its design targets nuanced discrimination in concealed images.
- 5 Conclusions: Experiments on multiple WSCOS tasks confirm superiority over the baseline and existing methods.