Source-linked AI summary
SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anything
Ge-Peng Ji, Deng-Ping Fan, Peng Xu, Ming-Ming Cheng, Bowen Zhou, Luc Van Gool
TL;DR
SAM’s promising segmentation capabilities prompt investigation of where it performs poorly. This study evaluates SAM against cutting-edge models and visualizes three concealed scenes; SAM often segments occluded objects into multiple separated masks.
Problem
The paper asks where SAM performs poorly despite its promising segmentation capabilities across varied scenarios.
Source: §1 · p. 1Method
The study quantitatively compares SAM with cutting-edge models on camouflaged object segmentation and visualizes camouflaged animals, industrial defects, and medical lesions.
Source: §1 · p. 1Results
SAM often segments occluded objects into multiple separated masks; the φ score gap between SAM (ViT-H) and CamoFormer-S reaches 13.8% on COD10K, 25.6% on CAMO, and 16.9% on NC4K.
Source: §2.3 · p. 2, §2.5 · p. 3Takeaways & Limitations
The study aims to help readers understand SAM’s performance traits in concealed scenes and bring new ideas to computer vision researchers.
Source: §3 · p. 4SAM’s granularity and uncertainty are bottlenecks limiting applications requiring high accuracy, such as autonomous driving and clinical diagnosis.
Source: §2.5 · p. 3
Abstract
from arXiv · showhide
Segmenting anything is a ground-breaking step toward artificial general intelligence, and the Segment Anything Model (SAM) greatly fosters the foundation models for computer vision. We could not be more excited to probe the performance traits of SAM. In particular, exploring situations in which SAM does not perform well is interesting. In this report, we choose three concealed scenes, i.e., camouflaged animals, industrial defects, and medical lesions, to evaluate SAM under unprompted settings. Our main observation is that SAM looks unskilled in concealed scenes.
1 INTRODUCTION
SAM’s promising segmentation capabilities prompt investigation of where it performs poorly. This study evaluates SAM in concealed scenes and finds it unskilled there.
- Large models’ generalization and emergent capabilities make it easy to believe they can solve anything.Source: §1 · p. 1
- SAM is a large ViT-based model trained on SA-1B, with promising segmentation capabilities across varied scenarios.Source: §1 · p. 1
- The study compares SAM quantitatively on camouflaged object segmentation and visualizes camouflaged animals, industrial defects, and medical lesions, finding it unskilled in concealed scenes.Source: §1 · p. 1
2 EXPERIMENT
The experiment compares unprompted SAM with leading camouflaged-object-segmentation models and examines its performance across three concealed-scene types. SAM improves with model size but remains behind leading COS models, with qualitative failures linked to concealed or ambiguous regions.
2 EXPERIMENT
- The experiment evaluates SAM on three COS benchmarks against transformer-based COS models and presents visualizations across camouflaged animals, industrial defects, and medical lesions.The benchmarks are CAMO, COD10K, and NC4K; comparison models include CamoFormer-P/-S and HitNet.Source: §2 · p. 1, §2.2 · p. 1
2.1 Mask Selection Strategy
- For each image, the mask with the highest IoU against its ground-truth mask is selected from SAM’s unprompted binary predictions.This selection strategy chooses the most appropriate mask for evaluation of the task’s interesting regions.Source: §2.1 · p. 1
2.2 Evaluation Protocols
- Evaluation follows standard practice, using CAMO, COD10K, and NC4K and five commonly used segmentation metrics with adaptive, mean, or maximum thresholding where applicable.The metrics are Sα, Eφ, Fβ, Fwβ, and mean absolute error.Source: §2.2 · p. 1
2.3 Quantitative Evaluation
- The Eφ difference between SAM (ViT-H) and CamoFormer-S [7] is 13.8% on COD10K, 25.6% on CAMO, and 16.9% on NC4K.SAM’s Fwβ on CAMO rises from 0.353 to 0.655 from ViT-B to ViT-L, then to 0.700 with ViT-H; the paper describes the improvement from ViT-L to ViT-H as limited.Source: §2.3 · pp. 1–2, Table 1 · p. 2
2.4 Qualitative Comparison
- Across scenes, SAM misses camouflaged animals, often segments industrial products rather than defects, and roughly segments organs but misses amorphous medical lesions.For animals, similar foreground and background appearance makes targets difficult to distinguish; the paper notes SAM is trained on natural objects with standard sizes and high-contrast attributes, and suggests anatomical priors for medical cases.Source: §2.4 · pp. 2–3
2.5 Discussion
- SAM often splits an occluded object into multiple separated masks, which the authors say indicates room to improve its semantic capabilities in concealed scenes..Source: §2.5 · p. 3
- Although SAM uses supervised training, the authors did not observe emergent or reasoning abilities in their experiments.They suggest exploring whether more challenging training tasks improve performance.Source: §2.5 · p. 3
- The authors identify granularity and uncertainty as bottlenecks limiting SAM’s use in high-accuracy settings such as autonomous driving and clinical diagnosis.They propose prior knowledge as one potential way to alleviate this issue.Source: §2.5 · p. 3
3 CONCLUSION
The paper empirically studies SAM through quantitative comparison on camouflaged-object segmentation and failure cases across three concealed scenes.
3 CONCLUSION
- The study quantitatively evaluates SAM against cutting-edge models on the camouflaged-object segmentation task.Source: §3 · p. 4
- It presents failure cases in camouflaged animals, industrial defects, and medical lesions.Source: §3 · p. 4
- The authors expect the study to help readers understand SAM’s performance traits in concealed scenes and inspire new ideas for computer vision researchers.Source: §3 · p. 4