PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Segment Anything Model for Vision and Beyond

Chunhui Zhang, Li Liu, Yawen Cui, Guanjie Huang, Weilin Lin, Yiqian Yang, Yuehong Hu

arXiv:2305.08196v2cs.CVcs.AI

TL;DR

The paper addresses the need to understand SAM as a foundation model for segmentation and its expanding role across vision and beyond. It surveys SAM’s development, architecture, applications, and follow-up work, concluding with broad potential, limitations, and directions for stronger foundation models.

  • Problem

    The survey addresses the need for a comprehensive account of SAM’s progress, applications, and limitations across vision and beyond.

  • Method

    The paper synthesizes SAM’s foundation-model background, task and model design, concurrent methods, applications across tasks and data types, and follow-up research.

  • Results

    The survey finds that SAM has broad potential across image processing, vision-related tasks, and applications beyond vision, while documenting advantages and limitations.

  • Takeaways & Limitations

    The review provides insights for developing stronger foundation models and improving SAM’s robustness and generalization capabilities.

  • Takeaways & Limitations

    The surveyed multimodal visualization method does not fully explain how CLIP generates pixel-level results from text input.

Abstract

from arXiv · show

Artificial intelligence (AI) is evolving towards artificial general intelligence, which refers to the ability of an AI system to perform a wide range of tasks and exhibit a level of intelligence similar to that of a human being. This is in contrast to narrow or specialized AI, which is designed to perform specific tasks with a high degree of efficiency. Therefore, it is urgent to design a general class of models, which we term foundation models, trained on broad data that can be adapted to various downstream tasks. The recently proposed segment anything model (SAM) has made significant progress in breaking the boundaries of segmentation, greatly promoting the development of foundation models for computer vision. To fully comprehend SAM, we conduct a survey study. As the first to comprehensively review the progress of segmenting anything task for vision and beyond based on the foundation model of SAM, this work focuses on its applications to various tasks and data types by discussing its historical development, recent progress, and profound impact on broad applications. We first introduce the background and terminology for foundation models including SAM, as well as state-of-the-art methods contemporaneous with SAM that are significant for segmenting anything task. Then, we analyze and summarize the advantages and limitations of SAM across various image processing applications, including software scenes, real-world scenes, and complex scenes. Importantly, many insights are drawn to guide future research to develop more versatile foundation models and improve the architecture of SAM. We also summarize massive other amazing applications of SAM in vision and beyond. Finally, we maintain a continuously updated paper list and an open-source project summary for foundation model SAM at \href{https://github.com/liliu-avril/Awesome-Segment-Anything}{\color{magenta}{here}}.

1 INTRODUCTION

The survey situates SAM within the rise of foundation models and reviews its development, applications, and continuing research directions for vision and beyond.

  • Foundation models use broad pre-training to support zero-shot generalization across diverse downstream tasks, motivating analogous large-scale models in computer vision.
  • SAM is a task-agnostic foundation model for general image segmentation, trained on more than 1 billion masks and designed for broad downstream use.
  • The survey reviews SAM-related applications spanning medical analysis, image editing, style transfer, detection, captioning, tracking, 3D reconstruction, counting, and adversarial attacks.
  • This work presents a comprehensive review of SAM for vision and beyond, including historical development, applications across tasks and data types, and an updated paper and project summary.
  • The survey analyzes SAM across software, real-world, and complex scenes while identifying insights for more versatile foundation models and improved SAM architectures.

2 BACKGROUND AND TERMINOLOGY

The background introduces segmentation and foundation-model concepts, then explains how SAM combines promptable tasks, a modular architecture, and a data engine to support zero-shot transfer.

  • 2.1 Segmentation: Segmentation includes semantic, instance, and panoptic tasks, while interactive segmentation refines masks from user-provided points, strokes, or boxes.
  • 2.2 Foundation Models: The SA project targets promptable segmentation that can adapt quickly to tasks such as edge detection, object proposals, instance segmentation, and free-form text segmentation.
  • 2.2 Foundation Models: SAM is designed to return a valid mask for any segmentation prompt, including ambiguous prompts that may correspond to multiple reasonable objects.
  • 2.2 Foundation Models: SAM combines a MAE-pretrained ViT image encoder, sparse and dense prompt encoders, and a bidirectional Transformer mask decoder.
  • 2.3 Data: The SA data engine iteratively combines annotation and training, ultimately producing the SA-1B dataset with 11M images and 1.1B masks.
  • 2.3 Data: Reported zero-shot transfer performs strongly across point segmentation, edge detection, object proposals, instance and interactive segmentation, and Text-to-Mask segmentation.

2.4 Concurrent Works

Concurrent work pursued universal segmentation through task-conditioned multi-task learning, in-context coloring, and flexible multi-prompt visual-semantic modeling.

  • Overview: These methods address general segmentation alongside SAM through universal models, in-context inference, or broader prompt compatibility.
  • OneFormer: OneFormer unifies semantic, instance, and panoptic segmentation in one task-conditioned model and reports stronger performance than specialized models on ADE20K, Cityscapes, and COCO.
  • SegGPT: SegGPT frames segmentation as in-context coloring and uses random coloring to unify different segmentation data formats.
  • SEEM: SEEM supports points, boxes, scribbles, masks, text, and referred regions through a joint visual-semantic space that composes multiple prompts.

3.1 Software Scenes

SAM extends software-scene image editing and style transfer by providing promptable region masks that connect segmentation with inpainting, generation, and localized stylization.

  • 3.1.1 Image Editing: SAM addresses the annotation burden in image editing by generating accurate masks from simple prompts such as points or boxes.
  • 3.1.1 Image Editing: Inpaint Anything uses SAM masks with LaMa or Stable Diffusion to remove, fill, or replace selected objects.
  • 3.1.1 Image Editing: Edit Everything segments an image, uses CLIP to select the target region, and replaces it with an object generated by Stable Diffusion from a text prompt.
  • 3.1.2 Style Transfer: Any-to-Any Style Transfer lets users select style and content regions with SAM prompts, increasing interaction beyond applying one holistic style.
  • 3.1.2 Style Transfer: The style-transfer pipeline is reported as a plug-and-play component for local, global, and diffusion-based style-transfer methods.

3.2 Real-World Scenes

SAM is applied to real-world detection, counting, and moving-object segmentation, showing strong generalization in common scenes but uneven performance in unusual or low-contrast settings.

  • SAM generalizes well across common natural-image, agriculture, manufacturing, remote-sensing, and healthcare scenes, but is less effective in low-contrast conditions.
  • Detection: SAM outperforms U-Net on longitudinal concrete cracks but underperforms on unusual spalling cracks.
  • Detection: SAM enables universal crater detection by segmenting unrestricted image inputs, filtering circular-elliptical masks, and removing duplicates and artifacts.
  • Counting: SAM supports few-shot object counting by matching prompt-generated reference-object features against features from masks produced across the image.
  • Moving Object: SAM-generated preparatory masks support event-based moving-object segmentation datasets with 16 sequences and 13,314 annotated frames.

3.3 Complex Scenes

In complex scenes, SAM supports anomaly, concealed-object, thermal-infrared, and overhead-image applications, but performance depends strongly on domain characteristics and task-specific adaptation.

  • Low-Contrast Scene: SAA+ achieves state-of-the-art zero-shot anomaly segmentation using hybrid prompt regularization, while WS-SAM uses scribble supervision, pseudo-labeling, and multi-scale feature grouping for concealed objects.
  • Low-Contrast Scene: SAM fails to recognize glass objects themselves, performs worse than specialized methods, and is not ready for safety-critical glass-containing deployments.
  • Thermal Infrared Image: SATIR contains over 100,000 SAM-labeled thermal-infrared images, and its pretrained backbone improves SODA segmentation by approximately 1.3% mIoU.
  • Thermal Infrared Image: SAM performs strongly for chicken segmentation with total-point prompts, but thermal-image inputs introduce challenges.
  • Overhead Imagery: SAM often generalizes to overhead imagery but fails on targets with unique characteristics, motivating modest design changes and domain-specific decoders.
  • Overhead Imagery: SAM also supports large-scale remote-sensing annotation through the SAMRS dataset pipeline.

4.1 Vision Related Applications

This section surveys SAM applications across medical imaging, video tracking, image and video processing, and remote-sensing data annotation. The reviewed works show both broad zero-shot or adapted segmentation utility and extensions that support tracking, refinement, and large-scale dataset construction.

  • Medical Imaging: SAM-based medical imaging studies span CT, MRI, colonoscopy, histology, and multiple-format data, with evaluations covering diverse modalities and anatomies.The survey organizes medical-image use by imaging format and reports evaluations involving up to 11 medical imaging datasets or modalities.
  • Medical Imaging: SAM outperforms BET on MRI brain extraction across Dice, IoU, and accuracy, particularly with compromised image quality, non-isotropic voxels, or peripheral lesions.The comparison also reports finer separation of tissue compartments and brain structures.
  • Medical Imaging: Medical adaptations such as MedSAM, SAMM, and MSA improve domain-specific segmentation, with MedSAM surpassing default SAM across 21 3D and 9 2D tasks.MSA incorporates medical domain knowledge and reports performance across 19 tasks and varied modalities.
  • Vision Applications: SAM extensions broaden vision applications through interactive or automated video tracking, SAM-guided refinement, scene-text mask annotation, and remote-sensing segmentation.SAM-Track combines Grounding-DINO, DeAOT, and SAM, while SEEM is designed as an integrable refinement module.
  • Data Annotations: SAMText-9M contains over 2,400 video clips and more than 9 million scene-text mask annotations, while SAMRS supports semantic, instance, and object-detection research in remote sensing.SAMRS is constructed from existing remote-sensing object-detection datasets and is reported to exceed prior high-resolution segmentation datasets in size.

4.2 Beyond Vision

SAM is extended beyond conventional 2D segmentation through 3D reconstruction, multimodal explanation, graph analysis, video text annotation, and controllable captioning. These applications combine SAM with cross-view prompting, other foundation models, or downstream pipelines to broaden its scope.

  • 3D Reconstruction: SA3D iteratively projects SAM’s 2D masks into a 3D mask grid and uses cross-view self-prompting to obtain reliable novel-view segmentation.The resulting 3D mask grid can also determine occupied object space for reconstruction.
  • 3D Reconstruction: NeRF-based reconstruction remains constrained by high memory and computational costs, limiting current methods to relatively small scenes rather than large-scale outdoor environments.Additional depth maps and surface normals have been proposed to improve efficiency and accuracy.
  • Beyond Vision: SNA adapts SAM’s prompt-based framework to non-Euclidean graph analysis using slimmable graph convolutions and meta-learning for diverse graph samples and tasks.The survey presents this paradigm as a foundation-model direction for universal graph analysis.
  • Video Text Spotting: SAMText applies SAM to bounding-box annotations to generate fine-grained masks for large-scale video text datasets, including SAMText-9M.The resulting annotations support more accurate video text spotting models and may inspire segmentation-based approaches elsewhere.
  • Open-Vocabulary Interactive Segmentation: Specific-task adaptation remains challenging because SAM segments objects into separate masks, motivating pipelines that use Grounding DINO or CLIP to supply semantic visual prompts.Text-to-points is described as more readily achievable than text-to-boxes, which requires fine-tuning or additional supervision.
  • Multimodal Applications: Beyond segmentation, SAM supports controllable captioning and concept-based explanation through mask-guided controls and visual concepts supplied to surrogate models.CAT is training-free and adaptable, while EAC uses SAM-derived concepts to explain DNN predictions.

4.3 More Directions

SAM is applied to weakly supervised segmentation, adversarial robustness, one-shot personalization, and explainable AI. These studies show reduced annotation or training requirements alongside task-specific limitations and safety concerns.

  • Weakly-Supervised Semantic Segmentation: SAM-based weakly supervised segmentation can generate competitive pseudo-labels without fine-tuning, with reported improvements over prior methods on PASCAL VOC and MS-COCO.The cited study also found high-quality masks that could surpass human annotations in some cases.
  • Weakly-Supervised Semantic Segmentation: SAM-enhanced pseudo-label selection improved mIoU by an average of 6.2% across five SOTA WSSS methods on the PASCAL VOC 2012 training set.The approach is described as compatible with existing WSSS models without modifying their base networks or pipelines.
  • Adversarial Robustness: Attack-SAM finds that SAM is vulnerable to white-box attacks but relatively robust in black-box settings, with small objects tending to resist black-box attacks more strongly.The study also examines attack transferability among prompts in segment-everything mode.
  • One Shot: PerSAM creates personalized segmentation models from one-shot data without training, using target-guided attention, semantic prompting, and cascaded postrefinement.PerSAM-F fine-tunes only two parameters, and the approach is evaluated on PerSeg and video object segmentation.
  • Explainable AI: SAM supports concept-based XAI through EAC, which uses SAM-generated visual concepts and a per-input surrogate model to explain DNN predictions.The method is presented as an adaptable approach for explaining predictions using arbitrary concepts.
  • Explainable AI: EAC may produce misleading explanations in sensitive domains if unrelated concepts are used for medical images, creating risks for professional decision-making without adequate safety checks.The survey identifies mitigation of these societal risks as a direction for future work.

5 CONCLUSION

The survey comprehensively reviews SAM’s development, applications, advantages, and limitations across computer vision and beyond. It also distills insights for future foundation-model research and summarizes open-source SAM applications.

  • 5 CONCLUSION: The survey comprehensively reviews SAM’s progress for computer vision and beyond, including applications across tasks and data types.It also compares concurrent and follow-up works and discusses SAM in software, real-world, and complex scenes.
  • 5 CONCLUSION: It analyzes SAM’s advantages and limitations across applications to guide development of stronger foundation models and improved robustness and generalization.
  • 5 CONCLUSION: The survey summarizes additional SAM applications in vision and beyond and provides a preliminary open-source project summary.

A PRELIMINARY SUMMARY OF OPEN SOURCE PROJECTS ON SAM

The open-source project summary catalogs diverse SAM-based tools and applications spanning segmentation, editing, medical imaging, 3D perception, video, earth observation, and multimodal workflows.

  • A PRELIMINARY SUMMARY OF OPEN SOURCE PROJECTS ON SAM: The table summarizes open-source projects that extend SAM across segmentation, editing, tracking, medical imaging, 3D perception, and other applications.
  • A PRELIMINARY SUMMARY OF OPEN SOURCE PROJECTS ON SAM: Other projects adapt SAM to video tracking, semantic or instance segmentation, 3D perception, and interactive labeling interfaces.
  • A PRELIMINARY SUMMARY OF OPEN SOURCE PROJECTS ON SAM: Several projects combine SAM with language or vision-language models for text-prompted segmentation, object grounding, counting, captioning, and semantic labeling.
  • A PRELIMINARY SUMMARY OF OPEN SOURCE PROJECTS ON SAM: SAM-based tools support image editing and generation workflows including inpainting, style transfer, painting, and Stable Diffusion integration.
  • A PRELIMINARY SUMMARY OF OPEN SOURCE PROJECTS ON SAM: The catalog includes domain-specific tools for medical imaging, anomaly detection, earth observation, OCR, and large spatial files.
Loading 2305.08196v2…