PaperlayerView exact revision on arXiv

Source-linked AI summary

Fast Segment Anything

Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, Jinqiao Wang

arXiv:2306.12156v1cs.CVcs.AI

TL;DR

SAM’s computational demands limit its wider industrial use, motivating a faster alternative. FastSAM reformulates segment anything as CNN-based all-instance segmentation plus prompt-guided selection, achieving comparable performance with substantially higher speed while retaining a documented mask-quality weakness.

  • Problem

    SAM’s Transformer-based computation, especially at high-resolution inputs, makes wider industrial and real-time application challenging.

  • Method

    FastSAM trains a CNN instance-segmentation detector on 1/50 of SA-1B, then uses generated masks and prompts to select the desired objects.

  • Results

    FastSAM achieves comparable performance with SAM while running 50x faster than SAM (32×32) and 170x faster than SAM (64×64).

  • Takeaways & Limitations

    FastSAM provides a practical real-time alternative for segment anything and supports downstream tasks and industrial applications.

  • Takeaways & Limitations

    FastSAM’s mask generation performance is below SAM, with tiny-object masks tending toward squares and large-object masks showing border artifacts.

Abstract

from arXiv · show

The recently proposed segment anything model (SAM) has made a significant influence in many computer vision tasks. It is becoming a foundation step for many high-level tasks, like image segmentation, image caption, and image editing. However, its huge computation costs prevent it from wider applications in industry scenarios. The computation mainly comes from the Transformer architecture at high-resolution inputs. In this paper, we propose a speed-up alternative method for this fundamental task with comparable performance. By reformulating the task as segments-generation and prompting, we find that a regular CNN detector with an instance segmentation branch can also accomplish this task well. Specifically, we convert this task to the well-studied instance segmentation task and directly train the existing instance segmentation method using only 1/50 of the SA-1B dataset published by SAM authors. With our method, we achieve a comparable performance with the SAM method at 50 times higher run-time speed. We give sufficient experimental results to demonstrate its effectiveness. The codes and demos will be released at https://github.com/CASIA-IVA-Lab/FastSAM.

1. Introduction

FastSAM addresses SAM’s deployment challenge by replacing its computationally demanding Transformer-based approach with a real-time CNN solution for segment anything. It reports comparable performance while substantially reducing computational demands and preserving usefulness across vision tasks.

  • Motivation: SAM’s Transformer architecture requires substantial computation, making practical deployment, especially in real-time applications, challenging.The computational burden is identified as a central barrier to wider industrial use.
  • Approach: FastSAM decouples segment anything into all-instance segmentation followed by prompt-guided selection.A CNN detector first generates masks for all instances, then selects the prompt-relevant region.
  • Approach: FastSAM uses a CNN-based detector to produce segmentation masks and then outputs the region of interest corresponding to a prompt.This design leverages CNN computational efficiency to target real-time segmentation without much performance compromise.
  • Results: FastSAM is trained on only 2% (1/50) of SA-1B and achieves 63.7 at AR1000 on COCO, 1.2 points above SAM with 32×32 point prompts.The reported comparison also states that FastSAM runs 50 times faster on a single NVIDIA RTX 3090.
  • Implications: The method is presented as a real-time solution intended to facilitate industrial applications and provide an alternative perspective on efficient vision-model design.The paper connects this efficiency-oriented design with specific-task models and reduced computational effort.
  • Evaluation: The study evaluates FastSAM against SAM on multiple benchmarks to examine strengths and weaknesses in the segment anything domain.The comparison includes speed and downstream benchmark evaluations.

2. Preliminary

The paper presents SAM as a class-agnostic, promptable foundation model for zero-shot segmentation, then defines segment anything as prompt-conditioned mask generation. It frames the task as two stages that can reduce complexity and support real-time processing.

  • Segment Anything Model: SAM provides high-precision, class-agnostic segmentation guided by varied prompts and demonstrates adaptability across segmentation tasks.The model is described as a foundation model with zero-shot capabilities.
  • Segment Anything Model: SAM’s lack of real-time processing capability restricts applications requiring immediate segmentation results.The limitation is presented despite the model’s satisfying segmentation performance.
  • Segment Anything Task: The segment anything task produces an effective segmentation mask from prompts such as points, boxes, masks, text, or other content-indicating information.The definition covers both user interaction and free-form textual prompts.
  • Segment Anything Task: The task can be divided into detecting and segmenting all image objects, followed by selecting the prompted object or objects from the resulting segmentation.The first stage resembles panoptic segmentation, while the second uses the supplied prompt.
  • Segment Anything Task: This decoupling significantly reduces task complexity and creates the possibility of a real-time segment anything model.

3. Methodology

FastSAM implements segment anything as an all-instance segmentation stage followed by prompt-guided selection. It uses YOLOv8-seg and prompt-specific matching to select regions efficiently from generated masks.

  • 3.1. Overview: FastSAM consists of All-instance Segmentation and Prompt-guided Selection, with the second stage serving as task-oriented post-processing.The design differs from end-to-end Transformers by using convolutional local connections and receptive-field-related object assignment.
  • 3.1. Overview: YOLOv8-seg generates masks for all objects or regions before prompts identify the specific object or objects of interest.The supported prompts include points, boxes, and text, with text prompting based on CLIP.
  • 3.2. Model Architecture: YOLOv8-seg combines detection and instance-segmentation branches, producing bounding boxes alongside mask prototypes and coefficients.FastSAM directly uses this method for its all-instance segmentation stage.
  • 3.3. Prompt-guided Selection: Point prompts select masks containing foreground points and use background points to filter irrelevant masks before merging selected masks.
  • 3.3. Prompt-guided Selection: Box prompts select the mask whose bounding box has the highest IoU with the selected box.
  • 3.3. Prompt-guided Selection: Text prompts use CLIP text embeddings and image-feature similarity to select the mask with the highest matching score.
  • 3.3. Prompt-guided Selection: These prompt-guided techniques select specific objects from the segmented image and support real-time execution.The paper leaves more effective prompt-guided selection for future exploration.

4. Experiments

FastSAM is evaluated for runtime efficiency and four zero-shot tasks, including edge detection, object proposal generation, instance segmentation, and text-prompted segmentation. Results show comparable edge performance and strong object-proposal performance, while small-object masks and instance-segmentation accuracy remain limitations.

  • Experimental scope: FastSAM is evaluated across runtime efficiency and four zero-shot tasks spanning edge detection, object proposals, instance segmentation, and text-prompted segmentation.The experiments compare capabilities with SAM and examine real-world applications, speed, and deployment.
  • Implementation: FastSAM uses a YOLOv8-x CNN with 1024-pixel inputs and is trained for 100 epochs on 2% of SA-1B.The bounding-box regression reg max is increased from 16 to 26 for predicting large instances.
  • Runtime efficiency: FastSAM surpasses SAM at every tested prompt count, while its running speed remains unchanged as the number of prompts increases.The comparison uses milliseconds per image on a single NVIDIA GeForce RTX 3090 GPU; SAM also has an Everything mode.
  • Zero-Shot Edge Detection: FastSAM produces generally good BSDS500 edge maps with 68M parameters and achieves similar performance to SAM, with higher R50 and lower AP.Both methods predict more edges than the BSDS500 ground truth, including some logical edges that are not annotated.
  • Zero-Shot Object Proposal Generation: On COCO object proposals, FastSAM and SAM transfer fully zero-shot; FastSAM outperforms OLN at AR@1000 but trails supervised methods at AR@10.The evaluation covers all 80 COCO classes and reports average recall and AUC for learning-free, supervised, and zero-shot methods.
  • Zero-Shot Object Proposal Generation: FastSAM substantially surpasses SAM-H E64 by over 5% on LVIS bbox AR@1000, but its mask-proposal recall is lower because small-object masks are insufficiently fine-grained.FastSAM still falls short of LVIS-trained ViTDet-H, and its text-to-mask speed is limited because each mask region is processed by CLIP.

5. Real-world Applications

FastSAM is evaluated in anomaly detection, salient object segmentation, and building extraction, using point, box, and everything prompts. Across these applications, it generally produces results comparable to SAM, with specific differences in precision, background segmentation, and object-shape handling.

  • Evaluation Setup: The application evaluations compare FastSAM’s point-prompt, box-prompt, and everything modes with SAM and ground truths.The showcased scenarios include anomaly detection, salient object segmentation, and building extraction.
  • Anomaly Detection: FastSAM segments nearly all anomaly-detection regions similarly to SAM in everything mode, but with lower precision and incomplete background coverage.Foreground/background points or box-guided selection can isolate defective regions.
  • Salient Object Segmentation: Under everything mode, FastSAM segments fewer irrelevant background objects than SAM in salient object segmentation.The difference is described as minor for the ReDWeb-S evaluation.
  • Salient Object Segmentation: FastSAM’s point-guided salient-object masks are nearly identical to SAM-point and ground truth, with only minor edge details lost.A single box cannot select multiple objects, a limitation shared with SAM-box.
  • Building Extraction: FastSAM performs well on regularly shaped buildings but segments fewer shadow-related regions than SAM.Point and box prompts can select regions of interest, and merging can recover a building mask after placing a point in a shadow.

6. Discussion

FastSAM offers substantial speed advantages and strong box generation, while its mask quality remains below SAM in several cases. The discussion identifies confidence scoring, mask-generation artifacts, and limited training data as improvement opportunities.

  • Strengths: FastSAM achieves comparable performance with 50x higher speed than SAM (32×32) and 170x higher speed than SAM (64×64).The authors identify this speed as useful for industrial applications and report better masks for some large objects.
  • Weaknesses: FastSAM’s box generation has a significant advantage, but its mask generation performance is below SAM.The authors visualize representative bad cases in Figure 11.
  • Scoring: Low-quality small-object masks can receive large confidence scores because YOLOv8’s bounding-box score is not strongly related to mask quality.Predicting mask IoU or another quality indicator is proposed as a possible improvement.
  • Mask Generation: Tiny-object masks tend toward square shapes, while large-object masks may show artifacts along bounding-box borders.The authors attribute this weakness to YOLACT and suggest stronger mask prototypes or a reformulated mask generator.
  • Training Data: Using more training data could further enhance performance because FastSAM uses only 1/50 of the SA-1B dataset.The stated training-data limitation concerns the amount of SA-1B data used.

7. Conclusion

FastSAM is presented as an alternative to SAM that substantially accelerates the Segment Anything task while supporting multiple downstream tasks. The conclusion also acknowledges unresolved weaknesses in scoring and instance-mask generation.

  • Contribution: FastSAM provides a running speed 50 times faster than SAM-ViT-H (32×32).The conclusion frames this speedup as the central architectural alternative presented by the paper.
  • Results: Experiments show that FastSAM can solve multiple downstream tasks well.The conclusion does not enumerate those tasks in the supplied passage.
  • Limitations: FastSAM retains weaknesses in its scoring mechanism and instance mask-generating paradigm.The authors leave these problems for future study.
Loading 2306.12156v1…