PaperlayerView exact revision on arXiv

Source-linked AI summary

Segment Anything

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, Ross Girshick

arXiv:2304.02643v1cs.CVcs.AIcs.LG

TL;DR

A central question for image segmentation is how to achieve zero-shot generalization across new distributions and downstream tasks. The authors develop the promptable Segment Anything Model (SAM) and automatically collect the SA-1B dataset using a data engine. SAM shows consistently strong zero-shot results across several downstream tasks.

  • Problem

    The central question is which task, architecture, and data can support zero-shot segmentation across new distributions and downstream problems.

    Source: §1 · p. 1

  • Method

    The authors develop the promptable Segment Anything Model (SAM) to generate valid masks and automatically collect SA-1B through a data engine.

    Source: §1 · p. 2

  • Results

    SAM shows consistently strong zero-shot results across downstream tasks, including edge detection, object proposal generation, and instance segmentation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SAM can serve as a component in larger systems, including MCC for reconstructing unseen objects from a single RGB-D image.

    Source: §8 · p. 12

    SAM can miss fine structures and produce less crisp boundaries than more computationally intensive methods that zoom in [18].

    Source: §8 · p. 12

Abstract

from arXiv · show

We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive -- often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at https://segment-anything.com to foster research into foundation models for computer vision.

1. Introduction

Segment Anything combines a promptable segmentation task, SAM, and a model-in-the-loop data engine to pursue a foundation model that generalizes to new segmentation tasks and data distributions through prompting.

Task (§2)

  • The promptable segmentation task requires a valid mask for any segmentation prompt, including ambiguous prompts that could identify multiple objects.Spatial or text prompts specify what to segment; an ambiguous prompt must produce a reasonable mask for at least one interpretation. The task serves both pre-training and downstream segmentation through prompt engineering.Source: §1 · p. 2

Model (§3)

  • SAM combines an image encoder, prompt encoder, and lightweight mask decoder, reusing image embeddings for interactive segmentation and predicting multiple masks to handle ambiguity.Given an image embedding, SAM predicts a mask in ∼50ms in a web browser. It supports point, box, and mask prompts, with initial free-form text results.Source: §1 · p. 2

Data engine (§4)

  • The data engine co-develops SAM and annotations through assisted-manual, semi-automatic, and fully automatic stages.SAM first assists annotators, then generates masks at likely object locations while annotators label remaining objects to increase diversity. Finally, a regular grid of foreground points yields ∼100 high-quality masks per image.Source: §1 · p. 2

Responsible AI (§6)

  • SA-1B spans geographically and economically diverse countries, and SAM performs similarly across different groups of people.The authors study potential fairness concerns and biases and provide model and dataset cards in the appendix.Source: §1 · p. 2

Experiments (§7)

  • 23 segmentation datasets show SAM produces high-quality masks from a single foreground point, often only slightly below manually annotated ground truth.Zero-shot transfer also yields consistently strong quantitative and qualitative results for edge detection, object proposals, instance segmentation, and preliminary text-to-mask prediction. Room for improvement remains.Source: §1 · p. 2

Release

  • The authors release SA-1B for research purposes and SAM under the permissive Apache 2.0 license, alongside an online demonstration.Source: §1 · p. 2

Dataset (§5)

  • 1.1B high-quality masks and 11M diverse, licensed, privacy-preserving images constitute SA-1B, collected fully automatically by SAM.SA-1B has 400× more masks than existing segmentation datasets [66] [44]. Human ratings and numerous experiments verify mask quality and diversity.Source: §1 · p. 2, Figure 2 · p. 3

2. Segment Anything Task

Promptable segmentation requires a valid mask for any prompt, including ambiguous prompts, providing a basis for pre-training and zero-shot transfer. Prompt engineering and composition aim to extend one model to new segmentation tasks rather than a fixed task set.

Task

  • Promptable segmentation returns a valid mask from points, boxes, masks, text, or other prompts, even when ambiguity permits multiple objects.For an ambiguous prompt, a valid output must reasonably segment at least one of the possible objects.Source: §2 · p. 4

Pre-training

  • Pre-training simulates prompt sequences and compares predicted masks with ground truth, targeting a valid mask after every prompt rather than only after sufficient interaction.Adapted from interactive segmentation, this objective supports ambiguity-sensitive uses such as automatic annotation. It requires specialized modeling and training losses.Source: §2 · p. 4

Zero-shot transfer

  • Zero-shot transfer casts downstream segmentation tasks as prompting, such as using an existing detector’s bounding boxes to obtain instance masks.The experiments explore five example tasks in addition to automatic dataset labeling.Source: §2 · p. 4

Related tasks

  • Prompt-based task generalization [26] targets new inference-time tasks, unlike multi-task segmentation systems whose training and test tasks remain the same.Prior systems jointly perform semantic, instance, and panoptic segmentation [19] [54]. Promptable segmentation instead aims to support many, though not all, existing and new tasks through composition with other components.Source: §2 · p. 4

Discussion

  • Prompting and composition allow a segmentation model to operate within larger algorithmic systems, potentially supporting tasks unknown at model-design time.Unlike interactive segmentation’s focus on human users, promptable segmentation explicitly supports algorithmic composition. The authors anticipate broader applications than systems trained for fixed task sets.Source: §2 · p. 4

3. Segment Anything Model

SAM combines an image encoder, a flexible prompt encoder, and a fast mask decoder for promptable segmentation. It predicts multiple masks to handle ambiguous prompts.

Image encoder

  • The image encoder uses an MAE [47] pre-trained Vision Transformer (ViT) [33], minimally adapted for high-resolution inputs [62], and runs once per image.Image encoding can be completed before prompting.Source: §3 · p. 5

Prompt encoder

  • The prompt encoder supports sparse points, boxes, and text, alongside dense mask prompts.Points and boxes combine positional encodings with learned prompt-type embeddings; text uses an off-the-shelf CLIP text encoder. Masks are embedded with convolutions and summed element-wise with the image embedding.Source: §3 · p. 5

Mask decoder

  • The mask decoder combines image embeddings, prompt embeddings, and an output token through two modified Transformer blocks and a dynamic mask prediction head.Prompt self-attention and bidirectional prompt–image cross-attention update the embeddings. After image-embedding upsampling, an MLP turns the output token into a dynamic linear classifier that predicts foreground probabilities.Source: §3 · p. 5

Resolving ambiguity

  • 3 mask outputs suffice for most common ambiguous prompts, avoiding the averaging of multiple valid masks into one output.These alternatives accommodate nested whole, part, and subpart masks. Predicted confidence scores estimate each mask’s IoU for ranking.Source: §3 · p. 5

Efficiency

  • ∼50ms is the runtime of SAM’s prompt encoder and mask decoder in a web browser on CPU, given a precomputed image embedding.The three-component architecture is designed for amortized real-time performance, enabling interactive prompting.Source: §3 · p. 5

Losses and training

  • 11 rounds per mask simulate interactive prompting during training with geometric prompts, while only the minimum loss across candidate masks is backpropagated.Mask supervision combines focal loss [65] and dice loss, as used in [14]. Simulated interaction follows [37] and supports integration into the data engine.Source: §3 · p. 5

4. Segment Anything Data Engine

The data engine progressed from model-assisted manual annotation through semi-automatic collection to fully automatic mask generation. Iterative model improvements and ambiguity-aware predictions enabled the final stage to produce 1.1B masks across 11M images.

Assisted-manual stage

  • Professional annotators used foreground/background clicks and pixel-precise editing in a real-time, browser-based SAM tool powered by precomputed image embeddings.Without semantic constraints, annotators labeled both “stuff” and “things”; suggested object names or descriptions were not collected. They labeled objects by prominence and were encouraged to move to another image once a mask took over 30 seconds.Source: §4 · p. 5
  • 34 to 14 seconds per mask marked the assisted-manual annotation speedup, while average masks per image increased from 20 to 44.Across six retrainings, SAM moved from public segmentation datasets to newly annotated masks alone and scaled its encoder from ViT-B to ViT-H; this stage collected 4.3M masks from 120k images. At 14 seconds, annotation was 6.5× faster than COCO mask annotation [66] and only 2× slower than extreme-point bounding-box labeling.Source: §4 · p. 6

Semi-automatic stage

  • 5.9M additional masks from 180k images brought the semi-automatic stage’s cumulative collection to 10.2M masks, emphasizing less prominent objects.A bounding-box detector, trained on first-stage masks with a generic object category, prefilled confident masks so annotators could label remaining objects. Five retrainings accompanied this stage; manual annotation averaged 34 seconds per mask excluding automatic masks, while masks per image increased from 44 to 72 including automatic masks.Source: §4 · p. 6

Fully automatic stage

  • 1.1B high-quality masks were generated fully automatically across all 11M dataset images.This stage became feasible through improvements from accumulated annotations, including diverse semi-automatic masks, and an ambiguity-aware model that predicts valid masks in ambiguous cases.Source: §4 · p. 6
  • A 32×32 point grid prompted multiple masks per point, allowing predictions to represent subparts, parts, and whole objects.Predicted IoU selected confident masks, stability required similar masks at probability thresholds 0.5 − δ and 0.5 + δ, and non-maximal suppression removed duplicates. Overlapping zoomed-in crops improved smaller-mask quality.Source: §4 · p. 6

5. Segment Anything Dataset

SA-1B contains 11M licensed, privacy-protecting images and 1.1B segmentation masks. Its predominantly automatic masks closely match professional corrections, while its scale substantially exceeds existing segmentation datasets.

Images

  • 1500 pixels is the shortest-side resolution of released images, with faces and vehicle license plates blurred.Images were licensed from a provider working directly with photographers and downsampled to ease storage and accessibility challenges. SA-1B is being released under a favorable license for certain research uses, with protections for researchers.Source: §5 · p. 6

Masks

  • 99.1% of the data engine’s masks were generated fully automatically, and SA-1B includes only automatically generated masks.Section 7 reports high mask quality through human ratings relative to several datasets. Training on automatic masks is nearly as good as using all masks produced by the data engine.Source: §5 · p. 6

Mask quality

  • 94% of automatically predicted and professionally corrected mask pairs exceed 90% IoU, and 97% exceed 75% IoU.Professional annotators corrected approximately 50k masks from 500 randomly sampled images. Prior work estimates inter-annotator consistency at 85-91% IoU [44].Source: §5 · p. 6

Dataset scale

  • 11M images and 1.1B masks make SA-1B larger than Open Images by 11× in images and 400× in masks.The dataset was collected using the authors’ data engine and is being released to support foundation-model research in computer vision.Source: §5 · pp. 6–7, Figure 6 · p. 7

Mask properties

  • 36× more masks per image distinguish SA-1B from Open Images; even ADE20K, the closest dataset on this measure, has 3.5× fewer.SA-1B also contains a greater percentage of small and medium image-relative masks.Source: §5 · p. 7, Figure 6 · p. 7
  • SA-1B covers image corners better than LVIS v1 [44] and ADE20K, while COCO [66] and Open Images V5 show stronger center bias.Photographer biases remain present in all datasets. After controlling for mask-size distributions, SA-1B’s mask-concavity distribution is broadly similar to those of other datasets.Source: §5 · p. 7

6. Segment Anything RAI Analysis

SA-1B has uneven geographic and income representation, while SAM shows similar people-segmentation performance across perceived gender groups and no significant skin-tone differences. Clothing segmentation nevertheless shows indications of perceived-gender bias.

Geographic and income representation

  • SA-1B represents Europe, Asia & Oceania, and middle-income countries more than COCO [66] and Open Images; all underrepresent Africa, Latin America & Caribbean, and low-income countries.Countries where images were photographed are inferred using standard methods.Source: Table 1 · p. 7, §6 · p. 7
  • 28 million masks or more cover every SA-1B region, including Africa—10× the total number of masks in any previous dataset.Average mask counts remain fairly consistent across region and income, ranging from 94–108 per image.Source: §6 · p. 7

Fairness in segmenting people

  • SAM performs similarly across perceived gender groups, shows no significant skin-tone differences, and performs best on people perceived as older.Within each attribute grouping, 95% confidence intervals overlap except for older versus middle-aged people; the older group’s confidence interval is large.Source: Table 2 · p. 8, §6 · p. 8
  • Clothing segmentation shows indications of perceived-gender bias, and biases may arise when SAM is used within larger systems.People-segmentation tests use one or three randomly sampled interaction points, with MIAP for perceived gender and age and a proprietary dataset for perceived skin tone.Source: §6 · p. 8

7. Zero-Shot Transfer Experiments

SAM demonstrates zero-shot transfer across single-point segmentation, edge detection, object proposals, and instance segmentation, while a text-aware variant enables text-to-mask segmentation. Human ratings sometimes favor SAM despite lower automatic scores, and ablations show comparable performance with simpler training or substantially less data.

7.1. Zero-Shot Single Point Valid Mask Evaluation

  • 16 of 23 datasets favor SAM over RITM [92] in single-point mIoU, with gains reaching ∼47 IoU.A single point can identify multiple objects, making metrics unreliable when ground truth omits valid alternatives. Selecting the ground-truth-best of SAM’s three masks instead of its most confident prediction outperforms RITM on every dataset.Source: §7.1 · pp. 8–9, Figure 9 · p. 9
  • 7 to 9 mean human mask-quality ratings favor SAM over RITM [92], even on datasets where SAM has worse automatic scores.An ambiguity-unaware, single-output SAM receives lower ratings than standard SAM but still exceeds RITM. Baseline gaps narrow as prompts increase from 1 to 9 points, while random-point sampling widens SAM’s advantage with performance comparable to center sampling.Source: Figure 9 · p. 9, §7.1 · p. 9

7.2. Zero-Shot Edge Detection

  • SAM achieves high recall at 50% precision (R50), surpassing older zero-shot edge detectors but trailing BSDS500-trained state-of-the-art methods.Grid-prompted masks become edge maps through Sobel filtering and nonmaximum suppression. SAM performs well relative to HED, also trained on BSDS500, but predicts additional sensible, unannotated edges at the cost of precision.Source: Table 3 · p. 10, §7.2 · p. 10

7.3. Zero-Shot Object Proposals

  • SAM outperforms ViTDet-H on medium, large, rare, and common objects in LVIS proposal generation, although ViTDet-H using DMP [16] performs best overall.SAM outputs automatically generated masks as proposals and underperforms ViTDet-H only on small and frequent objects. Its ambiguity-unaware variant performs significantly worse on every average recall (AR) metric.Source: Table 4 · p. 10, §7.3 · p. 10

7.4. Zero-Shot Instance Segmentation

  • SAM trails fully supervised ViTDet in COCO and LVIS mask AP, but produces crisper boundaries and consistently higher human quality ratings.Benchmark segmentation uses ViTDet box prompts, whereas the human comparison uses LVIS ground-truth boxes. The AP gap shrinks on higher-quality LVIS masks; the authors hypothesize that ViTDet exploits annotation biases unavailable to zero-shot SAM.Source: §7.4 · pp. 10–11, Table 5 · p. 11, Figure 11 · p. 11

7.5. Zero-Shot Text-to-Mask

  • A text-aware SAM variant segments objects from simple and nuanced text prompts, with additional point prompts often correcting failures, similar to.This proof-of-concept trains with CLIP image embeddings extracted for manually collected masks, requiring no new text annotations. Aligned CLIP text embeddings provide prompts at inference.Source: Figure 12 · p. 11, §7.5 · p. 11

7.6. Ablations

  • ∼0.5 mIoU separates automatic-mask-only training from all-stage training, favoring the latter while motivating the simpler automatic-only default.Each successive data-engine stage improves mIoU. Training on all stages works best when manual and semi-automatic masks are oversampled by 10×.Source: §7.6 · p. 11, Figure 13 · p. 12
  • 1M training images, containing approximately 100M masks, yield performance comparable to all 11M images, whereas 0.1M images cause a large mIoU decline.Encoder scaling also saturates: ViT-H improves substantially over ViT-B but only marginally over ViT-L. The default nevertheless uses all 11M images.Source: §7.6 · pp. 11–12, Figure 13 · p. 12

8. Discussion · Appendix

The Segment Anything project contributes promptable segmentation, SAM, and SA-1B toward foundation models for image segmentation, with a prompt interface designed for composition. The discussion emphasizes supervised scaling while acknowledging mask-quality, runtime, and task limitations.

8. Discussion

  • SAM derives most capabilities from large-scale supervised training, despite self-supervised MAE [47] initialization.The work aligns with the foundation-model definition [8], although segmentation covers only part of computer vision. Data engines that scale annotations make supervised training effective.Source: §8 · p. 12
  • SAM's valid-mask interface across diverse prompts is designed to make it a reusable component in larger systems.MCC can use SAM for single-RGB-D-image 3D reconstruction with strong generalization to unseen objects. Wearable gaze prompts enable applications in new domains such as egocentric imagery without additional training.Source: §8 · p. 12
  • SAM can miss fine structures, hallucinate disconnected components, and produce less-crisp boundaries than computationally intensive zoom-in methods [18].The authors expect dedicated interactive methods [67] to outperform SAM with many points and domain-specific tools to outperform it in their domains. SAM prioritizes generality over high-IoU interactive segmentation.Source: §8 · p. 12
  • SAM processes prompts in real time, but heavy image encoding prevents real-time overall performance; text-to-mask remains exploratory and not entirely robust.It is also unclear how to design simple prompts for semantic and panoptic segmentation.Source: §8 · p. 12
  • Over 1B released masks accompany the project's principal contributions: promptable segmentation, SAM, and the SA-1B dataset.Whether SAM achieves foundation-model status remains dependent on how the community uses it.Source: §8 · p. 12

Appendix

  • The appendix lists model and task details, automatic mask generation, responsible-AI details, experiment implementation, human-study design, dataset/annotation/model cards, and annotation guidelines.These materials are organized into appendices A–G.Source: Appendix · p. 16

A. Segment Anything Model and Task Details · B. Automatic Mask Generation Details

SAM combines a once-per-image image encoder, prompt encoders, and a lightweight, ambiguity-aware mask decoder trained through simulated interaction. Automatic SA-1B generation uses a specialized SAM variant with multiscale prompting, quality filtering, and mask cleanup.

A. Segment Anything Model and Task Details

  • SAM computes image embeddings once per image using an MAE [47]-pretrained ViT-H/16, allowing subsequent prompts to reuse the expensive encoder output.The encoder uses windowed attention and four global-attention blocks. Rescaling and padding inputs to 1024×1024 produces a 64×64 embedding, projected to 256 channels with convolutions and layer normalization.Source: A. Segment Anything Model and Task… · p. 16
  • SAM represents sparse prompts as embeddings and adds spatially encoded mask prompts directly to image embeddings.Points combine positional encodings with foreground/background embeddings, boxes encode their corners, and text uses CLIP; sparse embeddings are 256-dimensional. Mask prompts are convolutionally downscaled to match image embeddings, while absent masks are represented by a learned “no mask” embedding.Source: A. Segment Anything Model and Task… · p. 16
  • A lightweight two-layer decoder updates prompt tokens and image embeddings through bidirectional cross-attention, then dynamically predicts masks from upscaled image features.A learned output token joins the prompt tokens; positional encodings and original prompt tokens are repeatedly reintroduced to preserve geometry. After 4× upsampling and another attention step, a three-layer MLP transforms the output token into weights for spatially point-wise mask prediction.Source: Figure 14 · p. 16, A. Segment Anything Model and Task… · pp. 16–17
  • Three masks handle single-prompt ambiguity, while multiple prompts return only a separate fourth mask.For ambiguous prompts, training backpropagates only the lowest mask loss, and an additional head predicts IoU to rank outputs. The separate multiple-prompt mask avoids degenerate losses and receives a regular gradient signal.Source: A. Segment Anything Model and Task… · p. 17
  • 20:1 focal-to-dice loss weighting supervises masks, while mean-square-error loss trains predicted IoU against actual mask IoU.The IoU loss is added with scaling factor 1.0. Auxiliary deep supervision after each decoder layer was unhelpful.Source: A. Segment Anything Model and Task… · p. 17
  • 11 training iterations simulate interaction: an initial point or box, eight error-driven correction points, and two mask-only refinements.Initial points and boxes are equally likely; boxes receive coordinate noise, and later iterations reuse unthresholded mask logits, selecting the highest-predicted-IoU mask when necessary. Each decoder iteration costs less than 1% of encoder compute; default training uses AdamW for 90k iterations with batch size 256 and no data augmentation.Source: A. Segment Anything Model and Task… · p. 17

B. Automatic Mask Generation Details

  • Automatic mask generation combines multiscale point grids and crop-aware suppression with a specialized SAM variant that trades inference speed for mask-generation properties.A 32×32 full-image grid and 20 overlapping crops with 16×16 or 8×8 grids generate candidates; crop-boundary masks are removed, and box NMS uses threshold 0.7, ranking by predicted IoU within crops and favoring zoomed-in crops across crops. The specialized model uses only manual and semi-automatic data, 177656 rather than 90k training iterations, large-scale jitter [40], three rather than two decoder layers, and four rather than nine training points without boxes; fewer points preserve one-point performance but harm evaluation with more points.Source: B. Automatic Mask Generation Details · p. 18
  • Automatic candidates undergo confidence, stability, and image-coverage filtering, followed by removal of small components and filling of small holes.Filters use predicted IoU threshold 88.0, stability IoU at least 95.0 between masks thresholded at -1 and +1, and rejection of masks covering at least 95% of an image. Spurious components and holes each affect an estimated 4% of masks; components below 100 pixels are removed and holes below 100 pixels are filled.Source: B. Automatic Mask Generation Details · p. 18

C. RAI Additional Details

The additional responsible-AI analysis estimates dataset geography and income and evaluates segmentation across perceived gender presentation and age. Clothing segmentation shows a gender-presentation disparity that narrows with more prompt points, while perceived-age differences are not significant.

Inferring geographic information for SA-1B

  • SA-1B’s approximate locations are inferred from captions using an Elmo-based named entity recognition model; ambiguity and potential bias restrict their use to dataset-level analysis.Location entities are matched to countries, provinces, and cities, prioritizing these levels in that order; inferred countries determine World Bank income levels, combining upper-middle and lower-middle categories. Inferred locations are withheld, and the image provider prohibits public caption release.Source: C. RAI Additional Details · p. 18

Inferring geographic information for COCO and Open Images

  • 24% of COCO [66] training images (19,562) and 18% of Open Images masked training images (493,517) received approximate locations through Flickr metadata, following.These datasets do not provide geolocations, and the retrieved samples may not represent their full dataset distributions.Source: C. RAI Additional Details · p. 18

Fairness in segmenting clothing

  • 95% confidence intervals are disjoint for clothing segmentation across perceived gender presentation, favoring predominantly masculine presentation; perceived-age differences are not significant.The analysis uses 6.5k Open Images clothing masks within MIAP person boxes. The gender-presentation gap narrows when moving from 1 to 3 prompt points.Source: Table 6 · p. 19, C. RAI Additional Details · p. 19

Fairness in segmenting people

  • 3.9k person masks support perceived-gender and perceived-age fairness analysis, matching Open Images mask bounding boxes to MIAP boxes within a 1% relative side-length margin.MIAP provides the demographic annotations and bounding boxes, while Open Images supplies the ground-truth masks.Source: C. RAI Additional Details · p. 19

D. Experiment Implementation Details … D.6. Probing the Latent Space of SAM

These experiments adapt SAM to point-prompted segmentation, edge detection, object proposals, instance segmentation, and text-to-mask prediction, and probe its latent representations. Qualitative findings include cleaner instance-mask boundaries than ViTDet and imperfect shape and semantic similarity without explicit semantic supervision.

D.1. Zero-Shot Single Point Valid Mask Evaluation

  • Point prompts begin farthest from the object boundary, or randomly in harder experiments, then target prediction-error regions.Each subsequent point is farthest from the error-region boundary. Random initial points model cases where prompts are not reliably near mask centers, such as eye-gaze input.Source: §D.1 · p. 19
  • 23 diverse segmentation datasets form the zero-shot point-prompt benchmark, with top-line performance averaging per-dataset mIoUs rather than pooling masks.Evaluation measures IoU after N = {1, 2, 3, 5, 9} prompts, emphasizing few-prompt performance rather than clicks needed to reach an IoU threshold. Datasets exceeding 15k masks are subsampled to approximately 10k masks, and people’s faces are blurred.Source: §D.1 · p. 19, Table 7 · p. 20
  • RITM is the default single-point baseline, outperforming FocalClick [18] and SimpleClick [67] on the evaluation suite.The baselines use their largest publicly released models trained on their broadest datasets, with default preprocessing and unchanged parameters; all are reported for multiple points. Oracle single-point evaluation selects the ground-truth-best mask among SAM’s three predictions instead of its top-ranked mask, accounting for prompt ambiguity.Source: §D.1 · p. 19

D.2. Zero-Shot Edge Detection

  • SAM’s zero-shot edge maps often align with human annotations but include extra edges, reflecting its lack of edge-detection training.A 16×16 foreground-point grid generates 768 masks; after duplicate suppression, Sobel filtering, boundary restriction, pixel-wise maximization, normalization, and edge NMS produce the edge map. Evaluation uses the 200-image BSDS500 test subset and ODS, OIS, AP, and R50 metrics.Source: Figure 15 · p. 21, §D.2 · p. 21

D.3. Zero-Shot Object Proposals

  • 64×64 point grids with a 0.9 NMS threshold yield approximately 900 proposals per image, without image crops or predicted-IoU/stability filtering.Evaluation uses LVIS v1 [44] mask AR@1000, including frequency-category subsets, against cascade ViTDet-H [62]; excess proposals are ranked by average confidence and stability and truncated to 1000. To test the value of multiple outputs, single-output SAM instead uses a 128×128 grid, 0.95 NMS threshold, and random ranking, yielding approximately 950 masks per image.Source: §D.3 · p. 21

D.4. Zero-Shot Instance Segmentation

  • SAM tends to produce higher-quality instance masks with cleaner boundaries than ViTDet, although its predictions can differ from dataset annotation conventions.SAM receives boxes from fully supervised ViTDet-H and refines its most confident mask by feeding it back with the box prompt. Its valid modal predictions can disagree with LVIS annotations, which prohibit holes in masks.Source: Figure 16 · p. 22, §D.4 · p. 22

D.6. Probing the Latent Space of SAM

  • SAM’s nearest-neighbor mask embeddings show imperfect shape and semantic similarity despite lacking explicit semantic supervision.Embeddings average mask-weighted image features from crops and their horizontally flipped versions. These preliminary observations suggest possible uses in data labeling, dataset understanding, and downstream features.Source: Figure 17 · p. 22, §D.6 · pp. 22–23

D.5. Zero-Shot Text-to-Mask

  • Aligned CLIP image and text embeddings let SAM train on image-derived prompts and accept text prompts without explicit text supervision.Training uses normalized ViT-L/14@336px embeddings, masks from the data engine’s first two stages, and 120k iterations with batch size 128. Image prompts use expanded object crops, optional background zeroing, and final-layer masked attention, followed by iterative point prompts; inference uses the unchanged CLIP text encoder.Source: §D.5 · p. 22

E. Human Study Experimental Design · F. Dataset, Annotation, and Model Cards

Human evaluation found that SAM received significantly higher mask-quality ratings than baseline models and single-output SAM, addressing limitations of ground-truth-based metrics. The paper also provides dataset, annotation, and model cards.

E. Human Study Experimental Design

  • Human review avoids IoU penalties for valid alternative objects and ground-truth annotation biases, while testing whether SAM’s disambiguation improves single-point masks.Single-output SAM may average ambiguous masks, while in-domain models can improve IoU by learning annotation biases rather than producing better masks. Instance-segmentation comparisons test whether SAM produces visually better masks despite lower AP.Source: E. Human Study Experimental Design · p. 23
  • 7 of 23 datasets were selected for single-point review, covering diverse imagery and cases where either SAM or RITM wins on IoU.Coverage includes scene-level, ego-centric, drawn, overhead, underwater, and synthetic imagery. Instance segmentation uses LVIS v1 validation, matching the baseline’s training dataset.Source: E. Human Study Experimental Design · p. 23
  • 1-10 mask-quality ratings assessed object validity, boundary cleanliness, and agreement with the input, supported by five diagnostic views.Scores ranged from no corresponding object (1) to no visible errors (10). Jobs were randomized among 30 annotators and rated once after a five-annotator pilot yielded an average score standard deviation of 0.83; quality-assurance testers spot-checked guideline adherence.Source: E. Human Study Experimental Design · p. 23
  • 1000 randomly selected masks per dataset supported single-point evaluation, while 1000 LVIS v1 validation masks supported instance-segmentation evaluation.Models received identical inputs: centermost points defined by maximal distance from mask edges, or ground-truth boxes; dataset ground-truth masks were also rated. Small masks were excluded, and large images were rescaled to a maximum side length of 2000 before prediction.Source: E. Human Study Experimental Design · p. 23
  • SAM received significantly higher human mask-quality ratings than RITM for single-point segmentation, ViTDet [62] for instance segmentation, and single-output SAM.Paired t-tests and 10k-sample paired bootstraps produced strongly significant results and 99% confidence intervals excluding zero. Instance-segmentation tests used all 1000 ratings; comparisons with COCO ground truth matched 794 inputs per model and ground truth.Source: E. Human Study Experimental Design · pp. 23–24, Table 8 · p. 24

F. Dataset, Annotation, and Model Cards

  • Documentation includes an SA-1B Dataset Card following, an annotation card following CrowdWork-Sheets, and a Model Card following.The annotation card covers the first two data-engine stages; the dataset and annotation cards use question-and-answer formats, and Table 9 contains the Model Card.Source: F. Dataset, Annotation, and Model Cards · p. 25

F.1. Dataset Card for SA-1B

SA-1B provides 11M licensed images and 1.1B segmentation masks with privacy blurring, while retaining annotation errors, representation imbalances, and potentially sensitive content.

Dataset purpose

  • 11M images and 1.1B masks make SA-1B the largest segmentation dataset to date, with greater geographic diversity than its predecessors.The images are licensed, and faces and license plates are blurred for privacy.Source: §F.1 · p. 25

Composition

  • ∼100 masks per image have no associated categories or text, and model-generated masks may contain errors or inconsistencies.The distinct photos cover locations, objects, and scenes, although some depict the same subject close together in time. Provider captions describing content and location supported geographic analysis but cannot be released under the agreement.Source: §F.1 · p. 25
  • Potentially sensitive protest and gathering scenes remain despite requested objectionable-content filtering, even though faces are blurred.These scenes may suggest religious beliefs, political opinions, or union memberships. Users can report objectionable images for removal.Source: §F.1 · p. 25

Collection Process

  • Released masks are automatically inferred by SAM; masks collected through model-assisted manual annotation are not released.All licensed images and their generated masks are included except ∼2k randomly selected images withheld for testing. Mask quality was validated as described in §5.Source: §F.1 · p. 25

Preprocessing / Cleaning / Labeling

  • 1500 pixels is the resized shorter side of images; unaltered photos are not released for safety and privacy.Faces are detected using RetinaFace; the license-plate blurring model is not public.Source: §F.1 · p. 26

Uses

  • SA-1B is intended for large-scale segmentation, but geographic and income coverage lacks parity across groups, requiring attention to learned biases.The dataset was used to train SAM, and researchers are encouraged to gather additional annotations.Source: §F.1 · p. 26

Distribution

  • Users must accept the dataset license and terms of use before downloading or using SA-1B; the terms specify prohibited uses.The dataset license and restrictions are available through the Meta AI dataset site.Source: §F.1 · p. 26

Maintenance

  • SA-1B has no data-retention limits; updates only remove reported images, and older versions containing potentially harmful content are not retained.Meta AI maintains the dataset, and users can report content for potential removal. Contributors must host and distribute any additional annotations themselves.Source: §F.1 · pp. 26–27

F.2. Data Annotation Card · G. Annotation Guidelines

The annotation process treated segmentation as subjective, prioritizing data diversity over completeness while using interactive tools, training, and recurring quality feedback. Human-review guidelines distinguished mask errors from pixelation that reflected the underlying image.

F.2. Data Annotation Card

  • Segmentation was treated as inherently subjective, with per-mask annotation prioritizing data diversity rather than completeness.Annotators could disagree about grouping objects, and their skills affected both mask quality and the number of masks per image.Source: §F.2 · p. 27
  • 30 annotation tasks completed by the research team helped identify tool challenges, resolve complex cases, and refine visually illustrated instructions.Weekly feedback sessions, live task demonstrations, Q&A, and asynchronous feedback let annotators raise unclear aspects of the instructions.Source: §F.2 · p. 27
  • Annotators were instructed to mask every identifiable potential object using corrective foreground/background clicks or bounding boxes, with pixel-precise tools available for refinement.This broad annotation objective coexisted with the practical emphasis on diversity rather than exhaustive coverage.Source: §F.2 · p. 27
  • 130 annotators, all based in Kenya, were selected for prior vision-annotation experience rather than sociodemographic characteristics.The authors did not believe annotator sociodemographics meaningfully affected the annotated data.Source: §F.2 · p. 27
  • One week of training on average preceded production, with manual mask spot-checks determining readiness and weekly reviews assessing production quality.Training began with a vendor-led one-day session and practice annotations; full-time staffing and very low attrition supported continuing instruction. Daily QA feedback, weekly research feedback, and spreadsheet/chat communication reportedly improved annotation speed and quality.Source: §F.2 · p. 27
  • Manual annotations trained early SAM versions and were not planned for release; annotators had no process to withdraw their data.Access to SA-1B required agreement to license terms permitting certain research uses. Dataset updates were not planned except to remove objectionable images.Source: §F.2 · p. 27

G. Annotation Guidelines

  • Review guidelines flagged unrelated-object combinations, point–mask disagreement, and inconsistent treatment of foreground occluders, while accepting pixelation that matched the underlying image.Foreground occluders should be consistently included or excluded. A small mask could still have an imperfect boundary extending beyond the object even when its blocky appearance was acceptable.Source: G. Annotation Guidelines · p. 30

Related papers

Loading 2304.02643v1…