PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM 3: Segment Anything with Concepts

Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, Christoph Feichtenhofer

arXiv:2511.16719v2cs.CVcs.AI

TL;DR

Promptable concept segmentation lacks a unified, high-performing approach for detecting, segmenting, and tracking concept-matched objects across images and videos. SAM 3 combines concept-conditioned detection and video tracking with scalable data generation, doubling prior PCS performance and improving visual segmentation capabilities.

  • Problem

    Promptable segmentation needs stronger, unified support for concept-matched objects across images and videos.

    Source: §1 · p. 2

  • Method

    SAM 3 combines concept-conditioned detection, memory-based video tracking, and a human- and model-in-the-loop data engine using diverse labeled media.

    Source: §1 · p. 2

  • Results

    SAM 3 doubles prior PCS performance across images and videos, reaching 48.8 mask AP on LVIS versus 38.5 for the previous best.

    Source: §1 · p. 2, §8 · p. 11

  • Takeaways & Limitations

    SAM 3 establishes a new state of the art for PCS while retaining visual segmentation capabilities and provides the SA-Co benchmark and released model resources.

    Source: §1 · p. 2, §8 · p. 11

    PCS remains intrinsically ambiguous because prompts can have multiple interpretations, subjective or vague meanings, and unclear object boundaries.

    Source: §2 · p. 3

Abstract

from arXiv · show

We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts, which we define as either short noun phrases (e.g., "yellow school bus"), image exemplars, or a combination of both. Promptable Concept Segmentation (PCS) takes such prompts and returns segmentation masks and unique identities for all matching object instances. To advance PCS, we build a scalable data engine that produces a high-quality dataset with 4M unique concept labels, including hard negatives, across images and videos. Our model consists of an image-level detector and a memory-based video tracker that share a single backbone. Recognition and localization are decoupled with a presence head, which boosts detection accuracy. SAM 3 doubles the accuracy of existing systems in both image and video PCS, and improves previous SAM capabilities on visual segmentation tasks. We open source SAM 3 along with our new Segment Anything with Concepts (SA-Co) benchmark for promptable concept segmentation.

1 Introduction

SAM 3 introduces Promptable Concept Segmentation, finding, segmenting, and tracking every instance matching text or image-based concepts, while improving prior promptable visual segmentation. Its detector-tracker architecture and human/model-in-the-loop data engine support state-of-the-art results on broad PCS benchmarks.

Task

  • SAM 3 formalizes PCS to predict masks for every object matching a concept from noun phrases, image exemplars, or both, while preserving identities across video frames.Text prompts are restricted to atomic noun phrases, and interactive refinement resolves ambiguities; complex language can be handled by combining SAM 3 with an MLLM.Source: Figure 1 · p. 1, §1 · p. 2

Model

  • SAM 3 combines a text-, geometry-, and exemplar-conditioned detector with a SAM 2-based tracker sharing one vision encoder.A separate presence head decouples recognition from localization, while detector-tracker separation avoids conflicts between identity-agnostic detection and identity-aware tracking.Source: §1 · p. 2

Data engine

  • The data engine produces 4M unique phrases and 52M masks, plus synthetic data with 38M phrases and 1.4B masks, using curated media, ontology-driven hard negatives, and human/AI verification.Human annotators correct difficult mask and exhaustivity errors identified from noisy pseudo-labels, while fine-tuned MLLMs provide near-human verification at higher throughput.Source: §1 · p. 2

Benchmark

  • The SA-Co benchmark contains 207K unique concepts with exhaustive masks across 120K images and 1.7K videos, exceeding existing benchmarks by more than 50× in concept coverage.The benchmark is designed for promptable concept segmentation.Source: §1 · p. 2

Results

  • 48.8 mask AP on LVIS versus 38.5 for the current best, at least 2× higher performance than baselines on SA-Co, and improved visual-prompt segmentation over SAM 2 establish SAM 3’s state of the art.Figure 2 illustrates improved open-vocabulary concept segmentation over OWLv2 on SA-Co.Source: §1 · p. 2, Figure 2 · p. 2
  • SAM 3 is open-sourced with SA-Co, checkpoints, and inference code, running at 30 ms per image with 100+ detected objects and near real-time video inference for approximately 5 concurrent objects.Ablations attribute gains to the backbone, presence head, and hard negatives, while scaling laws hold for both high-quality and synthetic datasets.Source: §1 · p. 2

2 Promptable Concept Segmentation (PCS)

Promptable Concept Segmentation detects, segments, and tracks every instance of visually grounded simple noun-phrase concepts in images or short videos using text, image exemplars, or both. Because open-vocabulary concepts are intrinsically ambiguous, the task uses consistent prompts and annotation, evaluation, and modeling strategies to accommodate multiple valid interpretations.

Task definition

  • PCS detects, segments, and tracks all instances of a visual concept in an image or video up to 30 seconds.Concepts are specified by short noun phrases, image exemplars, or their combination.Source: §2 · p. 3

Prompting

  • Noun-phrase prompts apply globally across frames, whereas positive or negative exemplar boxes can iteratively refine masks on individual frames.Exemplars are especially useful for missed instances or rare concepts.Source: §2 · p. 3
  • All prompts must preserve the same category definition; exemplars cannot redefine a concept, such as refining “fish” with a prompt referring only to its tail.The text prompt must instead be updated when the intended category changes.Source: §2 · p. 3

Ambiguity and evaluation

  • The unrestricted simple-noun-phrase vocabulary creates ambiguity from polysemy, subjective or vague descriptions, boundary uncertainty, occlusion, and blur.The authors address this through expert annotations, evaluation allowing multiple interpretations, clearer annotation guidelines, and an ambiguity module.Source: §2 · p. 3

3 Model

SAM 3 generalizes SAM 2 to promptable concept and visual segmentation across images and videos, combining a detector with a memory-based tracker over a shared Perception Encoder backbone. Its design adds global presence recognition, interactive exemplars and refinement, and temporally disambiguated tracking.

Model Overview

  • SAM 3 accepts noun-phrase concepts, image exemplars, and visual prompts to segment objects individually across space and time, supporting both PCS and PVS.Image exemplars and visual prompts can be added iteratively to remove false positives, add false negatives, or refine individual masklets.Source: §3 · p. 3

Detector Architecture

  • A shared Perception Encoder feeds a DETR-like image detector and a memory-based video tracker, while the detector uses prompt-conditioned fusion and query-based decoding.The detector combines image and prompt tokens through cross-attention before learned object queries predict prompt-matching objects, boxes, and masks.Source: §3 · pp. 3–4

Presence Token

  • A learned global presence token decouples recognition from localization by predicting concept presence, while each proposal query estimates whether it matches the present concept.The final proposal score is the product of the proposal score and the presence score.Source: §3 · p. 4

Tracker and Video Architecture

  • For videos, the detector finds new objects while the tracker propagates prior masklets, matches them to detections, spawns unmatched objects, and periodically refreshes memory with reliable detections.Temporal consistency scores suppress unreliable masklets, and three candidate masks with confidence scores address per-frame ambiguity.Source: §3 · pp. 4–5

Instance Refinement with Visual Prompts

  • SAM 3 refines individual masks or masklets with positive and negative clicks, propagates refined video masks across the sequence, and trains through four progressively expanding stages.The stages are Perception Encoder pre-training, detector pre-training, detector fine-tuning, and tracker training with a frozen backbone.Source: §3 · p. 5

4 Data Engine

SAM 3’s data engine iteratively combines AI models, human annotators, and model feedback to mine difficult media–concept pairs and expand coverage from images to videos. Across four phases, it more than doubles annotation throughput while building large-scale SA-Co/HQ and SA-Co/VIDEO datasets.

Overview

  • The feedback-loop data engine more than doubles annotation throughput by using AI annotators to steer human effort toward challenging failures while expanding visual-domain coverage.The four phases progressively increase AI involvement; Phases 1–3 process images, while Phase 4 adds videos.Source: §4 · p. 5

Data Engine Components

  • Candidate masks undergo Mask Verification and Exhaustivity Verification, with manual correction for incomplete annotations and rejection of ambiguous or ungroundable phrases.The pipeline mines media using a curated ontology, proposes noun phrases, generates masks, and verifies both mask quality and instance completeness.Source: §4 · p. 6

Phase 1: Human Verification

  • Phase 1 collects 4.3M image-NP pairs using human verification, with SAM 2 and an open-vocabulary detector providing initial mask proposals before SAM 3 takes over.The resulting SA-Co/HQ data trains SAM 3 for subsequent data-engine phases.Source: §4 · p. 6

Phase 2: Human + AI Verification

  • Phase 2 adds 122M image-NP pairs and roughly doubles throughput by fine-tuning Llama 3.2 AI verifiers for mask quality and exhaustivity, while mining hard-negative phrases.SAM 3 is retrained six times, and increasing auto-generation progressively focuses human annotation on difficult cases.Source: §4 · p. 6

Phase 3: Scaling and Domain Expansion

  • Phase 3 adds 19.5M image-NP pairs while expanding SA-Co/HQ to 15 datasets and long-tail concepts through alt-text mining and a 22.4M-node Wikidata-based ontology.The phase iterates SAM 3 training seven times and AI verifiers three times; exhaustivity verification requires modest domain-specific human supervision in new domains.Source: §4 · p. 6

Phase 4: Video Annotation

  • Phase 4 produces SA-Co/VIDEO with 52.5K videos and 467K masklets by targeting crowded scenes, tracking failures, and other video-specific annotation challenges.The pipeline samples frames, applies scene and motion filters, sends frames through the image annotation flow, and deduplicates or removes trivial masklets.Source: §4 · p. 7

5 Segment Anything with Concepts (SA-Co) Dataset

SA-Co provides large-scale image and video training data for promptable concept segmentation, together with a multi-split benchmark for open-vocabulary recognition. Its evaluation emphasizes calibrated classification and localization, combining them through classification-gated F1 while accounting for annotation ambiguity.

Training Data

  • SA-Co/HQ annotates 5.2M images and 4M unique noun phrases, while SA-Co/VIDEO contains 52.5K videos, 24.8K unique noun phrases, and 134K video–noun-phrase pairs.Training data also includes synthetic SA-Co/SYN and 15 external datasets in SA-Co/EXT enriched with hard negatives.Source: §5 · p. 7

SA-Co Benchmark

  • The SA-Co benchmark spans 207K unique phrases, 121K images and videos, and over 3M media–phrase pairs across Gold, Silver, Bronze, and Bio splits, with hard-negative labels.SA-Co/Gold supports human-performance measurement through three annotations per pair, while SA-Co/VEval covers video–noun-phrase pairs across three domains.Source: §5 · p. 7

Metrics

  • Predictions are evaluated only above confidence 0.5, enforcing calibration for downstream use while measuring localization with positive micro F1 and classification with image-level Matthews Correlation Coefficient.The classification metric evaluates object presence without regard to mask quality.Source: §5 · p. 7
  • Classification-gated F1 combines localization and classification as cgF1 = 100 ∗pmF1 ∗IL_MCC.This is the main metric for the PCS evaluation.Source: §5 · p. 7

Handling Ambiguity

  • For ambiguous noun phrases in SA-Co/Gold, oracle accuracy compares each prediction with all three ground truths and selects the best score.This protocol addresses multiple valid annotations per noun phrase.Source: §5 · p. 7

6 Experiments

SAM 3 achieves strong results across image and video promptable concept segmentation, few-shot transfer, counting, visual prompting, and complex-language segmentation. Ablations further show gains from the presence head, hard negatives, high-quality and synthetic data, and AI verification, including scalable adaptation to new domains.

Image PCS with Text

  • SAM 3 sets state-of-the-art results on closed-vocabulary image benchmarks and more than doubles the strongest baseline’s cgF1 on open-vocabulary SA-Co/Gold.It also reaches 74% of estimated human performance, improves further on other SA-Co splits, and outperforms APE on open-vocabulary semantic segmentation.Source: §6 · p. 7, Table 1 · p. 8

Few-Shot Adaptation

  • SAM 3 achieves state-of-the-art 10-shot transfer on ODinW13 and RF100-VL, surpassing Gemini in-context prompting and object-detection experts.On RF100-VL, fine-tuning adapts efficiently despite specialized prompts outside SAM 3’s current scope.Source: COCO LVIS ODinW13 · p. 8, Table 2 · p. 8

PCS with 1 Exemplar

  • SAM 3 outperforms T-Rex2 by +18.3 AP+ on COCO, +10.3 on LVIS, and +20.5 on ODinW with one exemplar.The evaluation compares text-only, image-only, and combined text-and-image prompting settings.Source: COCO LVIS ODinW13 · p. 8, Table 3 · p. 8

PCS with K Exemplars

  • After 3 interactive exemplar clicks, SAM 3 improves over text-only prompting by +21.6 cgF1 and over PVS refinement by +2.0 points.Performance plateaus after 4 clicks because exemplars cannot repair poor-quality masks, while switching to PVS then provides complementary gains.Source: COCO LVIS ODinW13 · p. 8, Figure 7 · p. 8

Video PCS with Text

  • SAM 3 largely outperforms video PCS baselines, especially on benchmarks with many noun phrases, and exceeds 80% of human pHOTA on SA-Co/VEval.The comparisons include GLEE, LLMDet plus the SAM 3 tracker, and SAM 3’s detector plus tracking-by-detection.Source: COCO LVIS ODinW13 · p. 9, Table 5 · p. 9

PVS

  • SAM 3 improves over SAM 2 on most VOS benchmarks, including a 6.5-point gain on MOSEv2, and achieves higher average mIoU on the SA-37 interactive segmentation benchmark.These evaluations cover visual prompting tasks including video object segmentation and interactive image segmentation.Source: COCO LVIS ODinW13 · p. 9, Table 6 · p. 9, Table 7 · p. 9

SAM 3 Agent

  • The SAM 3 Agent surpasses prior zero-shot results on ReasonSeg, OmniLabel, RefCOCO+, and RefCOCOg without training on referring-expression or reasoning-segmentation data.It uses MLLMs to propose noun-phrase queries, analyze masks, and iterate; consistent system prompts across MLLMs demonstrate robustness.Source: COCO LVIS ODinW13 · p. 9

Selected Ablations and Domain Adaptation

  • The presence head adds +1.5 cgF1, hard negatives raise image-level IL_MCC from 0.44 to 0.68, and synthetic plus high-quality data add +8.8 and +14.6 cgF1.AI verifiers add +7.2 cgF1 and another 1.1 points, closing half the gap to human performance; synthetic domain data eventually catches up to human-annotated data without annotation cost.Source: COCO LVIS ODinW13 · p. 10, Figure 8 · p. 10, Table 9 · p. 10

7 Related Work

SAM 3 extends promptable segmentation from interactive image and video refinement to concept segmentation using text and image-exemplar prompts. It relates to open-vocabulary segmentation, visual grounding, multimodal models, and multi-object tracking while combining their relevant capabilities.

Promptable and Interactive Visual Segmentation

  • SAM 3 extends SAM and SAM 2’s geometry-based interactive segmentation to text and image-exemplar prompts that segment every concept instance in images and videos.SAM introduced promptable image segmentation, while SAM 2 extended refinement to arbitrary video frames.Source: §7 · p. 11

Open-Vocabulary Detection and Segmentation

  • Open-vocabulary detectors and segmenters use language encoders for unseen categories, while SAM 3 introduces a benchmark with more than 100× prior unique concepts.Image exemplars offer a practical alternative to text but convey abstract object concepts less effectively; prior exemplar methods include DINOv (Li et al., 2023a) and T-Rex2.Source: §7 · p. 11

Visual Grounding

  • Visual grounding unifies phrase relevance and region localization, with later systems supporting text, referring expressions, and visual prompts across images and videos.GLIP (Li et al., 2022b), GroundingDino, MQ-GLIP, and GLEE exemplify this progression.Source: §7 · p. 11
  • Multimodal models generate boxes, points, or masks and may combine language with segmentation, whereas SAM 3 additionally supports exemplars and interactive refinement and can serve as an MLLM vision tool.GLEE (Wu et al., 2024a) lacks exemplars and interactive refinement; GLaMM accepts textual and optional image prompts, while Gemini2.5 (Comanici et al., 2025) and Molmo output visual predictions.Source: §7 · p. 11

Multi-Object Tracking and Segmentation

  • Multi-object tracking either associates framewise detections or jointly detects and tracks, but joint models face detection–tracking conflicts that SAM 3 addresses by tightly integrating image detection with video tracking.Tracking-by-detection methods include SORT, Tracktor, ByteTrack, SAM2MOT (Jiang et al., 2025), and OC-SORT.Source: §7 · p. 11

8 Conclusion … A Ablations

SAM 3 introduces promptable concept segmentation through text or image-exemplar prompts, establishes the SA-Co benchmark, and reports state-of-the-art PCS performance while noting out-of-domain generalization as a limitation. The appendix organizes ablations, model and data-engine details, benchmark analyses, additional experiments, agent studies, and model documentation.

8 Conclusion

  • SAM 3 enables interactive open-vocabulary segmentation from text and image-exemplar prompts through the PCS task and SA-Co benchmark.Its contributions include decoupled recognition, localization, and tracking, plus a human–AI data engine.Source: §8 · p. 11
  • SAM 3 achieves state-of-the-art PCS results on SA-Co, doubling prior-system performance in both images and videos.The model also retains visual segmentation capabilities, while its reported limitation is weaker generalization to out-of-domain terms.Source: §8 · p. 11

Ablations

  • The appendix includes model ablations covering architecture, image and video training data, automatic domain adaptation, and annotation speed.These are organized under model ablations and separate image/video data-engine speed analyses.Source: §A.1 · p. 12

Appendix

  • The appendix documents model architecture, implementation, training, data-engine construction, SA-Co data and metrics, and human performance on the benchmark.Data-engine details include the media pool, ontology, human and AI verification phases, domain expansion, and video annotation.Source: §A.1 · pp. 12–13

A.1 Model

  • Additional experiments examine noun-phrase PCS, visual exemplars and interactivity, few-shot fine-tuning, object counting, video PCS, and visual segmentation details.The appendix also reports qualitative and full quantitative results for the SAM 3 agent.Source: §A.1 · pp. 12–13
  • The appendix further covers the SAM 3 agent, including its design, qualitative analysis, and full quantitative results.These sections are followed by object multiplex formulation and results.Source: §A.1 · p. 12
  • The appendix concludes with object-multiplex results and model and annotation cards documenting the data annotation and model.The limitations section is separately identified alongside these supporting materials.Source: §A.1 · p. 12

A.1 Model Ablations

The ablations show that decoupling concept presence from object localization improves PCS performance, while PE’s aligned vision–text encoders provide the strongest overall backbone choice and robustness. The implementation standardizes token counts and training conditions across encoder comparisons.

Presence Token

  • The presence token significantly improves performance, particularly on IL_MCC, by decoupling concept recognition from object localization.The presence score is trained with binary cross-entropy on concept presence or absence, while object queries localize matching instances.Source: §A.1 · p. 13
  • With a presence token, supervising mask scores only for positive concepts and supervising presence and mask scores separately is the best strategy, while their product forms the inference score.When a concept is image-level negative, box/mask object scores receive no gradients; positive concepts still receive classification supervision through Hungarian matching.Source: §A.1 · p. 13, Table 10 · p. 14

Vision and Text Encoder

  • PE’s aligned image and text encoders are the best overall choice, outperforming Hiera, DINOv2, and PE with an unaligned text encoder while improving robustness to domain shifts.Robustness is measured by AP on COCO-O across shifts such as sketches, cartoons, and paintings.Source: §A.1 · p. 14, Table 11 · p. 14

Implementation Details

  • The encoder comparisons use matched detector token counts, subset global attention, separately optimized recipes, and 2 × 2 token downsampling to 1296 fusion tokens.Input resolutions are 1008 px for PE, 1008 px for DINOv2, and 1152 px for Hiera; all models train on 5 million SA-Co/HQ samples.Source: §A.1 · p. 14

A.2 Image Training Data Ablations … A.6 Video Training Data Ablations

Across image and video ablations, performance improves with higher-quality, larger, and harder training data, while synthetic data enables domain adaptation without human annotation. AI verifiers increase image annotation throughput, whereas video pre-labeling trades speed for exhaustivity and improves video segmentation when used for training.

A.2 Image Training Data Ablations

  • SA-Co/SYN and SA-Co/HQ both scale image performance, with synthetic data adding gains beyond high-quality data but showing larger out-of-domain quality gaps.SA-Co/HQ improves in-domain subsets; SA-Co/SYN benefits from large-scale caption concepts, while weaker annotation quality and limited coverage hurt Wiki-Food&Drink performance.Source: §A.2 · pp. 14–15
  • Increasing hard negative noun phrases improves all image metrics, particularly IL_MCC, by training against adversarial distractors absent from the image.These negatives are phrases for which an earlier SAM 3 generation predicts masks, making them difficult classification cases.Source: §A.2 · p. 15
  • AI verifiers improve image PCS by +7.2 cgF1 points through presence-score replacement and another +1.1 points after mask verification.The gains arise from better object-score calibration and rejected-mask removal, nearly closing half the SAM 3–human gap.Source: §A.2 · p. 16

A.3 Automatic Domain Adaptation

  • Domain-specific synthetic data approaches human-annotated adaptation as it scales, and mixing high-quality general-domain pretraining data is especially beneficial for synthetic fine-tuning.SA-Co/SYN-Food eventually surpasses its teacher when mixed with pretraining data, whereas fine-tuning only on new-domain synthetic data performs worse.Source: §A.3 · p. 16, Figure 9 · p. 17

A.4 Image Data Engine Annotation Speed

  • AI verifiers double image data-engine throughput, including a 5x speedup for negative phrases and a 36% speedup for positive phrases.They handle mask and exhaustivity verification while humans focus on correction; annotation quality is reported as similar to human verification.Source: Table 15 · p. 17, §A.4 · pp. 17–18

A.5 Video Data Engine Annotation Speed

  • The all-human video engine is 40% slower than direct human annotation but yields 9% more masklets because pre-labels improve exhaustivity and add visual cues.Starting from PL increases annotation time while helping annotators locate candidate masklets.Source: Table 16 · p. 18, §A.5 · p. 18

A.6 Video Training Data Ablations

  • Adding noun-phrase masklets as Stage 4 video training data improves J &F on SA-Co/VEval, DAVIS17, and SA-V.The scaling study evaluates varying amounts of SA-Co/VIDEO masklets under the VOS task.Source: §A.6 · p. 18, Table 17 · p. 18

B Limitations · C Model Details · C.1 Model Architecture

SAM 3 performs strongly on promptable concept segmentation but remains limited by fine-grained generalization, prompt expressivity, video-tracking efficiency, and concept–instance interaction. Its architecture generalizes SAM 2 to PCS and PVS with multimodal prompts through a dual encoder–decoder transformer containing approximately 850M parameters.

B Limitations

  • SAM 3 shows strong PCS performance in images and videos but retains limitations across many scenarios.These limitations include domain generalization, prompt complexity, multi-object tracking, and concept–instance interaction.Source: B Limitations · p. 18
  • SAM 3 struggles with zero-shot generalization to fine-grained out-of-domain concepts, especially in niche domains such as aircraft types, medical terms, and thermal imagery.Small amounts of human annotation enable rapid adaptation, while domain-specific synthetic data can improve new-domain performance without human involvement.Source: B Limitations · p. 18
  • Video inference scales linearly with tracked objects because SAM 3 assigns each object a SAM 2-style masklet, requiring multiple GPUs for real-time operation.The reported configurations support up to 10 objects on 2 H200s, 28 on 4 H200s, and 64 on 8 H200s at 30 FPS; shared global memory is proposed to improve ambiguity resolution and efficiency.Source: B Limitations · p. 19
  • SAM 3 enforces a hard mode switch between concept and instance interaction so instance-level modifications do not affect other instances of the same concept.Future work could more seamlessly interleave concept and instance prompts.Source: B Limitations · p. 19

C.1 Model Architecture

  • The model uses a dual encoder–decoder transformer architecture broadly based on the SAM series (Ravi et al., 2024) and DETR.Figure 10 provides an overview of the architecture and identifies newly introduced components alongside SAM 2 and PE components.Source: §C.1 · p. 19, Figure 10 · p. 19

C Model Details

  • SAM 3 generalizes SAM 2 to support both PCS and PVS with multimodal prompts, including text, boxes, and points, across images and videos.The architecture also supports interactive use across both tasks.Source: §C.1 · p. 19

C.1 Model Architecture

  • SAM 3 has approximately 850M parameters: roughly 450M in the vision encoder, 300M in the text encoder, and 100M in detector and tracker components.The detector handles images, while tracker components built on top of it handle video.Source: §C.1 · p. 19

C.2 Image Implementation Details … D.3 Phase 1: Human Verification

This section describes SAM 3’s image and video architecture, staged training, and implementation details, alongside data-engine strategies for augmenting training, organizing concepts, mining media, and human-verifying phrase masks.

C.2 Image Implementation Details

  • The image system encodes text, exemplars, and geometric prompts into conditioned frame embeddings, then uses transformer fusion and decoder stacks to predict boxes, scores, and segmentation masks.Image and text encoders are Transformers trained on 5.4 billion image-text pairs; exemplar and geometry tokens can be absent for text-only prompts, while the segmentation head shares semantic and instance segmentation components and uses SimpleFPN multi-scale features.Source: §C.2 · p. 20
  • The image model separates global noun-phrase presence from local query classification, improving robustness to false positives and supporting counting when presence is known.A shared presence token supplies p(NP appears in image), while per-query classification and positive-only mask training condition on that presence; the presence score can be set to 1 for counting or single-frame visual segmentation.Source: §C.2 · pp. 20–21
  • To resolve ambiguous phrase interpretations, SAM 3 trains a winner-takes-all mixture of experts and adds an expert-selection head, reducing overlapping instances by 15%.The ambiguity head leaves masks, boxes, and presence scores unchanged, and Intersection-over-Minimum is used to identify nested overlaps more effectively than Intersection-over-Union.Source: §C.2 · pp. 21–22

C.3 Video Implementation Details

  • The tracker conditions current-frame tokens on spatial memories and object pointers; temporal disambiguation delays output to validate candidate masklets and remove spurious or duplicate masklets, while periodic re-prompting recovers temporary failures.The default confirmation delay is T = 15; periodic re-prompting uses N = 16, with overlap and confidence thresholds of 0.8.Source: §C.3 · pp. 22–23

C.4.1 Training Stages

  • Training proceeds through four stages: perception-encoder pre-training, detector and visual-prompt pre-training, high-quality data and interactivity refinement with a presence loss, and frozen-backbone video-tracker training.Stage 1 uses 5.4 billion image-text pairs; Stage 2 broadens open-vocabulary detection and segmentation; Stage 3 retains exhaustivity-verified data and trains a presence loss; Stage 4 focuses on spatial-temporal tracking with longer temporal contexts.Source: §C.4.1 · p. 24

C.4.2 Additional Training Settings

  • For PCS, training additionally applies data augmentation transformations, while the tracker stage uses multi-mask supervision, occlusion supervision on every frame, and longer-context fine-tuning.The supplied passage identifies PCS data augmentation but does not enumerate its transformations; tracker fine-tuning uses 16-frame and 32-frame videos.Source: §C.4.2 · p. 25, §C.4.1 · p. 25
  • Additional training settings use ontology-derived synonyms, negatives, and hierarchy closure, alongside fixed-square geometric augmentation, safety negatives, and selectively applied mosaics.Inputs are usually 1008 × 1008; mosaics are limited to at most 3 × 3 and restricted to datasets unlikely to contain missing annotations.Source: §C.4.2 · pp. 25–26

D Data Engine Details

  • The data engine combines image and video components across phases, with datasets and performance improvements reported for each phase.Its media pool spans web-scraped and specialized-domain sources, and targeted mining covers 12 media domains.Source: D Data Engine Details · p. 26, §D.1 · p. 26

D.2 SA-Co Ontology

  • The SA-Co ontology expands selected Wikidata nodes recursively into 22.4 million nodes across 17 top-level and 72 sub-categories, supporting concept distribution tracking and negative-phrase creation.A Sentence-BERT retriever proposes candidate nodes and an AI annotator selects the closest match.Source: §D.2 · pp. 26–27

D.3 Phase 1: Human Verification

  • Phase 1 generates noun phrases and candidate masks, then human annotators separately verify mask quality and exhaustivity before correction or final annotation.Captions from BLIP-2 are parsed into noun phrases; FIBER or OWLv2 boxes prompt SAM 2, while correction uses SAM 1 clicks and group masks for inseparable instances.Source: §D.3 · pp. 27–28
  • Correction produces complete instance or group-mask coverage, while absent phrases receive no masks and overly complex jobs may be rejected.Annotators can reject vague or uncertain image–phrase pairings, improving consistency and agreement.Source: §D.3 · p. 28

D.4 Phase 2: Human + AI Verification … E.2 SA-Co Evaluation Benchmark

This section describes the data engine and datasets supporting SAM 3, spanning taxonomy-guided mining, annotation generation, human/AI verification, and targeted handling of challenging image and video scenarios. It also introduces the SA-Co benchmark, with exhaustive instance annotations, hard negatives, and broad concept coverage for promptable concept segmentation.

D.4 Phase 2: Human + AI Verification

  • AI verifiers reach or surpass human performance, and integrating them with SAM 3 consistently improves PCS while more than doubling data-engine throughput.AI verifiers focus human effort on poor-quality or missing masks, increasingly restricting manual work to SAM 3 failure cases.Source: §D.4 · p. 30

D.5 Phase 3: Scaling and Domain Expansion

  • Phase 3 expands mining to novel, long-tail, fine-grained, crowded, and small-object domains while relying primarily on AI verification to increase throughput.It identifies difficult images using SA-1B mask statistics, including pairwise-mask IoU, object counts, and mask areas.Source: §D.5 · p. 30

D.6 Phase 4: Video Annotation

  • The video pipeline mines challenging single-shot clips, proposes and filters frame-level concepts, generates and deduplicates masklets, and uses human verification and correction for exhaustive tracking annotations.It targets crowded multi-object videos by retaining phrases with at least N=3 objects and videos containing at least M=2 such phrases.Source: §D.6 · pp. 30, 33
  • Video annotation expands coverage by randomly sampling frames and by selecting object-dense frames identified through the video data-engine pipeline.Both strategies increase the diversity and volume of annotated video data by applying the image data engine to sampled frames.Source: §D.6 · p. 34

E.1 SA-Co Training Data

  • SAM 3’s training data combines human- or AI-verified high-quality images, large-scale synthetic images, external annotated datasets, and human-verified videos.The synthetic set contains 39M images, 1.7B image-NP pairs, and 1.4B masks, while external datasets are mapped to the SA-Co ontology and enriched with negative labels.Source: §E.1 · p. 34

E.2 SA-Co Evaluation Benchmark

  • SA-Co evaluates promptable concept segmentation in images and videos using exhaustively masked matching instances, federated labels, hard negatives, and approximately 50× more unique phrases than LVIS-test.The benchmark spans common objects, fine-grained concepts, food, art, robotics, and other sub-domains, with hard negatives throughout.Source: §E.2 · pp. 35–36

E.3 Metrics … F.4 Object Counting

This section defines and examines evaluation protocols for promptable concept segmentation and object counting, covering metrics, human agreement, dataset examples, prompting settings, and counting-specific post-processing. It also describes how exemplar ambiguity and overlapping individual/group predictions affect these evaluations.

E.3 Metrics

  • The proposed cgF1 metric combines localization and classification to evaluate PCS under large open-vocabulary label spaces, where conventional AP is noisy, costly, and poorly calibrated.Localization uses matched mask F1 across IoU thresholds, with positive micro F1 as the main localization metric; classification uses image-level MCC based on whether predictions are present.Source: §E.3 · pp. 36–37

E.4 Human Performance on SA-Co

  • PCS is intrinsically ambiguous, so SA-Co/Gold uses three independent annotations and oracle evaluation selects the annotation pair maximizing local F1.Random-pair evaluation shows a noticeable gap from oracle performance, while hard-negative human MCC is estimated from additional annotations and video performance uses an additional annotation.Source: §E.4 · pp. 38–39

E.5 Additional Dataset Examples

  • The paper illustrates SA-Co’s visual domains, annotations, and synthetic positive and negative noun phrases across image and video datasets.Examples include independent masks, disagreements over applicability or instance boundaries, and AI-verified exhaustive or negative phrases.Source: §E.5 · p. 39, Figure 14 · p. 40, Figure 19 · p. 44

F.1 PCS with NPs on Images

  • Image PCS experiments compare SAM 3 with box detectors, segmentation models, and Gemini across established datasets and SA-Co, using cgF1 except pmF1 for negative-free SA-Co/Bio.On SA-Co/Gold, evaluation reports oracle metrics and estimated human performance; box-based methods are converted to masks with SAM 1.Source: §F.1 · p. 45

F.2 Visual Exemplars and Interactivity

  • Combined text and visual prompts are better suited than visual-only prompts because text reduces ambiguity and the additional box helps identify unfamiliar concepts.SAM 3 performs worse with visual-only prompts than text-only prompts on SA-Co/Gold, whereas combining both inputs addresses the ambiguity.Source: §F.2 · p. 46, Table 35 · p. 47

F.3 Few-Shot Fine-tuning

  • SAM 3 consistently surpasses prior state-of-the-art methods on OdinW13 few-shot detection, while surpassing leading methods on Roboflow-100VL after few-shot or full fine-tuning but underperforming zero-shot.OdinW13 reports mean BoxAP across 13 datasets; Roboflow-100VL averages results across 100 datasets, where minimal fine-tuning closes the zero-shot gap.Source: §F.3 · pp. 46–47

F.4 Object Counting

  • Object-counting evaluation compares an internal SAM 3 checkpoint with MLLMs and detection experts on CountBench and PixMo-Count using accuracy and MAE.The evaluation excludes unavailable links, leaving 487 CountBench images and 529 PixMo-Count images; the presence token is omitted because it does not help counting.Source: §F.4 · p. 47
  • SAM 3 can output predictions for both individual objects and the group as a whole, contradicting the counting objective.Source: §F.4 · p. 48
  • SAM 3 uses IoM-based NMS at threshold 0.5, retains predictions with confidence above 0.5, and counts the remaining predictions.IoM divides overlap by the smaller mask area, helping distinguish whole-versus-part detections where IoU can be misleading.Source: §F.4 · p. 48

F.5 Video PCS Details … G.1 Agent Design

The paper details SAM 3’s video PCS evaluation and strong performance across video segmentation tasks, then introduces an agent that iteratively plans, queries SAM 3, examines masks, and returns results for complex visual requests.

F.5 Video PCS Details

  • Video PCS evaluates noun-phrase prompts on SA-Co/VEval subsets and public benchmarks using cgF1, pHOTA, TETA, and official dataset metrics.The protocol prompts each video with its supplied noun phrases or all dataset category names, covering benchmarks with up to 5.1K noun phrases and 1,196 categories.Source: §F.5 · pp. 48–49
  • The video metrics combine video-level existence and localization quality, match masklets by volume IoU, and adapt HOTA to class-agnostic video–noun-phrase pairs.cgF1 multiplies VL_MCC by positive macro F1, while pHOTA reports detection and association accuracy; TETA is also evaluated.Source: §F.5 · p. 49
  • SAM 3 largely outperforms video PCS baselines across SA-Co/VEval and public benchmarks, achieving state-of-the-art results on LVVIS and OVIS while excelling in pDetA and pAssA.GLEE and LLMDet are zero-shot on SA-Co/VEval, and both SAM 3’s detector and tracker contribute critically to performance.Source: §F.5 · p. 50, Table 39 · p. 51
  • Temporal disambiguation strategies improve video PCS performance, especially pHOTA, and produce notably better qualitative video outputs.The comparison disables all temporal disambiguation strategies in the reported alternative setting.Source: §F.5 · p. 50

F.6 PVS Details

  • SAM 3 improves VOS on every evaluated dataset, including challenging MOSEv2 and long-video LVOSv2, and outperforms prior methods on interactive image and video segmentation.Interactive image segmentation uses 1 or 5 clicks; interactive video segmentation improves both offline and online evaluation under the nine-dataset, three-click protocol.Source: §F.6 · p. 50, Figure 21 · p. 52

G.1 Agent Design

  • SAM 3 Agent converts natural-language segmentation requests into masks through an MLLM-controlled perception–action loop that plans, calls tools, inspects feedback, and terminates with selected masks or no-mask reporting.Its four tools are segment_phrase, examine_each_mask, select_masks_and_return, and report_no_mask; intermediate tools can be repeated before a return tool ends the task.Source: §G.1 · pp. 55–56

F.6.1 Additional Model Outputs for Different Tasks

  • The agent prunes intermediate trial-and-error states, discards prior masks after segment_phrase, and retains used prompts to control context growth during difficult queries requiring up to 60 steps.Additional examples show successful ReasonSeg and RefCOCOg outputs alongside a failure caused by incorrect final mask selection.Source: §G.1 · p. 56, Figure 25 · p. 57, Figure 26 · p. 58

G.2 Qualitative Analysis … H.1 Formulation

SAM 3 Agent handles diverse reasoning and referring-expression segmentation queries, achieving strong zero-shot results while SAM 3 Multiplex reduces multi-object tracking computation through shared bucketed memory. Multiplex preserves object identity with slot-specific embeddings while jointly processing spatial memory.

G.2 Qualitative Analysis

  • A complete reasoning trace shows the agent simplifying an unsuccessful prompt, generating four masks for “pixel character,” and selecting the mask corresponding to “the stronger Mario.”The trace illustrates SAM 3 providing grounding while the MLLM performs visual reasoning and final mask selection.Source: §G.2 · pp. 58–60, Figure 27 · p. 60
  • Qualitatively, SAM 3 Agent handles free-form queries with varied spatial and logical complexity, rejects unmatched queries, returns multiple masks, and segments fine-grained concepts.The MLLM backbone supports varied query length and format and improves segmentation of concepts such as aircraft types and food categories beyond SAM 3 alone.Source: §G.2 · p. 58

G.3 Full Quantitative Results

  • SAM 3 Agent achieves the best zero-shot results on ReasonSeg and OmniLabel, surpasses prior zero-shot state of the art on RefCOCO+ and RefCOCOg, and approaches trained methods on RefCOCO.The RefCOCO gap is attributed to dataset-specific annotation biases that training-based methods can exploit but the zero-shot agent cannot.Source: §G.3 · p. 60, Table 40 · p. 61, Table 41 · p. 62

H Object Multiplex

  • SAM 3 Multiplex addresses per-object memory processing costs that grow linearly with object count and limit shared context in crowded scenes.The method introduces shared-memory joint multi-object processing within SAM 3’s existing tracking pipeline.Source: H Object Multiplex · p. 64

H.1 Formulation

  • Multiplex groups objects into fixed-capacity buckets of M slots, reducing bucket-level memory forward passes from O(N) to O(⌈N/M⌉); with M=16, 16 objects require one pass.Joint encoding remains effective because object memory features are usually spatially disjoint and overlaps are sparse.Source: §H.1 · p. 64
  • Multiplex separates shared spatial memory from object-specific embeddings, preserving identity during temporal association and mask decoding while leaving the overall tracking pipeline unchanged.This factorization adapts established shared-representation and identity-cue principles to SAM 3.Source: §H.1 · p. 64
  • Mux converts per-object masks and embeddings into bucket-slot tensors, while demux restores slot-indexed outputs to per-object predictions using the recorded assignment.The operators translate between object-indexed data space and bucket-and-slot-indexed multiplex space.Source: §H.1 · p. 64, Figure 29 · p. 64
  • Each slot predicts three candidate masks, producing 3M masks per bucket, while fixed five-token decoding per slot yields 5M tokens; slot randomization encourages permutation robustness.For M=16, decoding uses 80 tokens per bucket without significantly increasing runtime.Source: §H.1 · p. 64

H.2 Results · I Model and annotation cards

SAM 3 Multiplex improves SAM 3 across SA-Co/VEval video PCS and most VOS benchmarks, while scaling more efficiently for videos with multiple tracked objects. Its main tradeoff is modest single-object latency overhead and mixed performance on public video PCS benchmarks.

H.2 Results

  • SAM 3 Multiplex improves video PCS across all three SA-Co/VEval splits, including +2.7 cGF and +0.9 pHOTA on YT-Temporal-1B, while public-benchmark results are mixed.It improves OVIS by +1.8 mAP but slightly regresses on LVVIS, BURST, and YTVIS21; comparisons use the same detector as SAM 3.Source: §H.2 · p. 65, Table 43 · p. 65
  • SAM 3 Multiplex improves SAM 3 on six of seven VOS benchmarks, with gains of +1.2 on MOSEv1 and +2.0 on MOSEv2.SAM 3 already provides state-of-the-art VOS results across the benchmark suite.Source: §H.2 · p. 65, Table 44 · p. 65
  • At 128 tracked objects, SAM 3 Multiplex achieves a 5.2× speedup over SAM 3 and scales more efficiently as object counts increase.The advantage begins beyond the crossover point of at least two objects, while SAM 3 throughput degrades sharply and Multiplex remains stable.Source: §H.2 · p. 65, Figure 30 · p. 65
  • For single-object tracking, bucketization introduces approximately 7% latency overhead before Multiplex becomes faster at two or more tracked objects.Throughput measurements use a single H100 GPU across varying numbers of tracked video objects.Source: §H.2 · p. 65, Figure 30 · p. 65
  • The reported FPS uses optimized implementations with batched detection and postprocessing, reduced CUDA synchronization, and improved torch.compile compatibility.These optimizations are planned for inclusion in the SAM 3.1 release.Source: §H.2 · p. 65

I.1 Data annotation card · I.2 Model card

The data annotation card documents ambiguity-aware, quality-controlled annotation for promptable concept segmentation, while noting regional, temporal, and coverage limitations. The model card is presented following the structure of Mitchell et al. (2019).

I.1 Data annotation card

  • I.1 Data annotation card: Three independent reviews in SA-Co/Gold capture valid alternative interpretations of the inherently ambiguous segmentation task.Annotators could disagree about which objects a phrase should include, so the benchmark preserves multiple responses and measures agreement.Source: §I.1 · pp. 66–67
  • I.1 Data annotation card: Detailed, frequently updated guidelines, vendor training, weekly feedback, and logged clarifications were used to resolve corner cases and reduce repeat errors.The research team QA’ed vendor quality teams and annotators while maintaining communication channels for ambiguous jobs.Source: §I.1 · p. 66
  • I.1 Data annotation card: The annotation pool may introduce regional variation and does not represent all communities that will use SAM 3.Annotators were based in APAC and EMEA, and regional differences may affect word meanings and visual interpretations of concepts.Source: §I.1 · p. 66
  • I.1 Data annotation card: Annotation quality required valid, complete masks, correct concept assignment, and exhaustive coverage, with training and ongoing quality assurance.Annotators completed two-day training; vendor QA reviewed 5%–20% of annotations depending on task complexity, averaging 10%.Source: §I.1 · p. 67
  • I.1 Data annotation card: Image labels combine mask verification, exhaustivity checks, and correction, while video labels retain only manually validated annotations meeting all three quality criteria.Annotators could research unfamiliar concepts and reject vague or low-confidence jobs; all video annotations were manually reviewed.Source: §I.1 · p. 67
  • I.1 Data annotation card: Dataset utility may change as public source images disappear, concepts evolve visually, or new phrases and item types emerge.The benchmark is intended for model evaluation and should not be used for training.Source: §I.1 · p. 67

I.2 Model card

  • I.2 Model card: SAM 3’s model card follows the structure proposed by Mitchell et al. (2019).The supplied passage identifies the model-card structure but provides no additional model-card content.Source: Table 45 · p. 68

Related papers

Loading 2511.16719v2…