PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM 3: Segment Anything with Concepts

Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, Christoph Feichtenhofer

arXiv:2511.16719v2cs.CVcs.AI

TL;DR

Existing promptable segmentation systems focus on individual objects, leaving open the task of finding and segmenting every instance of a concept across images and videos. SAM 3 introduces PCS with text or image-exemplar prompts, using a shared-encoder detector-tracker architecture and scalable data engine. It doubles prior PCS accuracy on SA-Co while improving visual segmentation, with ambiguity and out-of-domain concepts remaining important scope boundaries.

  • Problem

    Prior promptable segmentation focused on single objects per visual prompt, leaving the general task of finding and segmenting all instances of a concept unaddressed.

  • Method

    SAM 3 combines a shared-encoder detector and video tracker with decoupled recognition-localization via a presence head and a human- and AI-assisted data engine.

  • Results

    SAM 3 doubles prior systems’ PCS performance on SA-Co in images and videos, while improving previous SAM visual segmentation capabilities.

  • Takeaways & Limitations

    SAM 3 provides open-vocabulary interactive segmentation through text and image exemplars and releases the SA-Co benchmark, checkpoints, and inference code.

  • Takeaways & Limitations

    The model struggles to generalize to out-of-domain terms, and the broad noun-phrase vocabulary is intrinsically ambiguous.

Abstract

from arXiv · show

We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts, which we define as either short noun phrases (e.g., "yellow school bus"), image exemplars, or a combination of both. Promptable Concept Segmentation (PCS) takes such prompts and returns segmentation masks and unique identities for all matching object instances. To advance PCS, we build a scalable data engine that produces a high-quality dataset with 4M unique concept labels, including hard negatives, across images and videos. Our model consists of an image-level detector and a memory-based video tracker that share a single backbone. Recognition and localization are decoupled with a presence head, which boosts detection accuracy. SAM 3 doubles the accuracy of existing systems in both image and video PCS, and improves previous SAM capabilities on visual segmentation tasks. We open source SAM 3 along with our new Segment Anything with Concepts (SA-Co) benchmark for promptable concept segmentation.

1 Introduction

SAM 3 extends promptable segmentation from single-object visual prompts to finding, segmenting, and tracking every instance of concepts specified by text or image exemplars. It combines a shared-encoder detector and tracker with a scalable human- and AI-assisted data engine, achieving strong benchmark results.

  • Motivation and task: SAM 3 addresses the gap between single-object promptable visual segmentation and segmenting all instances of a concept across images and videos.Concepts are specified with short noun phrases, image exemplars, or both, while object identities are preserved across video frames.
  • Model: The model combines a text-, geometry-, and exemplar-conditioned detector with a video tracker sharing one vision encoder.The tracker supports video segmentation and interactive refinement, while the decoupled design avoids conflict between identity-agnostic detection and identity separation.
  • Data engine: A human- and model-in-the-loop data engine expands media diversity, label difficulty, and verification efficiency using ontology-guided phrases, hard negatives, and AI annotators.Human annotators focus on correcting challenging cases identified by human and AI verification.
  • Data engine: 4M unique phrases and 52M masks support high-quality training, alongside synthetic data containing 38M phrases and 1.4B masks.The engine starts from noisy media-phrase-mask pseudo-labels and filters or corrects errors before training.
  • Benchmark: 207K unique concepts in the SA-Co benchmark cover 120K images and 1.7K videos, exceeding existing benchmarks by more than 50× in concept count.The benchmark evaluates promptable concept segmentation with exhaustive masks.
  • Results: 48.8 zero-shot mask AP on LVIS exceeds the current best of 38.5, while SA-Co performance surpasses baselines by at least 2×.The experiments also report improvements over SAM 2 on visual prompts and near-real-time inference for approximately five concurrent video objects.

2 Promptable Concept Segmentation (PCS)

Promptable Concept Segmentation asks a model to detect, segment, and track every instance matching a short noun-phrase or exemplar-defined visual concept. Its broad vocabulary creates ambiguity, addressed through clearer annotation, multiple expert labels, flexible evaluation, and model-side handling.

  • Task definition: PCS detects, segments, and tracks all instances of a visual concept in an image or video up to 30 seconds long.Prompts use short noun phrases, image exemplars, or both; exemplars may be positive or negative boxes on individual frames.
  • Task definition: Concepts are restricted to simple noun phrases consisting of a noun and optional modifiers.Text prompts apply globally across frames, whereas image exemplars can iteratively refine target masks.
  • Prompt constraints: All prompts must preserve a consistent category definition, so an exemplar showing only part of an object cannot refine a broader concept such as “fish.”The text prompt should instead be updated when the intended category changes.
  • Ambiguity: A vocabulary covering any visually groundable noun phrase is intrinsically ambiguous because of polysemy, subjective descriptors, vague concepts, boundary ambiguity, occlusion, and blur.Examples include “mouse,” “cozy,” “large,” “brand identity,” and whether “mirror” includes its frame.
  • Ambiguity: The ambiguity protocol uses three expert annotations, evaluation allowing multiple valid interpretations, clearer annotation guidelines, and an ambiguity module.These measures target conflicting interpretations that can otherwise produce overlapping masks.

3 Model

SAM 3 generalizes SAM 2 by combining concept-conditioned image detection with memory-based video tracking, while supporting interactive visual refinement. Its detector separates global concept recognition from local object localization through a presence head.

  • Model scope: SAM 3 supports PCS and PVS using simple noun phrases, image exemplars, or conventional visual prompts to define objects for segmentation.Image exemplars can be added iteratively to refine target masks, including false positives and false negatives.
  • Architecture: The architecture combines a DETR-based detector and a SAM 2-style tracker that share a vision encoder.The detector handles image-level capabilities, while the tracker and memory support video segmentation.
  • Detection: A global presence token predicts whether the noun-phrase concept appears, allowing proposal queries to focus on localizing matching objects.Each proposal score is multiplied by the global presence score, decoupling recognition from localization.
  • Video tracking: The video tracker propagates prior masklets, matches them with current detections, and spawns masklets for newly detected objects.Temporal suppression and periodic re-prompting with reliable detections address ambiguities and stale tracker predictions.
  • Inference and refinement: During inference, the tracker retains confidently present frames and selects the highest-confidence mask among three outputs for each tracked object.Visual prompts can refine individual masks, with video refinements propagated across the entire video.

4 Data Engine

SAM 3 uses a human- and model-in-the-loop data engine that iteratively mines difficult media-concept pairs, verifies masks and exhaustivity, and expands coverage across domains and video.

  • Engine design: The data engine actively mines media-phrase pairs where SAM 3 produces low-quality data, then uses human and AI annotators in a feedback loop.Delegating selected tasks to AI annotators more than doubles throughput compared with human-only annotation.
  • Verification: Candidate masks undergo Mask Verification for quality and relevance, followed by Exhaustivity Verification to check whether all concept instances are annotated.The pipeline generates noun phrases, candidate masks, and verification decisions before human correction of remaining errors.
  • Human–AI collaboration: AI verifiers roughly double data-engine throughput and let human effort focus on challenging cases while hard-negative noun phrases are added adversarially.The verifiers are fine-tuned from human accept/reject labels and the process is repeatedly retrained as SAM 3 improves.
  • Scaling and domain expansion: Domain expansion broadens SA-Co/HQ to 15 datasets and adds long-tail concepts mined from alt-text and a 22.4M-node ontology.The process uses modest domain-specific human supervision to improve exhaustivity verification in new domains.
  • Video annotation: Video annotation adds scene and motion filtering, targeted searches, frame sampling, spatio-temporal masklets, deduplication, and removal of trivial masks.Video frames enter the image annotation flow, while the extended SAM 3 system produces masklets for video-specific challenges.

5 Segment Anything with Concepts (SA-Co) Dataset

SA-Co provides large-scale training and evaluation resources for promptable concept segmentation, combining diverse concepts, hard negatives, exhaustive masks, and calibration-aware metrics.

  • Training data: SA-Co/HQ contains 5.2M images and 4M unique noun phrases, alongside synthetic and external datasets for PCS training.The training collection includes high-quality human-verified data, synthetic labels, and external instance-mask datasets enriched with hard negatives.
  • Benchmark composition: The SA-Co benchmark contains 207K unique phrases, 121K images and videos, and over 3M media-phrase pairs with hard-negative labels.Its splits include Gold, Silver, Bronze, and Bio variants spanning human-annotated and existing datasets.
  • Metrics: SA-Co evaluates calibrated predictions by retaining only confidence-above-0.5 outputs and measuring image-level concept presence with IL_MCC.Average precision alone does not account for calibration, motivating the thresholded evaluation design.
  • Metrics: The main classification-gated F1 metric is defined as cgF1 = 100 ∗pmF1 ∗IL_MCC.This combines mask-quality performance with image-level presence classification.
  • Ambiguity handling: SA-Co/Gold uses three annotations per noun phrase and reports oracle accuracy by selecting the best score against the available ground truths.This setup measures performance under annotation ambiguity.

6 Experiments

SAM 3 is evaluated across image and video concept segmentation, visual prompting, counting, transfer, and agent-assisted segmentation. It generally outperforms prior systems, with especially strong gains on open-vocabulary PCS, while ablations examine architectural and data-engine effects.

  • Image PCS with Text: SAM 3 achieves more than double the cgF1 of the strongest baseline on open-vocabulary SA-Co/Gold image concept segmentation.It also reaches 74% of estimated human performance and performs better on additional SA-Co splits.
  • Few-Shot Adaptation: +18.3, +10.3, and +20.5 are SAM 3’s margins over T-Rex2 on COCO, LVIS, and ODinW few-shot adaptation, respectively.The evaluation covers text-only, image-only, and combined text-and-image prompting.
  • PCS with K Exemplars: SAM 3’s PCS cgF1 improves more quickly than an ideal PVS baseline as interactive exemplar prompts are added.The experiment adds positive prompts for missed ground truths and negative prompts for false positives.
  • Object Counting: SAM 3 provides object segmentation alongside competitive object counting accuracy, unlike most evaluated MLLMs.Counting is evaluated on CountBench and PixMo-Count using Accuracy and MAE.
  • PVS: SAM 3 improves visual segmentation over SAM 2 on most VOS benchmarks, including a 6.5-point gain over prior work on MOSEv2.The model is also evaluated on interactive image segmentation using the SA-37 benchmark.
  • Selected Ablations: The presence head boosts cgF1 by +1.5, while hard negatives improve IL_MCC from 0.44 to 0.68 in selected ablations.Replacing the presence score with exhaustivity-verifier scores boosts cgF1 by +7.2, and mask verification adds another 1.1 points.

7 Related Work

Related work spans promptable visual segmentation, open-vocabulary detection and segmentation, visual grounding, and multi-object tracking and segmentation. SAM 3 combines these directions by supporting concept prompts, exhaustive instance segmentation, interactive refinement, and video identity tracking.

  • Promptable Visual Segmentation: Earlier SAM systems introduced promptable segmentation for images and videos, but focused on segmenting a single object per point, box, or mask prompt.SAM 3 extends this setting to finding and segmenting all instances of a concept.
  • Open-Vocabulary Detection and Segmentation: Open-vocabulary detection and segmentation exhaustively label open-vocabulary object categories with boxes or pixel masks using arbitrary text descriptions.Recent methods commonly leverage large-scale vision-language encoders such as CLIP.
  • Visual Grounding: Visual grounding localizes a language expression to an image region, while related systems increasingly support multiple prompt modalities and video inputs.GLEE supports text phrases, referring expressions, and visual prompts for category and instance grounding in images and videos.
  • Visual Grounding: Unlike GLEE, SAM 3 supports exemplars and interactive refinement, while unlike reasoning-oriented systems it focuses on concept segmentation.MLLM-based systems can interleave natural-language responses with segmentation masks or output boxes and masks.
  • Multi-Object Tracking and Segmentation: Multi-object tracking and segmentation methods associate detected video objects with unique identities, using either tracking-by-detection or jointly trainable architectures.Joint models face a conflict between semantic detection and identity disentanglement, which SAM 3 addresses through decoupled design.

8 Conclusion

SAM 3 introduces Promptable Concept Segmentation and the SA-Co benchmark, combining concept prompting, decoupled detection and tracking, and human–AI data curation. It reports state-of-the-art PCS results while retaining visual segmentation capabilities, but remains limited on out-of-domain terms.

  • Conclusion: SAM 3 and SA-Co introduce open-vocabulary concept segmentation with text and image exemplars for interactive segmentation.The benchmark and model target concept-level segmentation across images and videos.
  • Conclusion: The architecture decouples recognition, localization, and tracking while extending SAM 2’s visual segmentation capabilities.This design is paired with a high-quality data engine using complementary human and AI annotators.
  • Conclusion: SAM 3 doubles prior-system performance for PCS on SA-Co in both images and videos.The paper presents this as a state-of-the-art result.
  • Limitations: SAM 3 struggles to generalize to out-of-domain terms, although automatic domain expansion can mitigate this limitation with extra training.The paper identifies this as one of several model limitations.

A.1 Model Ablations

The ablations test presence-based recognition–localization decoupling, supervision choices, encoder selection, and implementation details. Results support a separate presence score, aligned vision–text encoders, and positive-only object-score supervision.

  • Presence Token: A presence token predicts concept presence with binary cross-entropy supervision and significantly improves performance, particularly on IL_MCC.The token is trained only on the PCS task during joint training.
  • Presence Token: The best supervision strategy supervises mask scores only for positive concepts while training presence and mask scores separately.Their product is used as the total object score during inference.
  • Vision and Text Encoder: PE-aligned image and text encoders work well because SAM 3 requires both semantic visual and linguistic understanding.The encoder choice is motivated by broad concept coverage and aligned image–text representations.

A.2 Image Training Data Ablations

SAM 3’s image performance improves as training data becomes larger, more diverse, higher quality, and enriched with hard negatives. AI verifiers and synthetic data further improve annotation throughput, domain adaptation, and downstream segmentation.

  • Training Data: Adding SA-Co/SYN and then SA-Co/HQ progressively improves performance beyond SA-Co/EXT alone on SA-Co/Gold.SA-Co/SYN adds scale, while SA-Co/HQ contributes quality and distributional similarity; SA-Co/SYN still adds gains on top of SA-Co/HQ.
  • Scaling: SAM 3 scales with SA-Co/HQ data on both in-domain MetaCLIP and Wiki-Food&Drink subsets, although the simplified ablation setting leaves a larger human-performance gap.The subsets are in-domain for SA-Co/HQ because they share its distribution.
  • Scaling: SA-Co/SYN scales on both in-domain and out-of-domain subsets, but weaker annotation quality and limited domain coverage create larger gaps out of domain.The MetaCLIP gap is 42.5 vs. 49.0, while the Wiki-Food&Drink gap is 37.4 vs. 59.9.
  • Hard Negatives: More hard-negative noun phrases improve SAM 3 across all metrics, with the largest effect on IL_MCC.Hard negatives are absent concepts that earlier SAM 3 versions incorrectly predicted masks for, making them difficult classification distractors.
  • AI Verifiers: +7.2 cgF1 points from the Exhaustivity Verification presence score and a further +1.1 points from Mask Verification improve SAM 3 on PCS.The exhaustivity score also improves pmF1 by calibrating object scores, and the combined verifier system closes nearly half the gap to human performance.
  • Automatic Domain Adaptation: Domain-specific synthetic data can improve a new domain without human annotation, with SA-Co/SYN-Food eventually surpassing its teacher when mixed with high-quality pre-training data.Mixing new-domain synthetic data with pre-training data performs better than fine-tuning on synthetic new-domain data alone.
  • AI Verifiers: AI verifiers more than double data-engine throughput and achieve similar or better Mask Verification and Exhaustivity Verification performance than human verifiers.As SAM 3 and the verifiers improve, human annotators correct fewer errors and increasingly challenging data can be collected.
  • Video Data: Adding video-engine masklets improves J &F on SA-Co/VEval, DAVIS17, and SA-V.These masklets are used as additional Stage 4 video-level training data.

B Limitations

SAM 3 extends SAM 2 to concept segmentation while retaining visual-prompt capabilities, but its scope and efficiency remain bounded. Limitations include restricted language prompts, weak zero-shot fine-grained domain generalization, linear multi-object tracking cost, and a hard concept-to-instance mode switch.

  • Generalization: Zero-shot generalization remains weak for fine-grained out-of-domain concepts, especially in niche visual domains such as thermal imagery.Small quantities of human-annotated data enable rapid adaptation, and domain-specific synthetic data can improve new-domain performance without human involvement.
  • Language Prompts: PCS is constrained to simple noun phrases and does not support longer referring expressions or multi-attribute queries beyond one or two attributes.Combining SAM 3 with an MLLM enables handling more complex phrases.
  • Video Tracking: Video inference cost scales linearly with the number of tracked objects because each object uses a separate SAM 2-style masklet.Real-time inference is supported by distributing up to 10, 28, or 64 objects across 2, 4, or 8 H200 GPUs, respectively.
  • Video Tracking: The current video tracker lacks shared object-level context for resolving ambiguities in multi-object tracking.Shared global memory is proposed as future work that could also improve inference efficiency.
  • Interactivity: Concept-level and instance-level interactivity require a hard mode switch, preventing seamless interleaving of concept and instance prompts.Future work could interleave the two prompt types more seamlessly.
  • Scope: SAM 3 is a SAM 2 generalization supporting PCS and PVS with multimodal prompts and interactivity in images and videos.Its architecture uses roughly 850M parameters across vision and text encoders, detector, and tracker components.

C.2 Image Implementation Details

SAM 3’s image implementation combines transformer-based prompt encoding, detection, and segmentation with ambiguity-aware prediction. A shared presence mechanism separates concept recognition from object localization, while winner-takes-all experts model multiple valid interpretations.

  • Encoders: SAM 3 uses transformer-based image and text encoders, with visual prompts represented as geometry or exemplar tokens.The encoders use contrastive vision-language training, while geometry and exemplar tokens attend to frame embeddings.
  • Fusion Encoder: A six-block fusion encoder conditions frame embeddings on text and geometry/exemplar prompt tokens.Its self- and cross-attention layers produce conditioned frame embeddings for downstream prediction.
  • Detector: The image detector is a six-block encoder-decoder transformer with learned object queries, cross-attention to prompts, and iterative box refinement.Dedicated MLPs predict bounding boxes and scores from the object queries.
  • Presence Head: The presence head factorizes concept matching into conditional query localization and a shared probability that the noun phrase appears in the image.The shared presence score is computed from a presence token and can be set to 1 when the target is known to be present.
  • Segmentation Head: SAM 3’s segmentation head shares semantic and instance segmentation components, using conditioned features for semantic masks and object queries for instance masks.SimpleFPN supplies multi-scale features to the head.
  • Ambiguity Handling: An ambiguity head trains K experts with winner-takes-all supervision so experts can specialize in different interpretations of an ambiguous phrase.The model uses K = 2 best-performing experts, while K > 3 is harder to train because of mode collapse.

D.4 Phase 2: Human + AI Verification

Phase 2 uses retrieval, ontology-guided concept selection, automated proposal and filtering, and human-plus-AI verification to expand and clean PCS training data. AI verifiers improve end-to-end system performance and more than double data-engine throughput while reserving human effort for difficult cases.

  • Concept Mining: Retrieval models mine challenging concepts, while ontology-guided selection balances coverage and targets long-tail and fine-grained phrases.Offline concept indexing and online image- and text-based retrieval support concept-specific mining.
  • Phrase Curation: Captioning and phrase parsing generate diverse noun phrases, followed by filtering of non-groundable, redundant, frequent, or easy phrases.The pipeline also singularizes phrases, deduplicates near-duplicates, and removes possessives.
  • Hard Negatives: Hard negative phrases are proposed from ontology relatives or visually similar concepts and retained when SAM 3 produces adversarial predictions overlapping positive annotations.This process creates negatives that challenge the current model rather than merely absent phrases.
  • Mask Proposal: SAM 3 proposes instance and semantic masks, including coarse-concept masks for fine-grained domains where the target concept is difficult to detect.Masks are deduplicated by IoU and then cleaned through verification.
  • Verification: AI verifiers perform mask-quality and exhaustivity verification on image–phrase–mask data, trained from human annotations.The verification tasks are formulated as multiple-choice image-text questions.
  • Verification Results: AI verifiers match or surpass human performance on verification tasks, and their end-to-end combination with SAM 3 always outperforms SAM 3 alone.Mask verification is typically human-level zero-shot on new domains, while exhaustivity verification reaches human performance with thousands of domain-specific examples.
  • Throughput: More than 2× throughput results from using AI verifiers to handle routine cases and directing human annotators toward SAM 3 failure cases.The authors expect throughput to increase as both SAM 3 and verifier models improve.
  • Benchmark: The resulting SA-Co benchmark exhaustively annotates matching instances across diverse images and videos and includes challenging hard negatives for open-vocabulary evaluation.SA-Co/Gold contains about 50× more unique phrases than LVIS-test.

E.3 Metrics

The paper introduces classification-gated F1 and separate localization and classification measures for PCS, addressing the instability of AP in very large open vocabularies. Evaluation also accounts for annotation ambiguity through multiple ground truths and oracle matching protocols.

  • Motivation: AP becomes impractical and noisy for open-vocabulary detection with tens of thousands of labels and does not measure confidence calibration.The paper therefore replaces it with metrics designed for PCS.
  • Metric Components: The PCS metrics separate localization quality from image-level classification of whether any matching mask is predicted.Localization uses IoU-based bipartite matching and F1 aggregation across thresholds, while classification uses image-level confusion counts.
  • Classification-Gated F1: Classification-gated F1 combines positive micro F1 with Matthews Correlation Coefficient to evaluate localization and classification jointly.The metric is designed as the main PCS score for images.
  • Ambiguity-Aware Evaluation: SA-Co/Gold uses three independent annotations per datapoint and an oracle protocol that selects the ground truth yielding the best local F1.This accommodates valid disagreements caused by ambiguous phrases.
  • Prompt Evaluation: Visual exemplars alone are ambiguous, whereas combining text and visual prompts better resolves the intended concept.SAM 3 performs worse with visual prompts alone than with text-only prompts on SA-Co/Gold.
  • Generalization Results: SAM 3 achieves superior mean BoxAP across all 13 OdinW13 datasets and surpasses leading methods on Roboflow-100VL in few-shot and full fine-tuning settings.It underperforms the current state of the art in Roboflow-100VL zero-shot evaluation.

F.5 Video PCS Details

The video PCS evaluation tests SAM 3 on dedicated and public benchmarks using concept prompts and tracking-aware metrics. Results show strong performance across benchmarks, with temporal disambiguation improving video PCS and broader visual segmentation capabilities extending beyond PCS.

  • Benchmarks and metrics: SAM 3 is evaluated on SA-Co/VEval and public video benchmarks using cgF1, pHOTA, TETA, and official dataset metrics.SA-Co/VEval includes SA-V, YT-Temporal-1B, and SmartGlasses subsets with thousands of noun phrases.
  • Benchmark results: SAM 3 largely outperforms the evaluated baselines, including on frame-level detection and cross-frame association in SA-Co/VEval.Module replacement comparisons indicate that both SAM 3’s detector and tracker contribute to final video performance.
  • Temporal disambiguation: Temporal disambiguation strategies boost video PCS performance, especially under pHOTA, and improve qualitative video outputs.The comparison disables all temporal disambiguation strategies to measure their contribution.
  • Additional video segmentation tasks: SAM 3 also brings gains on all evaluated VOS datasets, including challenging and long-video benchmarks, and improves over SAM 2 in interactive video segmentation.The interactive evaluation uses nine zero-shot datasets and three clicks per interacted frame.
  • Agent extension: SAM 3 Agent extends the system to complex natural-language segmentation by iteratively querying an MLLM and SAM 3.The agent can plan, invoke segmentation, inspect visual and textual feedback, revise its plan, and finalize masks.

G.2 Qualitative Analysis

The qualitative analysis examines SAM 3 Agent’s reasoning-based segmentation behavior and its detailed interaction with SAM 3. Results indicate strong zero-shot performance, while annotation bias explains many low-IoU cases on RefCOCO-Seg.

  • Agent capabilities: SAM 3 Agent handles free-form queries with spatial and logical complexity, rejects unmatched queries, and produces multiple masks when several objects match.The MLLM backbone supports queries varying in length and format and helps segment fine-grained concepts.
  • Reasoning trace: In the complete reasoning trace, the agent simplifies unsuccessful prompts before using SAM 3’s intermediate masks to identify the requested object.The original query “the stronger Mario” yields no masks directly; “pixel character” produces four masks for selection.
  • Quantitative and error analysis: SAM 3 Agent achieves the best reported zero-shot results on ReasonSeg and OmniLabel and surpasses previous zero-shot state-of-the-art results on RefCOCO+ and RefCOCOg.On RefCOCO, the analysis attributes many low-IoU predictions to annotation bias and ground-truth errors rather than solely model behavior.

H.2 Results

SAM 3 Multiplex replaces independent per-object video memory processing with shared-memory processing. It improves many video segmentation results and scales substantially better as the number of tracked objects increases, though public PCS changes are mixed.

  • Video PCS: +2.7 cgF1 and +0.9 pHOTA are the largest reported SA-Co/VEval gains for SAM 3 Multiplex, occurring on YT-Temporal-1B.Multiplex improves across all three SA-Co/VEval test splits.
  • Video PCS: SAM 3 Multiplex improves OVIS by +1.8 mAP but slightly regresses on LVVIS, BURST, and YTVIS21.The public-benchmark results are mixed despite consistent gains on the SA-Co/VEval splits.
  • Video object segmentation: SAM 3 Multiplex improves VOS performance on six of seven benchmarks, including +1.2 on MOSEv1 and +2.0 on MOSEv2.The largest gains occur on the challenging MOSE benchmarks.
  • Inference efficiency: At 128 tracked objects, SAM 3 Multiplex achieves a 5.2× speedup over SAM 3 on a single H100 GPU.Multiplex adds approximately 7% latency for single-object tracking but outperforms SAM 3 beyond two objects.

I.1 Data annotation card

The annotation card describes PCS as an inherently ambiguous task requiring completeness, validity, and exhaustivity checks. SA-Co/Gold uses multiple annotator reviews, while the annotator pool and regional coverage impose representation limits.

  • Task ambiguity: PCS annotations may admit multiple valid interpretations, such as whether associated objects should be included in a person mask.SA-Co/Gold records reviews from three annotators to capture this ambiguity.
  • Annotator assumptions: Annotators were expected to be English-proficient, experienced in segmentation, detail-oriented, and willing to research unfamiliar concepts.They were instructed to reject objectionable content and flag harmful or offensive concepts.
  • Representation and bias: Annotators were based in APAC and EMEA, and their regional language and concept differences may affect the annotated data.The annotation pool does not represent all communities that will use SAM 3.
  • Quality criteria: Annotation quality is defined by mask quality, concept correctness, and exhaustivity.These criteria cover mask integrity, matching the requested concept, and including all relevant instances.
  • Quality control: The dataset uses detailed guidelines, rejection of uncertain or vague jobs, manual review, and correction workflows to validate annotations.Video annotations are accepted only after meeting the stated criteria across all three quality axes.
Loading 2511.16719v2…