PaperlayerView exact revision on arXiv

Source-linked AI summary

SAM 2: Segment Anything in Images and Videos

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer

arXiv:2408.00714v2cs.CVcs.AIcs.LG

TL;DR

Existing video segmentation systems do not provide a capability comparable to segmenting anything in videos, despite the need for temporal localization beyond images. SAM 2 unifies promptable image and video segmentation with streaming memory, achieving better video accuracy with 3× fewer interactions and better image performance than SAM while being 6× faster.

  • Problem

    Existing video segmentation models and datasets fall short of providing image-model-like capability for segmenting arbitrary entities across challenging video frames.

  • Method

    SAM 2 unifies promptable image and video segmentation through a streaming architecture with memory attention over prior object interactions and frames.

  • Results

    SAM 2 delivers better video segmentation accuracy with 3× fewer interactions and outperforms SAM on image benchmarks while being 6× faster.

  • Takeaways & Limitations

    SAM 2 provides a unified segmentation capability across images and videos with strong performance across diverse zero-shot benchmarks.

  • Takeaways & Limitations

    The data-engine approach initially required annotating intermediate-frame masks from scratch without previous memory before the fully featured SAM 2 was developed.

Abstract

from arXiv · show

We present Segment Anything Model 2 (SAM 2), a foundation model towards solving promptable visual segmentation in images and videos. We build a data engine, which improves model and data via user interaction, to collect the largest video segmentation dataset to date. Our model is a simple transformer architecture with streaming memory for real-time video processing. SAM 2 trained on our data provides strong performance across a wide range of tasks. In video segmentation, we observe better accuracy, using 3x fewer interactions than prior approaches. In image segmentation, our model is more accurate and 6x faster than the Segment Anything Model (SAM). We believe that our data, model, and insights will serve as a significant milestone for video segmentation and related perception tasks. We are releasing our main model, dataset, as well as code for model training and our demo.

1 Introduction

SAM 2 is a unified foundation model for promptable visual segmentation in images and videos, built with streaming memory and a model-in-the-loop data engine. It targets video’s temporal and visual challenges while improving interaction efficiency and segmentation performance across video and image benchmarks.

  • Contribution: SAM 2 unifies promptable segmentation for images and videos, treating an image as a single-frame video.The work includes a task, model, and dataset.
  • Task: Promptable Visual Segmentation predicts a video entity’s spatio-temporal mask from points, boxes, or masks on any frame, with iterative refinement through additional-frame prompts.The predicted spatio-temporal mask is called a masklet.
  • Model: Streaming memory stores object information and prior interactions, enabling masklet prediction throughout videos and correction using previously observed frames.The architecture extends image segmentation to video while processing video frames.
  • Data engine: 8.4× faster at comparable quality, the model-in-the-loop data engine interactively annotates new and challenging data for a dataset targeting any object with a valid boundary.Unlike most existing video segmentation datasets, it includes objects, parts, and subparts rather than restricting categories.
  • Results: 3× fewer interactions, SAM 2 achieves better video segmentation accuracy than prior approaches; it also outperforms prior work on video benchmarks and is 6× faster than SAM on image segmentation benchmarks.The experiments evaluate multiple settings across video and image segmentation.

2 Related work

Prior work established promptable image segmentation, interactive video segmentation, and semi-supervised video object tracking, but existing methods and datasets have important limitations. SAM 2 addresses these gaps with a unified prompt-driven video model and a larger, more diverse dataset covering object parts.

  • Image segmentation: SAM introduced promptable image segmentation from points or bounding boxes, with zero-shot segmentation enabled by training on the SA-1B dataset.Subsequent work extended SAM toward fine-grained masks.
  • Interactive Video Object Segmentation: Interactive video segmentation uses scribbles, clicks, or bounding boxes to obtain object masklets, evolving from graph-based optimization toward newer prompt-guided approaches.Click-based input is easier to collect, but SAM-plus-tracker systems may fail on some objects, perform poorly on video frames, or lack interactive error refinement.
  • Interactive Video Object Segmentation: SAM 2 pursues interactive video object segmentation with a unified model that directly accepts prompts, supported by a large and diverse dataset.This goal is shared with earlier interactive segmentation work, while the proposed system unifies the model and data around the task.
  • Video Object Segmentation: Semi-supervised VOS tracks an object through a video from a first-frame mask, making it a special case of promptable visual segmentation but requiring a costly high-quality annotation.The first-frame mask serves as the supervision signal for the target object.
  • Video segmentation datasets: Existing video segmentation datasets often lack coverage for segmenting anything in videos, focusing on whole objects and classes such as people, vehicles, and animals.SA-V also covers object parts and contains over an order of magnitude more masks than these datasets.

3 Task: promptable visual segmentation

The promptable visual segmentation task lets users define or refine an object with prompts on any video frame, while the model immediately produces and propagates masks across the video. SAM 2 supports data collection for SA-V and evaluation in interactive video, semi-supervised VOS, and image-segmentation settings.

  • Task definition: Prompts on any video frame can define or refine an object using positive/negative clicks, boxes, or masks.The model should immediately return a valid segmentation mask on the prompted frame.
  • Task definition: After initial prompts, SAM 2 propagates them across the entire video to localize the target mask on every frame.Additional prompts on any frame can refine the segment throughout the video.
  • Applications and evaluation: SAM 2 serves as a data-collection tool for building the SA-V dataset and is evaluated in interactive video segmentation, semi-supervised VOS, and image segmentation.The semi-supervised VOS setting limits annotations to the first frame, while image segmentation uses the SA benchmarks.

4 Model

SAM 2 generalizes SAM to spatio-temporal segmentation by combining frame prompts with streaming memory of past predictions and prompted frames. Its architecture uses a lightweight SAM-like decoder, memory attention, hierarchical image features, and video-specific mechanisms for ambiguity and object absence.

  • Promptable segmentation: SAM 2 accepts point, box, and mask prompts on individual frames to define an object’s spatio-temporal extent, while iterative prompts refine frame masks.A lightweight mask decoder combines image embeddings with prompts to produce each frame’s segmentation mask.
  • Memory attention: Streaming memory conditions each frame embedding on past predictions, prompted frames, and object pointers before mask decoding.A memory encoder writes predictions into banks that memory attention cross-attends to alongside the current image encoding.
  • Image encoder: The image encoder processes each video frame once as it becomes available and supplies unconditioned multiscale features from an MAE-pre-trained hierarchical Hiera encoder.This streaming design supports real-time processing of arbitrarily long videos.
  • Prompt encoder and mask decoder: The decoder predicts multiple masks for ambiguous prompts and adds an object-presence head for frames where no valid target exists, such as during occlusion.Hierarchical encoder skip connections also provide high-resolution embeddings for mask decoding.
  • Memory bank: The memory bank retains up to N recent-frame memories and up to M prompted-frame memories in separate FIFO queues, supplemented by lightweight object pointers.Recent-frame memories encode temporal position information to represent short-term object motion, whereas prompted-frame memories do not.
  • Training: Training jointly uses image and video data with simulated interactive prompting over sequences of 8 frames and up to 2 prompted frames.Initial prompts are sampled as ground-truth masks with probability 0.5, positive clicks with probability 0.25, or bounding boxes with probability 0.25.

5 Data

The authors build an interactive, human-in-the-loop data engine that evolves from per-frame SAM annotation to memory-enabled SAM 2 refinement, producing a large and diverse video segmentation dataset. Iteratively adding data from each phase improves segmentation accuracy across in-domain and zero-shot benchmarks.

  • Data engine: The data engine collects diverse masklets without semantic constraints, covering both whole objects and object parts through interactive human annotation.The engine uses an interactive model-in-the-loop setup and focuses on video segmentation data diversity.
  • Data engine: The three phases progress from per-frame SAM annotation, to SAM 2 mask propagation, to fully featured SAM 2 with temporal memory and occasional refinement clicks.Phase 2 still requires annotating intermediate masks from scratch, whereas Phase 3 uses object memories across time.
  • Data engine: The pipeline adds quality verification, returning unsatisfactory masklets for refinement, and automatically generates masklets from grid-point prompts to increase coverage and expose failure cases.A separate annotator verifies whether each masklet tracks the target consistently across all frames.
  • Data engine: Iteratively adding data from each phase consistently improves J &F accuracy on the SA-V validation set and 9 zero-shot benchmarks under fixed iterations.The evaluation uses the standard J &F metric with 3-click prompts on the first frame.
  • SA-V dataset: 50.9K videos and 642.6K masklets comprise SA-V, with 53× more annotated masks than any existing VOS dataset, or 15× excluding automatic annotations.The dataset includes 190.9K manual and 451.7K automatic masklets; SA-V Manual has a 42.5% disappearance rate.

6 Zero-shot experiments

SAM 2 outperforms prior methods across zero-shot interactive and semi-supervised video segmentation tasks. On zero-shot image segmentation, it improves accuracy over SAM, runs 6× faster, and reaches higher accuracy with combined image-video training data.

  • Promptable video segmentation: >3× fewer interactions are required for SAM 2 to generate better video segmentation accuracy than the two baseline methods.SAM 2 maintains high-quality segmentation while allowing continued refinement with prompts.
  • Semi-supervised video object segmentation: SAM 2 outperforms XMem++ and Cutie on 17 video datasets across click, box, and mask prompts in semi-supervised video object segmentation.This includes the conventional non-interactive VOS setting with mask input.
  • Segment Anything task: 58.9 mIoU with 1 click exceeds SAM’s 58.1 mIoU with 1 click on 23 datasets, while SAM 2 is 6× faster without extra data.The speed improvement is mainly attributed to SAM 2’s smaller but more effective Hiera image encoder.
  • Segment Anything task: 61.4% average accuracy is achieved on the 23 datasets when training uses the combined SA-1B and video data mix.The combined training data also produces exceptional gains on video benchmarks.

7 Comparison to state-of-the-art in semi-supervised VOS

SAM 2 is evaluated under the historically common semi-supervised VOS protocol and shows significant improvements over existing methods. Larger image encoders improve accuracy, while SAM 2 also demonstrates gains on open-world and long-term video benchmarks.

  • Protocol and models: SAM 2 evaluates semi-supervised VOS using ground-truth masks on the first frame, with Hiera-B+ and Hiera-L encoders spanning speed–accuracy tradeoffs.FPS is measured on a single A100 GPU with batch size one.
  • VOS comparison: SAM 2 shows significant improvement over the best existing methods under standard VOS comparison protocols.Table 6 reports video-segmentation accuracy using J &F and G.
  • VOS comparison: Larger image encoders bring significant accuracy gains across the board.The comparison evaluates SAM 2 variants with different image encoder sizes.
  • Open-world and long-term VOS: SAM 2 significantly outperforms prior work on SA-V val/test and brings notable gains in long-term video object segmentation on LVOS.SA-V measures open-world segmentation of any object class, while prior methods peak at approximately the same accuracy and remain below SAM 2.

8 Conclusion · Appendix · A Data and model ablations

The conclusion presents SAM 2 as a video-domain evolution of Segment Anything built around promptable video segmentation, memory-equipped architecture, and the SA-V dataset. The appendix introduces ablations evaluating design choices across multiple video benchmarks under a standardized 3-click J &F comparison.

  • 8 Conclusion: SAM 2 extends the promptable segmentation task from images to video.
  • 8 Conclusion: SAM 2 equips the SAM architecture to use memory when applied to video.
  • 8 Conclusion: The diverse SA-V dataset supports training and benchmarking for video segmentation.
  • 8 Conclusion: The authors characterize SAM 2 as a significant advancement in visual perception and a milestone for further research and applications.
  • Appendix: The appendix lists data and model ablations, annotation guidelines, zero-shot experiment details, and dataset, annotation, and model cards.
  • A Data and model ablations: The ablation section evaluates SAM 2 design decisions on SA-V val, Internal-test, MOSE dev, and the average across 9 zero-shot video datasets.
  • A Data and model ablations: 3-click input on the first frame is used with J &F as a comparison metric balancing the 1-click regime and VOS-style mask prompts.

A.1 Data ablations … B Details on the PVS Task

The ablations show that data composition, scaling, model capacity, positional encoding, and memory design govern SAM 2’s image and video segmentation performance. The PVS formulation unifies image segmentation and video object segmentation through interactive clicks, boxes, and masks across video frames.

  • A.1 Data ablations: A model trained only on existing VOS datasets performs well on in-domain MOSE but poorly on the nine zero-shot VOS datasets and other evaluation sets.The reported MOSE score is 59.7 J &F.
  • A.1 Data ablations: +12.1% average performance improvement on nine zero-shot datasets results from adding data-engine components to the training mixture, while SA-1B images improve image segmentation without degrading VOS.The best results come from mixing VOS, SA-1B, and data-engine data; SA-V plus SA-1B is strong except on MOSE.
  • A.1 Data ablations: A consistent power-law relationship links the quantity of SA-V training data to video segmentation accuracy on SA-V val, nine zero-shot datasets, and MOSE dev.Accuracy is measured with 3-click first-frame prompts using average J &F across the three benchmark groups.
  • A.1 Data ablations: Filtering SA-V masklets by the number of edited frames outperforms random sampling with 50k masklets, performs strongly using 25% of the data, but remains below all 190k masklets.The comparison uses SA-V Manual data subsets.
  • A.2 Model architecture ablations: The architecture ablations evaluate video J &F, image mIoU, and relative video throughput under a smaller 512-input-resolution setup, revealing largely decoupled image and video design choices.The model is modular, enabling separate image and video component decisions.
  • A.2.1 Capacity ablations: Higher resolution improves image and video tasks, more frames improve video benchmarks, and the final model uses 1024^2 resolution with a default of 8 frames.The frame count balances speed and accuracy.
  • A.2.1 Capacity ablations: More memory capacity and larger image encoders generally improve performance, while fewer memory channels reduce storage requirements 4× with little regression; removing image-encoder RPB preserves performance while accelerating 1024-resolution inference.Scaling memory-attention improves video metrics, whereas scaling the image encoder improves both image and video metrics; 2d-RoPE is used in memory attention by default.
  • A.2.3 Memory architecture ablations: GRU-based recurrent memory does not improve average performance except slightly on LVOSv2, whereas object pointers help SA-V val and LVOSv2 but not average performance across nine zero-shot datasets.SAM 2 therefore defaults to cross-attending to object pointers together with memory.

C Limitations … D.2.1 Pre-training

SAM 2 has limitations in recovering object identity and handling multiple objects, while its architecture and pre-training introduce mechanisms for high-resolution decoding, memory-based propagation, and improved IoU supervision.

  • C Limitations: SAM 2 may fail across shot changes or lose and confuse objects in crowded scenes, after long occlusions, or in extended videos.Refinement clicks on additional frames can quickly recover the correct prediction when errors occur.
  • C Limitations: SAM 2 processes each tracked object separately, using shared per-frame embeddings without inter-object communication.Shared object-level contextual information could improve efficiency.
  • C Limitations: Human annotators verify masklet quality and select frames requiring correction in the data engine.Automating this process could enhance efficiency.
  • D SAM 2 details: The architecture details expand the model description in §4.The subsequent components describe image encoding, memory attention, prompt encoding, mask decoding, and memory storage.
  • D.1 Architecture: Stride 4 and 8 image-encoder features are added to mask-decoder upsampling to produce high-resolution segmentation details.Stride 16 and 32 features are fused by a feature pyramid network for per-frame image embeddings.
  • D.1 Architecture: L = 4 memory-attention layers use sinusoidal absolute positional embeddings and 2d spatial RoPE, excluding object pointer tokens from RoPE.Object pointer tokens lack specific spatial correspondence.
  • D.1 Architecture: The mask token becomes the frame’s object pointer in the memory bank, while an additional token and MLP head produce an occlusion score.The occlusion score indicates whether the object of interest is visible in the current frame.
  • D.2.1 Pre-training: During SA-1B pre-training, masks covering more than 90% of an image are filtered and training is restricted to 64 randomly sampled masks per image.The image encoder is initialized from MAE-pre-trained Hiera.

D.2.2 Full training … E.1 SA-V dataset details

SAM 2 is jointly trained on image and video data using alternating batches, interactive prompt simulation, and long-sequence fine-tuning. Benchmarking uses standardized GPU settings, while SA-V combines diverse videos with automatically generated masklets.

  • D.2.2 Full training: SAM 2 is trained on SA-V, Internal data, 10% of SA-1B, and open-source video datasets including DAVIS, MOSE, and YouTubeVOS.The released model uses SA-V manual labels, Internal data, and SA-1B.
  • D.2.2 Full training: Training jointly alternates full batches from image and video datasets, with sampling probabilities proportional to dataset size.This strategy optimizes data usage and computational resources while supporting the PVS and SA tasks.
  • D.2.2 Full training: 8-frame sequences simulate interactive segmentation using ground-truth masklets, predictions, and initial prompts comprising masks, positive clicks, or bounding boxes.Initial prompts are sampled as ground-truth masks with 50% probability, positive clicks with 25%, and bounding boxes with 25%.
  • D.2.2 Full training: 16-frame fine-tuning targets challenging videos with the highest number of edited frames to improve segmentation quality on long videos.This addresses the limitation of training primarily on relatively short 8-frame sequences.
  • D.2.2 Full training: 20:1:1:1 weights combine focal-plus-dice mask loss, MAE IoU loss, and cross-entropy object loss during full training.For multi-mask predictions, supervision selects the mask with the lowest segmentation loss.
  • D.3 Speed benchmarking: Benchmarking runs on a single A100 GPU with PyTorch 2.3.1, CUDA 12.1, bfloat16 mixed precision, and compiled image encoders.SA-task FPS uses batch size 10, while video-task benchmarking uses batch size 1.
  • E.1 SA-V dataset details: SA-V videos span 240p to 4K resolution and 4 seconds to 2.3 minutes, totaling 4.2M frames across 196 hours.The average resolution is 1,401 × 1,037, and average duration is 13.8 seconds.
  • E.1 SA-V dataset details: 47-country collection gives SA-V diverse coverage, with more than 88% of masks occupying less than 0.1 of normalized video area.The dataset compares mask-size distributions with DAVIS, MOSE, and YouTubeVOS.

E.1.1 Fairness evaluation … F.3.1 Dataset details

The appendices evaluate SAM 2’s fairness, annotation data engine, and zero-shot transfer across diverse video and image benchmarks. They report interactive protocols, dataset composition, baseline comparisons, and performance caveats.

  • E.1.1 Fairness evaluation: SAM 2’s fairness evaluation uses Ego-Exo4D third-person 20-second clips with self-reported demographics and tests 1-click, 3-click, and first-frame ground-truth mask prompts.At 3 clicks and with ground-truth mask prompts, demographic discrepancy is minimal; among correctly segmented clips, 1-click J &F is 94.3 for males and 92.7 for females.
  • E.2 Data engine details; E.2.1 Annotation protocol: The annotation protocol separates object selection, masklet tracking, and quality verification among annotators, with SAM 2 deployed interactively to target challenging objects.Annotators were asked to select objects requiring at least 2 edited frames, while selection and consistent video-wide masklet annotation were decoupled.
  • E.2.2 Data engine phase comparison: The data-engine phase comparison is a controlled experiment on 169 videos and 452 masklets, categorized by first-frame mask area into small, medium, and large buckets.Phase 1 provides the quality reference because its masks were produced through frame-by-frame annotation.
  • F Details on zero-shot transfer experiments; F.1 Zero-shot video tasks; F.1.1 Video dataset details: Unless otherwise noted, zero-shot experiments use the SAM 2 Hiera-B+ configuration with 1024 resolution and the full dataset combination.The video evaluations cover 17 diverse zero-shot datasets, including medical, robotic, long-term, open-vocabulary, open-world, and challenging-motion data.
  • F.1.2 Interactive offline and online evaluation details: Interactive offline evaluation repeatedly prompts the lowest-IoU frame, whereas online evaluation propagates prompts in one pass and pauses when prediction IoU falls below 0.75.Both settings use 3 clicks per interacted frame, evaluate independently per object on 9 densely annotated datasets, and report J &F over 1 through 8 interacted frames.
  • F.1.4 SAM+XMem++ and SAM+Cutie baseline details: SAM 2 outperforms SAM+XMem++ and SAM+Cutie with a notable margin across all interactive offline and online datasets and settings.The baselines obtain an initial or corrected mask from SAM and track it with XMem++ or Cutie.
  • F.2 DAVIS interactive benchmark; F.3 Zero-shot image tasks; F.3.1 Dataset details: SAM 2 also outperforms MiVOS and CiVOS on DAVIS under click inputs, although first-click part segmentation can penalize performance on whole-object videos; the image evaluation suite contains 37 datasets.The DAVIS comparison reports J &F@60s and AUC-J &F, while the image suite combines 23 datasets previously used by SAM with additional datasets listed in the appendix.

F.3.2 Detailed zero-shot experiments

Detailed zero-shot experiments compare SAM 2 with SAM and HQ-SAM across 37 datasets using 1- and 5-click mIoU. SAM 2 improves accuracy and speed, with further gains from mixed image-video training and strong per-dataset performance.

  • Experimental setup: SAM 2 is evaluated against SAM and HQ-SAM using 1- and 5-click mIoU across 23 established and 14 newly introduced video datasets.Results are categorized by dataset domain and model image-encoder size.
  • Image-only models: 6x faster, SAM 2 (Hiera-B+) trained only on SA-1B surpasses SAM (ViT-H) at 1-click accuracy and both SAM (ViT-H) and HQ-SAM (ViT-H) at 5-click accuracy.SAM 2 (Hiera-L) further improves 1-click accuracy by 1 point on average, while remaining 3.4x faster than SAM (ViT-H) and 1.5x faster than SAM (ViT-B).
  • Mixed image-video training: 61.4% average accuracy is achieved across the 23 datasets when using the Hiera-B+ image encoder with mixed image and video training.The mixed-data models also show substantial improvements on the SA-23 video benchmarks and the 14 newly introduced video datasets.
  • Limitations: The experiments do not scale beyond Hiera-L, although larger models are expected to provide better performance.This establishes the explored model-size limit rather than a measured result beyond Hiera-L.
  • Per-dataset analysis: 29 datasets show SAM 2 (Hiera-B+) surpassing SAM by up to 53.9 mIoU in 1-click accuracy despite its smaller image encoder.Figure 15 color-codes each dataset’s 1-click mIoU delta relative to SAM by image or video data type.

G Details on comparison to state-of-the-art in semi-supervised VOS · H Model, data and annotation cards · H.1 Model card

The appendix compares SAM 2 with prior semi-supervised VOS methods, showing strong qualitative and quantitative performance while noting an OVIS data-overlap caveat. It also provides model, data, and annotation-card materials, including a model card for SAM 2.

  • G Details on comparison to state-of-the-art in semi-supervised VOS: SAM 2 restricts the propagated masklet to the target object, whereas Cutie-base+ incorrectly propagates a shirt-only first-frame mask to the whole person.The comparison uses a mask prompt in the first frame.
  • G Details on comparison to state-of-the-art in semi-supervised VOS: SAM 2 outperforms prior work on all evaluated semi-supervised VOS benchmarks and metrics.The comparison covers additional semi-supervised VOS metrics.
  • G Details on comparison to state-of-the-art in semi-supervised VOS: Unlike previous approaches, SAM 2 is not specialized for semi-supervised VOS and supports more general promptable segmentation without restriction to specific object classes.These properties accompany the reported benchmark and metric performance.
  • G Details on comparison to state-of-the-art in semi-supervised VOS: OVIS is not strictly zero-shot for SAM 2 because its videos are used in MOSE, part of the training-data mix.This is a stated evaluation caveat.
  • H Model, data and annotation cards: The appendix documents video segmentation datasets used for zero-shot evaluation and provides examples from the SAM 2 zero-shot video benchmark suite.These materials are presented as Table 16 and Figure 17.
  • G Details on comparison to state-of-the-art in semi-supervised VOS: The appendix reports SAM 2 comparisons on the SA-V, LVOS, and LVOSv2 benchmarks, including evaluations using open-sourced code and checkpoints or author-reported results.The comparisons cover SAM 2 versus previous work for semi-supervised VOS.
  • H.1 Model card: The appendix includes a model card for SAM 2 following the structure proposed by Mitchell et al. (2019).The model card appears as Table 18.

H.2 Dataset card for SA-V dataset · H.3 Data annotation card

SA-V is a large public video segmentation dataset designed for promptable video segmentation, comprising 50.9K videos and 642.6K masklets. Its documentation describes annotation procedures, collection and preprocessing, intended uses, release terms, and limitations.

  • H.2 Dataset card for SA-V dataset: 50.9K videos and 642.6K masklets make SA-V the largest publicly available video segmentation dataset, created for the PVS task.The dataset is released under a permissive Creative Commons license.
  • H.3 Data annotation card: Each video is annotated at 6 FPS with object-tracking masklets, averaging 3.8 manual and 8.9 auto masklets per video without categories or text labels.Auto masklets are generated using SAM 2, while manual masklets are produced by annotators.
  • H.3 Data annotation card: The dataset documentation notes human errors in manual masklets and model inconsistencies in auto masklets as potential annotation noise.Reviewers were advised to refuse explicit imagery, and videos were subjected to face blurring.
  • H.3 Data annotation card: Crowdworkers recorded the videos through a contracted third-party vendor, while annotators from another vendor manually collected masklets and were paid hourly wages.Videos were filmed from November 2023 to March 2024, and masklet annotations were collected from April to July 2024.
  • H.3 Data annotation card: The videos were resampled to 24 fps and converted to MP4 before release, while the underlying recording equipment was unknown.The dataset’s raw data was not saved in addition to the processed data.
  • H.2 Dataset card for SA-V dataset: SA-V has been used to train and evaluate SAM 2 and can support VOS, iVOS, PVS, and image segmentation when frames are sampled.The dataset consists entirely of videos and encourages subject-matter diversity without specific collection themes.
  • H.2 Dataset card for SA-V dataset: The dataset is not parity-balanced across all geographic and demographic groups, so users are encouraged to consider these limitations when applying it.The documentation reports an analysis of dataset geography and crowdworker demographics.
  • H.2 Dataset card for SA-V dataset: SA-V is distributed under the Creative Commons Attribution 4.0 International Public License, hosted and maintained by Meta FAIR, with no support for older versions after updates.External contributions are encouraged but will not be validated, hosted, maintained, or supported by Meta.
Loading 2408.00714v2…