Source-linked AI summary
Semantic-SAM: Segment and Recognize Anything at Any Granularity
Feng Li, Hao Zhang, Peize Sun, Xueyan Zou, Shilong Liu, Jianwei Yang, Chunyuan Li, Lei Zhang, Jianfeng Gao
TL;DR
Semantic-SAM targets the limited exploration of universal pixel-level image understanding by enabling segmentation and recognition at arbitrary granularities. It unifies diverse datasets and uses decoupled object-part classification plus multi-choice learning so one click can produce multiple semantically labeled masks. Joint training achieves broad segmentation performance improvements and demonstrates semantic awareness and granularity abundance.
Problem
Universal pixel-level image understanding remains less explored, while existing architectures and datasets limit semantic-aware, multi-granularity segmentation.
Method
Semantic-SAM consolidates seven datasets and combines decoupled object-part classification with multi-choice learning and many-to-many matching for multi-granularity masks.
Results
Joint SA-1B and COCO panoptic training gains 2.3 box AP and 1.2 mask AP, while granularity completeness exceeds SAM by more than 3.4 1-IoU.
Takeaways & Limitations
Semantic-SAM demonstrates semantic awareness and granularity abundance, and jointly trains SA-1B with classic segmentation datasets.
Takeaways & Limitations
Single-input-single-output architectures inherently restrict end-to-end prediction of multi-granularity masks.
Abstract
from arXiv · showhide
In this paper, we introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity. Our model offers two key advantages: semantic-awareness and granularity-abundance. To achieve semantic-awareness, we consolidate multiple datasets across three granularities and introduce decoupled classification for objects and parts. This allows our model to capture rich semantic information. For the multi-granularity capability, we propose a multi-choice learning scheme during training, enabling each click to generate masks at multiple levels that correspond to multiple ground-truth masks. Notably, this work represents the first attempt to jointly train a model on SA-1B, generic, and part segmentation datasets. Experimental results and visualizations demonstrate that our model successfully achieves semantic-awareness and granularity-abundance. Furthermore, combining SA-1B training with other segmentation tasks, such as panoptic and part segmentation, leads to performance improvements. We will provide code and a demo for further exploration and evaluation.
1 Introduction
Semantic-SAM addresses the limited exploration of universal pixel-level image understanding by combining semantic awareness with multi-granularity segmentation. It unifies diverse training data and introduces architecture and learning designs that let one click produce semantically labeled masks at multiple levels.
- Universal pixel-level image understanding remains less explored, requiring versatile representations, semantic meaning, and masks at multiple granularities.
- Model Architecture: Existing models fall short because fixed single-input-single-output architectures restrict end-to-end multi-granularity prediction.Postprocessing can produce multiple masks, but is described as inefficient and ineffective for granularity-aware segmentation.
- Semantic-SAM uses one click to predict masks at multiple granularities with semantic labels for both objects and parts.The demonstrated levels range from a person’s head to an entire truck.
- Multi-choice learning represents each click with multiple level-specific queries trained against ground-truth masks at different granularities through many-to-many matching.
- Decoupled classification independently models objects and parts with a shared text encoder, adapting the loss to each data type.Generic segmentation data lacks part-classification loss, while SAM data lacks classification loss.
- The model consolidates seven datasets across generic, part, and SA-1B segmentation, and joint training improves performance across datasets.Joint SA-1B and COCO panoptic training yields gains of 2.3 box AP and 1.2 mask AP, while granularity completeness exceeds SAM by more than 3.4 1-IoU.
2 Data Unification: Semantics and Granularity
Semantic-SAM unifies seven datasets spanning object-level, part-level, and multi-level mask annotations to support semantic and granularity-aware training.
- Seven datasets provide three granularity types: COCO panoptic, ADE20k panoptic, PASCAL part, PACO, PartImageNet, SA-1B, and Objects365.COCO and ADE20k provide object-level masks and labels; three part datasets provide part-level masks and labels; SA-1B provides up to six mask levels without labels; Objects365 provides abundant object labels.
- Table 1 reports the data statistics for the datasets used in Semantic-SAM.
3 Semantic-SAM
Semantic-SAM is a universal segmentation framework that combines semantic recognition with multi-granularity mask prediction. Its architecture and training scheme use multiple prompt queries, decoupled object-part classification, and many-to-many matching to produce diverse masks.
- Model: Each click is converted into an anchor-box representation and combined with K = 6 trainable content embeddings for different granularity levels.The six-query choice reflects up to six mask levels per user click.
- Model: The deformable decoder maps query features, reference boxes, and image features to outputs containing semantic categories and masks.Each output supports concept-recognition and mask-prediction losses.
- Model: Semantic-SAM accepts generic, part, and class-agnostic segmentation data through a vision encoder and query-based mask decoder.The decoder supports generic segmentation and point- or box-prompted segmentation.
- Training: Decoupled object and part classification uses a shared text encoder while adapting losses to the annotation type.Generic data omits part-classification loss, whereas SA-1B data omits classification loss.
- Training: Many-to-many matching trains multiple predictions from one click against ground-truth masks at different granularities, unlike single-match training.The method is designed to avoid restricting a click to one ground-truth mask and to generate richer output granularities.
- Training: For box prompts, noised ground-truth boxes train reconstruction of original boxes and masks; generic segmentation follows the Mask DINO pipeline.The box content uses a learnable general prompt rather than a ground-truth label embedding.
4 Experiments
Experiments evaluate Semantic-SAM on generic, part, single-granularity interactive, and multi-granularity interactive segmentation. Results show improved segmentation performance, richer mask granularities, effective many-to-many matching, and saturation beyond 15% of SA-1B training data.
- Evaluation: The model is evaluated on COCO Val2017 and a 1,000-image SA-1B subset using PQ, AP, Point, and 1-IoU metrics.Multi-granularity evaluation matches all possible ground-truth masks to predictions and averages IoU across granularities.
- Generic Segmentation: +2.2 AP box and +1.3 AP mask result from jointly training SA-1B with COCO panoptic segmentation.The improvement is reported using SA-1B at 1/10 data, with gains mainly affecting small-object detection.
- Part Segmentation: Adding SA-1B data brings a decent performance improvement on Pascal Part part segmentation.The joint-training setting uses SA-1B at 1/10 data.
- Single-granularity Interactive Segmentation: Semantic-SAM outperforms SAM on COCO Val2017 one-click mIoU under the same settings.The comparison reports 1-IoU for single-granularity interactive segmentation.
- Multi-granularity Interactive Segmentation: Many-to-many matching significantly improves 1-IoU@All Granularity and validates the strategy for learning complete granularities.The evaluation matches all possible ground-truth masks for each click with predicted masks.
- Ablations: Performance on COCO Val2017 saturates after using more than 15% of the total SA-1B data.The model therefore achieves good zero-shot performance without training on the whole SA-1B dataset.
- Visualization: Semantic-SAM produces more diverse, higher-quality masks across levels than SAM, which can output duplicate masks or artifacts such as holes and islands.Visualizations also show more meaningful granularity than the original SA-1B annotation and fixed prompt embeddings corresponding to semantic granularity levels.
5 Related works
Related work spans generic, part, open-vocabulary, and interactive segmentation. Prior systems and datasets advance individual capabilities, while Semantic-SAM combines multi-granularity data and interactive segmentation into a unified model.
- Generic Segmentation: Generic segmentation includes semantic, instance, and panoptic subdivisions supported by datasets such as MSCOCO, LVIS, Objects365, PASCAL, CityScapes, and ADE20k.Semantic segmentation labels pixels by class, instance segmentation separates same-class objects, and panoptic segmentation combines both.
- Part Segmentation: Part segmentation targets finer-grained understanding through bottom-up, top-down, and part-aware panoptic approaches.Related datasets cover domains including cars, birds, fashion, and general objects.
- Open-vocabulary Segmentation: Open-vocabulary segmentation methods use foundation-model knowledge from CLIP, ALIGN, and diffusion models across semantic, instance, panoptic, and part-level tasks.Examples include LSeg, OpenSeg, GroupViT, DenseCLIP, MaskCLIP, X-Decoder, and OpenSeeD.
- Interactive Segmentation: Interactive segmentation integrates user inputs to separate objects, with SAM scaling training data from 0.12M COCO images to 10M fine-grained images.SEEM further adds language and function for generic and grounded segmentation.
6 Conclusion
Semantic-SAM provides universal open-vocabulary segmentation with semantic awareness and abundant granularity. Joint training with SA-1B and classic segmentation datasets also improves panoptic and part segmentation.
- Semantic-SAM segments and recognizes anything at any desired granularity within a universal framework.
- The model combines datasets from multiple granularities and semantic levels with multi-choice learning and a universal modeling framework.
- Comprehensive experiments and visualizations verify Semantic-SAM’s semantic awareness and granularity abundance.
- Training jointly with SA-1B and classic segmentation datasets improves panoptic and part segmentation tasks.