Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively

• SAM cannot recognize its predicted segments, while CLIP representations are challenging to adapt to dense prediction; naive combinations are costly and weak on small-object recognition.Prior classification-head solutions are constrained to specific datasets or closed-set settings, and prior dense-prediction approaches tend to be dataset-specific.Source: §1 · pp. 1–2
• SAM2CLIP distills SAM knowledge into a lightweight transformer-like adapter while CLIP stays frozen, and CLIP2SAM transfers frozen CLIP features to the SAM decoder.SAM2CLIP uses pixel-wise distillation on multi-scale features; CLIP2SAM uses a jointly trained feature-pyramid adapter with RoIAlign.Source: §1 · p. 2
• Training with COCO, LVIS, and ImageNet-22k is presented as enhancing recognition, while the SAM adaptation supports integration with various detectors in closed- and open-set environments.The paper describes the model as able to segment and recognize objects and as suitable for both settings.Source: §1 · pp. 2–3
• The proposed architecture is a single-encoder design, and its SAM encoder is discarded during inference.Fig. 2 contrasts it with two simple SAM–CLIP combination baselines.Source: Figure 2 · p. 3
• Open-Vocabulary SAM improves COCO IoU by over 2% and mAP by 3% with various detectors, and improves LVIS recognition over previous adapters by over 20%.The introduction also reports higher recognition performance and lower computational cost than direct SAM–CLIP combinations with image or feature cropping on the COCO open-vocabulary benchmark.Source: Figure 1 · p. 2, §1 · p. 3

• Vision-language models align image and text representations; open-vocabulary dense prediction uses that alignment to recognize text-described region concepts across detection and segmentation tasks.Prior approaches include CLIP distillation, frozen-CLIP detectors, and mask-proposal classification, while most are trained and tested on specific datasets.Source: §2 · pp. 3–4
• Visual prompting spans in-context methods and SAM’s point, box, and mask prompts, but SAM itself predicts class-agnostic masks.Other work on efficient parameter tuning typically focuses on a single model; this paper bridges CLIP and SAM.Source: §2 · p. 4
• Prior work has used SAM and CLIP for segmentation, but the paper identifies its approach as the first attempt to unify VLMs and SAM for segmentation and recognition of novel classes.The related-work discussion also describes SAM as an interactive tool used in grounding, tracking, distillation, medical analysis, and generation.Source: §2 · p. 4

3.1 Preliminaries and Baselines

• The combined baselines classify SAM masks using CLIP, while adapter-based fine-tuning addresses their dataset-specific knowledge gap.One baseline classifies resized masked images; the other pools CLIP features cropped by SAM masks.Source: §3.1 · p. 6

3.2 Open Vocabulary SAM

• A unified architecture uses a frozen CLIP visual encoder with SAM’s prompt encoder and mask decoder, addressing the cost of two independent backbones.SAM2CLIP transfers SAM knowledge to CLIP, while CLIP2SAM transfers CLIP knowledge to SAM’s decoder.Source: §3.2 · p. 6
• SAM2CLIP distills SAM-Huge features into frozen CLIP with an adapter, while CLIP2SAM uses CLIP knowledge to improve SAM-decoder recognition.CLIP2SAM adds a label token that is refined against CLIP text embeddings; a lightweight feature pyramid addresses small-object recognition using multi-scale features and RoI-Align.Source: §3.2 · p. 7
• Open-vocabulary inference uses the frozen CLIP backbone to recognize novel objects and geometrically combines CLIP and CLIP2SAM class scores.The method generates masks and labels from visual prompts such as boxes and points, targeting foreground objects.Source: §3.2 · p. 8

3.3 Training and Application

• Training first distills SAM knowledge with SA-1B (1%), then jointly trains CLIP2SAM and the mask decoder on COCO or LVIS annotations.ImageNet joint training supports classification over 22,000 categories; inference uses point or box prompts.Source: §3.3 · p. 8

4 Experiments

• The evaluation uses COCO and LVIS, including COCO's 48 annotated base classes and 17 unannotated target classes, and reports recognition accuracy and prompt-mask IoU.Training proceeds from SAM2CLIP on 1% of SA-1B to COCO or LVIS fine-tuning; joint SAM and COCO training is omitted due to computation costs.Source: §4 · p. 9

4.1 Main Results

• On LVIS, Open-Vocabulary SAM achieves over 20% accuracy improvement over previous adapters; on COCO with Detic prompts, it gains 3.0 mask mAP over previous works.It also reports 2% IoU improvements with ground-truth box prompts and performs best on boxes and points among the combined baselines.Source: §4.1 · p. 11
• Open-Vocabulary SAM performs strongly against open-vocabulary instance-segmentation methods, compares favorably with SAM across detectors, and scales to over 22,000 classes.The large-vocabulary setting adds V3Det, Object365, and ImageNet-22k data; the authors report no baseline comparisons for it because of cost.Source: §4.1 · p. 12

4.2 Ablation Studies and Analysis

• Adding CLIP2SAM and joint mask-classification and mask-prediction co-training improves segmentation and classification after shared-backbone adaptation, with negligible additional compute cost.The shared-backbone variant reduces parameters and FLOPs but slightly lowers segmentation performance, which the authors attribute to the SAM–COCO domain gap during SAM2CLIP training.Source: Table 6 · p. 11, §4.2 · p. 13
• SAM2CLIP benefits most from multi-scale features, which combine high-resolution features with high-level semantics for aligning CLIP and SAM features.The CLIP2SAM alternatives outperform the combined baseline but struggle with small objects; the proposed design improves performance considerably.Source: Table 7 · p. 12, §4.2 · p. 13
• ConvNext large performs best among the tested CLIP backbones, while RN50x16 offers comparable performance with better efficiency.The study motivates CNN-based CLIP backbones by their structural information and the location-sensitive visual prompts.Source: §4.2 · pp. 13–14

5 Conclusion

• Open-Vocabulary SAM takes box or point inputs to interactively segment and label visual prompts, and supports open-vocabulary and closed-set detectors.Source: §5 · p. 14
• With more data, the method performs large-vocabulary segmentation and recognition over 22K classes and can serve as an annotation tool for segmentation and instance labeling.Source: §5 · p. 14
• The authors present Open-Vocabulary SAM as a baseline for combining different forms of vision foundation models and encouraging further research.Source: §5 · p. 14

Appendix

• Section S1 provides further details and discussion of the method.Source: Appendix · p. 15
• Section S2 reports additional results.Source: Appendix · p. 15
• Section S3 presents a demo and discusses future work.Source: Appendix · p. 15

S1.1 Comparison with Recent Joint SAM and CLIP Models

• Unlike Recognize Anything, which focuses on image tagging and uses SAM as an external module, Open-Vocabulary SAM supports box- or mask-level recognition.The passage says Recognize Anything cannot perform well with box- or mask-level recognition [83].Source: S1.1 Comparison with Recent Joint SAM… · p. 15
• Unlike SAM-CLIP, Open-Vocabulary SAM supports instance-level classification and interactive segmentation while retaining the original CLIP and scaling beyond its frozen capability.The passage characterizes SAM-CLIP's classification as image-level and says it falls short of original CLIP [59].Source: S1.1 Comparison with Recent Joint SAM… · p. 15
• Unlike SSA, Open-Vocabulary SAM takes visual prompts rather than coarse semantic-segmentation results and does not require an external segmentor.Open-Vocabulary SAM focuses on instance segmentation, whereas SSA refines semantic-segmentation results.Source: S1.1 Comparison with Recent Joint SAM… · p. 15
• Unlike Sambor, Open-Vocabulary SAM supports interactive segmentation without retaining both foundation models and their extra computational costs [20].The supplementary illustration shows SAM2CLIP transferring SAM knowledge to CLIP and CLIP2SAM transferring CLIP knowledge to the SAM decoder.Source: S1.1 Comparison with Recent Joint SAM… · p. 15, Figure S1 · p. 16

S1.2 Comparison with Open-Vocabulary Methods

• The method takes human visual prompts or proposals from detectors for segmentation and recognition, rather than relying on proposal generation to recall all scene objects.It can also be deployed with open-vocabulary detectors or class-agnostic detectors.Source: S1.2 Comparison with Open-Vocabulary… · p. 16
• Open-Vocabulary SAM is intended for interactive prompting, whereas prior open-vocabulary semantic-segmentation methods focus on cross-dataset evaluation.Source: S1.2 Comparison with Open-Vocabulary… · p. 16
• Table S2 compares mask mean AP on LVIS across detectors, using masks generated by each method and labels from the corresponding detectors.Source: Table S2 · p. 17

S1.3 Training and Inference Details

• Only SAM2CLIP, CLIP2SAM, and the SAM decoder are learned; the remaining parameters are frozen.The learned components have 129M, 3.8M, and 4.0M parameters, respectively, out of 304M total.Source: S1.3 Training and Inference Details · p. 17
• After SAM2CLIP distillation, the heavy SAM encoder is dropped during detection and classification fine-tuning, speeding up CLIP2SAM.The SAM encoder has 637M parameters.Source: S1.3 Training and Inference Details · p. 17
• CLIP2SAM is co-trained on detection, segmentation, and classification data, after which the model can recognize and segment over twenty thousand categories.Segmentation training uses mask and classification labels, while detection and classification training focus on the classification head.Source: S1.3 Training and Inference Details · p. 17

S2.1 More Comparison with SAM

• Open-Vocabulary SAM has comparable or slightly worse mask quality than SAM, but lower computational cost; it outperforms TAP with lower cost.Source: S2.1 More Comparison with SAM · p. 17

S2.2 More Visualization Comparison

• Open-Vocabulary SAM has an advantage over the feature-crop baseline in object-level recognition, especially for small objects, while that baseline can confuse adjacent objects.The visualization prints the class name assigned to each corresponding mask.Source: S2.2 More Visualization Comparison · pp. 17–18, Figure S2 · p. 18

S2.3 Segment and Recognize Everything

• With box prompts, the model segments identifiable objects and recognizes them, demonstrating its ability to handle diverse objects in an image.Source: S2.3 Segment and Recognize Everything · p. 18, Figure S3 · p. 19

S3.1 Failure Cases

• Failure cases include confusing subtly distinct classes, extreme occlusion, and overlapping objects.Examples include mistaking a truck for a minivan and a vase for a bowl.Source: S3.1 Failure Cases · pp. 18–19

S3.2 Short Introduction To Demo

• The online demo shows the model segmenting and recognizing varied classes across many scenes.Source: S3.2 Short Introduction To Demo · p. 19

S3.3 Future Work

• Future work may add coarse masks or language descriptions as prompts and consider background categories alongside the current foreground focus.Source: S3.3 Future Work · p. 20