Source-linked AI summary
SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding
Haoxiang Wang, Pavan Kumar Anasosalu Vasu, Fartash Faghri, Raviteja Vemulapalli, Mehrdad Farajtabar, Sachin Mehta, Mohammad Rastegari, Oncel Tuzel, Hadi Pouransari
TL;DR
Separate vision foundation models provide complementary semantic and spatial capabilities but impose deployment costs. The paper merges SAM and CLIP using multi-task distillation with memory rehearsal, producing SAM-CLIP, which retains their zero-shot abilities and achieves state-of-the-art zero-shot semantic segmentation while using one unified backbone.
Problem
Vision foundation models provide complementary capabilities, but maintaining separate models is inefficient and multitask training is costly.
Method
The paper merges SAM and CLIP through multi-task distillation and memory rehearsal using limited replay data.
Results
SAM-CLIP retains the zero-shot capabilities of SAM and CLIP and achieves state-of-the-art zero-shot semantic segmentation across five datasets.
Takeaways & Limitations
The unified model combines semantic and spatial capabilities in one backbone while reducing inference memory and computational costs.
Takeaways & Limitations
The method assumes access to limited subsets of the base and auxiliary VFMs’ training datasets or their proxies.
Abstract
from arXiv · showhide
The landscape of publicly available vision foundation models (VFMs), such as CLIP and Segment Anything Model (SAM), is expanding rapidly. VFMs are endowed with distinct capabilities stemming from their pre-training objectives. For instance, CLIP excels in semantic understanding, while SAM specializes in spatial understanding for segmentation. In this work, we introduce a simple recipe to efficiently merge VFMs into a unified model that absorbs their expertise. Our method integrates techniques of multi-task learning, continual learning, and distillation. Further, it demands significantly less computational cost compared to traditional multi-task training from scratch, and it only needs a small fraction of the pre-training datasets that were initially used to train individual models. By applying our method to SAM and CLIP, we obtain SAM-CLIP: a unified model that combines the capabilities of SAM and CLIP into a single vision transformer. Compared with deploying SAM and CLIP independently, our merged model, SAM-CLIP, reduces storage and compute costs for inference, making it well-suited for edge device applications. We show that SAM-CLIP not only retains the foundational strengths of SAM and CLIP, but also introduces synergistic functionalities, notably in zero-shot semantic segmentation, where SAM-CLIP establishes new state-of-the-art results on 5 benchmarks. It outperforms previous models that are specifically designed for this task by a large margin, including +6.8% and +5.9% mean IoU improvement on Pascal-VOC and COCO-Stuff datasets, respectively.
1. Introduction
Vision foundation models provide complementary capabilities, but maintaining separate models is inefficient and jointly training new multitask foundations is costly. The paper proposes merging VFMs with limited replay data and distillation, instantiated as SAM-CLIP.
- Motivation: Vision foundation models support diverse downstream tasks and prompt-based zero-shot capabilities shaped by their pre-training objectives.CLIP supports semantic classification from text prompts, while SAM supports segmentation from geometric prompts.
- Motivation: Maintaining separate models increases memory and runtime costs and prevents cross-model learning, especially on edge devices.Multitask training can address this issue but often requires simultaneous access to all tasks and substantial computation and data.
- Research gap: Existing training-free merging methods can avoid additional training but may cause decreased performance or poor generalization, while naive weight interpolation causes significant forgetting.The paper targets merging models trained with different objectives and capabilities.
- Approach: The proposed approach treats merging as continual learning, using limited pre-training data or surrogates with multitask distillation to preserve the original model knowledge.This bridges training-free merging and multitask training through replay during the merging process.
- Instantiation: SAM-CLIP merges SAM and CLIP into one multi-task model intended for edge deployment.The unified model combines the capabilities of both vision foundation models in a single architecture.
- Results: SAM-CLIP retains CLIP and SAM zero-shot capabilities while enabling zero-shot semantic segmentation through combined semantic and spatial skills.The paper reports state-of-the-art performance on zero-shot semantic segmentation across multiple datasets with a single-stage inference setup.
2. Background
The background covers vision-language and segmentation foundation models, model merging, distillation, continual learning, and zero-shot semantic segmentation. It highlights the efficiency advantage of SAM-CLIP over composing SAM and CLIP separately.
- Vision-language models: CLIP-style vision-language models learn aligned image-text embeddings for zero-shot image-text retrieval and classification through contrastive training.Their image and text encoders produce modality-specific embeddings aligned for positive image-text pairs.
- Segmentation models: SAM provides promptable segmentation using an image encoder, prompt encoder, and mask decoder trained on image, geometric-prompt, and mask triplets.Its geometric prompts include points, mask regions, and bounding boxes.
- Segmentation models: Follow-up SAM methods mainly improve quality or efficiency for the original segmentation task, whereas SAM-CLIP targets capability merging across tasks.The cited variants include high-quality, faster, and distilled SAM models.
- Model composition: Composing SAM and CLIP for semantic segmentation requires two models, K + 1 forward passes per image, and twice the memory footprint.SAM generates K object segments, after which CLIP processes each segment for labeling.
- Related methods: Knowledge distillation and continual learning provide relevant foundations for transferring capabilities while addressing catastrophic forgetting.Continual learning commonly uses replay or regularization to preserve previously learned knowledge.
- Zero-shot semantic segmentation: Zero-shot semantic segmentation predicts dense masks from free-form text prompts without task-specific class knowledge or fine-tuning.Prior approaches commonly use image-text datasets and pretrained vision-language models.
- Efficiency: SAM-CLIP uses one ViT model and a single backbone forward pass per image, reducing inference memory and computational costs relative to model composition.This supports deployment scenarios where loading and executing two separate models is inefficient.
3. Proposed Approach
The approach merges SAM and CLIP image encoders into one backbone using limited replay data, multi-head learning, and rehearsal-based multi-task distillation. Two-stage training transfers auxiliary knowledge while preserving the base model’s capabilities.
- Model setup: SAM serves as the base VFM and CLIP as the auxiliary VFM because their spatial and semantic capabilities are complementary.The method retains SAM’s high-resolution processing while integrating CLIP’s semantic image representation.
- Replay data: Limited subsets of the SAM and CLIP training datasets, or proxies, provide memory replay during continual-learning-based merging.These replay sets are denoted D_SAM and D_CLIP.
- Model architecture: The unified architecture merges both image encoders into EncSAM-CLIP while retaining lightweight task-specific heads for SAM and CLIP.The SAM, prompt, and mask-decoder components and CLIP text encoder remain associated with their respective heads.
- Evaluation scope: The resulting model targets unified zero-shot evaluation across classification, retrieval, and instance segmentation while demonstrating minimal forgetting on the original tasks.Table 1 compares SAM-CLIP with ViT-B baseline foundation models on these three task categories.
- Training procedure: The method first freezes the merged image backbone to probe HeadCLIP, then jointly trains the encoder and heads with L_CLIP + λL_SAM.The second stage uses samples from both replay datasets, while separate losses train the corresponding capabilities.
- Forgetting mitigation: A smaller learning rate for EncSAM-CLIP and HeadSAM helps reduce forgetting while CLIP and SAM knowledge are distilled jointly.The learning rate for these parameters is an order of magnitude smaller than for HeadCLIP.
4. Experiments
SAM-CLIP is evaluated as a unified model for inherited CLIP and SAM capabilities, learned representations, and zero-shot semantic segmentation. Its two heads can also be composed to refine CLIP-based masks with SAM-based spatial detail using a single encoder pass.
- Implementation Details: SAM-CLIP integrates a lightweight three-layer CLIP head into a ViT-B/16 SAM backbone and uses max-pooled patch tokens for image-level representations.The architecture retains the SAM backbone while adding a CLIP-specific representation pathway.
- Implementation Details: SAM-CLIP uses a two-stage training procedure combining CLIP-head probing with joint backbone and head fine-tuning under CLIP and SAM distillation losses.The first stage freezes the backbone for 20 epochs; the second unfreezes it for 16 epochs with distillation-loss ratio 1:10.
- Zero-Shot Evaluations: SAM-CLIP establishes state-of-the-art zero-shot semantic segmentation results on all five evaluated datasets.The evaluation uses image features, text-template embeddings as mask classifiers, and linearly upscaled mask logits.
- Head-Probing Evaluations on Learned Representations: SAM-CLIP attains comparable image-classification probing performance with CLIP while providing richer visual features than SAM and CLIP in head-probing evaluations.The representation evaluations cover semantic segmentation and image classification tasks with ViT-B models.
- Composing Both CLIP and SAM Heads for Better Segmentation: Composing the CLIP and SAM heads refines text-prompted low-resolution masks into more fine-grained high-resolution semantic masks.The pipeline generates point prompts from CLIP predictions, feeds them with the mask to the SAM prompt encoder, and produces 256 × 256 masks after one 1024px encoder pass.
5. Conclusion
The paper proposes merging SAM and CLIP into one unified vision architecture using multi-task distillation and memory rehearsal. SAM-CLIP retains original zero-shot capabilities, supports richer downstream representations, and develops synergistic zero-shot functionality.
- SAM-CLIP merges SAM and CLIP into a single vision backbone using multi-task distillation and memory rehearsal.
- The merged model retains the parent models’ zero-shot capabilities with minimal forgetting and is suitable for edge-device deployment.
- SAM-CLIP produces richer representations for diverse downstream tasks and demonstrates a synergistic new zero-shot capability.
A. More Experimental Details
The experiments use a two-stage multi-task distillation process with resolution- and learning-rate choices designed to balance CLIP learning against retention of SAM abilities. Additional comparisons examine implementation settings and visual outputs.
- Training procedure: Training first probes HeadCLIP, then jointly trains HeadSAM, HeadCLIP, and the ViT backbone with a multi-task distillation loss.The first stage uses CLIP distillation; the second combines CLIP distillation with SAM self-distillation.
- Data sampling: The second stage samples separate CLIP and SAM batches, using batches of 2048 and 32 images respectively.
- Training schedule: Training early-stops after 16 epochs because later training increases forgetting measured by COCO instance-segmentation performance.
- Loss balancing: A 1:10 CLIP-to-SAM distillation-loss ratio provides the best reported trade-off between learning CLIP abilities and mitigating SAM forgetting.A 1:1 ratio causes greater forgetting of SAM’s instance-segmentation ability, while 1:10 and 1:100 preserve it relatively well.
- Resolution: SAM-CLIP benefits from 336px inputs for zero-shot classification, whereas CLIP models show worse accuracy at that resolution.Table 1 reports both models at 224px; the 336px SAM-CLIP results appear in Figure 1.
B. Visual Comparisons of SAM and SAM-CLIP in Segmentation Tasks
Visual and quantitative comparisons indicate that SAM-CLIP preserves SAM-like instance segmentation while enabling semantic segmentation through coordinated CLIP-head and SAM-head processing.
- Instance segmentation: SAM-CLIP maintains instance-segmentation performance comparable to SAM on COCO and LVIS.
- Instance segmentation: Given identical geometric prompts, SAM and SAM-CLIP produce highly similar masks with only slight differences.
- Semantic segmentation: For semantic segmentation, SAM-CLIP uses CLIP-head masks to generate automatic point prompts that the SAM-head refines into fine-grained masks.
- Semantic segmentation: SAM’s point-prompting approach often segments only part of an object, whereas SAM-CLIP uses text prompts to automatically generate semantic masks.
C. Inference Experiments
Inference follows the original CLIP and SAM protocols for classification and instance segmentation, while zero-shot semantic segmentation derives patch-level classes from text-image similarities and combines both heads for mask refinement.
- CLIP and SAM tasks: Zero-shot classification and instance-segmentation inference follow the original CLIP and SAM evaluation protocols.Classification uses 224px images, while instance segmentation uses 1024px images.
- Semantic segmentation inference: Zero-shot semantic segmentation assigns each patch the class whose text embedding has the highest cosine similarity with its CLIP-head feature.Class names are inserted into 80 prompt templates to obtain text embeddings.
- Head composition: Combining the heads first produces CLIP mask logits, then uses each class mask with 1–3 automatically sampled points as SAM prompt-encoder input.
C.1. SAM-CLIP vs. SAM+CLIP
The SAM+CLIP pipeline composes separate SAM and CLIP models for zero-shot semantic segmentation, while SAM-CLIP is evaluated as an alternative unified model.
- The SAM+CLIP pipeline passes SAM ViT-B segmentation masks to CLIP ViT-B/16 for class prediction.It uses SAM in segment-everything mode and CLIP trained on DataComp-1B.
- The comparison between SAM-CLIP and SAM+CLIP is reported in Table 7.
- Figure 7 visualizes the SAM+CLIP pipeline described in Section C.1.
- Figure 8 presents Wise-FT applied to a CLIP-distilled SAM ViT-B model, with the CLIP teacher performance marked by a red dashed line.
D. Weight Averaging
The study tests Wise-FT as a weight-averaging approach for mitigating forgetting when distilling CLIP into SAM, but finds an unresolved learning–forgetting tradeoff.
- Wise-FT linearly interpolates pretrained and fine-tuned parameters using coefficient α to mitigate forgetting.
- CLIP-distilled SAM gains zero-shot classification but forgets its original zero-shot instance-segmentation abilities at α = 1.The student is SAM ViT-B, the teacher is CLIP ViT-B/16, and training uses ImageNet-21k for 20 epochs.
- No Wise-FT point simultaneously achieves classification accuracy above 60% on ImageNet and mAP above 35 on COCO.
E. Limitations
The merged model may inherit limitations from its source models and adds overhead through its auxiliary-model head, while robustness and fairness remain unassessed.
- SAM-CLIP may inherit biases in the data distributions of the original SAM and CLIP models.
- The study does not assess the robustness or fairness of the proposed merging method.
- The merged image encoder requires an additional CLIP-Head, increasing overall size by approximately 25% compared with a single ViT-B.
- The base VFM architecture must be adopted from an existing model, although SAM offers ViT-B, ViT-L, and ViT-H variants.