Source-linked AI summary
EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything
Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest Iandola, Raghuraman Krishnamoorthi, Vikas Chandra
TL;DR
SAM’s large image encoder limits practical deployment despite strong zero-shot versatility. The paper introduces SAMI, which transfers SAM features into lightweight encoders, and uses them to build EfficientSAMs; across downstream tasks, SAMI consistently outperforms other masked-pretraining methods while EfficientSAMs reduce complexity.
Problem
SAM’s expensive image encoder creates high computation and memory costs, limiting its use in practical and real-time applications.
Method
SAMI trains lightweight encoders to reconstruct features from SAM’s image encoder, after which the pretrained encoders and SAM mask decoder are finetuned on SA-1B as EfficientSAMs.
Results
SAMI consistently outperforms other masked image pretraining methods across image classification, object detection, instance segmentation, and semantic segmentation, while EfficientSAMs perform favorably against other fast SAM models.
Takeaways & Limitations
SAMI-pretrained lightweight encoders support efficient SAM models with reduced complexity and useful performance for practical deployment.
Abstract
from arXiv · showhide
Segment Anything Model (SAM) has emerged as a powerful tool for numerous vision applications. A key component that drives the impressive performance for zero-shot transfer and high versatility is a super large Transformer model trained on the extensive high-quality SA-1B dataset. While beneficial, the huge computation cost of SAM model has limited its applications to wider real-world applications. To address this limitation, we propose EfficientSAMs, light-weight SAM models that exhibits decent performance with largely reduced complexity. Our idea is based on leveraging masked image pretraining, SAMI, which learns to reconstruct features from SAM image encoder for effective visual representation learning. Further, we take SAMI-pretrained light-weight image encoders and mask decoder to build EfficientSAMs, and finetune the models on SA-1B for segment anything task. We perform evaluations on multiple vision tasks including image classification, object detection, instance segmentation, and semantic object detection, and find that our proposed pretraining method, SAMI, consistently outperforms other masked image pretraining methods. On segment anything task such as zero-shot instance segmentation, our EfficientSAMs with SAMI-pretrained lightweight image encoders perform favorably with a significant gain (e.g., ~4 AP on COCO/LVIS) over other fast SAM models.
1. Introduction
SAM delivers versatile zero-shot segmentation but its large image encoder creates substantial computation and memory costs that hinder practical deployment. The paper addresses this bottleneck with SAMI-pretrained lightweight encoders and EfficientSAM models, which retain useful performance with reduced complexity.
- SAM achieves strong zero-shot transfer across segmentation tasks using a prompt-based Vision Transformer trained on the 11M-image, 1B-mask SA-1B dataset.
- 632M parameters in SAM’s ViT-H image encoder, versus 3.87M in its prompt-based decoder, make the image encoder the main efficiency bottleneck.
- SAMI trains lightweight ViT encoders to reconstruct feature embeddings generated by SAM’s ViT-H encoder, producing backbones for downstream vision tasks.
- 82.7% top-1 accuracy is achieved by SAMI-pretrained ViT-Small after 100 ImageNet-1K epochs, outperforming other state-of-the-art image-pretraining baselines.
- EfficientSAMs are lightweight SAM models designed to provide state-of-the-art quality-efficiency trade-offs and complement SAM for practical deployment.
2. Related Work
Related work spans segment-anything foundation models, efficient Vision Transformers, knowledge distillation, and self-supervised masked image modeling. EfficientSAM builds on masked autoencoding while using SAM features as the reconstruction target.
- Segment Anything Models: SAM is a versatile vision foundation model that segments objects from interaction prompts and supports diverse downstream applications.
- Vision Transformers: Efficient Vision Transformer research develops smaller ViT variants and convolutional hybrids to reduce deployment parameters and computation.
- Knowledge Distillation: Knowledge distillation transfers information from larger teacher models to smaller students through hard labels, soft labels, or intermediate features.
- Masked Image Pretraining: Masked image modeling pretrains visual representations by reconstructing masked content, with methods differing in whether they predict tokens, gradients, or image patches.
- Masked Image Pretraining: SAMI extends MAE by reconstructing features from the SAM image encoder, enabling masked pretraining to leverage knowledge from a vision foundation model.
3. Approach
SAMI adapts masked autoencoding to reconstruct SAM image-encoder features with lightweight encoders, then reuses the pretrained encoder in EfficientSAM. The decoder and projection head support reconstruction during pretraining but are not both retained for the final segmentation model.
- Masked Autoencoders: MAE separates unmasked image tokens for encoder processing from masked tokens reconstructed by its decoder during self-supervised learning.
- SAMI: SAMI uses SAM image-encoder latent features as reconstruction targets, transferring SAM representations into lightweight ViT encoders through masked modeling.
- SAMI Components: Cross-attention decoding reconstructs masked-token features using queries from masked tokens and keys and values from masked and unmasked features.
- SAMI Components: A linear projection head aligns the feature dimension of the MAE output with that of the SAM image encoder.
- Reconstruction Objective: SAMI training compares SAM image-encoder outputs with projected MAE outputs to compute the reconstruction loss.
- EfficientSAM: After pretraining, the decoder is discarded; the lightweight encoder becomes EfficientSAM’s image encoder alongside SAM’s default mask decoder and is finetuned on SA-1B.
4. Experiments
Experiments evaluate SAMI-pretrained lightweight encoders across classification, detection, segmentation, and segment-anything tasks. SAMI consistently improves downstream performance, while EfficientSAMs provide competitive segmentation with substantially lower complexity.
- Image Classification: SAMI achieves 84.8% top-1 accuracy on ImageNet-1K with SAMI-B, outperforming MAE-B by 1.2%.It also reports substantial gains for lightweight ViT-Tiny and ViT-Small models over several pretraining and distillation baselines.
- Object Detection and Instance Segmentation: SAMI-B gains 0.9 APbbox and 0.6 APmask over MAE-B on COCO object detection and instance segmentation.SAMI-S significantly outperforms DeiT-S by 2.6 APbbox and 2.3 APmask.
- Semantic Segmentation: SAMI-pretrained backbones improve ADE20K semantic-segmentation results over MAE by 2.5, 4.7, and 3.7 mIoU points.The experiments use ViT backbones in Mask2former and support generalization across downstream tasks.
- Segment Anything Task: EfficientSAM-Ti exceeds MobileSAM by 1.9 mIoU for one-click and 1.5 mIoU for one-box interactive segmentation on COCO with comparable complexity.EfficientSAM-S remains within 1.5 mIoU of SAM on COCO box prompting while using 20x fewer parameters.
- Segment Anything Task: EfficientSAM-S gains more than 6.5 AP on COCO and 7.8 AP on LVIS over FastSAM for zero-shot instance segmentation.EfficientSAM-Ti also exceeds FastSAM by 4.1 AP on COCO and 5.3 AP on LVIS, and uses 9.8M parameters versus FastSAM’s 68M.
- Ablation Studies: Ablations favor direct MSE reconstruction, cross-attention over all-token decoding, high mask ratios, and powerful reconstruction targets.SAMI-Ti is 3% better with masked-token cross-attention than with all tokens, while CLIP targets outperform MAE by 0.8% for ViT-Tiny on ImageNet-1K.
5. Conclusion
SAMI uses SAM’s foundation-model guidance to improve masked image pretraining for ViTs and supports efficient SAM models with lightweight encoders. Experiments across multiple vision tasks consistently validate SAMI’s advantages, while its broader applications remain preliminary.
- SAMI reconstructs latent features from the SAM image encoder to transfer foundation-model knowledge to ViTs.
- Experiments on image classification, object detection, instance segmentation, semantic segmentation, and segment anything consistently validate SAMI’s advantages.
- SAMI also helps build efficient SAMs with pretrained lightweight encoders.
- The authors describe SAMI’s applications beyond efficient segment anything as preliminary.
Anything
The supplementary material provides additional instance-segmentation results for the efficient SAM model. It also reports inference-efficiency measurements using a standardized one-box-prompt setup on an NVIDIA A100.
- Table 8 reports inference efficiency with all models prompted by one ViTDet box on a single NVIDIA A100.
- The supplementary material provides more results demonstrating the efficient SAM model’s instance-segmentation capabilities.
6. Efficiency Evaluation
Inference efficiency is evaluated by measuring throughput and parameter count under a fixed hardware, prompt, and input-resolution setting.
- Throughput and parameter counts are recorded for the models.
- Throughput is measured in images per second on a single NVIDIA A100 with one box prompt at 1024 × 1024 resolution.
7. Qualitative Evaluation
The qualitative evaluation tests EfficientSAM on prompt-based segmentation, segment everything, and salient instance segmentation. The examples show reasonable or decent masks across these settings, while occasional noisy segmentation remains visible.
- The qualitative evaluation covers point-based and box-based prompt segmentation, segment everything, and salient instance segmentation without manually created prompts.
- Point-prompt examples show reasonable instance-segmentation results.
- Box-prompt examples generate expected object segmentation.
- Segment-everything examples provide decent segmentation performance.
- Salient-instance examples generate masks automatically without manually creating points or boxes.
- The model may sometimes produce noisy segmentation, as illustrated in the noise examples.