Source-linked AI summary
Personalize Segment Anything Model with One Shot
Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Xianzheng Ma, Hao Dong, Peng Gao, Hongsheng Li
TL;DR
SAM does not automatically segment user-designated concepts without labor-intensive prompting. The paper introduces PerSAM and PerSAM-F, which personalize SAM from one-shot data through semantic guidance and lightweight scale-aware fine-tuning, and reports competitive results across segmentation and personalized image synthesis tasks.
Problem
SAM lacks the ability to automatically segment specific visual concepts in new images or videos without additional human prompting.
Method
PerSAM uses one reference image and mask with location priors, target-guided attention, target-semantic prompting, and post-refinement, while PerSAM-F trains 2 scale-weighting parameters.
Results
PerSAM and PerSAM-F achieve competitive or leading performance across personalized segmentation benchmarks, one-shot segmentation, video object segmentation, and DreamBooth-assisted personalized text-to-image synthesis.
Takeaways & Limitations
The approach expands SAM personalization to private one-shot segmentation and supports foreground-focused personalization of diffusion-model image generation.
Takeaways & Limitations
PerSAM remains challenged by hierarchical objects whose local parts and global shapes create ambiguity about the appropriate mask scale.
Abstract
from arXiv · showhide
Driven by large-data pre-training, Segment Anything Model (SAM) has been demonstrated as a powerful and promptable framework, revolutionizing the segmentation models. Despite the generality, customizing SAM for specific visual concepts without man-powered prompting is under explored, e.g., automatically segmenting your pet dog in different images. In this paper, we propose a training-free Personalization approach for SAM, termed as PerSAM. Given only a single image with a reference mask, PerSAM first localizes the target concept by a location prior, and segments it within other images or videos via three techniques: target-guided attention, target-semantic prompting, and cascaded post-refinement. In this way, we effectively adapt SAM for private use without any training. To further alleviate the mask ambiguity, we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the entire SAM, we introduce two learnable weights for multi-scale masks, only training 2 parameters within 10 seconds for improved performance. To demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for personalized evaluation, and test our methods on video object segmentation with competitive performance. Besides, our approach can also enhance DreamBooth to personalize Stable Diffusion for text-to-image generation, which discards the background disturbance for better target appearance learning. Code is released at https://github.com/ZrrSkywalker/Personalize-SAM
1 INTRODUCTION
The paper addresses SAM’s inability to automatically segment user-designated concepts and introduces training-free and lightweight fine-tuning approaches for personalized segmentation and image synthesis.
- SAM requires precise human prompts for each image, making personalized segmentation labor-intensive in complex visual contexts.
- PerSAM personalizes SAM from one reference image and rough mask by estimating target locations and using positive-negative location priors.The location prior is derived from feature similarities across foreground pixels.
- PerSAM handles varied poses, scenes, multiple similar objects, identical objects, and video tracking, but can fail when objects contain hierarchical subparts.The ambiguity arises because both local parts and global shapes may be valid SAM masks.
- PerSAM-F freezes SAM and trains 2 parameters within 10 seconds for scale-aware mask selection, improving segmentation accuracy while avoiding one-shot over-fitting.It combines multiple potential mask scales using learnable relative weights.
- The paper introduces PerSeg and reports competitive results across personalized segmentation, one-shot part and semantic segmentation, and video object segmentation.PerSAM also enhances DreamBooth by segmenting foreground targets so background information does not disturb personalized text-to-image generation.
- Three techniques—target-guided attention, target-semantic prompting, and cascaded post-refinement—guide SAM toward the target concept.Attention guidance focuses feature aggregation on target regions, while semantic prompting adds high-level visual cues.
2 RELATED WORK
Related work situates the paper among foundation models, large-scale segmentation models, and parameter-efficient fine-tuning methods that adapt pretrained models to downstream tasks.
- Foundation Models: Foundation models use large-scale data and computation to achieve strong generalization and transfer across downstream scenarios.
- Large Models in Segmentation: Segmentation research spans semantic, instance, panoptic, and interactive segmentation, while newer vision models use extensive mask pre-training for broad image-distribution generalization.
- Parameter-efficient Fine-tuning: Parameter-efficient fine-tuning freezes foundation-model weights and adds small trainable modules to reduce computational and memory costs.
3 METHOD
PerSAM personalizes SAM from one reference image and mask by locating the target, injecting target semantics, and refining predictions without training. PerSAM-F addresses segmentation-scale ambiguity by combining three-scale masks with two learnable weights.
- Task Definition: PerSAM defines personalized segmentation as automatically segmenting a designated object in new images or videos from one reference image and mask.The reference mask may be accurate or a rough sketch, and evaluation uses the PerSeg dataset.
- Location Prior: PerSAM builds a location confidence map from local feature similarities, then selects highest- and lowest-confidence points as positive-negative prompts.The confidence map aggregates appearance evidence from foreground pixels and object parts.
- Target-guided Attention: Target-guided attention uses the confidence map to bias every decoder cross-attention layer toward foreground target regions.The attention bias is controlled by a balancing factor α and is intended to improve feature aggregation without training.
- Target-semantic Prompting: Target-semantic prompting adds the reference object's global embedding to decoder input tokens, complementing positional prompts with high-level visual cues.The target embedding is repeated and incorporated element-wise into the test-image tokens.
- Scale-aware Fine-tuning: PerSAM-F outputs three mask scales and combines them with two learnable weights to adaptively select an appropriate segmentation scale.Its weighted output is M = w1 · M1 + w2 · M2 + (1 − w1 − w2) · M3.
4 EXPERIMENT
The experiments evaluate PerSAM and PerSAM-F on personalized image, video, and part segmentation, plus DreamBooth assistance and component ablations. Across these settings, the methods show strong performance, efficient personalization, and robustness to relaxed reference inputs.
- 4.1 PERSONALIZED EVALUATION: PerSeg contains 40 objects with 5–7 images and masks per object, using one image-mask pair as one-shot input and mIoU and bIoU for evaluation.
- 4.1 PERSONALIZED EVALUATION: PerSAM-F improves PerSAM by +2.7% overall mIoU and +5.9% overall bIoU on PerSeg.
- 4.1 PERSONALIZED EVALUATION: PerSAM-F surpasses SegGPT by +2.4% overall mIoU and +4.1% overall bIoU, while training-free PerSAM outperforms Painter, VP, and SEEM.
- 4.2 EXISTING SEGMENTATION BENCHMARKS: On DAVIS 2017 validation, PerSAM surpasses Painter by +32.3% J&F score, while PerSAM-F exceeds SegGPT by +0.5%.PerSAM-F also outperforms methods trained with extensive video data.
- 4.2 EXISTING SEGMENTATION BENCHMARKS: PerSAM-F performs better than Painter and comparably to SegGPT across four one-shot image segmentation datasets, including category-wise and part-wise tasks.The datasets are FSS-1000, LVIS-92i, PASCAL-Part, and PACO-Part.
- 4.3 PER-SAM-ASSISTED DREAMBOOTH: PerSAM-assisted DreamBooth decouples generated backgrounds from the reference sofa and better matches newly prompted contexts such as jungle and snow.
- 4.4 ABLATION STUDY: Ablations report gains of +3.6% from negative point prompting, +11.4% from cascaded post-refinement, +1.9% from attention guidance, and +3.5% from semantic prompting.
- 4.4 ABLATION STUDY: Scale-aware fine-tuning boosts the ablation score by +6.0%, while prompt tuning and Adapter over-fit the one-shot data and degrade accuracy.
5 CONCLUSION
The paper personalizes SAM for specific visual concepts using one-shot data through PerSAM and PerSAM-F. It reports leading benchmark performance, mask-scale disambiguation, and assistance for personalized diffusion-model image synthesis.
- PerSAM personalizes SAM with one-shot data through training-free high-level target-semantic techniques, while PerSAM-F adds scale-aware fine-tuning.
- PerSAM-F uses only 2 learnable parameters to alleviate mask-scale ambiguity and achieves leading performance across various benchmarks.
- The approach also assists DreamBooth in fine-tuning personalized text-to-image diffusion models.