Source-linked AI summary
Text-to-seed generation: Training-free open-vocabulary seeded semantic segmentation via re-purposing diffusion as text-guided seed generator
Kumju Jo, Heesun Jung, Sungyong Baik
TL;DR
OVSS remains difficult because SAM has limited semantic understanding and coarse-mask refinement depends on inaccurate initial predictions. T2S uses Stable Diffusion attention maps to localize text-described objects as seeds, then prompts SAM to expand them into masks. Across standard benchmarks and qualitative settings, it reports strong performance without additional training or annotations, while remaining vulnerable to severe occlusion, faint boundaries, and spatial overlap.
Problem
SAM struggles with semantic text prompts, and refinement-based approaches depend on the quality of coarse masks or bounding boxes.
Method
T2S is a training-free framework that uses Stable Diffusion attention maps for text-guided seed localization and SAM point prompts for seed-based region expansion.
Results
T2S achieves strong performance on standard OVSS benchmarks and produces high-quality masks across datasets and qualitative settings without additional training or annotations.
Takeaways & Limitations
The results support using Stable Diffusion for semantic seed generation and SAM as a seed-driven spatial expansion model for flexible open-vocabulary segmentation.
Takeaways & Limitations
T2S may fail under severe occlusion, faint object boundaries, or target/non-target spatial overlap, causing unsegmented regions or boundary leakage.
Abstract
from arXiv · showhide
Open-vocabulary semantic segmentation (OVSS) aims to segment image regions corresponding to arbitrary text queries. Although the Segment Anything Model (SAM) is a powerful foundation model for segmentation, its standalone performance on OVSS remains limited. Existing methods therefore often use SAM to refine coarse masks predicted by other models, but this strategy is unreliable when the initial masks are inaccurate. In this work, we argue that more reliable segmentation can be achieved by exploiting SAM as a region expansion module guided by accurate object points (i.e., seeds) rather than inaccurate coarse masks. Inspired by classical seeded segmentation, we reformulate OVSS as text-guided seed localization followed by seed-based region expansion. To realize this idea, we propose Text-to-Seed (T2S), a training-free framework that leverages the text-to-region correspondence of Stable Diffusion to generate attention-based seed points for target categories described by text. These sparse seeds are then used as point prompts for SAM to produce full object masks. Without task-specific training or additional annotations, T2S achieves strong performance on standard OVSS benchmarks, demonstrating the effectiveness of combining semantic grounding with seed-driven spatial segmentation.
1. Introduction
OVSS requires generalization to arbitrary text-described categories, but SAM struggles with semantic prompts and refinement methods inherit errors from coarse masks. T2S instead uses Stable Diffusion for seed localization and SAM for region expansion, achieving strong performance without additional training.
- OVSS aims to segment open-set categories specified by arbitrary text queries.
- SAM provides spatially coherent representations but struggles with text prompts and semantic understanding, while refinement approaches depend on coarse-mask or bounding-box quality.
- Stable Diffusion-based segmentation methods can struggle to generate high-quality masks and recognize multiple subjects because SD was designed for image generation.
- T2S reformulates OVSS as seed initialization followed by region expansion, using Stable Diffusion attention maps to localize queried categories and SAM point prompts to obtain masks.
- T2S achieves strong performance across datasets as a plug-and-play framework using SAM and Stable Diffusion off the shelf without additional training.
2. Related Works
Related work applies CLIP, diffusion models, and SAM to open-vocabulary segmentation, but each component has limitations in spatial coherence, mask quality, or semantic grounding. T2S uses diffusion for point-level localization and SAM for region expansion rather than coarse-mask refinement.
- Open-vocabulary segmentation extends pixel labeling beyond predefined categories to arbitrary text-prompted categories.
- CLIP-based methods provide image-level text correspondence but poor spatial coherence, motivating fine-tuning approaches that still struggle with arbitrary categories.
- Stable Diffusion encodes semantic and spatial cues in latent representations and attention maps, enabling its use in open-vocabulary segmentation and related tasks.
- SAM supports class-agnostic segmentation but has limited semantic understanding for arbitrary open-vocabulary queries.
- T2S produces point-level localization with diffusion and uses SAM for region expansion rather than conventional coarse-mask or bounding-box refinement.
3. Background
Stable Diffusion generates images through text-conditioned denoising with U-Net attention layers. Its cross-attention and self-attention maps encode text–visual correspondence and spatial similarity, respectively, and are aggregated by resolution.
- 3.1. Diffusion Model: Stable Diffusion generates an image from Gaussian noise using a text embedding derived from the input prompt.
- 3.1. Diffusion Model: The diffusion process follows a Markov chain that incrementally adds noise over T timesteps, with β_t specifying the noise or variance schedule.
- 3.1. Diffusion Model: A text-conditioned denoising network predicts the added Gaussian noise at each timestep to recover the image.
- 3.2. Aggregation of attention maps: Each transformer block contains cross-attention and self-attention layers operating on text embeddings and latent features.
- 3.2. Aggregation of attention maps: Cross-attention maps encode text–visual correspondence, whereas self-attention maps capture spatial similarities among latent features.
- 3.2. Aggregation of attention maps: The U-Net includes attention layers in both encoder and decoder sections, producing the attention maps used by the framework.
- 3.2. Aggregation of attention maps: Attention maps are generated at four spatial resolutions and aggregated by grouping layers by resolution, then averaging and normalizing maps to a 0–1 range.
4. Proposed Method
T2S repurposes Stable Diffusion attention maps for text-guided seed localization, then uses iterative seed spreading and SAM-based region expansion to produce segmentation masks.
- Overview: T2S comprises iterative seed initialization, region expansion, and classification, connecting Stable Diffusion for seed generation with SAM for region expansion.The framework is designed as a plug-and-play combination of off-the-shelf SD and SAM components.
- Initialization of seed generator: Stable Diffusion attention maps are enhanced by up-weighting target-category EoT embeddings by a factor of two.The method uses EoT embeddings because their attention maps contain semantic information about objects in the text prompt.
- Initialization of seed generator: Attention maps are aggregated from t = 0 to t = T/2, avoiding early denoising steps whose noise produces less reliable maps.Aggregation uses cross-attention and self-attention maps at resolutions 8 × 8, 16 × 16, 32 × 32, and 64 × 64.
- Seed generation and spreading: Initial seeds are extracted from 8 × 8 ACA maps using a threshold, while higher-resolution ASA maps provide finer attention for subsequent refinement.Lower-resolution ACA offers better localization, whereas higher-resolution ASA supplies fine-grained attention.
- Iterative seed generation and spreading: The SGS process spreads current seeds through self-attention, combines attention information at 16 × 16, and bilinearly upsamples masks toward 64 × 64.Iterations continue until the highest ASA resolution, sK = 64, is reached.
- Region expansion via SAM: Positive category seeds and negative background or irrelevant-feature seeds are fed to SAM as point prompts for region expansion.The resulting masked image is classified with CLIP to filter false-positive segmented regions.
- Iterative seeded semantic segmentation: Iteratively masking detected regions makes attention maps more fine-grained and sparse, helping localize multiple subjects at different image locations.The procedure repeats until CLIP rejects regions more than n = 10 times consecutively or all image regions are covered.
5. Experiments
T2S is evaluated across standard OVSS benchmarks and qualitative scenarios, with ablations examining its design choices. The reported results show strong performance, broad qualitative robustness, and limitations under especially difficult visual conditions.
- 5.3. Quantitative results: T2S reports outstanding performance across several OVSS datasets compared with methods based on CLIP, diffusion, SAM, or their combination.The comparison includes results reported by prior papers and Grounded-SAM results reported in prior work.
- 5.4. Ablation Study: Higher-resolution attention maps decrease performance because attention disperses and loses category-localization information.This ablation concerns ACA-map resolution used to initialize seed generation.
- 5.5. Qualitative Results: T2S produces high-quality masks in multi-object scenes through iterative seeded semantic segmentation despite Stable Diffusion’s limitation in modeling multiple objects.The iterative procedure is presented as overcoming this limitation in the qualitative results.
- 5.5. Qualitative Results: T2S produces high-quality masks for arbitrary categories, including fictional characters, by combining Stable Diffusion localization with SAM region expansion off the shelf.The paper contrasts this with approaches fine-tuned on fixed categories using bounding-box or mask annotations.
- 5.5. Qualitative Results: T2S recovers precise boundaries for small occluded targets, whereas CaR misses several small objects.The paper also reports robustness under intricate background structures and other complex visual conditions.
- 5.5. Qualitative Results: T2S may fail under severe occlusion, faint object boundaries, or target/non-target spatial overlap, causing unsegmented regions or boundary leakage.These failure modes are illustrated in the qualitative evaluation.
- 5.6. Evaluation on Out-of-Domain images: T2S outperforms CaR+SAM in the reported out-of-domain evaluation, demonstrating strong generalization capability.
6. Conclusion
T2S recasts open-vocabulary semantic segmentation as text-guided seed localization followed by seed-based region expansion. It combines Stable Diffusion-generated attention-based seeds with SAM to produce full object masks without additional training or annotations.
- T2S recasts open-vocabulary semantic segmentation as text-guided seed localization and seed-based region expansion.
- Stable Diffusion generates attention-based seed points, which SAM expands into full object masks.
- T2S avoids the limitations of coarse-mask refinement by combining semantic grounding with spatial segmentation.
- T2S achieves strong performance on standard open-vocabulary semantic segmentation benchmarks without additional training or annotations.