Source-linked AI summary
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP
Daniil Pakhomov, Sanchit Hira, Narayani Wagle, Kemar E. Green, Nassir Navab
TL;DR
Semantic segmentation can require slow, domain-informed annotation, and consistent labels may be difficult when classes lack clear boundaries. This paper discovers semantic regions through StyleGAN feature clustering and prompt-guided discovery, then trains on generated images and masks. It reports outperforming recent semi-supervised methods and competitive performance with fully supervised methods.
Problem
Semantic segmentation annotation is slow, can require domain knowledge, and may be inconsistent when classes lack clear visual boundaries.
Source: §1 · p. 1Method
The method clusters StyleGAN feature representations to discover regions, uses CLIP text prompts to surface rare classes, and trains a segmentation model on generated images and masks.
Source: §2 · p. 3, §2.2 · p. 3, §2.3 · p. 3Results
The authors report that the method outperforms recently introduced semi-supervised models and achieves results competitive with fully supervised methods.
Source: §1 · pp. 2–3, §4 · p. 6Takeaways & Limitations
The method discovers meaningful, consistent semantic classes even when people may find them difficult to define and label consistently.
Source: §4 · p. 6
Abstract
from arXiv · showhide
We introduce a method that allows to automatically segment images into semantically meaningful regions without human supervision. Derived regions are consistent across different images and coincide with human-defined semantic classes on some datasets. In cases where semantic regions might be hard for human to define and consistently label, our method is still able to find meaningful and consistent semantic classes. In our work, we use pretrained StyleGAN2 generative model: clustering in the feature space of the generative model allows to discover semantic classes. Once classes are discovered, a synthetic dataset with generated images and corresponding segmentation masks can be created. After that a segmentation model is trained on the synthetic dataset and is able to generalize to real images. Additionally, by using CLIP we are able to use prompts defined in a natural language to discover some desired semantic classes. We test our method on publicly available datasets and show state-of-the-art results.
1. Introduction
Semantic segmentation is limited by slow annotation and inconsistent labeling, while many deep-learning approaches require substantial training data. This work discovers semantic regions without annotations and reports results competitive with fully supervised methods.
1. Introduction
- Image annotation is slow, may require domain knowledge, and can be inconsistent when semantic classes lack clear visual boundaries.The authors note that heterogeneous labels can affect overall model performance.Source: §1 · p. 1, Figure 2 · p. 2
- The method outperforms recent semi-supervised methods without annotations and is competitive with fully supervised segmentation methods.The authors also describe it as state of the art relative to recently introduced semi-supervised models.Source: §1 · pp. 2–3, Figure 2 · p. 2
- The method discovers regions that coincide with human-defined classes and proposes consistent regions that may be difficult for people to define and label.Examples include samples with no clear visual border between most semantic classes.Source: §1 · p. 2, Figure 2 · p. 2
- Natural-language prompts let the method discover rare semantic regions such as beards, glasses, and hats.The authors also describe assigning natural-language prompts to discovered regions through automatic classification.Source: §1 · p. 2
2. Method
The method uses StyleGAN2 feature maps to discover semantic regions, then creates annotated synthetic images for training a segmentation model. CLIP-guided latent manipulation helps surface rare classes, while CLIP embeddings assign natural-language prompts to discovered clusters.
2. Method
- StyleGAN2 feature maps support clustering to discover semantic regions consistent across generated samples, and the model can be pretrained or trained on a target dataset.StyleGAN2 also provides realistic samples and latent manipulation, which the method uses to discover additional classes.Source: §2 · p. 3, §2.1 · p. 3
2.2. Clustering
- K-means clusters flattened feature maps; assigning new samples to their closest cluster yields segmentation masks with minimal additional computational overhead.Earlier layers produce coarser but more semantically meaningful clusters, while later layers produce finer clusters with less semantic meaning.Source: §2.2 · p. 3
2.3. Class Discovery with Image Manipulation
- For rare classes missed during clustering, CLIP-prompted latent manipulation adds the desired attribute to generated images so it can emerge as a cluster.The manipulation direction G is learned from latent-image pairs classified against a natural-language prompt, and α controls manipulation strength.Source: §2.3 · p. 3, §2 · p. 3, Figure 4 · p. 4
2. Method
- The synthetic dataset is built by clustering feature maps from an initial sample set, then labeling pixels in a much larger generated set with the discovered clusters.A segmentation network is subsequently trained on those synthetic image-mask pairs.Source: Figure 3 · p. 4
2.4. Cluster Classification
- CLIP assigns each discovered cluster the text prompt with the largest dot product between its image-region embedding and the prompt embedding.Cluster embeddings are averaged across images; higher spatial resolution is reported to improve results for smaller clusters.Source: §2.4 · pp. 4–5, Figure 5 · p. 5
2.5. Knowledge Distillation
- A segmentation model trained on synthetic images and their masks is used to obtain segmentation predictions for real images, and the authors report that it generalizes.The described student model uses ResNet-18 and dilated convolutions.Source: §2 · p. 3, §2.5 · p. 5
3. Datasets and Results
The evaluation covers face parsing and eye segmentation, with results varying by dataset and class complexity. On OpenEDS, the discovered iris, pupil, and sclera classes match the dataset categories, and segmentation is reported as close to fully supervised performance.
Face segmentation
- On CelebA-Mask8, the method achieves state-of-the-art results against semi-supervised methods; on CelebAMask-HQ, it performs worse than the fully supervised method.CelebAMask-HQ evaluation merges left and right eyes and ears and excludes earrings and necklaces because StyleGAN struggles to generate those attributes realistically.Source: §3 · p. 5
Beard segmentation
- For the beard class, the authors collected a small labeled dataset to evaluate the discovered semantic class against a fully supervised method.The passage does not report the comparison outcome.Source: §3 · p. 5
OpenEDS
- On OpenEDS, iris, pupil, and sclera were discovered, and segmentation performance was very close to the fully supervised method because the semantic classes are simpler.StyleGAN2 was trained from scratch on OpenEDS images for this evaluation.Source: §3 · pp. 5–6
4. Conclusion
The authors conclude that unsupervised learning can discover meaningful, consistent semantic classes that mostly coincide with human-labeled classes. They also report competitive segmentation results and prompt-guided discovery of classes such as beard.
4. Conclusion
- The method discovers meaningful, consistent semantic classes that mostly coincide with human-defined labels and can identify classes that are difficult for people to define and label consistently.The authors report that synthetic image-mask training generalizes to real images, is competitive with fully supervised methods, and outperforms recently introduced semi-supervised models.Source: §4 · p. 6