Source-linked AI summary
Zero-Shot Semantic Segmentation
Maxime Bucher, Tuan-Hung Vu, Matthieu Cord, Patrick Pérez
TL;DR
Semantic segmentation lacks a way to classify categories with no training examples, motivating zero-shot semantic segmentation. The paper introduces ZS3Net, which generates unseen-class visual features for joint seen/unseen pixel classification, then adds self-training and graph context; it reports competitive results on Pascal-VOC and Pascal-Context, with improvements from these extensions.
Problem
Zero-shot semantic segmentation targets pixel-wise classification of categories absent from training, using unseen labels without manual annotations or real samples.
Method
ZS3Net combines a deep segmentation backbone with a class-conditioned generative model that synthesizes unseen-class features for training a classifier alongside seen-class samples.
Results
ZS3Net reports significant gains over a ZSL baseline on unseen-class mIoU, including +32.2% mIoU in the 2-split, while remaining comparable on seen classes.
Takeaways & Limitations
Self-training can bring overall performance on the 2-unseen split in both datasets close to supervised performance, while graph-context encoding improves representations for complex scenes.
Takeaways & Limitations
ZS5Net uses labelled and unlabelled training data containing unseen-class objects, so it is not purely transductive zero-shot learning.
Abstract
from arXiv · showhide
Semantic segmentation models are limited in their ability to scale to large numbers of object classes. In this paper, we introduce the new task of zero-shot semantic segmentation: learning pixel-wise classifiers for never-seen object categories with zero training examples. To this end, we present a novel architecture, ZS3Net, combining a deep visual segmentation model with an approach to generate visual representations from semantic word embeddings. By this way, ZS3Net addresses pixel classification tasks where both seen and unseen categories are faced at test time (so called "generalized" zero-shot classification). Performance is further improved by a self-training step that relies on automatic pseudo-labeling of pixels from unseen classes. On the two standard segmentation datasets, Pascal-VOC and Pascal-Context, we propose zero-shot benchmarks and set competitive baselines. For complex scenes as ones in the Pascal-Context dataset, we extend our approach by using a graph-context encoding to fully leverage spatial context priors coming from class-wise segmentation maps.
1 Introduction
The paper extends zero-shot learning from recognition to semantic segmentation, targeting unseen categories without manual annotations or real samples. ZS3Net generates unseen-class visual features and combines them with seen-class features, with self-training and contextual cues further improving performance.
- Zero-shot semantic segmentation addresses training settings where some object classes are entirely absent, using only unseen labels rather than manual annotations or real samples.
- Prior zero-shot learning mainly focused on image classification, while extensions to detection existed but segmentation lacked an established approach.
- ZS3Net combines a deep image-embedding backbone with a class-conditioned generative model to synthesize visual features for unseen categories.
- The method further incorporates spatial context because semantically related objects tend to share characteristic scene contexts, such as fields for cows and horses or urban scenes for motorbikes and bicycles.
- Evaluations on Pascal-VOC and Pascal-Context across varying numbers of unseen classes report strong performance, further boosted by self-training and semantic contextual cues.
2 Zero-shot semantic segmentation
Zero-shot semantic segmentation learns pixel-wise classifiers for unseen categories using semantic class descriptions rather than unseen training examples. ZS3Net generates unseen-class features, combines them with seen-class features, and extends the pipeline with self-training and spatial context.
- Problem: Zero-shot learning transfers semantic information from class descriptions to recognize categories absent from training examples.Semantic embeddings place classes with related linguistic contexts closer in representation space.
- Architecture: ZS3Net combines a supervised DeepLabv3+ backbone with a generator conditioned on semantic class embeddings.The backbone extracts pixel-wise features, while the generator models class-dependent visual representations.
- Architecture: Synthetic features for unseen classes are combined with real seen-class features to fine-tune a classifier that predicts both class types.At test time, the retrained classifier operates on real DeepLab features from images containing seen and unseen categories.
- Self-training: Self-training extends ZS3Net by using the most confident pseudo-labels from unlabelled images containing unseen objects as additional training features.The resulting model is called ZS5Net and uses a relaxed setting with unlabelled unseen-class images available during training.
- Context encoding: Graph-context encoding represents semantic connected components and their adjacency relations, conditioning feature generation on spatial object arrangements.The method uses true segmentation masks during generative-model training, while images containing unseen classes are not used in that training.
3 Experiments
Experiments evaluate zero-shot semantic segmentation across Pascal-VOC and Pascal-Context using generalized protocols, multiple unseen-class splits, and standard segmentation metrics. ZS3Net improves unseen-class performance over embedding-based baselines, while self-training and graph-context cues provide further gains.
- Experimental setup: Evaluation uses Pascal-VOC and Pascal-Context with incremental 2-, 4-, 6-, 8-, and 10-unseen-class splits.Pascal-VOC contains 20 object classes, while Pascal-Context contains 59 densely annotated object/stuff classes.
- Experimental setup: Generalized evaluation jointly reports seen, unseen, and all-class performance using PA, MA, mIoU, and harmonic mean hIoU.The harmonic mean emphasizes balanced seen- and unseen-class performance because seen-class mIoUs often dominate arithmetic averages.
- Pascal-VOC: ZS3Net improves unseen-class segmentation over embedding-based baselines, including a +32.2% mIoU gain on the Pascal-VOC 2-unseen split.On seen classes, ZS3Net performs comparably to the ZSL baseline, while its hIoU also exceeds the baseline by significant margins.
- Pascal-Context: On Pascal-Context, ZS3Net outperforms the ZSL baseline by significant margins across evaluation metrics despite the dataset's denser annotations and lower overall difficulty.Qualitative results show unseen cow and boat objects recognized by ZS3Net when they are misclassified as seen classes without zero-shot learning.
- Pascal-Context: Graph-context enrichment consistently improves ZS3Net on Pascal-Context by exploiting spatial arrangement priors in complex scenes.The paper motivates these priors using object-context associations such as cows and horses in fields or motorbikes and bicycles in urban scenes.
- Self-training: Self-training strongly boosts seen-, unseen-, and all-class performance, approaching supervised performance on the 2-unseen split in both datasets.The procedure uses p = 25% high-scoring unseen pixels for Pascal-VOC and p = 75% for Pascal-Context, helping disambiguate pixels initially classified as seen classes.
4 Conclusion
ZS3Net addresses zero-shot semantic segmentation by combining text and image embeddings, generative modeling, and classifiers to segment seen and never-seen classes. The paper reports competitive benchmarks and extensions for unlabeled unseen pixels and complex scenes.
- ZS3Net combines text and image embeddings, generative modeling, and classic classifiers for segmentation of seen and never-seen classes at test time.
- The model establishes competitive baselines across zero-shot semantic segmentation benchmarks.
- A self-training extension uses unlabeled pixels from unseen classes when they are available during training.
- Graph-context encoding improves semantic class representation for complex scenes.