Source-linked AI summary
Language-driven Semantic Segmentation
Boyi Li, Kilian Q. Weinberger, Serge Belongie, Vladlen Koltun, René Ranftl
TL;DR
Semantic segmentation is constrained by fixed label sets and costly pixel-level annotation, motivating models that can recognize unseen categories without new labeled samples. LSeg aligns dense pixel embeddings with language-based label embeddings and supports flexible label sets at test time. It achieves strong zero-shot and few-shot performance, remains competitive with fixed-label methods, and has identified failure cases when labels omit the true class.
Problem
Semantic segmentation commonly relies on restricted, costly-to-annotate label sets, limiting recognition of the broader range of semantic categories.
Method
LSeg trains an image encoder to align dense pixel embeddings with text embeddings of semantic labels and uses label-equivariant spatial regularization for flexible predictions.
Results
LSeg provides strong zero-shot segmentation, competes with few-shot methods without additional training samples, and incurs only a negligible loss on fixed-label segmentation.
Takeaways & Limitations
LSeg enables zero-shot segmentation with arbitrary label sets on the fly while retaining competitive performance on existing fixed-label tasks.
Takeaways & Limitations
When test-time labels omit the true class, LSeg assigns the pixel to the closest available text label; dataset and language-model biases may also be inherited.
Abstract
from arXiv · showhide
We present LSeg, a novel model for language-driven semantic image segmentation. LSeg uses a text encoder to compute embeddings of descriptive input labels (e.g., "grass" or "building") together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class. The text embeddings provide a flexible label representation in which semantically similar labels map to similar regions in the embedding space (e.g., "cat" and "furry"). This allows LSeg to generalize to previously unseen categories at test time, without retraining or even requiring a single additional training sample. We demonstrate that our approach achieves highly competitive zero-shot performance compared to existing zero- and few-shot semantic segmentation methods, and even matches the accuracy of traditional segmentation algorithms when a fixed label set is provided. Code and demo are available at https://github.com/isl-org/lang-seg.
1 INTRODUCTION
LSeg addresses the restricted label sets and annotation costs of semantic segmentation by using language-informed representations and flexible test-time labels. It supports arbitrary label-set changes while achieving competitive zero- and few-shot performance without additional training samples.
- Motivation: Existing segmentation models use limited label sets because pixel-level annotation is costly, labor intensive, and difficult to keep consistent across fine-grained or hierarchical labels.The restricted label sets limit recognition relative to the much larger vocabulary of English nouns.
- Approach: LSeg leverages modern language models to increase the flexibility and generality of semantic segmentation models.Its approach is inspired by CLIP-style language-assisted recognition.
- Approach: The model synthesizes zero-shot segmentation models on the fly, allowing users to expand, shrink, or reorder labels at test time.An output module spatially regularizes predictions while preserving label-order equivariance.
- Approach: LSeg can segment additional objects when labels are added, while assigning objects without matching labels to an available alternative such as “other.”Figure 1 illustrates different segmentation maps produced from different provided label sets.
- Results: Across zero- and few-shot tasks, LSeg outperforms existing zero-shot methods, remains competitive on few-shot benchmarks, and requires no additional training samples.Text embeddings incur only a negligible performance loss relative to standard fixed-label segmentation methods.
2 RELATED WORK
Related work addresses generalized segmentation, language-driven recognition, open-set recognition, and cross-domain adaptation. LSeg builds on language-assisted recognition while targeting flexible prediction of unseen semantic classes rather than merely detecting unknown samples or adapting visual domains.
- Generalized semantic segmentation: Few-shot segmentation transfers to novel classes using a small number of labeled support images, whereas zero-shot methods aim to segment unseen objects without additional novel-class samples.Few-shot methods still require labeled data containing the novel classes.
- Generalized semantic segmentation: Zero-shot segmentation methods commonly use text embeddings to generate or align visual features for unseen categories.Prior approaches include generative models and projections into word-embedding spaces.
- Open-set recognition: Open-set recognition determines whether a sample lies outside the training distribution but does not predict labels for entirely new classes.This distinguishes open-set recognition from generalized semantic segmentation.
- Cross-domain adaptation: Cross-domain adaptation improves transfer to novel visual domains through feature alignment, self-training, or information propagation, but does not address restricted label sets.The related methods are therefore orthogonal to LSeg’s target problem.
- Language-driven recognition: Language-assisted recognition has improved zero-shot classification and flexible object detection by combining language models with visual feature encoders.LSeg is inspired by this CLIP-based line of work.
3 LANGUAGE-DRIVEN SEMANTIC SEGMENTATION
LSeg embeds text labels and image pixels in a shared space, then assigns labels by pixel–label similarity. Its flexible label handling supports variable label sets, while spatial regularization restores predictions to input resolution.
- LSeg embeds text labels and image pixels into a common space and assigns the closest label to each pixel.
- The text encoder produces label vectors whose number and ordering can vary freely.
- The image encoder produces a dense embedding for every downsampled input pixel using a DPT-based architecture.The embedding has spatial dimensions determined by the input size and downsampling factor.
- Pixel–label correlations are computed by inner products, producing an H̃ × W̃ × N tensor of scores across labels.
- Training maximizes the score for each pixel’s ground-truth label through a temperature-scaled per-pixel softmax cross-entropy objective.The temperature is set to t = 0.07.
- A label-equivariant post-processing module spatially regularizes and upsamples low-resolution predictions to the original image resolution.The module addresses the image encoder’s lower-resolution outputs caused by memory constraints.
4 EXPERIMENTS
LSeg is evaluated across few-shot and zero-shot semantic segmentation benchmarks, with comparisons against established baselines and analysis of its generalization to unseen categories.
- Experimental setup: The evaluation primarily targets zero-shot segmentation, although few-shot benchmarks are used because standardized zero-shot protocols and sufficient baselines are unavailable.Few-shot methods have access to labeled samples and therefore more information than LSeg in these comparisons.
- Experimental setup: Experiments use PASCAL-5i, COCO-20i, and FSS-1000 few-shot segmentation benchmarks under standard evaluation protocols.PASCAL-5i and COCO-20i use mIoU and FB-IoU; FSS-1000 contains 1,000 object classes split into 520 training, 240 validation, and 240 test classes.
- Experimental setup: LSeg is compared with state-of-the-art few-shot models and zero-shot baselines across the benchmark datasets.The comparisons include OSLSM, Co-FCN, AMP-2, PANet, PGNet, FWB, PPNet, DAN, PFENet, RePRI, HSNet, ZS3Net, and Xian et al.
- PASCAL-5i and COCO-20i: LSeg outperforms the zero-shot baseline across PASCAL-5i and COCO-20i folds and remains competitive with several few-shot methods.The comparison is conducted using the same ResNet101 backbone, with an additional performance advantage observed for the larger ViT-L/16 backbone.
- FSS-1000: On FSS-1000, LSeg reaches 87.8 mIoU versus 86.5 mIoU for HSNet with the larger ViT-L/16 backbone.Under the same ResNet101 backbone, LSeg achieves comparative results to the state-of-the-art one-shot method.
5 EXPLORATION AND DISCUSSION
LSeg’s exploration covers training configurations, spatial regularization, text encoders, fixed-label performance, and qualitative generalization and failure cases. The experiments show that regularization improves outputs, text-encoder choice matters, and flexible labeling remains competitive on fixed-label segmentation.
- Ablation studies: Using two BottleneckBlocks produced the strongest spatial-regularization improvement with little architectural overhead.The ablation varied block types and depths N ∈ [0, 1, 2, 4].
- Ablation studies: RN50×16 achieved the best text-encoder performance, surpassing the weakest ViT-B/32 encoder by 2.5%.The authors conjecture that the gain reflects the larger embedding dimension.
- Fixed-label comparison: On ADE20K’s fixed label set, LSeg with RN50 × 16 incurred only a negligible loss compared with DPT.The comparison included OCNet, ACNet, DeeplabV3, and DPT.
- Qualitative findings: LSeg transferred segmentation to related unseen labels such as “building,” “greenery,” “dessert,” and “cake” without those labels appearing in training.It also suppressed the semantically related “bread” label in the described example.
- Qualitative findings: LSeg implicitly handled hierarchical labels by merging grass into plant and recognizing furry as a parent category for cats.Replacing “sofa” with unseen “furniture” also grouped the sofa and small shelf under furniture.
- Qualitative findings: Failure cases arose when true labels were absent or multiple label explanations were plausible, causing nearest-label assignments or overly coarse object predictions.Examples include labeling a dog as “toy” and house windows as “house.”
6 CONCLUSION
LSeg is a language-driven semantic segmentation architecture that represents labels and visual concepts in a shared embedding space. It supports arbitrary label sets for zero-shot segmentation while remaining competitive on fixed-label tasks.
- LSeg maps semantically similar labels to similar embedding regions and correlates visual concepts with those representations for segmentation.
- LSeg synthesizes zero-shot segmentation models with arbitrary label sets on the fly.
- The resulting models are strong zero-shot baselines and can rival few-shot models without sacrificing fixed-label accuracy.
ETHICS STATEMENT
The ethics statement reports no inherent concerns for the method or code across several listed risk categories, while acknowledging that source datasets and language models may contain bias.
- The authors report no inherent concerns about discrimination, fairness, privacy, security, legal compliance, or research integrity in the method or code.
- Bias in image datasets and language models may be inherited by models trained with this approach.
REPRODUCIBILITY
The authors state that LSeg is reproducible from the method and training descriptions and provide an interactive demo for user-selected images.
- The code can be implemented from the method description and training details in Sections 3–5.
- An interactive demo lets users try LSeg with images of their choosing.