PaperlayerView exact revision on arXiv

Source-linked AI summary

Scaling Open-Vocabulary Image Segmentation with Image-Level Labels

Golnaz Ghiasi, Xiuye Gu, Yin Cui, Tsung-Yi Lin

arXiv:2112.12143v2cs.CV

TL;DR

Open-vocabulary segmentation needs to localize arbitrary text concepts, while CLIP and ALIGN primarily provide image-level representations and pixel-supervised alternatives require costly annotations. OpenSeg predicts mask regions, aligns caption words to those regions, and achieves strong transfer performance, including a +19.9 mIoU gain over LSeg on PASCAL.

  • Problem

    Open-vocabulary classification models struggle to localize text queries to pixels, while pixel-wise segmentation supervision is costly to scale.

  • Method

    OpenSeg represents images with predicted segmentation masks and features, then learns visual-semantic alignments between caption words and mask regions.

  • Results

    +19.9 mIoU on PASCAL: OpenSeg significantly outperforms the strongest LSeg model and reports transfer gains on other holdout segmentation datasets.

  • Takeaways & Limitations

    Mask representations support weakly supervised learning from captions, enabling larger training datasets and vocabulary sizes for open-vocabulary segmentation.

  • Takeaways & Limitations

    OpenSeg cannot segment visual concepts that lack associated segmentation proposals.

Abstract

from arXiv · show

We design an open-vocabulary image segmentation model to organize an image into meaningful regions indicated by arbitrary texts. Recent works (CLIP and ALIGN), despite attaining impressive open-vocabulary classification accuracy with image-level caption labels, are unable to segment visual concepts with pixels. We argue that these models miss an important step of visual grouping, which organizes pixels into groups before learning visual-semantic alignments. We propose OpenSeg to address the above issue while still making use of scalable image-level supervision of captions. First, it learns to propose segmentation masks for possible organizations. Then it learns visual-semantic alignments by aligning each word in a caption to one or a few predicted masks. We find the mask representations are the key to support learning image segmentation from captions, making it possible to scale up the dataset and vocabulary sizes. OpenSeg significantly outperforms the recent open-vocabulary method of LSeg by +19.9 mIoU on PASCAL dataset, thanks to its scalability.

1 Introduction

Open-vocabulary segmentation must connect arbitrary text queries to pixel regions, but image-level models lose localization and pixel-supervised methods are costly to scale. OpenSeg uses mask regions for weak caption supervision and reports strong transfer performance.

  • Problem: Open-vocabulary segmentation aims to identify image regions using arbitrary text queries, beyond the closed-set categories learned by conventional semantic segmentation.The task must indicate both what concepts are present and where they occur.
  • Problem: CLIP and ALIGN perform well for image-level retrieval and classification but remain challenged when localizing arbitrary text queries to image regions.Their single image feature loses much location information, limiting pixel-level localization.
  • Motivation: Pixel-wise semantic supervision is expensive, whereas image captions provide scalable labels for learning visual-semantic alignments.OpenSeg targets caption supervision to scale training data and vocabulary sizes.
  • Method: OpenSeg first predicts class-agnostic segmentation masks and features, then aligns caption words with predicted mask regions.It omits a no-object label so the model can predict alternative image organizations beyond the single annotated organization.
  • Evaluation: OpenSeg evaluates transfer on holdout segmentation datasets, contrasting with specialist models trained and tested within one labeled dataset distribution.The paper describes this as the first language-based zero-shot transfer evaluation across image-segmentation datasets.
  • Results: +19.9 mIoU on PASCAL: OpenSeg significantly outperforms the strongest LSeg model, while also exceeding LSeg+ by 2.7 mIoU on PASCAL-Context and 1.9 mIoU on ADE-20k.The reported gains are attributed to using image caption data for larger vocabularies and more training examples.

2 Related Work

Prior work develops grouping, per-pixel segmentation, caption-based grounding, and referring segmentation, but these approaches differ in semantic supervision, scalability, or supported region types. OpenSeg combines mask proposals with caption-based region-word alignment.

  • Grouping for visual recognition: Grouping methods organize pixels into mid-level regions that support later semantic segmentation or object detection, while some methods segment visual entities without semantic labels.OpenSeg additionally assigns open-vocabulary semantics to segmented regions.
  • Fully-supervised segmentation: Per-pixel open-vocabulary methods use CLIP text embeddings with pixel embeddings, but require expensive per-pixel semantic supervision that is difficult to scale.OpenSeg instead uses cheap image-level caption supervision.
  • OpenSeg: OpenSeg represents images with N proposal masks and features, unlike ALIGN/CLIP's global vector or per-pixel models' H×W feature maps.This representation supports weakly supervised learning from captions.
  • Concurrent work: Concurrent approaches use text supervision, CLIP-derived pseudo-labels, or self-training, with differences in training requirements and inference speed.The cited examples include GroupViT, single-image segmentation from interpretability, and CLIP adaptation with pseudo per-pixel labels.
  • Visual grounding: Caption-based visual grounding often relies on object detectors and therefore cannot handle stuff or produce one segmentation map covering everything.OpenSeg also uses captions as semantically rich supervision while targeting segmentation regions.
  • Referring image segmentation: Referring image segmentation computes a binary mask for a referring expression, but its fully supervised datasets make training data difficult to scale.This distinguishes it from caption-supervised open-vocabulary segmentation.

3 Method

OpenSeg represents images with multiple mask proposals and region features, then learns caption-based region-word alignments for open-vocabulary segmentation. Its training combines class-agnostic mask supervision with weak image-caption supervision and supports scaling through pseudo-labels.

  • Representation: OpenSeg represents each image with N proposal masks and corresponding D-dimensional features instead of a single vector or dense feature map.This representation is designed to support precise segmentation from image captions through weak supervision.
  • Mask Proposal Learning: A feature pyramid and cross-attention module use position-augmented image features and learned queries to predict N segmentation masks.Mask queries are compared with image features through dot products followed by a sigmoid.
  • Mask Proposal Learning: Predicted masks are trained against class-agnostic labeled masks by maximizing the Dice coefficient of the best-matched proposal for each labeled mask.Because N typically exceeds the number of labeled masks, unmatched proposals can segment unlabeled regions without penalty.
  • Visual-Semantic Alignment: Caption nouns and adjectives are aligned to mask-pooled region features using cosine similarities and a grounding loss over image-caption pairs in each minibatch.The similarity function encourages each word to match one or a few regions while avoiding penalties for regions without similar words.
  • Joint Training: The total training objective sums the grounding loss and segmentation loss with weight α.When α = 0, the model relies only on visual-semantic grounding and performs poorly, indicating class-agnostic mask annotations are critical for learning masks.
  • Scaling and Inference: To scale training, a segmentation-trained teacher generates pseudo masks for a large image-text dataset, which OpenSeg trains alongside human labels.At inference, cosine similarities between category word embeddings and region features are multiplied by masks to produce pixel-level category logits.

4 Experiments

OpenSeg is evaluated for proposal recall, open-vocabulary segmentation, cross-dataset transfer, and ablations. Results emphasize generalization from class-agnostic masks and caption supervision, while proposals are important at inference.

  • Experimental setup: OpenSeg uses EfficientNet-B7 or ResNet101 backbones, FPN multi-scale fusion, and a frozen ALIGN BERT-Large text encoder.Feature pyramids P2–P5 are merged into a 640-dimensional representation.
  • Training and evaluation datasets: The experiments use COCO, Localized Narrative, PASCAL Context, PASCAL VOC, and ADE20k, including open-vocabulary ADE20k settings with 847 or 150 categories.ADE20k is kept unseen in Localized Narrative training, while COCO provides 118k training images and 5k validation images.
  • Mask proposals: OpenSeg evaluates mask proposals with recall at IoU 50%, 70%, and 90%, comparing 128 proposals against MCG and COB.The model is trained on COCO and additionally evaluated across datasets; Figure 4 illustrates semantically meaningful proposals in an underwater scene absent from COCO.
  • Open-vocabulary segmentation: On A-847, LSeg+ is worse than ALIGN by 1.0 mIoU and 7.3 Grounding mIoU, while OpenSeg exceeds LSeg+ by +2.5 mIoU and +11.3 Grounding mIoU.On PC-459, OpenSeg exceeds LSeg+ by +1.2 mIoU and +15.0 Grounding mIoU.
  • Open-vocabulary segmentation: +19.9 mIoU separates the strongest OpenSeg model from the strongest LSeg model on PASCAL-20.The strongest OpenSeg initializes its EfficientNet-B7 backbone from ALIGN and uses sync loss.
  • Ablation experiments: Without mask proposals, OpenSeg mIoU drops from 42.1 to 32.1 on PC-59 and from 21.1 to 16.4 on A-150.Using nouns and adjectives in captions performs best, whereas using all words produces 0.2–1.1 worse mIoU.

5 Conclusion

The conclusion presents OpenSeg as an open-vocabulary segmentation model that combines mask regions with language-based visual-semantic alignment. It reports transfer gains against strong ALIGN-initialized baselines and motivates language-mediated generalist segmentation.

  • 5 Conclusion: OpenSeg organizes images into regions described by arbitrary text queries, unlike closed-vocabulary semantic segmentation methods.Its image representation consists of mask regions followed by visual-semantic alignments.
  • 5 Conclusion: Mask-region representations support weakly supervised grounding of caption words to predicted mask proposals, making training data scalable.The conclusion links this representation to scalable learning rather than pixel-wise supervision.
  • 5 Conclusion: OpenSeg is evaluated on holdout image segmentation datasets and achieves significant gains against strong baselines initialized by a pre-trained ALIGN model.The authors frame language as an interface for transferring a generalist segmentation model across datasets.
  • 5 Conclusion: The language interface supports recognition and localization of visual concepts for applications such as interactive assistants, content creation, and robot instruction.The authors caution that deployment requires studying biases and calibrating predictions.

B Limitations of our approach

OpenSeg’s mask-based approach depends on the availability of suitable segmentation proposals. Concepts lacking associated proposals cannot be segmented, motivating improved mask generalization or adaptation.

  • B Limitations of our approach: OpenSeg cannot segment visual concepts that do not have associated segmentation proposals.The limitation follows from representing images through a small number of predicted mask regions.

C Architecture of the cross-attention module

The cross-attention module repeatedly updates mask queries using position-augmented image features to generate segmentation mask queries. The experiments stack this module three times.

  • C Architecture of the cross-attention module: The region-image cross-attention module interacts mask queries with position-augmented image features to produce updated mask queries.The first query set is randomly initialized, and the module is stacked repeatedly for region segmentation.
  • C Architecture of the cross-attention module: The module is stacked three times in the experiments, with T = 3.Adding query self-attention before query-image cross-attention does not improve performance.

D Mask generalization on MSeg dataset

The MSeg experiments test whether a model trained on one dataset generalizes across multiple datasets, with MSeg training providing an upper-bound reference. The paper also visualizes mask proposals in unseen scenes.

  • D Mask generalization on MSeg dataset: MSeg aggregates training images and annotations from all datasets and provides the best results, serving as a performance upper bound.Training on COCO performs slightly worse, indicating reduced cross-dataset generalization relative to the aggregated setup.
  • D Mask generalization on MSeg dataset: The experiments evaluate whether a model trained on a single dataset can generalize to multiple datasets using recall at an IoU of 0.5.The setup uses curated MSeg annotations and summarizes results in Table 6.
  • D Mask generalization on MSeg dataset: Figure 7 displays all 128 mask proposals generated for the same unseen scene.Figure 4 presents a subset of predicted masks for that scene.

F Ablation on batch size

The batch-size ablation compares unsynchronized and synchronized contrastive-loss training while distinguishing global, local, and effective loss batch sizes.

  • F Ablation on batch size: The experiments compare OpenSeg with unsynchronized and synchronized contrastive loss.Synchronized loss uses examples from all cores, whereas unsynchronized loss uses each core’s local examples.
  • F Ablation on batch size: The default global batch size is 1024, with a local batch size of 16 across 64 cores.These settings are used unless otherwise stated.
  • F Ablation on batch size: The effective contrastive-loss batch size is 16 for unsynchronized training and 1024 for synchronized training.The distinction follows from computing the loss locally or across all cores.

G Importance of segmentation loss

The ablations show that segmentation loss is necessary for learning OpenSeg’s visual grouping, while mask pseudo-labels and segmentation loss on all examples improve generalization beyond COCO categories.

  • G Importance of segmentation loss: Grounding loss alone produces significantly worse performance than training with both grounding and segmentation losses.This result is presented as evidence that segmentation loss matters for learning visual grouping.
  • G Importance of segmentation loss: Mask pseudo-label training worsens mIoU on PC-59 but improves performance on A-847, PC-459, and A-150.The latter datasets include categories outside COCO, and the results indicate improved OpenSeg generalization.
  • G Importance of segmentation loss: The Table 8 ablation sets segmentation loss to zero on COCO examples or Loc. Narr. examples when training on the combined dataset.In both cases, the table caption reports a performance drop.

H Ablation on randomly dropping words

OpenSeg randomly drops nouns from captions during training, and intermediate keep probabilities produce the best reported performance.

  • H Ablation on randomly dropping words: The best results occur when the noun keep probability is 0.5 or 0.75.Table 9 compares different probabilities for retaining extracted caption nouns.
  • H Ablation on randomly dropping words: Randomly dropping caption nouns within a certain probability range prevents overfitting and improves performance.The nouns are extracted from captions and retained independently according to the keep probability.
  • H Ablation on randomly dropping words: Table 9 denotes the noun-retention probability as kp and reports kp = 0.5/0.75 as the best setting.The table caption summarizes the same ablation using the abbreviation kp.

I Ensembling and prompt engineering

OpenSeg improves open-vocabulary segmentation through ensembling and prompt engineering, while visualizations illustrate its predicted masks and per-pixel outputs across random A-150 examples.

  • I Ensembling and prompt engineering: Ensembling combines multiple descriptions for the same dataset category because image captions and test labels may use different names.Examples include person-related synonyms such as man, woman, child, and people.
  • I Ensembling and prompt engineering: Prompt engineering adds short contextual phrases to polysemous class names to target the intended dataset meaning.For example, fan becomes ceiling fan or floor fan when the dataset denotes a cooling machine.
  • I Ensembling and prompt engineering: Overlapping categories can penalize predictions that are semantically valid, and this problem becomes more frequent as vocabulary size increases.Examples include roof versus building and clothes versus person.
  • I Ensembling and prompt engineering: Grounding mIoU reduces ambiguity-related evaluation issues compared with standard mIoU.The metric uses ground-truth classes present in an image as text queries and has less of the class-overlap ambiguity.
  • I Ensembling and prompt engineering: OpenSeg’s improvement is smaller for Grounding mIoU than mIoU because Grounding mIoU has less class-name ambiguity.With COCO+Loc. Narr. training, the reported averages are +1.2 versus +1.8 across four benchmarks.
  • I Ensembling and prompt engineering: The model predicts a full set of 128 segmentation masks for possible image organizations.The visualization displays the complete set of predicted segmentation masks.
  • J Visualization of segmentation predictions: Figures 8–11 compare OpenSeg outputs with ground-truth masks and per-pixel predictions without segmentation proposals on random A-150 images.Figures 8–9 show Parts 1–2, while Figures 10–11 show Parts 3–4 of the visualization set.
Loading 2112.12143v2…