PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning to Generate Text-grounded Mask for Open-world Semantic Segmentation from Only Image-Text Pairs

Junbum Cha, Jonghwan Mun, Byungseok Roh

arXiv:2212.00785v2cs.CV

TL;DR

Open-world segmentation must localize arbitrary concepts using only image-text pairs, yet prior contrastive methods learn image-text rather than region-text alignment. TCL embeds text grounding within contrastive learning to learn region-text alignment directly, and it achieves state-of-the-art zero-shot segmentation on all 8 benchmarks. The paper also introduces a unified evaluation protocol for fair comparison.

  • Problem

    Open-world segmentation lacks dense pixel-level supervision, while existing methods train image-text alignment although inference requires region-text alignment.

  • Method

    TCL generates text-grounded segmentation masks, extracts grounded region embeddings, and applies contrastive learning between regions and their texts without dense annotations.

  • Results

    TCL achieves state-of-the-art zero-shot segmentation performance on all 8 benchmarks, substantially surpassing previous methods under a unified evaluation protocol.

  • Takeaways & Limitations

    Explicitly learning region-text alignment provides a framework for open-world segmentation from image-text pairs and motivates a research direction beyond transferred image-level alignment.

  • Takeaways & Limitations

    Semi-supervised alternatives remain constrained by costly dense annotation and domains where such annotation is available.

Abstract

from arXiv · show

We tackle open-world semantic segmentation, which aims at learning to segment arbitrary visual concepts in images, by using only image-text pairs without dense annotations. Existing open-world segmentation methods have shown impressive advances by employing contrastive learning (CL) to learn diverse visual concepts and transferring the learned image-level understanding to the segmentation task. However, these CL-based methods suffer from a train-test discrepancy, since it only considers image-text alignment during training, whereas segmentation requires region-text alignment during testing. In this paper, we proposed a novel Text-grounded Contrastive Learning (TCL) framework that enables a model to directly learn region-text alignment. Our method generates a segmentation mask for a given text, extracts text-grounded image embedding from the masked region, and aligns it with text embedding via TCL. By learning region-text alignment directly, our framework encourages a model to directly improve the quality of generated segmentation masks. In addition, for a rigorous and fair comparison, we present a unified evaluation protocol with widely used 8 semantic segmentation datasets. TCL achieves state-of-the-art zero-shot segmentation performances with large margins in all datasets. Code is available at https://github.com/kakaobrain/tcl.

1. Introduction

Open-world segmentation seeks arbitrary-concept segmentation from image-text pairs without dense annotations, but existing methods face a training–testing alignment discrepancy. TCL directly learns region-text alignment and achieves state-of-the-art performance across eight benchmarks.

  • Open-world segmentation targets arbitrary semantic concepts rather than a small set of predefined categories.
  • Image-text pairs provide broad conceptual knowledge, but precise localization remains difficult without dense annotations.Dense annotations improve benchmark performance but limit scalability and applicable domains.
  • Existing methods learn image-text alignment during training and transfer it to region-text alignment at inference.This transfer creates a discrepancy between the alignment learned during training and that required for segmentation.
  • TCL incorporates text grounding into contrastive learning to generate masks, compute grounded region embeddings, and align regions with text.Its contrastive objective is directly affected by segmentation quality, enabling end-to-end training of the grounder.
  • TCL achieves state-of-the-art zero-shot segmentation performance with large margins on all 8 datasets under a unified evaluation protocol.The protocol supports fair, direct comparison with existing methods.

2. Related Works

Related work distinguishes semi-supervised methods that use dense masks from unsupervised methods trained only on image-text pairs. Without dense supervision, existing approaches transfer image-level alignment to regions through heuristics or clustering, while dense annotations constrain scalability and applicable domains.

  • Open-world segmentation extends zero-shot recognition to arbitrary concepts rather than predefined target categories.
  • Semi-supervised approaches combine dense segmentation annotations with image-text pairs to learn segmentation and expand vocabulary.
  • Dense annotation is costly, and its availability limits the applicable domains of semi-supervised methods.
  • Unsupervised methods learn segmentation from image-text pairs alone, avoiding human annotators and broadening potential domains.
  • Existing unsupervised methods transfer image-level alignment to region-level alignment using heuristic modification or clustering.MaskCLIP uses a heuristic image-encoder modification, while other methods use clustering-based strategies.

3. Methods

TCL replaces image-level contrastive alignment with text-grounded region alignment by generating masks, extracting masked-region embeddings, and training the grounder through contrastive losses.

  • Overview: TCL addresses the train-test discrepancy by directly learning region-text alignment from image-text pairs without dense annotations.Conventional objectives learn image-text alignment, whereas segmentation requires region-text alignment at inference.
  • Grounder: The grounder uses frozen CLIP image and text encoders plus a trainable grounding decoder to produce pixel-level embeddings aligned with text.The image encoder supplies global and patch-level features, the text encoder supplies text embeddings, and the decoder converts dense features into pixel-level embeddings.
  • Grounder: The model predicts text-grounded masks from text and pixel-level embeddings, then uses those masks to extract text-grounded image embeddings for contrastive learning.The generated masks replace global image embeddings or weight pixel-level embeddings, allowing the grounding process to participate in end-to-end training.
  • Text-grounded Contrastive Learning: TCL applies image-level, feature-level, and area-level losses, with smooth regularization, to train masks using positive and negative image-text pairs.Image-level loss contrasts masked-image and text embeddings; feature-level loss uses negative masks; area loss constrains mask sizes.
  • Text-grounded Contrastive Learning: The area loss prevents whole-image mask collapse by matching positive and negative mask areas to priors.The negative prior is 0.0, while the positive prior is set to 0.4 based on the average text-described region area measured by MaskCLIP on CC3M.

4. Experiments

The experiments establish a unified evaluation setting across eight semantic segmentation datasets and compare TCL with existing open-world methods. TCL achieves large-margin gains across datasets, while ablations show that its specialized losses and grounding decoder are central to precise region-text alignment.

  • Experiment Settings: The unified protocol evaluates open-world segmentation on eight benchmarks using consistent class names, image processing, and mIoU.The benchmarks include datasets with and without background classes, while class-name expansion tricks and Dense CRF refinement are excluded.
  • Zero-shot Transfer to Semantic Segmentation: TCL outperforms all existing methods by large margins across all evaluated datasets.The comparison reports zero-shot segmentation performance using mIoU under the unified protocol.
  • Zero-shot Transfer to Semantic Segmentation: The grounding decoder produces more precise, finer, and less noisy text-grounded masks than the unrefined CLIP dense features.The visualization compares positive and negative prompts with and without the grounding decoder.
  • Qualitative Results: Qualitative comparisons show distinct baseline errors, including large-group segmentation, background-region errors, and imprecise target capture.Wild-image examples also include concepts absent from conventional segmentation datasets and free-form text prompts.
  • Ablation Studies: Adding the grounding decoder without TCL loss does not improve performance, whereas the complete TCL framework improves localization.The ablation compares a MaskCLIP-based baseline, a decoder-only variant, and the complete framework.
  • Ablation Studies: The proposed TCL loss improves performance from 61.1 to 77.4, while area TCL prevents model collapse and smooth regularization contributes substantially.Using image-level and feature-level TCL losses together provides further gains, and target-dataset fine-tuning is not required.

5. Conclusion

The paper addresses the training–testing alignment discrepancy by learning region-text alignment directly and evaluates TCL under a unified protocol. TCL attains state-of-the-art zero-shot segmentation across eight benchmarks.

  • TCL addresses the discrepancy between image-text alignment during training and region-text alignment during testing.
  • TCL incorporates grounding into contrastive learning to align text with text-grounded regions and segmentation masks.
  • The unified evaluation protocol enables fair comparison of existing open-world segmentation methods.
  • TCL achieves state-of-the-art zero-shot segmentation performance on all 8 benchmarks, surpassing previous methods.

B. Architecture Details

TCL adapts pretrained CLIP features with a grounding decoder while preserving pretrained knowledge through a separate inference-time branch. The decoder uses gated convolutions, upsampling, and mixed mask outputs.

  • The knowledge-preservation branch produces parameter-free masks from CLIP dense features and mixes them with decoder masks using w_kp = 0.3.This branch is used only during inference and functions as a cost-free ensemble.
  • The model freezes pretrained CLIP encoders and trains a grounding decoder to adapt image-text alignment to region-text alignment.Only the last image-encoder block is later unfrozen after 30,000 iterations.
  • The grounding decoder uses four gated convolution blocks, where learned gates modulate convolution outputs added to skip connections.The gated update is x' = x + tanh(g) · Conv(x).
  • Upsamplers increase feature-map resolution for high-resolution segmentation, using nearest-neighbor and bilinear interpolation.

C. Fair Comparison

The paper evaluates methods under unified and additional fairness settings, including standardized refinement and comparable dataset scale. TCL remains the strongest method in these comparisons.

  • The evaluation introduces a unified protocol and additional comparisons because fairness conditions can be controversial.
  • Fair comparison with refinement methods: TCL achieves state-of-the-art performance with significant margins both with and without the same PAMR refinement applied to all methods.
  • Fair comparison with refinement methods: +3.3 mIoU is TCL’s gain from PAMR, compared with +2.1 for ReCo, +0.6 for GroupViT, and −1.5 for MaskCLIP.
  • Fair comparison in dataset scale: Larger training datasets do not guarantee better performance, and dataset-size effects vary across benchmarks.
  • Fair comparison in dataset scale: TCL outperforms all evaluated GroupViT variants despite comparable training-data scale.

D. Efficiency Analysis

TCL uses efficiency-aware design choices to manage contrastive matching and additional grounded-image encoding. Its higher-resolution masks trade off against inference speed, while scaling remains more challenging than in CLIP.

  • Inference throughput: TCL has slightly lower FPS than GroupViT or MaskCLIP because it produces relatively high-resolution segmentation masks.The mask-resolution/FPS trade-off can be controlled through design choices.
  • PAMR is not relevant to the throughput comparison, while CLIP scores on masked images are reported for robustness analysis.
  • Efficiency-aware designs in TCL: TCL uses late fusion, positive-pair encoding, frozen encoders, and reused text embeddings to reduce computational cost.
  • Efficiency-aware designs in TCL: TCL training takes about 12 hours, with inference throughput comparable to MaskCLIP or GroupViT.
  • Efficiency-aware designs in TCL: Scaling TCL is more challenging than scaling CLIP because its B × B mask is a spatial tensor of size B × B × H × W.

E. Additional Details and Experiments

Additional experiments examine masked-image robustness, unified zero-shot evaluation, and TCL’s failure cases. TCL handles masked images effectively, achieves state-of-the-art performance under partial-class evaluation, but still struggles with precise boundaries and noisy predictions.

  • Masked-image robustness: CLIP scores improve for masked images, especially in complex contexts, suggesting that masking removes noisy context and focuses on target objects.The analysis evaluates masked images against positive texts and reports improved scores in several complex-image examples.
  • Unified evaluation: TCL achieves state-of-the-art zero-shot segmentation performance with a large margin on every dataset under the partial-classes protocol.The protocol evaluates selected classes from PASCAL VOC20, PASCAL Context59, and COCO-Stuff.
  • Failure cases: TCL’s qualitative errors include inaccurate segment boundaries, such as mountain regions containing sky and cell-phone regions containing hands or arms.The paper identifies precise boundary capture as a fundamental challenge without dense annotations.
  • Benchmark ambiguity: Benchmark labels can be ambiguous because classes have similar meanings, superset-subset relations, or multiple valid semantics in one segment.Examples include clouds versus fog, hill versus mountain, and broccoli, vegetable, and food-other.

G. Analysis on Model Behavior

TCL’s behavior depends on prompt specificity: more specific prompts better capture intended regions, while ambiguous cases may still be semantically acceptable despite differing from ground truth. This controllability can support class-name expansion, although the paper avoids dataset-specific tricks.

  • Prompt specificity: More specific text prompts produce better segmentation results and improve the model’s controllability.The model tends to capture the intended region more accurately as prompt specificity increases.
  • Ambiguous cases: Some predictions differ from ground truth yet remain semantically correct, whereas other errors are clearly incorrect predictions.Figure 9 distinguishes acceptable semantic alternatives from genuine errors.
  • Prompt controllability: Prompt controllability can be exploited through class-name expansion to maximize benchmark performance, but TCL avoids such dataset-dependent tricks.The authors omit these tricks to prevent overestimation of performance.

H. Additional Qualitative Results

Additional qualitative comparisons in complicated scenes show that TCL generally produces cleaner and better segmentations than the baselines, while image resolution limits recognition of small objects.

  • Complicated scenes: In complicated Cityscapes and COCO-Stuff scenes, GroupViT and MaskCLIP generate few segments, while ReCo produces noisy predictions.The comparisons include scenes with at least three segments.
  • Complicated scenes: TCL also produces partially noisy results but is relatively cleaner and better than the comparison baselines.The paper gives failures such as MaskCLIP missing buildings and GroupViT confusing roads with traffic lights.
  • Evaluation resolution: The unified protocol uses 448 × 896 Cityscapes images instead of the widely used 1024 × 2048 resolution, which can hinder recognition of small objects.The paper notes that higher resolution can help recognize persons in the qualitative examples.

H.2. Additional Qualitative Examples in the Wild

In-the-wild examples test TCL on proper nouns, colors, letters, and subclasses. The results show recognition and segmentation of varied concepts, while baseline behavior differs from the fixed-benchmark comparisons.

  • In-the-wild concepts: The in-the-wild evaluation tests proper nouns, same-object colors, letters, and subclasses.Examples include Frodo, colored bananas, MMU, and Corgi.
  • In-the-wild concepts: TCL recognizes and segments varied concepts in web images in the wild.The examples cover multiple semantic types rather than only standard benchmark classes.
  • Baseline comparison: ReCo produces relatively plausible segmentation maps in the wild, unlike its relative standing in the quantitative fixed-benchmark evaluation.The baseline behavior differs from the tendencies observed for GroupViT and MaskCLIP.
Loading 2212.00785v2…