PaperlayerView exact revision on arXiv

Source-linked AI summary

SED: A Simple Encoder-Decoder for Open-Vocabulary Semantic Segmentation

Bin Xie, Jiale Cao, Jin Xie, Fahad Shahbaz Khan, Yanwei Pang

arXiv:2311.15537v2cs.CV

TL;DR

Open-vocabulary semantic segmentation requires adapting image-level vision-language models to pixel-level predictions across arbitrary categories. SED uses hierarchical cost-map generation, gradual feature fusion, and category early rejection; it reports strong accuracy–speed trade-offs, including 4.7 times acceleration on PC-459. The method still struggles with near-synonym categories.

  • Problem

    Open-vocabulary semantic segmentation must assign pixels to arbitrary categories, but existing methods face limitations when adapting image-level vision-language models to pixel-level prediction.

  • Method

    SED combines hierarchical encoder-based pixel-level cost-map generation with a gradual fusion decoder and category early rejection.

  • Results

    SED achieves superior performance across multiple datasets and provides 4.7 times acceleration on PC-459 without significant segmentation-performance degradation.

  • Takeaways & Limitations

    SED provides an accuracy–speed trade-off for open-vocabulary segmentation by combining hierarchical features with early rejection of non-existing categories.

  • Takeaways & Limitations

    SED sometimes struggles to recognize near-synonym categories as distinct classes.

Abstract

from arXiv · show

Open-vocabulary semantic segmentation strives to distinguish pixels into different semantic groups from an open set of categories. Most existing methods explore utilizing pre-trained vision-language models, in which the key is to adopt the image-level model for pixel-level segmentation task. In this paper, we propose a simple encoder-decoder, named SED, for open-vocabulary semantic segmentation, which comprises a hierarchical encoder-based cost map generation and a gradual fusion decoder with category early rejection. The hierarchical encoder-based cost map generation employs hierarchical backbone, instead of plain transformer, to predict pixel-level image-text cost map. Compared to plain transformer, hierarchical backbone better captures local spatial information and has linear computational complexity with respect to input size. Our gradual fusion decoder employs a top-down structure to combine cost map and the feature maps of different backbone levels for segmentation. To accelerate inference speed, we introduce a category early rejection scheme in the decoder that rejects many no-existing categories at the early layer of decoder, resulting in at most 4.7 times acceleration without accuracy degradation. Experiments are performed on multiple open-vocabulary semantic segmentation datasets, which demonstrates the efficacy of our SED method. When using ConvNeXt-B, our SED method achieves mIoU score of 31.6\% on ADE20K with 150 categories at 82 millisecond ($ms$) per image on a single A6000. We will release it at \url{https://github.com/xb534/SED.git}.

1. Introduction

Open-vocabulary semantic segmentation addresses the limitation of closed-set models, while prior vision-language approaches still face backbone and efficiency constraints. SED combines hierarchical cost-map generation, gradual fusion, and category early rejection to improve the accuracy–speed trade-off.

  • Problem: Open-vocabulary segmentation aims to assign pixels to arbitrary semantic categories rather than a fixed closed set.
  • Motivation: Prior single-stage methods use pixel-level vision-language features or cost maps but commonly rely on plain-transformer backbones.
  • Contributions: SED uses a hierarchical encoder-based cost map, gradual fusion decoder, and category early rejection for open-vocabulary segmentation.The hierarchical encoder preserves spatial information, while early rejection removes non-existing categories during decoding.
  • Contributions: 4.7 times acceleration is obtained on PC-459 without significant segmentation-performance degradation.
  • Results: 35.2% mIoU on A-150 and 22.6% on PC-459 are achieved with ConvNeXt-L.

2. Related Work

Related work progresses from conventional segmentation and vision-language modeling toward two-stage and single-stage open-vocabulary segmentation. SED follows the single-stage cost-map direction while replacing plain-transformer processing with a hierarchical encoder and adding efficiency-oriented decoding.

  • Semantic Segmentation: Traditional semantic segmentation developed through FCN- and transformer-based approaches that extract local and nonlocal contextual information.
  • Vision-Language Models: Vision-language models learn aligned image-text representations and provide the foundation for recognizing open-vocabulary categories.
  • Open-Vocabulary Semantic Segmentation: Open-vocabulary methods use either two-stage mask-proposal classification or single-stage pixel-level prediction frameworks.
  • Open-Vocabulary Semantic Segmentation: Single-stage methods include MaskCLIP, SAN, FC-CLIP, and CAT-Seg, which respectively use pixel features, adapters, convolutional CLIP, or cost-map refinement.
  • Open-Vocabulary Semantic Segmentation: SED differs from CAT-Seg by using a simpler hierarchical-encoder framework, linear input-size complexity, and category early rejection for faster inference.

3. Method

SED generates pixel-level image-text cost maps with a hierarchical encoder and combines them with multi-scale features through a gradual fusion decoder. Category early rejection reduces decoder computation by filtering likely absent categories during inference.

  • 3.1. Hierarchical Encoder-based Cost Map: SED’s hierarchical encoder produces pixel-level image-text cost maps and multi-scale features for the decoder.ConvNeXt extracts F2, F3, F4, and F5 at strides 4, 8, 16, and 32 pixels, while text embeddings represent category prompts.
  • 3.1. Hierarchical Encoder-based Cost Map: The cost map computes cosine similarity between aligned visual features and category text embeddings across spatial positions and prompt templates.
  • 3.2. Gradual Fusion Decoder: The gradual fusion decoder combines feature aggregation and skip-layer fusion to turn noisy low-resolution cost maps into high-resolution segmentation features.FAM models local-region and class relationships, while SFM incorporates detail from shallow hierarchical features.
  • 3.2. Gradual Fusion Decoder: The decoder stops gradient back-propagation from skip-layer fusion directly to the image encoder to avoid degrading open-vocabulary segmentation.
  • 3.2. Gradual Fusion Decoder: Hierarchical encoding with skip-layer fusion significantly improves performance, likely because it supplies local information while stopped gradients preserve open-vocabulary ability.
  • 3.2. Gradual Fusion Decoder: Category early rejection selects top-k categories from intermediate segmentation maps and removes nonselected category features before later decoder layers.During training, auxiliary convolutional predictions are supervised by ground truth; during inference, selected categories are propagated onward.

4. Experiments

SED is evaluated across multiple open-vocabulary segmentation datasets, where it generally improves accuracy over comparable methods while maintaining fast inference. Ablations attribute these gains to the hierarchical encoder, gradual fusion decoder, and category early rejection.

  • Comparisons With State-of-the-art Methods: SED outperforms comparable methods on all five test sets without requiring additional backbone or dataset.The comparison uses mIoU across five test sets and contrasts SED with methods based on comparable vision-language models.
  • Comparisons With State-of-the-art Methods: SED-fast is 1.9% better than SAN at similar speed with the base model, while the large model is 0.9% better and 1.8 times faster.These accuracy and inference-time comparisons are reported on A-150.
  • Ablation Study: The full SED configuration reaches 11.2%, 18.6%, and 31.8% mIoU on A-847, PC-459, and A-150, respectively, after adding hierarchical encoding and gradual fusion.The full configuration improves over the baseline by 3.9%, 3.7%, and 8.1% on those datasets.
  • Ablation Study: Hierarchical encoding improves the baseline by 2.6%, 2.3%, and 4.5% mIoU on A-847, PC-459, and A-150, respectively.The ablation compares a hierarchical encoder against the original plain-transformer baseline.
  • Ablation Study: Using three decoder layers improves PC-459 by 1.6% over one layer, while fusing different feature maps also significantly improves performance.These findings come from the gradual fusion decoder ablations.

5. Conclusion

SED combines hierarchical cost-map generation with gradual fusion decoding and category early rejection for open-vocabulary semantic segmentation. Experiments show effectiveness in accuracy and speed, while near-synonym categories remain challenging.

  • SED combines hierarchical encoder-based cost-map generation, gradual fusion decoding, and category early rejection for open-vocabulary semantic segmentation.The decoder uses hierarchical encoder features and cost maps to produce high-resolution segmentation features while rejecting non-existing categories early.
  • Experiments on multiple datasets demonstrate SED’s effectiveness in both segmentation accuracy and inference speed.
  • Near-synonym categories remain difficult for SED to recognize, motivating future category-attention or fine-grained-data strategies.
Loading 2311.15537v2…