Refining Open-Vocabulary Semantic Segmentation via Regional Semantics and Visual Prototypes

1. INTRODUCTION

• CLIP-based segmentation builds on image-level visual-language alignment, but dense prediction requires pixel-level alignment and existing approaches can produce noisy masks.The introduction also identifies the vision-text modality gap as a constraint on existing methods.Source: §1 · p. 1
• The regional semantic preservation module modifies CLIP’s final attention layer to improve regional coherence in pixel-level visual features.It selectively reuses pixel-level features, calculating similarity between adjacent features while modifying only the final layer.Source: §1 · pp. 1–2, Figure 1 · p. 2
• Visual feature prototypes from the unlabeled set provide an auxiliary dense predictor that complements text-based predictions to mitigate the modality gap.The method also trains an additional feature extractor using model-generated masks of unlabeled samples and updates it with EMA.Source: §1 · pp. 1–2
• The method achieves state-of-the-art annotation-free semantic segmentation performance compared with existing methods.The introduction states the comparison without specifying dataset-level scores.Source: §1 · p. 2

2.1. Preliminaries

• The visual encoder uses attention-based pooling, with the averaged sub-feature as query and spatial sub-features as keys and values; CLIP compares image and text features using cosine similarity.The class probability calculation uses temperature τ.Source: §2.1 · p. 2

2.2. Regional Semantic Preservation Module

• Replacing q-k mapping with v-v mapping in CLIP’s final attention layer emphasizes similarity between adjacent pixel features and aggregates semantically related regions.The change is confined to the final self-attention block to preserve feature extraction functionality with minimal adjustments.Source: §2.2 · p. 2

2.3. Prediction Enhancement with Visual Prototypes

• The visual feature memory module selects top-K confident pixel features per category, averages them into visual prototypes, and combines its predictions with text-based predictions residually.The visual predictor is intended to alleviate the modality gap while retaining CLIP text embeddings’ semantic knowledge.Source: §2.3 · pp. 2–3, Figure 2 · p. 3

2.4. Self-training

• An auxiliary feature extractor is optimized using refined-predictor segmentation outcomes as pseudo-labels, and EMA smooths its parameters during inference.The reported inference setting is β = 0.9999.Source: §2.4 · p. 3

3.1. Implementation Details

• Evaluations cover ADE20K, Cityscapes, COCO Stuff164k, PASCAL Context, and VOC12AUG, using CLIP-ViT-B/16 and CLIP-ResNet-50 backbones.The reported setup includes 85 prompt templates and considers training-free prediction and self-training.Source: §3.1 · p. 3

3.2. Main Results

• 35.9% average mIoU is the highest among compared methods, and the method achieves state-of-the-art results on ADE20K (17.2%), Cityscapes (32.5%), and PASCAL Context (33.7%).Table 2 compares mIoU (%) on five semantic segmentation benchmarks; MaskCLIP† is the baseline without refinement.Source: Table 2 · p. 4, §3.2 · p. 4
• The method also improves dense prediction with CLIP-ResNet-50 across five datasets in both training-free prediction and self-training settings.Table 3 reports mIoU (%) under CLIP-ResNet-50, marking the training-free setting with ♢.Source: §3.2 · p. 4, Table 3 · p. 4
• Figure 4 presents qualitative predictions across five datasets, with examples showing object-boundary delineation, localization, and local-context understanding.Each image group pairs an input image and ground truth with the proposed method’s visualization.Source: §3.2 · p. 4, Figure 4 · p. 4
• Ablations across all five datasets with CLIP-ViT-B/16 report improvements from the proposed components in both training-free prediction and self-training.Table 4 identifies refined self-attention, visual-memory embeddings, and the optimization strategy as the components evaluated.Source: §3.2 · p. 4, Table 4 · p. 4

• The paper studies annotation-free open-vocabulary semantic segmentation, where CLIP visual features align with text embeddings for dense prediction.Source: §4 · p. 4
• Its attention computation encourages semantic consistency among adjacent regions, while a visual feature memory supplies prototypes as an additional predictor alongside text embeddings.Source: §4 · p. 4
• The approach achieves state-of-the-art performance across multiple benchmarks, which the conclusion presents as evidence for the proposed components.Source: §4 · p. 4

• The authors state that their numerical simulation study required no ethical approval.Source: §6 · p. 5