Source-linked AI summary
CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation
Seokju Cho, Heeseong Shin, Sunghwan Hong, Anurag Arnab, Paul Hongsuck Seo, Seungryong Kim
TL;DR
Open-vocabulary segmentation requires pixel-level labels from broad text-defined vocabularies, but image-level vision-language pretraining and prior fine-tuning can leave unseen classes misaligned. CAT-Seg adapts CLIP by aggregating image-text cosine-similarity costs with spatial and class aggregation while fine-tuning its encoders. It reports state-of-the-art benchmark performance, gains of +3.6 mIoU and +8.1 mIoU on two benchmarks, and over ×3.7 faster inference than region-text methods.
Problem
Image-level supervision in vision-language models creates a mismatch for pixel-level segmentation, while CLIP fine-tuning can overfit seen classes and misalign unseen classes.
Method
CAT-Seg forms and aggregates a multimodal cost volume from cosine similarities between CLIP image and text embeddings, using spatial and class aggregation while fine-tuning CLIP encoders.
Results
+3.6 mIoU on A-847 and +8.1 mIoU on PC-459 versus recent state of the art, with inference over ×3.7 faster than region-text methods.
Takeaways & Limitations
Cost aggregation adapts CLIP for segmentation across seen and unseen classes and achieves state-of-the-art results on standard benchmarks and large domain-shift scenarios.
Takeaways & Limitations
Performance in particular scientific domains is comparable to random predictions, possibly because CLIP has limited knowledge in those domains.
Abstract
from arXiv · showhide
Open-vocabulary semantic segmentation presents the challenge of labeling each pixel within an image based on a wide range of text descriptions. In this work, we introduce a novel cost-based approach to adapt vision-language foundation models, notably CLIP, for the intricate task of semantic segmentation. Through aggregating the cosine similarity score, i.e., the cost volume between image and text embeddings, our method potently adapts CLIP for segmenting seen and unseen classes by fine-tuning its encoders, addressing the challenges faced by existing methods in handling unseen classes. Building upon this, we explore methods to effectively aggregate the cost volume considering its multi-modal nature of being established between image and text embeddings. Furthermore, we examine various methods for efficiently fine-tuning CLIP.
1. Introduction
Open-vocabulary segmentation must transfer image-level vision-language knowledge to pixel-level labeling across seen and unseen classes. CAT-Seg addresses this mismatch by aggregating image-text costs and fine-tuning CLIP, achieving strong benchmark and domain-shift results.
- Motivation: Open-vocabulary semantic segmentation assigns pixel labels from an unbounded set of text-defined classes, while CLIP-like models primarily receive image-level supervision.This creates a disparity between pretraining and pixel-level segmentation.
- Motivation: Existing region-based approaches only partially bridge the gap because CLIP still distinguishes conceptualized regions from entire images.
- Motivation: Fine-tuning CLIP commonly overfits to seen classes, misaligning the joint embedding space for unseen classes and motivating frozen encoders in prior methods.
- Approach: CAT-Seg aggregates cosine similarities between image and text embeddings as a multimodal cost volume, then refines it through spatial and class aggregation.The cost volume can be viewed as rough class-grounded semantic masks that are refined into predictions.
- Approach: Cost aggregation operates on similarity scores rather than embeddings, reducing overfitting risk while preserving the direct image-text embedding alignment during CLIP adaptation.
- Results: +3.6 mIoU on A-847 and +8.1 mIoU on PC-459 versus recent state of the art, with inference over ×3.7 faster than region-text methods.CAT-Seg also reports state-of-the-art performance on standard and extreme domain-shift scenarios.
2. Related Work
Prior work adapts CLIP through pixel embeddings, regions, tokens, adapters, or mask prediction, but CAT-Seg instead fine-tunes CLIP encoders using a cost volume formed solely from CLIP embeddings. Its cost aggregation is multimodal because the matching costs connect image and text modalities.
- Open-vocabulary semantic segmentation: LSeg aligns pixel-level visual embeddings with CLIP text embeddings, while OpenSeg identifies local image regions and correlates them with text embeddings.
- Open-vocabulary semantic segmentation: ZegCLIP and SAN predict masks with CLIP embeddings but add learnable tokens or adapter layers trained on seen classes.
- Open-vocabulary semantic segmentation: CAT-Seg avoids external layers and fine-tunes CLIP encoders by aggregating a cost volume obtained solely from CLIP image and text embeddings.
- Fine-tuning vision-language models: Prompt-learning and adapter methods adapt CLIP without directly tuning its encoder, whereas CAT-Seg investigates encoder fine-tuning for segmentation.
- Cost aggregation: Unlike conventional cost aggregation between image pairs, CAT-Seg aggregates cosine-similarity costs generated jointly from image and text modalities.
3. Methodology
CAT-Seg constructs a CLIP-based segmentation pipeline by computing image–text cosine-similarity costs, then aggregating them spatially and across classes before decoding high-resolution predictions. The framework also uses CLIP embeddings for guidance and investigates efficient encoder fine-tuning.
- Open-vocabulary segmentation assigns every image pixel a label from a variable set of text-described categories.
- Cost computation and embedding: CAT-Seg computes a cost volume by cosine similarity between dense CLIP image embeddings and text embeddings for candidate classes.The dense image embeddings index spatial positions, while text embeddings index categories.
- Spatial cost aggregation: Spatial aggregation refines each class-specific cost slice using Transformer-based processing with global or semi-global receptive fields.The implementation uses Swin Transformer blocks, including local-window and shifted-window self-attention.
- Class cost aggregation: Class aggregation models relationships among categories with a position-free Transformer, supporting varying class counts and permutation invariance.A linear Transformer is used because this stage does not require spatial structure and benefits from linear complexity in the number of tokens.
- CAT-Seg framework: The aggregated cost volume is decoded through upsampling and aggregation with CLIP feature maps to produce high-resolution segmentation predictions.Intermediate CLIP ViT features provide detailed representations without adding a separate feature backbone.
- CAT-Seg framework: Embedding guidance incorporates projected CLIP image and text embeddings into spatial and class aggregation, supplying spatial structure and contextual information through query and key inputs.The method concatenates projected embeddings with the corresponding cost features and reports that query-key guidance is sufficient.
4. Experiments
Experiments evaluate CAT-Seg on standard and multi-domain benchmarks, then analyze aggregation, fine-tuning, components, generalization, and efficiency. CAT-Seg reports strong benchmark performance, improved unseen-class behavior, and efficient inference, while medical-domain results remain limited.
- Standard benchmarks: 16.0 mIoU on A-847 and 23.8 mIoU on PC-459 exceed the previous state of the art by 29% and 52%, respectively.These results are reported for models using the ViT-L/14 vision-language model.
- Multi-domain evaluation: CAT-Seg achieves the highest mean score on MESS and particularly excels in general, agriculture, and biology domains.Performance is inconsistent with respect to vision-language model size in medical sciences and engineering.
- Aggregation and fine-tuning: Cost aggregation benefits significantly from CLIP fine-tuning, whereas feature aggregation gains only marginally.The comparison uses shared baseline architectures differing in whether they aggregate concatenated features or image-text cosine similarities.
- Aggregation and fine-tuning: Cost aggregation identifies the unseen class “birdcage,” while feature aggregation overfits to the seen class “bucket.”The qualitative comparison visualizes predictions from the fine-tuned feature- and cost-aggregation variants.
- Component analysis: Adding spatial and class aggregation, embedding guidance, and the upsampling decoder improves performance across the evaluated benchmarks.The component analysis reports that embedding guidance further improves all benchmarks, while the decoder produces consistent improvements.
- Fine-tuning strategies: Fine-tuning only the query and value projections gives the best performance improvement among tested methods while remaining efficient.Fine-tuning both CLIP encoders performs better than fine-tuning only the image or text encoder in the authors’ framework.
- Efficiency and generalization: CAT-Seg is over 3.7× faster at inference than region-text methods and avoids an additional mask generator.Efficiency comparisons measure learnable and total parameters, training and inference time, and inference GFLOPs.
5. Conclusion
CAT-Seg adapts CLIP to open-vocabulary semantic segmentation by aggregating image-text cosine similarities and fine-tuning CLIP encoders. It surpasses prior state of the art on standard benchmarks and under substantial domain differences.
- CAT-Seg aggregates cosine-similarity scores between CLIP image and text embeddings for open-vocabulary semantic segmentation.
- The framework fine-tunes CLIP encoders to adapt the model to the downstream segmentation task.
- CAT-Seg surpasses previous state-of-the-art methods on standard benchmarks and in scenarios with substantial domain differences.
- Performance across diverse domains supports the promise of cost aggregation for open-vocabulary semantic segmentation.
A. More Results
The supplementary material describes CAT-Seg’s architecture, including embedding guidance and an upsampling decoder that combines cost embeddings with multi-scale CLIP features. These components provide architectural detail for the cost aggregation process and feature refinement.
- Embedding guidance extracts CLIP visual and text embeddings, projects them linearly, and concatenates them with the cost volume before aggregation-layer query and key projections.
- The upsampling decoder applies a transposed convolution to high-resolution CLIP ViT features to generate upsampled feature maps.
- The extracted feature maps increase from 24 × 24 to 48×48 pixels for the first feature map and 96×96 pixels for the second.
- For ViT-B/16 and ViT-L/14, decoder features are taken from different encoder depths for the first and second feature maps.The first feature uses the 8th or 16th layer, while the shallower second feature uses the 4th or 8th layer, respectively.
- The multi-scale features enhance cost embeddings with fine details through a U-Net-like architecture.
B.2. Other Implementation Details
Implementation details use fixed CLIP training settings and a patch-based inference strategy to address high-resolution limitations. Overlapping patches are combined with a resized full image, while ViT’s quadratic sequence-length complexity constrains conventional high-resolution outputs.
- Training constructs the cost volume at H = W = 24 with a 384 × 384 training resolution and official ViT-B/16 or ViT-L/14@336px CLIP weights.
- The proposed inference strategy avoids additional training while addressing the computational burden of positional-embedding upsampling.
- ViT’s quadratic complexity with sequence length limits CAT-Seg’s ability to output conventional resolutions such as 640 × 640 without sacrificing fine details.
- Patch inference divides an image into overlapping 384 × 384 sub-images with 128×128 overlaps and also processes the full image resized to 384 × 384.
- Patch and full-image predictions are merged, with overlapping regions averaged to produce the final prediction.
C.1. Ablation Study of Inference Strategy
The inference-strategy ablation compares using the training resolution with the proposed patch inference strategy. The patch strategy produces large performance gains relative to inference at the training resolution.
- Table 11 compares different inference strategies using CLIP with ViT-L.
- The proposed patch inference strategy yields large performance gains compared with using the training resolution at inference time.
C.2. Ablation on VLM
The ablation evaluates CAT-Seg across various vision-language models and finds that the framework generalizes beyond a single VLM. More powerful VLMs produce better results.
- Table 12 reports results for CAT-Seg with various vision-language models.
- CAT-Seg can be applied to various VLMs.
- Better results are obtained when a more powerful model is applied.
D. More Qualitative Results
The paper provides additional qualitative segmentation results across four benchmark configurations and compares CAT-Seg with other methods on A-847.
- Additional qualitative results are provided on A-847 with 847 categories.
- Additional qualitative results are provided on PC-459 with 459 categories, A-150 with 150 categories, and PC-59 with 59 categories.
- On A-847, CAT-Seg is further compared with ZegFormer, ZSseg, and OVSeg.
E. Limitations
The evaluation follows prior work and uses other segmentation datasets, but ambiguities in their ground-truth maps make evaluation reliability questionable. The section also presents qualitative results across benchmark configurations, including A-847, PC-459, A-150, and PC-59.
- Evaluation uses other segmentation datasets following prior work, but ambiguities in their ground-truth maps make the evaluation dataset somewhat unreliable.
- The paper identifies constructing a more reliable dataset with ground truths accounting for these ambiguities as an intriguing topic.
- Qualitative results cover A-847 with 847 categories, PC-459 with 459 categories, A-150 with 150 categories, and PC-59 with 59 categories.
- A-847 qualitative results include comparisons with ZegFormer, ZSseg, and OVSeg.