PaperlayerView exact revision on arXiv

Source-linked AI summary

Side Adapter Network for Open-Vocabulary Semantic Segmentation

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, Xiang Bai

arXiv:2302.12242v2cs.CVcs.AI

TL;DR

Open-vocabulary semantic segmentation must extend CLIP’s image-level recognition to pixel-level regions without sacrificing its broad category capability. SAN attaches a lightweight side network to frozen CLIP for mask proposals and attention-bias-guided recognition, with end-to-end training. It significantly outperforms prior methods on five benchmarks while using fewer trainable parameters and less computation.

  • Problem

    CLIP lacks the pixel-level recognition capability needed for semantic segmentation, while fine-tuning on smaller segmentation datasets can compromise open-vocabulary recognition.

  • Method

    SAN uses a lightweight side network with separate mask-proposal and attention-bias branches attached to frozen CLIP in an end-to-end, single-forward design.

  • Results

    SAN significantly outperforms previous state-of-the-art methods on five semantic segmentation benchmarks with fewer trainable parameters and lower computation cost.

  • Takeaways & Limitations

    SAN provides a lightweight framework for leveraging frozen CLIP features and adapting CLIP-aware mask prediction and recognition to open-vocabulary segmentation.

  • Takeaways & Limitations

    The feature-fusion design is intentionally simple, and a more sophisticated structure may improve performance.

Abstract

from arXiv · show

This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN). Our approach models the semantic segmentation task as a region recognition problem. A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias which is applied in the CLIP model to recognize the class of masks. This decoupled design has the benefit CLIP in recognizing the class of mask proposals. Since the attached side network can reuse CLIP features, it can be very light. In addition, the entire network can be trained end-to-end, allowing the side network to be adapted to the frozen CLIP model, which makes the predicted mask proposals CLIP-aware. Our approach is fast, accurate, and only adds a few additional trainable parameters. We evaluate our approach on multiple semantic segmentation benchmarks. Our method significantly outperforms other counterparts, with up to 18 times fewer trainable parameters and 19 times faster inference speed. We hope our approach will serve as a solid baseline and help ease future research in open-vocabulary semantic segmentation. The code will be available at https://github.com/MendelXu/SAN.

1. Introduction

Open-vocabulary semantic segmentation aims to recognize and segment arbitrary categories despite CLIP’s image-level training and the limited category coverage of labeled segmentation datasets. SAN addresses this gap with a lightweight, end-to-end side network that generates CLIP-aware mask proposals and attention biases, achieving strong benchmark performance with modest computation and trainable parameters.

  • Labeled segmentation datasets usually cover only tens to hundreds of categories, while data collection and annotation limit further expansion.
  • CLIP’s image-level contrastive training lacks the pixel-level recognition capability required for semantic segmentation.Fine-tuning can address the granularity gap but may compromise open-vocabulary recognition because segmentation datasets are much smaller than vision-language pre-training data.
  • SAN attaches a side adapter to frozen CLIP, predicting mask proposals and attention biases for proposal-wise class recognition.The attention biases guide deeper CLIP layers, and mask proposals are combined with proposal logits during inference.
  • End-to-end training makes SAN’s mask prediction CLIP-aware while reusing CLIP features to keep the side network lightweight.The design also decouples mask prediction from mask recognition and uses shallow CLIP features with deeper CLIP attention biases in a single-forward pipeline.
  • +1.8 mIoU improvements on 5 datasets for ViT-B/16 and +2.3 mIoU improvements for ViT-L/14 are reported over prior methods.With ensemble, the average gaps increase to +2.9 mIoU and +3.7 mIoU, respectively.
  • 8.4M trainable parameters with 64.3 GFLOPs represent only 13% and 20% of [10], and 6% and less than 1% of [22], respectively.

2. Related Works

Related work applies vision-language pre-training and parameter-efficient tuning to open-vocabulary recognition, but segmentation remains difficult because image-level representations do not align with pixel-level recognition. SAN differs from prior approaches through end-to-end, lightweight, CLIP-aware mask prediction and decoupled recognition.

  • Vision-language pre-training learns generic vision-language representations, while CLIP and ALIGN extend this paradigm with large-scale pre-training.
  • Parameter-efficient tuning methods train prompts or adapter modules in fixed CLIP models, mainly for image-level recognition or vision-language tasks.
  • Side-Tuning attaches a side network to a pre-trained model and combines the side and pre-trained representations, but related efforts do not directly support open-vocabulary semantic segmentation.
  • Open-vocabulary segmentation methods have explored joint pixel–class embeddings and fine-tuned vision-language models, with fine-tuning requiring additional data or compromising open-vocabulary capability.
  • SimSeg generates masked crops before frozen-CLIP recognition, but its heavy mask generator, repeated CLIP forwarding, and CLIP-unaware mask prediction reduce efficiency and performance.
  • SAN uses an end-to-end framework with lightweight, CLIP-aware mask prediction and decoupled mask recognition to better leverage CLIP than two-stage approaches.

3. Side Adapter Network

SAN is an end-to-end side network attached to frozen CLIP that jointly produces mask proposals and attention biases for CLIP-based mask recognition. It reuses CLIP features and applies the biases to deeper CLIP layers in a lightweight single-forward design.

  • Architecture: SAN uses a lightweight vision transformer that fuses shallow CLIP features and outputs mask proposals plus attention biases.The biases are applied to deeper CLIP layers for mask recognition.
  • Mask prediction: Mask proposals are generated from inner products between projected query tokens and projected visual tokens.The query and visual tokens are separately projected through 3-layer MLPs before mask generation.
  • Mask recognition: Attention biases are generated from projected attention queries and visual features, then resized and applied across selected CLIP self-attention layers.The attention-query and mask-query projections may be shared, while the biases can be used in multiple self-attention layers.
  • Design rationale: The decoupled design accommodates cases where the region useful for recognition differs from the predicted mask region.The paper identifies this difference as the motivation for separating mask prediction from recognition.
  • Mask recognition: Shadow [CLS] copies, called [SLS] tokens, receive updates from visual tokens while attention biases guide their region-focused interactions in CLIP.The [SLS] tokens are not allowed to update visual or [CLS] tokens, and their resulting features are compared with CLIP text embeddings for class prediction.
  • Training: SAN is trained end-to-end with Dice, binary cross-entropy, and classification losses, making its mask proposals and attention biases CLIP-aware.The CLIP model remains frozen while the side adapter adapts to it during training.

4. Experiments

SAN is evaluated across six datasets using COCO Stuff for training and multiple Pascal and ADE20K benchmarks for testing. It achieves strong segmentation performance while remaining parameter- and computation-efficient, and ablations support its CLIP-aware, feature-fused design.

  • Datasets and evaluation: Six datasets are used: COCO Stuff for training, with ADE20K-150, ADE20K-847, Pascal Context-59, Pascal Context-459, and Pascal VOC for evaluation.Performance is measured by mean class-wise intersection over union (mIoU).
  • System-level comparison: +1.8 mIoU for ViT-B/16 and +2.3 mIoU for ViT-L/14 are achieved over comparable methods, increasing to +2.9 and +3.7 mIoU with ensembling.These averages compare methods using CLIP ViT models and COCO Stuff under the same setting.
  • Ablation studies: Removing CLIP feature fusion drops mIoU from 27.8 to 21.1, while multi-layer fusion adds +0.8 mIoU over single-layer fusion.The study also finds that deeper-layer fusion outperforms shallower-layer fusion, and the best single-forward split uses the first 9 and last 3 layers.
  • Ablation studies: End-to-end training improves the two-stage baseline by +4.5 mIoU, while the decoupled head improves performance by +1.7 mIoU over a single head.Both head variants are trained end-to-end, so their mask predictions remain CLIP-aware.

5. Conclusion

SAN leverages frozen CLIP features in an end-to-end framework for open-vocabulary semantic segmentation. It outperforms previous state-of-the-art methods across five benchmarks with fewer trainable parameters and lower computation cost.

  • SAN maximally adopts frozen CLIP features through an end-to-end open-vocabulary semantic segmentation pipeline.
  • SAN significantly outperforms previous state-of-the-art methods on five semantic segmentation benchmarks.
  • The framework achieves these results with much fewer trainable parameters and much less computation cost.

A.1. Attention Bias Format

The attention-bias ablation compares increasingly fine-grained bias formats. Per-head biases improve performance, whereas adding layer-wise biases provides no further gain.

  • Per-head attention biases achieve better segmentation performance than using the same bias across attention heads.
  • Adding layer-wise attention biases on top of per-head biases brings no gains.

A.2. [SLS] tokens.

SAN uses copies of the CLIP [CLS] token as [SLS] tokens, which are conceptually similar to MCTs but implemented differently. Learned initialization performs only marginally worse than [CLS]-based initialization.

  • [SLS] tokens are copies of the [CLS] token and are conceptually similar to Mask Class Tokens.
  • [SLS] tokens differ from MCTs in how they are updated and where they are introduced in CLIP.
  • Learned embedding initialization is only marginally worse than initializing [SLS] tokens from [CLS].

A.3. Prompt Engineering

SAN uses multiple prompt templates to decorate class names and averages their text embeddings for inference. Prompt engineering improves performance on ADE-150 and ADE-847.

  • Multiple templates decorate class names, whose text embeddings are averaged into the final inference embedding for each class.
  • 1.2 mIoU improvement is reported on ADE-150 from prompt engineering.
  • 0.7 mIoU improvement is reported on ADE-847 from prompt engineering.

B. Comparison with fine-tuned models

On ADE-847, increasing the CLIP fine-tuning learning rate worsens performance, while frozen CLIP preserves open-vocabulary capability.

  • As the CLIP learning rate increases, ADE-847 performance worsens.A learning rate of 0 denotes the frozen CLIP model.
  • Fine-tuning CLIP can disrupt its open-vocabulary capability.The reported comparison also notes better performance on COCO Stuff as evidence of this disruption.
Loading 2302.12242v2…