PaperlayerView exact revision on arXiv

Source-linked AI summary

Context-aware Feature Generation for Zero-shot Semantic Segmentation

Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, Liqing Zhang

arXiv:2008.06893v1cs.CV

TL;DR

Zero-shot semantic segmentation must recognize unseen objects despite training annotations for only seen categories. CaGNet inserts a contextual module into a segmentation-and-generation framework to guide diverse feature synthesis from semantic embeddings, and experiments report effectiveness across three benchmark datasets.

  • Problem

    Zero-shot semantic segmentation aims to segment unseen objects when training images provide pixel-wise annotations only for seen categories.

  • Method

    CaGNet uses a Contextual Module to encode pixel-wise contextual information into latent codes that guide feature generation alongside semantic word embeddings.

  • Results

    Qualitative and quantitative results on three benchmark datasets show the effectiveness of CaGNet.

  • Takeaways & Limitations

    CaGNet produces more diverse and context-aware features while avoiding explicit contextual information from unseen categories.

  • Takeaways & Limitations

    The contextual cues considered by prior ZS3Net (GC) are limited to object-level spatial arrangement, and unseen-category relational graphs may be inaccessible.

Abstract

from arXiv · show

Existing semantic segmentation models heavily rely on dense pixel-wise annotations. To reduce the annotation pressure, we focus on a challenging task named zero-shot semantic segmentation, which aims to segment unseen objects with zero annotations. This task can be accomplished by transferring knowledge across categories via semantic word embeddings. In this paper, we propose a novel context-aware feature generation method for zero-shot segmentation named CaGNet. In particular, with the observation that a pixel-wise feature highly depends on its contextual information, we insert a contextual module in a segmentation network to capture the pixel-wise contextual information, which guides the process of generating more diverse and context-aware features from semantic word embeddings. Our method achieves state-of-the-art results on three benchmark datasets for zero-shot segmentation. Codes are available at: https://github.com/bcmi/CaGNet-Zero-Shot-Semantic-Segmentation.

1 INTRODUCTION

Zero-shot semantic segmentation seeks to segment unseen categories when training annotations cover only seen categories. CaGNet addresses this by generating diverse, context-aware pixel features from semantic embeddings and pixel-wise contextual information.

  • Problem: Zero-shot semantic segmentation trains with pixel-wise annotations for seen categories, although test images may contain both seen and unseen objects.The task transfers knowledge across categories to reduce annotation requirements.
  • Method: CaGNet generates unseen-category features using semantic word embeddings together with pixel-wise contextual latent codes.The latent codes are produced by a Contextual Module inserted after the segmentation backbone.
  • Motivation: Pixel-wise features vary with surrounding context, including object location, posture, and background objects, in an interlaced and complicated way.The paper illustrates this variation by clustering cat features with K = 2 and K = 5.
  • Method: CaGNet uses contextual latent codes that support stochastic sampling, avoiding the need for explicit unseen-category relational graphs when generating features.This design is presented as an advantage over ZS3Net (GC).
  • Contributions: The method unifies the segmentation network and feature-generation network, using a shared classifier while incorporating a contextual module and feature generator.The contributions also include a context selector and experiments on Pascal-Context, COCO-stuff, and Pascal-VOC.

2 RELATED WORKS

Prior semantic segmentation methods incorporate multi-scale context but require annotations for all categories, while zero-shot methods transfer knowledge through semantic embeddings. CaGNet combines segmentation and feature generation to use pixel-wise context for zero-shot segmentation.

  • Semantic Segmentation: Semantic segmentation methods expand receptive fields and fuse contextual information from feature maps at different scales.Examples include specialized pooling in PSPNet and Deeplab and encoder-decoder designs such as U-Net and RefineNet.
  • Semantic Segmentation: These segmentation models still require annotations for all categories during training and therefore do not directly address zero-shot segmentation.CaGNet is presented as combining a segmentation network with a feature generator for this setting.
  • Zero-shot Learning: Zero-shot learning transfers knowledge from seen to unseen categories through category-level semantic embeddings.Many methods learn mappings between visual feature space and semantic embedding space.
  • CaGNet: CaGNet combines a segmentation network with feature generation to generate context-aware features for zero-shot semantic segmentation.Its overview includes backbone E, Contextual Module CM, generator G, discriminator D, and classifier C.
  • Zero-shot Learning: Feature-generation approaches for zero-shot classification synthesize image features from word embeddings and random vectors without contextual information.CaGNet instead uses pixel-wise contextual information because semantic segmentation requires pixel-level predictions.

3 METHODOLOGY

CaGNet augments a segmentation network with a Contextual Module that captures pixel-wise, multi-scale context and uses it to guide context-aware feature generation from semantic word embeddings. The resulting pipeline jointly links segmentation and feature generation through shared contextual and classifier components.

  • Overview: CaGNet inserts a Contextual Module after the backbone to capture pixel-wise contextual information for feature generation.The module produces contextual representations while preserving pixel-to-pixel correspondence.
  • Contextual Module: The Contextual Module uses serial dilated convolutions to preserve contextual information at multiple scales without pooling.Deeper context maps have larger receptive fields, allowing information from progressively larger neighborhoods to be collected.
  • Contextual Module: A context selector assigns pixel-wise weights to different context-map scales before concatenating them and projecting the result into a contextual latent code distribution.The selector accommodates pixels whose relevant context may come from small or large receptive fields.
  • Context-aware Feature Generator: The contextual latent code is sampled from a Gaussian distribution and used with semantic word embeddings to generate pixel-wise features for seen and unseen categories.The generator receives the latent code map and semantic embedding map, while reconstruction, classification, and adversarial objectives regulate generated features.
  • Network Integration: The contextual module also outputs residual-attended real features for classification, while the generator and segmentation network share the classifier.The residual attention output is X_n = F_n + F_n ⊙ϕ(Z_n), and the networks are optimized through linked training stages.

4 EXPERIMENTS

CaGNet is evaluated on three benchmark datasets under zero-shot segmentation settings that use annotations only for seen categories. Experiments compare its performance with prior methods and isolate the contributions of contextual modeling, feature generation, and training choices.

  • Datasets and settings: The experiments use Pascal-Context, COCO-stuff, and Pascal-VOC with pixel-wise annotations of unseen categories ignored during training.The evaluation follows standard train/test splits and predefined seen/unseen category partitions.
  • Datasets and settings: The comparison includes SPNet, ZS3Net, calibrated SPNet, and self-training variants using the same Deeplabv2 backbone.The study reports results with and without the self-training strategy and discusses differences in prior experimental settings.
  • Comparison with state of the art: CaGNet significantly improves unseen and overall performance over SPNet and ZS3Net across all three datasets, especially on mIoU and hIoU.It can underperform SPNet on seen categories because it sacrifices some seen pixels for substantially better unseen performance.
  • Ablation study: Feature generation with G and D substantially raises unseen-category performance, while the complete contextual module improves all reported metrics.Applying CM alone provides only marginal improvement, whereas adding feature generation reduces the gap between seen and unseen categories.
  • Ablation study: Dilated convolutions, multi-scale contextual information, and adaptive context selection each contribute to the contextual module’s performance.The proposed contextual module outperforms simpler convolutional variants, and the context selector improves over fixed multi-scale aggregation.
  • Training analysis: A feature generating ratio of 1 : 1 is reported as reasonable for balancing seen and unseen pixels during Pascal-VOC finetuning.Without constraints, pixel-level seen features greatly outnumber unseen features in the reported datasets.
  • Qualitative analyses: The contextual module improves generated features for both seen and unseen categories compared with random latent vectors without CM.The comparison includes examples such as person, potted plant, sheep, sofa, and tv monitor.

5 CONCLUSION

CaGNet unifies segmentation and feature generation for zero-shot semantic segmentation, using contextual information to generate diverse, context-aware features. Qualitative and quantitative results on three benchmark datasets demonstrate its effectiveness.

  • CaGNet unifies the segmentation network and feature generation for zero-shot semantic segmentation.
  • The method uses contextual information to generate diverse and context-aware features.
  • Qualitative and quantitative results on three benchmark datasets demonstrate the method’s effectiveness.

Supplementary for Context-aware Feature Generation For

The supplied supplementary front matter identifies the work as a zero-shot semantic segmentation paper from Shanghai Jiao Tong University. It lists the authors and an arXiv version dated 16 August 2020.

  • The supplementary document is titled “Supplementary for Context-aware Feature Generation For Zero-shot Semantic Segmentation.”
  • The listed authors are Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, and Liqing Zhang.
  • The arXiv record is version 1, dated 16 August 2020.
  • The document identifies Shanghai Jiao Tong University as the authors’ affiliation.

CCS CONCEPTS

The paper is classified under image segmentation within the Computing methodologies taxonomy. Its supplementary material is associated with an ACM Multimedia 2020 publication format.

  • The CCS concept classifies the work under Computing methodologies → Image segmentation.
  • The supplementary reference format identifies the venue as the 28th ACM International Conference on Multimedia.
  • The cited ACM Multimedia event is designated MM ’20 and scheduled for October 12–16, 2020.

1 COMPARISON IN THE SETTING OF ZS3NET

Under the ZS3Net setting, CaGNet is evaluated using the same general experimental framework and achieves stronger unseen-category performance. Its overall hIoU improves by 13.0 on Pascal-VOC and 4.9 on Pascal-Context.

  • The evaluation follows ZS3Net’s backbone, semantic word embedding method, seen/unseen splits, and evaluation metrics.
  • The experiments use word2vec embeddings of length 300 and Deeplabv3+ as the backbone.
  • CaGNet achieves comparable or better results on seen categories while significantly improving unseen-category results.
  • 13.0 and 4.9 are the overall hIoU improvements on Pascal-VOC and Pascal-Context, respectively.
  • The larger gain on Pascal-VOC may be related to the difficulty of segmenting more unseen categories.

2 MORE ABLATION STUDIES ON OUR CONTEXTUAL MODULE

Additional ablations examine removing residual attention and arranging the contextual module’s dilated convolutions in parallel rather than serially.

  • Ablation scope: These variants supplement the main contextual-module ablation study on Pascal-VOC.The supplemental cases are reported as additional rows associated with the contextual-module experiments.
  • Residual connection: The “w/o residual” variant removes the link from contextual latent code back to the segmentation network and reconstructs the original feature map instead of the enhanced map.Its loss functions also replace the enhanced feature map with the original feature map.
  • Convolution arrangement: The “Parallel” variant places three dilated convolutions in parallel after the input feature map to obtain context maps with different receptive fields.The resulting receptive fields are 3 × 3, 5 × 5, and 13 × 13, compared with 3×3, 7×7, and 17×17 in serial mode.

3 HYPER-PARAMETER ANALYSES

The supplementary material analyzes hyper-parameter sensitivity and provides visual and tabular evaluation references for the Pascal-VOC and Pascal-Context experiments.

  • Hyper-parameter sensitivity: λ1 has a larger impact than λ2, and performance drops sharply when λ1 is very small, supporting the necessity of feature reconstruction.The analysis varies each parameter from 0.1 to 1000 and reports hIoU (%) on Pascal-VOC.
  • Evaluation references: Table 1 reports zero-shot segmentation performances on Pascal-Context and Pascal-VOC under the ZS3Net setting.The table identifies the best results in boldface.
  • Evaluation references: Table 2 reports ablation studies of special contextual-module cases on Pascal-VOC.The supplementary figures also visualize parameter effects, segmentation results, and reconstruction loss maps.

4 MORE VISUALIZATIONS OF SEGMENTATION RESULTS

Supplementary visualizations compare methods on unseen-object segmentation in Pascal-VOC, where the proposed method succeeds on several objects that competing methods misclassify.

  • Segmentation visualizations: The supplementary figure extends the segmentation-result visualizations reported in the main paper.It provides additional examples on the Pascal-VOC dataset.
  • Segmentation visualizations: The proposed method segments unseen “tv”, “train”, “sofa”, and “sheep” more successfully than SPNet and ZS3Net in the visual comparisons.The comparison is presented as further evidence of the method’s advantage on unseen objects.
  • Segmentation visualizations: SPNet and ZS3Net misclassify “tv” and “sofa” as “table”, whereas the proposed method segments them successfully.These examples come from the first and third visualization rows.
  • Segmentation visualizations: The proposed method recognizes “sheep”, while ZS3Net and SPNet classify it as “cow”.The comparison is shown in the fourth visualization row.

5 MORE VISUALIZATIONS OF FEATURE GENERATION

Supplementary feature-generation visualizations compare reconstruction loss maps and indicate that the Contextual Module improves generated features for both seen and unseen categories.

  • Feature-generation visualizations: Darker reconstruction loss regions indicate better agreement between generated and corresponding real feature maps.The visualizations compare reconstruction loss maps computed from generated and real feature maps on Pascal-VOC test images.
  • Feature-generation visualizations: The supplementary figure extends the feature-generation visualizations from the main paper.It uses test images from Pascal-VOC as examples.
  • Feature-generation visualizations: The Contextual Module facilitates better feature generation for seen “person” and unseen categories such as “tv” and “potted plant”.The examples include “tv” in the first two rows and “potted plant” in the fourth row.
Loading 2008.06893v1…