PaperlayerView exact revision on arXiv

Source-linked AI summary

Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation

Guolei Sun, Wenguan Wang, Jifeng Dai, Luc Van Gool

arXiv:2007.01947v2cs.CVcs.LGeess.IV

TL;DR

Image-level WSSS needs classifiers whose localization maps cover more complete objects, but existing approaches primarily exploit intra-image information. The paper introduces complementary co-attentions that mine shared and unshared cross-image semantics, improve localization through related-image context, and generalize across three supervision settings, achieving first place in the CVPR2020 LID20 challenge.

  • Problem

    Image-level WSSS classifiers often produce localization maps focused on discriminative parts rather than complete objects, while cross-image semantic relations remain underused for object-pattern mining.

  • Method

    A paired-image co-attention classifier mines common semantics with co-attention and unshared semantics with contrastive co-attention, also using related-image context for localization-map inference.

  • Results

    The approach generalizes across precise image-level, extra single-label, and noisy web-crawled supervision, and ranked 1st in the weakly supervised semantic segmentation track of the LID20 challenge.

  • Takeaways & Limitations

    Cross-image semantic relations support more comprehensive object-pattern discovery and localization while providing a unified treatment of different WSSS supervision settings.

  • Takeaways & Limitations

    Traditional WSSS datasets use the stated settings, whereas LID20 requires separate training and testing details.

Abstract

from arXiv · show

This paper studies the problem of learning semantic segmentation from image-level supervision only. Current popular solutions leverage object localization maps from classifiers as supervision signals, and struggle to make the localization maps capture more complete object content. Rather than previous efforts that primarily focus on intra-image information, we address the value of cross-image semantic relations for comprehensive object pattern mining. To achieve this, two neural co-attentions are incorporated into the classifier to complimentarily capture cross-image semantic similarities and differences. In particular, given a pair of training images, one co-attention enforces the classifier to recognize the common semantics from co-attentive objects, while the other one, called contrastive co-attention, drives the classifier to identify the unshared semantics from the rest, uncommon objects. This helps the classifier discover more object patterns and better ground semantics in image regions. In addition to boosting object pattern learning, the co-attention can leverage context from other related images to improve localization map inference, hence eventually benefiting semantic segmentation learning. More essentially, our algorithm provides a unified framework that handles well different WSSS settings, i.e., learning WSSS with (1) precise image-level supervision only, (2) extra simple single-label data, and (3) extra noisy web data. It sets new state-of-the-arts on all these settings, demonstrating well its efficacy and generalizability. Moreover, our approach ranked 1st place in the Weakly-Supervised Semantic Segmentation Track of CVPR2020 Learning from Imperfect Data Challenge.

1 Introduction

Weakly supervised semantic segmentation reduces annotation demands but image-level classifiers often localize only discriminative object parts. The paper addresses this limitation by using cross-image semantic relations through complementary co-attentions, improving object-pattern discovery and localization-map inference across WSSS settings.

  • Pixel-wise supervision requires intensive manual labeling, motivating WSSS from weaker annotations such as image-level labels.
  • Classifier-derived localization maps often emphasize discriminative parts rather than entire objects, limiting their usefulness as segmentation supervision.
  • Cross-image semantic relations provide additional cues for object-pattern mining beyond single-image information.
  • Co-attention captures shared semantics while contrastive co-attention focuses on unshared semantics, enabling more comprehensive object-pattern learning.
  • Related images provide richer context for localization maps, while paired training images also support data augmentation and more accurate pseudo pixel-level annotations.
  • The framework handles precise image-level, extra single-label, and noisy web-crawled supervision, addressing domain gaps and data noise within one approach.

2 Related Work

Prior WSSS methods mainly expand classifier activation regions or incorporate additional image-level data, but often rely on single-image reasoning or setting-specific techniques. This work instead encodes cross-image semantics during classifier learning and localization inference while accommodating clean, single-label, and noisy web data.

  • Image-level WSSS commonly derives Class Activation Maps, but these maps typically identify small discriminative object parts rather than complete objects.
  • Additional approaches use simple single-label examples or noisy web and video data to strengthen weak supervision.
  • Earlier work rarely uses cross-image information, and one cited approach applies off-the-shelf co-segmentation instead of encoding semantic relations into network learning and inference.
  • The proposed framework uses cross-image semantics in both classifier training and localization-map inference, unlike methods restricted to single-image inference.
  • It is designed for clean image-level supervision and naturally incorporates simple single-label or noisy web-crawled data without setting-specific complicated optimization or heuristic constraints.

3 Methodology

The method trains a classifier on paired images to exploit shared and unshared cross-image semantics, complementing single-image recognition. Co-attention mines common object patterns, while contrastive co-attention isolates exclusive semantics to improve object understanding and localization.

  • Co-attention classification: Paired training images provide cross-image semantic relations that supplement single-image labels for more comprehensive object-pattern learning.The framework uses image pairs and their image-level labels as class-level context.
  • Co-attention classification: Co-attention captures shared semantics between paired images and uses common labels to associate object regions with their semantic classes.Affinity-based attention preserves common objects and produces class-aware activation maps for the shared labels.
  • Co-attention classification: Co-attention highlights multiple parts of common objects, extending beyond highly discriminative regions such as a person’s face to legs and arms.This encourages the classifier to recognize broader object content from cross-image context.
  • Localization and augmentation: The co-attention mechanism supports localization by using correlations between related images to infer richer object context and more accurate maps.The resulting pipeline also provides data augmentation through training image pairs.
  • Contrastive co-attention: Contrastive co-attention mines unshared semantics by filtering common objects and emphasizing exclusive regions in each image.For example, it highlights Table and Cow in paired images containing Table-Person and Cow-Person, respectively.
  • Complementary learning: The classifier jointly learns common and uncommon semantic recognition as complementary auxiliary tasks, using paired samples to exploit cross-image context.The contrastive branch can encourage discrimination when remaining regions are insufficient for direct recognition.

4 Experiment

Experiments evaluate the method across standard, extra single-label, extra web-data, challenge, inference, and ablation settings. The method outperforms competing WSSS approaches and shows that co-attention, related-image context, and contrastive co-attention improve localization and segmentation, with too many related images introducing noise.

  • Experimental Overview: The experiments cover standard PASCAL VOC supervision, extra single-label data, web-crawled data, the LID challenge, inference strategies, and ablations.These settings assess generalization across supervision regimes and examine localization-map generation and co-attention components.
  • Standard Setting: 66.2 and 66.9 mIoU on PASCAL VOC12 val and test sets outperform all competitors under image-level supervision.The val score is 1.0% higher than OAA+, while the method uses end-to-end single-phase training without the extra networks or multi-step training used by several alternatives.
  • Extra Single-Label Data: 5.0% and 4.2% improvements over AttnBN on val and test sets demonstrate gains with extra single-label data.The authors attribute this to capturing common cross-domain semantics and learning domain adaptation through co-attention.
  • Extra Web-Crawled Data: 67.7 and 67.5 mIoU on PASCAL VOC12 val and test sets establish new state-of-the-art results with web-crawled data.The method exceeds Hong et al. by 9.6% and 8.8% and Shen et al. by 3.6% on the test set, despite using static images rather than additional videos.
  • Inference Strategies: Using related images raises localization-map mIoU from 64.7 with the test image alone to 66.2 with other related images.Co-attention benefits inference by exploiting additional semantic context, whereas contrastive co-attention does not improve localization-map inference.
  • Ablation Studies: Co-attention improves the basic classifier by 3.8%, contrastive co-attention adds 0.7% mIoU, and performance peaks with three related images before degrading.Additional related images initially provide useful semantic information and more integral object regions, but imperfect localization introduces noise after the information bottleneck.

5 Conclusion

The proposed co-attention classification network mines common and exclusive semantics across paired images to discover integral object regions. It also improves localization map inference and generalizes across multiple WSSS settings.

  • The co-attention classification network mines common semantics, while contrastive co-attention focuses on exclusive, unshared semantics within paired samples.
  • Cross-image semantic cues provide complementary supervision for discovering more complete object patterns and integral object regions.
  • The approach leverages related-image context to improve localization maps and generalizes to settings with single-label or web images.
  • Experiments across three WSSS settings show consistently promising results, and the method ranked first in the LID20 weakly supervised semantic segmentation track.
Loading 2007.01947v2…