PaperlayerView exact revision on arXiv

Source-linked AI summary

Regional Semantic Contrast and Aggregation for Weakly Supervised Semantic Segmentation

Tianfei Zhou, Meijie Zhang, Fang Zhao, Jianwu Li

arXiv:2203.09653v2cs.CV

TL;DR

Weakly supervised segmentation must infer dense object regions from sparse image-level labels, while prior methods use limited single-image or pairwise context. RCA addresses this with a regional memory bank supporting semantic contrast and aggregation across weakly labeled training data, and reports leading results on PASCAL VOC 2012 and COCO 2014.

  • Problem

    Image-level supervision provides no direct correspondence between semantic labels and object regions, while existing methods exploit limited single-image or pairwise context for localization.

  • Method

    RCA stores historical pseudo-region features in a regional memory bank and uses semantic contrast plus semantic aggregation for dataset-level learning and inference.

  • Results

    RCA improves segmentation over OAA++ and EPS on PASCAL VOC 2012 and sets a new state-of-the-art with RCA+EPS.

  • Takeaways & Limitations

    Dataset-level regional context gives RCA a stronger basis for holistic object-pattern understanding and semantic localization under image-level supervision.

Abstract

from arXiv · show

Learning semantic segmentation from weakly-labeled (e.g., image tags only) data is challenging since it is hard to infer dense object regions from sparse semantic tags. Despite being broadly studied, most current efforts directly learn from limited semantic annotations carried by individual image or image pairs, and struggle to obtain integral localization maps. Our work alleviates this from a novel perspective, by exploring rich semantic contexts synergistically among abundant weakly-labeled training data for network learning and inference. In particular, we propose regional semantic contrast and aggregation (RCA) . RCA is equipped with a regional memory bank to store massive, diverse object patterns appearing in training data, which acts as strong support for exploration of dataset-level semantic structure. Particularly, we propose i) semantic contrast to drive network learning by contrasting massive categorical object regions, leading to a more holistic object pattern understanding, and ii) semantic aggregation to gather diverse relational contexts in the memory to enrich semantic representations. In this manner, RCA earns a strong capability of fine-grained semantic understanding, and eventually establishes new state-of-the-art results on two popular benchmarks, i.e., PASCAL VOC 2012 and COCO 2014.

1. Introduction

Weakly supervised semantic segmentation reduces annotation demands but makes complete object localization difficult. RCA addresses this gap by learning semantic relations across dataset-level pseudo regions through semantic contrast and aggregation.

  • Weak supervision reduces the burden of collecting pixel-level annotations but provides limited information for dense segmentation.
  • RCA performs dataset-level relation learning over pseudo regions to mine contextual knowledge from many training samples rather than individual images or pairs.
  • CAM-based localization is challenging because image-level labels do not specify image-label-to-object-region correspondence, producing sparse and incomplete estimates.
  • Prior approaches mainly exploit single-image information or limited image pairs, neglecting broader inter-image semantic context.
  • Semantic contrast separates object-region embeddings across categories, while semantic aggregation collects diverse contextual representations to enrich features.
  • RCA combines both components to explore weakly labeled training data, narrow the gap between image-level concepts and pixel-level regions, and improve WSSS performance.

2. Related Work

Related work uses weak annotations, contrastive representation learning, and contextual aggregation to reduce segmentation annotation costs and enrich features. RCA extends these directions by mining region-level semantic relations across many dataset images with a persistent memory bank.

  • Weakly supervised segmentation uses image labels, scribbles, boxes, or points, with image-level labels requiring the least annotation effort.
  • Earlier cross-image methods model semantic relations between pairs or small groups of images, limiting the amount of context they can exploit.
  • RCA stores region-level semantic embeddings for each category in a pseudo-region memory bank to support broader object-pattern mining.
  • Unlike conventional contrastive methods that compare image, pixel, or patch samples, RCA performs contrast over categorical pseudo regions from weak labels.
  • Prior relational-context methods capture local contexts within individual images, whereas RCA mines semantic contexts across the entire dataset.
  • RCA’s memory bank stores region-level semantics and remains active during inference to provide holistic contextual knowledge.

3.1. Problem Statement

The standard WSSS setup associates each image with only a multi-label class-presence vector. A classifier then produces dense embeddings, class activation maps, and image-level scores, but existing solutions exploit limited within-image context.

  • Each training image is paired with a binary vector indicating which of L predefined categories are present.
  • The classification network maps an image to a dense embedding F with spatial size W×H and D channels.
  • A class-aware convolutional layer converts F into L activation maps, one for each semantic category.
  • Global average pooling converts the activation maps into class scores used for multi-label classification.
  • RCA adds semantic contrast and aggregation over pseudo regions from many images, supported by an external pseudo-region memory bank.

3.2. Regional Semantic Contrast and Aggregation

RCA represents categorical pseudo regions, stores their dataset-level semantics in a dynamic memory bank, contrasts regions across categories, and aggregates prototype contexts into enriched features for final activation maps.

  • 3.2.1 Pseudo-Region Representation: Categorical pseudo-region embeddings are obtained by masked average pooling over strongly activated pixels in each present class.The mask selects pixels whose activation exceeds the class map’s mean value.
  • 3.2.2 Pseudo-Region Memory Bank: The dynamic memory bank maintains category-specific dictionaries of region-aware embeddings collected throughout training.Each current region feature is incorporated through momentum updating when its class is present and its score exceeds a threshold.
  • 3.2.3 Regional Semantic Contrast: Semantic contrast pulls each region embedding toward memory features from the same category and pushes it away from features from different categories.
  • 3.2.3 Regional Semantic Contrast: Region mixup regularizes contrastive learning by linearly combining regions from different categories, supporting robust representations under noisy weak labels.
  • 3.2.4 Regional Semantic Aggregation: Semantic aggregation compresses each category’s memory features into multiple prototypes, then uses feature-prototype affinities to compute contextual summaries.
  • 3.2.4 Regional Semantic Aggregation: Concatenating original and aggregated features preserves intra-image context while adding inter-image global context for semantic understanding.
  • 3.2.4 Regional Semantic Aggregation: The enriched feature is passed to a class-aware convolutional layer to produce the final activation maps.

3.3. Detailed Network Architecture

The network combines a backbone and class-wise attention maps with a three-term image-supervised objective. The objective includes region mixup contrastive learning, auxiliary intermediate-CAM supervision, and final-CAM supervision, with coefficients balancing the terms.

  • Network components: The backbone FFCN maps an input image I into a convolutional representation F, using VGG16 or ResNet38 in this implementation.Any FCN network can serve as the backbone.
  • Network components: The class-wise convolutional layer FCAM produces class-aware attention maps from feature embeddings, with separate FCAM instances used for intermediate and final predictions.
  • Training objective: Each image is supervised by a region mixup contrastive loss, an auxiliary cross-entropy loss for intermediate CAM prediction P, and a main cross-entropy loss for final CAM prediction O.The region mixup contrastive loss averages the loss over all regions appearing in the image.
  • Training objective: The coefficients α1 and α2 balance the region mixup contrastive term and the auxiliary and main cross-entropy terms.
  • Evaluation: Table 1 reports pseudo-label quality on the VOC 2012 train set and segmentation results on the validation set.

4. Experiment

RCA is evaluated on VOC 2012 and COCO 2014 using pseudo-label quality, semantic segmentation, and qualitative localization analyses. Its regional semantic contrast and aggregation components improve localization and segmentation over established baselines, with gains across datasets and backbone settings.

  • Ablation Studies: RCA ablations show that combining semantic contrast and aggregation improves pseudo-label quality from 69.5% to 71.4% and segmentation from 69.3% to 70.6%.Semantic contrast provides informative memory representations, enabling reliable context aggregation.
  • Ablation Studies: 0.8% mIoU is lost without region mixup, indicating that region mixup improves representations learned from noisy pseudo regions.
  • Ablation Studies: γ = 0.99 is optimal, while performance remains robust for γ in 0.8∼0.99 and degrades with overly slow or absent momentum updates.Momentum updating supports consistent and comprehensive memory representations for semantic contrast and aggregation.
  • Object Localization: RCA improves pseudo-segmentation labels over OAA++ by 3.2% with VGG16 and 3.8% with ResNet38, and improves over EPS from 71.4% to 74.1%.
  • Semantic Segmentation: On VOC 2012, RCA improves OAA++ by 2.9% or 3.0% on val and 3.6% or 3.4% on test, while RCA+EPS sets a new state-of-the-art.
  • Semantic Segmentation: On COCO 2014, RCA surpasses OAA+ and EPS by 2.1% and 1.1%, respectively, while qualitative results show stronger handling of small or large objects, multiple instances, and occlusions.

5. Conclusion

RCA learns semantic segmentation from image-level supervision by mining dataset-wide semantic contexts through a continuously updated memory bank. Its semantic contrast and aggregation mechanisms improve network learning and inference, with leading performance on PASCAL VOC 2012 and COCO 2014.

  • RCA uses a continuously updated memory bank to store historical pseudo-region features and exploit their semantic relations during learning and inference.Semantic contrast provides additional supervisory signals, while semantic aggregation supplies holistic contextual cues.
  • Semantic contrast and aggregation enable RCA to discover rich semantic contexts from weakly labeled training data under image-level supervision only.The approach addresses the limited knowledge carried by image labels by exploiting relations between memory contents and mini-batch samples.
  • RCA achieves leading performance on the PASCAL VOC 2012 and COCO 2014 benchmarks.
Loading 2203.09653v2…