Source-linked AI summary
Weakly Supervised Instance Segmentation using Class Peak Response
Yanzhao Zhou, Yi Zhu, Qixiang Ye, Qiang Qiu, Jianbin Jiao
TL;DR
Weakly supervised instance segmentation lacks an established image-level-only solution despite the cost of pixel-level masks. The paper exploits class peak responses, stimulating and back-propagating them into Peak Response Maps for instance mask extraction, and reports top results across pointwise localization, semantic segmentation, and image-level supervised instance segmentation.
Problem
Image-level supervised instance segmentation remains an unexplored and challenging problem, whereas pixel-level mask annotation is expensive and labor-intensive.
Method
The method stimulates peaks in class response maps, back-propagates them to informative instance regions, and uses Peak Response Maps for mask extraction.
Results
The method reports top results for pointwise localization and weakly supervised semantic segmentation, and first reports image-level supervised instance segmentation results.
Takeaways & Limitations
Instance-aware cues can be discovered in hierarchical convolutional response maps for weakly supervised instance-level problems.
Takeaways & Limitations
Weakly supervised instance segmentation performance remains limited by proposal quality.
Abstract
from arXiv · showhide
Weakly supervised instance segmentation with image-level labels, instead of expensive pixel-level masks, remains unexplored. In this paper, we tackle this challenging problem by exploiting class peak responses to enable a classification network for instance mask extraction. With image labels supervision only, CNN classifiers in a fully convolutional manner can produce class response maps, which specify classification confidence at each image location. We observed that local maximums, i.e., peaks, in a class response map typically correspond to strong visual cues residing inside each instance. Motivated by this, we first design a process to stimulate peaks to emerge from a class response map. The emerged peaks are then back-propagated and effectively mapped to highly informative regions of each object instance, such as instance boundaries. We refer to the above maps generated from class peak responses as Peak Response Maps (PRMs). PRMs provide a fine-detailed instance-level representation, which allows instance masks to be extracted even with some off-the-shelf methods. To the best of our knowledge, we for the first time report results for the challenging image-level supervised instance segmentation task. Extensive experiments show that our method also boosts weakly supervised pointwise localization as well as semantic segmentation performance, and reports state-of-the-art results on popular benchmarks, including PASCAL VOC 2012 and MS COCO.
1. Introduction
The paper addresses image-level supervised instance segmentation by using class peak responses to extract instance-level cues and masks with classification networks. The approach is simple, compatible with modern architectures, and reports top performance across benchmarks.
- Motivation: Image-level annotations are cheaper and easier to define than pixel-level masks, motivating weakly supervised segmentation.Dense pixel-level annotation is described as expensive and labor-intensive.
- Challenge: Class response maps capture class-relevant image regions but cannot distinguish different instances of the same category.This limitation makes standard weakly supervised semantic segmentation insufficient for instance segmentation.
- Method: Peak Response Maps encode spatial layouts and fine-detailed boundaries, enabling instance masks to be extracted with off-the-shelf methods.The representation is designed to preserve instance-specific visual cues rather than only class-level saliency.
- Practicality: The approach uses standard classification settings, supports modern network architectures, and incurs negligible computational overhead.The paper contrasts this simplicity with fully supervised approaches using CRFs, RNNs, or template matching.
- Method: The method stimulates peaks in class response maps and back-propagates them to informative instance regions, including boundaries.These maps are called Peak Response Maps and provide instance-level representations for mask extraction.
- Results: The method reports top performance on multiple benchmarks and first reports results for image-level supervised instance segmentation.The implementations include popular CNNs such as VGG16 and ResNet50.
2. Related Work
Prior weakly supervised segmentation methods mainly produce class-aware but instance-agnostic evidence, while instance segmentation requires separate instance labels and detailed masks. This paper instead uses instance-aware cues learned by classification networks, with image-level annotations alone.
- Weakly Supervised Semantic Segmentation: Weakly supervised semantic segmentation has explored points, bounding boxes, scribbles, and selected foreground as alternatives to dense annotations.These alternatives still involve more human effort than image-level supervision.
- Weakly Supervised Semantic Segmentation: Existing approaches use graphical models or external localization networks to infer object locations, often with additional or time-consuming processing.These methods may rely on repeated model learning or pre-generated object segment proposal priors.
- Instance Segmentation: Instance segmentation requires both instance-aware region labels and fine-detailed masks, making it more challenging than class-aware semantic segmentation.Fully supervised methods commonly add constraints from precise object bounding boxes.
- Instance Segmentation: The weakly supervised instance segmentation problem remains open, while this paper requires only image-level annotations by exploiting instance-aware cues learned in classification networks.This contrasts with approaches that obtain pseudo-ground-truth masks from bounding-box supervision.
- Image-Level Supervised Deep Activation: Class response aggregation methods such as global max pooling or global average pooling operate from a global perspective and make instance discrimination difficult.The paper motivates local peak responses as a way to account for spatial relevance.
- Weakly Supervised Semantic Segmentation: Top-down attention methods refine class saliency maps, but their class-aware and instance-agnostic cues are insufficient for instance segmentation.The proposed method is presented as providing fine-detailed instance-aware cues instead.
3. Method
The method converts a classification CNN into a peak-stimulated fully convolutional network, back-propagates class peaks into Peak Response Maps, and combines these cues with proposals to extract instance masks.
- Class Response Maps: Fully convolutional classifiers produce spatial class response maps while preserving image location information.The network is obtained by removing global pooling and adapting fully connected layers to 1x1 convolutions.
- Peak Stimulation: A peak stimulation layer detects local maxima in each class response map and aggregates responses from those peaks into class confidence scores.Peaks are defined within a window of radius r^2, with r set to 3 in the experiments.
- Peak Stimulation: Using peaks for the final decision concentrates learning on informative receptive fields rather than allowing easy negatives to dominate.The backward gradient is apportioned to peak locations, including potential positives and hard negatives.
- Peak Back-propagation: Peak back-propagation propagates each class peak through convolutional and pooling layers to localize fine-detailed, instance-aware visual cues as PRMs.The transition probabilities use ReLU-positive weights and normalization to propagate visiting probabilities through the network.
- Weakly Supervised Instance Segmentation: Instance masks are selected from proposal galleries by combining PRM overlap, proposal boundary similarity, and class-response suppression of irrelevant regions.The method then ranks proposals and applies non-maximum suppression to form the instance prediction set.
- Weakly Supervised Instance Segmentation: The overall inference procedure forwards an image, detects class peaks, back-propagates each peak, scores proposals, and applies non-maximum suppression.This pipeline uses a network trained with peak stimulation and outputs an instance segmentation prediction set.
4. Experiment
Experiments evaluate PRMs for pointwise localization, cue quality, semantic segmentation, and weakly supervised instance segmentation. Results show improved localization and fine-detailed instance cues, while proposal quality remains a practical limitation.
- Pointwise localization: Peak stimulation improves pointwise localization over the unstimulated baseline on VOC 2012 and MS COCO 2014.The evaluation fine-tunes ResNet50 with and without peak stimulation and reports validation-set performance.
- Peak response analysis: PRM quality is evaluated by correlation with same-class ground-truth masks, where scores above 0.5 count as true positives.A score of 0 indicates no valid object region, while 1 indicates perfect separation of instance cues from background.
- Peak response analysis: PRM energy inside instances averages 78% for single-object images and 67% for images containing 2–5 objects.Even crowded scenes with more than six objects collect more PRM energy inside instances than in the background.
- Semantic segmentation: The method reports competitive weakly supervised semantic segmentation without CRF post-processing or additional supervision.Models use image-level labels and standard classification settings rather than time-consuming training strategies.
- Instance segmentation: The proposed approach significantly outperforms weakly supervised localization baselines for instance segmentation across IoU thresholds.Gains at IoU 0.25 and 0.5 reflect object localization, while gains at 0.75 indicate fine-detailed instance cues from peak back-propagation.
5. Conclusions
The paper presents a simple technique that uses class peak responses to enable classification networks to extract instance masks under image-level supervision. Peak stimulation reinforces localization, while peak back-propagation extracts fine-detailed cues; the work reports strong results across localization and segmentation tasks.
- Contributions: The method enables classification networks to extract instance masks using class peak responses.It relies on instance-aware cues naturally learned by convolutional filters and encoded in hierarchical response maps.
- Mechanism: Peak stimulation reinforces object localization, while peak back-propagation extracts fine-detailed visual cues for each instance.The paper presents these mechanisms as the basis for weakly supervised instance-level problems.
- Results: The work reports top results for pointwise localization and weakly supervised semantic segmentation, and introduces image-level supervised instance segmentation results.The authors describe the instance-segmentation results as the first of their kind.