Source-linked AI summary
Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations
Jiwoon Ahn, Sunghyun Cho, Suha Kwak
TL;DR
Weakly supervised instance segmentation from image-level labels is ill-posed because those labels lack instance-specific information. The paper uses CAM seeds and IRNet to estimate instances and boundaries, generate pseudo labels, and train a fully supervised model; on PASCAL VOC 2012, it achieves state-of-the-art performance in instance and semantic segmentation.
Problem
Image-level supervision lacks instance-specific information, making weakly supervised instance segmentation significantly ill-posed.
Method
IRNet estimates class-agnostic instance maps and pairwise semantic affinities to assign instance labels to CAM seeds and propagate them into pseudo segmentation labels without additional supervision.
Results
On PASCAL VOC 2012, models trained with the pseudo labels achieve state-of-the-art performance in both instance and semantic segmentation.
Takeaways & Limitations
The approach enables instance segmentation from image-level class labels without additional supervision or segmentation proposals.
Takeaways & Limitations
IRNet relies on inter-pixel relations mined from CAMs, whose class-equivalence relations are unavailable directly because pixel-wise class labels are not provided.
Abstract
from arXiv · showhide
This paper presents a novel approach for learning instance segmentation with image-level class labels as supervision. Our approach generates pseudo instance segmentation labels of training images, which are used to train a fully supervised model. For generating the pseudo labels, we first identify confident seed areas of object classes from attention maps of an image classification model, and propagate them to discover the entire instance areas with accurate boundaries. To this end, we propose IRNet, which estimates rough areas of individual instances and detects boundaries between different object classes. It thus enables to assign instance labels to the seeds and to propagate them within the boundaries so that the entire areas of instances can be estimated accurately. Furthermore, IRNet is trained with inter-pixel relations on the attention maps, thus no extra supervision is required. Our method with IRNet achieves an outstanding performance on the PASCAL VOC 2012 dataset, surpassing not only previous state-of-the-art trained with the same level of supervision, but also some of previous models relying on stronger supervision.
1. Introduction
The paper addresses the ill-posedness and annotation cost of weakly supervised instance segmentation from image-level labels. It proposes IRNet to identify instances, estimate boundaries, and generate pseudo labels for training a fully supervised model.
- Manual instance-mask annotation is prohibitively time-consuming, limiting datasets in class diversity and annotated-data volume.
- Image-level labels reduce annotation cost but provide only class existence, while CAMs roughly localize classes without separating instances or recovering complete areas.
- The framework generates pseudo instance-segmentation labels from image-level labels and uses them to train a known CNN model.
- IRNet estimates a class-agnostic instance map and pairwise semantic affinities that complement CAMs for assigning instance labels and propagating attention.
- The approach learns and predicts semantic affinities through class-boundary detection using image-level supervision.
- On PASCAL VOC 2012, the model substantially outperforms prior work with the same supervision and surpasses some methods using bounding-box or full supervision.
2. Related Work
Prior work uses weak labels, iterative attention mining, affinity-based CAM propagation, or segmentation proposals, but image-level instance segmentation remains severely underdetermined.
- Weakly supervised semantic segmentation has used iterative complementary-attention mining and CAM propagation based on semantic affinities.
- Weakly supervised instance segmentation commonly uses bounding boxes, which provide object location and scale but require manual annotation.
- Image-level instance segmentation is rarely studied because class labels provide no instance-specific information.
- A prior approach detects peaks in class attention and combines them with high-quality segmentation proposals to recover entire instances.
- This approach shares affinity-propagation motivation with prior work while learning affinities through class-boundary detection.
3. Class Attention Maps
CAMs provide class-localization cues that seed pseudo-label generation and supervise IRNet through mined inter-pixel relations. The implementation uses a ResNet50 classifier with higher-resolution CAMs.
- CAMs define instance seed areas that are later propagated to recover entire instance areas.
- CAMs also supervise IRNet by providing reliable inter-pixel relations extracted from attention maps.
- For class c at coordinate x, CAM score Mc(x) is computed from the classification weights φc and feature map f(x), normalized by the maximum score.
- The classifier uses ResNet50 with the final downsampling stride reduced from 2 to 1, producing CAMs at 1/16 of the input resolution.
4. Inter-pixel Relation Network
IRNet learns a displacement field and class boundary map from inter-pixel relations derived from CAMs. These outputs provide instance grouping and semantic affinities for generating pseudo labels without extra supervision.
- Architecture: IRNet has two branches that predict a displacement vector field and a class boundary map, respectively.Both branches share a ResNet50 backbone and use multilevel backbone feature maps as input.
- Inter-pixel relation mining: CAMs provide confident foreground and background areas from which neighboring pixel pairs are sampled and assigned class-equivalence relations.Pixels with attention scores above 0.3 are foreground, while those below 0.05 are background; dense CRF refines these areas before pair sampling.
- Displacement field prediction: The displacement branch learns vectors pointing toward instance centroids by matching displacement differences between nearby same-class pixels to their coordinate differences.The method also uses a background loss to eliminate trivial centroid estimation from background pixels.
- Class boundary detection: The boundary branch learns semantic affinities by treating class boundaries between pixel pairs as latent evidence supervised through class-equivalence relations.For pixels xi and xj, affinity depends on the strongest predicted boundary along the line segment Πij; the resulting objective is a multiple-instance-learning formulation.
- Joint training: The two IRNet branches are jointly trained with the defined losses, while class-equivalence supervision rather than individual class labels improves relation usage and generalization.The losses are aggregated after normalization because the positive and negative relation populations are imbalanced.
5. Label Synthesis Using IRNet
IRNet outputs are converted into an instance map and semantic affinities, then combined with CAMs to propagate attention within estimated instances and boundaries. The resulting scores determine pseudo instance segmentation labels.
- Output conversion: IRNet outputs D and B are converted into a class-agnostic instance map and pairwise semantic affinities, respectively.The instance map groups pixels by estimated centroid, while affinities are derived directly from the boundary output.
- Instance-map synthesis: Because D may inaccurately estimate centroid offsets under incomplete CAM supervision, the displacement field is refined iteratively.Each vector receives the vector at its currently estimated centroid, and refinement converges as vectors near centroids approach zero.
- Instance-map synthesis: Candidate centroids are pixels with small displacement magnitudes, and each connected component of candidates is treated as one centroid.This groups neighboring pixels instead of relying on a single scattered centroid coordinate.
- Instance-wise CAM propagation: CAMs are combined with the instance map to form instance-wise CAMs, which are individually refined by random-walk propagation using the semantic affinity matrix.Boundary pixels are penalized with (1 −B) because isolated boundary pixels otherwise retain disproportionately high scores.
- Label assignment: Each pixel receives the class-instance combination with the maximum propagated score, while pixels below the bottom 25% are assigned to background.This converts propagated instance-wise attention into pseudo instance segmentation labels.
6. Experiments
Experiments on PASCAL VOC 2012 evaluate pseudo-label quality and downstream instance and semantic segmentation using only image-level class labels. IRNet improves pseudo labels and enables models that outperform prior weakly supervised methods and some methods using stronger supervision.
- Experimental Setting: The framework is trained and evaluated on PASCAL VOC 2012 using 10,582 training images and 1,449 validation images, while exploiting only image-level class labels.
- Pseudo-Label Analysis: Class boundary maps improve pseudo-instance-label quality by more than 25% by recovering object areas missing from CAMs.
- Pseudo-Label Analysis: Displacement fields further improve pseudo-label performance by 3.6% by distinguishing different instances of the same class.
- Pseudo-Label Analysis: IRNet produces substantially more accurate pseudo semantic segmentation labels than AffinityNet because it predicts higher-quality pairwise semantic affinities.
- Instance Segmentation: Mask R-CNN trained with the pseudo labels outperforms PRM and exceeds SDI by 1.9% and SDS by 2.9% in APr.
- Semantic Segmentation: DeepLab v2-ResNet50 trained with the pseudo semantic labels outperforms prior methods using the same supervision and recovers 88% of its fully supervised counterpart.
7. Conclusion
The conclusion frames image-level weakly supervised instance segmentation as ill-posed because it lacks instance-specific information. IRNet addresses this by identifying instances and estimating rough boundaries, enabling class attentions to support state-of-the-art instance and semantic segmentation on PASCAL VOC 2012.
- Image-level weak supervision makes instance segmentation significantly ill-posed because it lacks instance-specific information.
- IRNet identifies individual instances and estimates their rough boundaries, providing evidence for improving class attentions.
- Models trained with the resulting pseudo labels achieve state-of-the-art performance in both instance and semantic segmentation on PASCAL VOC 2012.
A. Appendix
The appendix supplies material omitted from the main sections, including a detailed centroid-detection algorithm, final evaluation models, and additional qualitative results.
- Section A.1 details centroid detection, Section A.2 introduces models trained with synthetic labels, and Section A.3 presents additional qualitative results.
A.1. Details of the Centroid Detection Algorithm
The centroid-detection procedure identifies low-magnitude displacement vectors as candidate centroids and groups them into connected components to form an instance map.
- Pixels whose displacement vectors in D have magnitudes below a threshold are treated as candidate centroids.
- Candidate centroids are represented by connected components, whose number is denoted K.
- The class-agnostic instance map I assigns each pixel a connected-component index.
A.2. Details of Our Segmentation Networks
The framework trains segmentation networks using synthetic labels generated from image-level supervision. Its qualitative and quantitative evaluations show accurate masks, instance separation, and performance close to a fully supervised semantic-segmentation model.
- Segmentation networks: Mask R-CNN and DeepLab v2 are finetuned with synthetic labels instead of ground-truth segmentation masks.Both models are first pretrained on ImageNet.
- Segmentation networks: The implementation uses Detectron’s Mask R-CNN with a ResNet50-FPN backbone and adjusted training steps for PASCAL VOC 2012.
- Segmentation networks: DeepLab v2 uses ResNet-50 with atrous spatial pyramid pooling and aggregates predictions from eight resized and flipped inputs.The evaluation scales are 0.5, 1.0, 1.5, and 2.0, with horizontal flipping.
- Evaluation: 89.4%: DeepLab trained with synthetic labels reaches this fraction of the reproduced fully supervised model’s performance on the PASCAL VOC 2012 test set.The reproduced fully supervised model is treated as the upperbound, though its measured performance is below the original report because dense CRF parameters were not carefully tuned.
- Qualitative results: The qualitative results show accurate masks, fine object shapes, and separation of adjacent instances sharing the same class.IRNet supplies class boundaries and displacement fields to instance centroids that CAMs do not directly provide.