Source-linked AI summary
Weakly Supervised Learning of Instance Segmentation with Inter-pixel Relations
Jiwoon Ahn, Sunghyun Cho, Suha Kwak
TL;DR
Instance segmentation relies on manually annotated instance masks, which are prohibitively time-consuming and leave datasets limited in class diversity and size. This paper uses IRNet to generate pseudo instance labels from image-level labels and reports state-of-the-art instance and semantic segmentation on PASCAL VOC 2012.
Problem
Manually annotating instance masks is prohibitively time-consuming, leaving instance segmentation datasets limited in class diversity and the amount of annotated data.
Source: §1 · p. 1Method
IRNet estimates class-agnostic instance maps and pairwise semantic affinities to assign instance labels to attention-map seeds and propagate them into pseudo labels for training.
Source: §1 · pp. 1–2Results
Takeaways & Limitations
Image-level supervision can support models that achieve state-of-the-art instance and semantic segmentation on PASCAL VOC 2012.
Source: §7 · p. 8IRNet's inter-pixel supervision requires reliable mining of class equivalence because pixel-wise class labels are unavailable in this weakly supervised setting.
Source: §4.2 · p. 4
Abstract
from arXiv · showhide
This paper presents a novel approach for learning instance segmentation with image-level class labels as supervision. Our approach generates pseudo instance segmentation labels of training images, which are used to train a fully supervised model. For generating the pseudo labels, we first identify confident seed areas of object classes from attention maps of an image classification model, and propagate them to discover the entire instance areas with accurate boundaries. To this end, we propose IRNet, which estimates rough areas of individual instances and detects boundaries between different object classes. It thus enables to assign instance labels to the seeds and to propagate them within the boundaries so that the entire areas of instances can be estimated accurately. Furthermore, IRNet is trained with inter-pixel relations on the attention maps, thus no extra supervision is required. Our method with IRNet achieves an outstanding performance on the PASCAL VOC 2012 dataset, surpassing not only previous state-of-the-art trained with the same level of supervision, but also some of previous models relying on stronger supervision.
POSTECH · 1. Introduction
The paper presents image-level-supervised instance segmentation by using IRNet to refine class attention maps into pseudo instance labels without extra supervision or segmentation proposals. The authors report that their method surpasses prior work trained with the same supervision and some methods using stronger supervision.
1. Introduction
- Manual instance-mask annotation is prohibitively time-consuming, limiting existing datasets in class diversity and annotated-data volume.Bounding boxes provide object properties other than shape but remain costly to annotate manually across many images and classes [24, 44].Source: §1 · p. 1
- CAMs provide rough class-region cues but have limited resolution, cover only partial objects, and cannot distinguish same-class instances; a prior proposal-based approach required external supervision.The paper proposes class-agnostic instance maps to avoid off-the-shelf segmentation proposals and class-boundary detection to predict pixel affinities [1].Source: §1 · pp. 1–2
- Image-level class labels are readily available in large-scale classification datasets such as ImageNet [45], and CAMs can be derived from them [46].These labels reduce annotation cost but indicate only whether object classes are present.Source: §1 · p. 1
- IRNet complements CAMs with class-agnostic instance maps and semantic affinities, enabling instance-wise CAMs to be refined into pseudo instance labels.Attention scores are propagated to relevant areas according to affinities between neighboring pixels, then each pixel receives the instance label with the highest score.Source: §1 · p. 2
- IRNet predicts instance maps from displacement vectors pointing to instance centroids and derives low semantic affinity across detected class boundaries.It is trained using displacements and class equivalence among high-attention pixels collected from CAMs, so image-level class labels provide all required supervision.Source: §1 · p. 2
- The method substantially outperforms the prior state of the art with the same supervision and surpasses stronger-supervision models SDI [24] and SDS [18] on PASCAL VOC 2012.The introduction also states that the approach requires neither additional supervision nor segmentation proposals.Source: §1 · pp. 1–2
2. Related Work
The section situates the method among weakly supervised semantic and instance segmentation, pixel-wise instance-location prediction, and pixel-affinity approaches. It highlights the challenge of recovering complete instances from image-level labels without extra supervision.
Weakly Supervised Semantic Segmentation
Weakly Supervised Instance Segmentation
- Image-level instance segmentation relies on high-quality proposals trained with extra supervision, whereas our approach uses neither and substantially surpasses.Image-level supervision is challenging because it provides no instance-specific information.Source: §2 · p. 3
Pixel-wise Prediction of Instance Location
3. Class Attention Maps
Class activation maps (CAMs) define instance seed areas for later propagation and provide supervision for learning IRNet from inter-pixel relations. They are generated with an image-classification CNN and kept at 1/16 of input-image width and height.
3. Class Attention Maps
- CAMs both define instance seed areas for later propagation [1] [26] and provide inter-pixel-relation supervision for training IRNet.Reliable inter-pixel relations are extracted from CAMs to train IRNet.Source: §3 · p. 3
- CAMs are generated using the method of with an image-classification CNN that uses global average pooling followed by a classification layer.Source: §3 · p. 3
- Using ResNet50 [20] with the last downsampling layer's stride reduced from 2 to 1 yields CAMs whose width and height are each 1/16 of the input image.CAMs for irrelevant classes are fixed to zero matrices.Source: §3 · p. 3
4. Inter-pixel Relation Network
IRNet predicts displacement and class-boundary maps used to estimate pseudo-instance masks from CAMs, and learns these outputs from inter-pixel relations without extra supervision.
4.1. IRNet Architecture
- IRNet jointly predicts a displacement field and class-boundary map using two branches that share a ResNet50 backbone and features from all five levels.The displacement branch merges features through a top-down pathway; the boundary branch resizes and concatenates features before producing its map.Source: §4.1 · p. 4
4.2. Inter-pixel Relation Mining from CAMs
- CAM mining selects confident foreground and background pixels, refines them with dense CRF, assigns each pixel its highest-scoring class, and samples nearby pairs by class equivalence.Foreground attention scores exceed 0.3 and background scores fall below 0.05; pixels outside these confident regions are ignored.Source: §4.2 · p. 4
4.3. Loss for Displacement Field Prediction
- For displacement learning, nearby same-class pixel pairs train their predicted-vector differences to match coordinate differences, making their implied centroids coincide.Random initialization is argued to help satisfy the per-instance centroid condition; background pixels receive a separate loss to prevent trivial centroid estimates.Source: §4.3 · pp. 4–5
4.4. Loss for Class Boundary Detection
- A multiple-instance-learning objective trains the boundary map from pixel-pair class equivalence, treating the line between differently labeled pixels as a bag of potential boundaries.Pairwise affinity is derived from the boundary map and trained with cross-entropy; the losses are normalized because the pair sets are imbalanced.Source: §4.4 · p. 5
4.5. Joint Learning of the Two Branches
- The two IRNet branches are jointly trained with all defined losses, using class-equivalence relations rather than individual class labels to use more relations per class and improve generalization.This relation-based supervision does not require additional supervision.Source: §4.5 · p. 5
5. Label Synthesis Using IRNet
IRNet’s displacement field D is converted into a class-agnostic instance map, while its boundary output B yields semantic affinities. The map and affinities are combined with class activation maps to synthesize pseudo instance-segmentation labels.
5. Label Synthesis Using IRNet
- IRNet’s displacement field D supplies the class-agnostic instance map, and its boundary output B supplies semantic affinities for pseudo-label synthesis.Semantic affinities are derived directly from B.Source: §5 · p. 5
5.1. Generating Class-agnostic Instance Map
- Because incomplete-CAM supervision makes displacement estimates inaccurate, the method iteratively refines D by following vectors at estimated centroids, then groups connected near-zero-displacement pixels as centroids to form I.The iterative refinement converges because displacement vectors near centroids tend to have almost zero magnitude.Source: §5.1 · pp. 5–6
5.2. Synthesizing Instance Segmentation Labels
- Powered, row-normalized semantic affinities drive random-walk propagation to refine instance-wise CAMs, while boundary scores are penalized.Each pixel receives the highest-scoring class-instance label; a pixel below the bottom-25% maximum-score cutoff is assigned to background.Source: §5 · p. 5, §5.2 · p. 6
6. Experiments
On PASCAL VOC 2012, the experiments evaluate both the quality of generated pseudo labels and the segmentation models trained with them. The framework is assessed for instance and semantic segmentation using image-level class labels.
6.1. Experimental Setting
- The framework uses image-level class labels on PASCAL VOC 2012, training on 10,582 images and validating on 1,449.The training set is expanded with the image set proposed in.Source: §6.1 · p. 6
6.2. Analysis of Pseudo Labels
- Class-boundary propagation improves pseudo-instance-label quality by more than 25%, and displacement fields add 3.6% by distinguishing same-class instances.The class-boundary map helps recover object areas missing from CAMs.Source: Table 1 · p. 7, §6.2 · p. 7
- Our pseudo semantic labels are substantially more accurate than AffinityNet’s, attributed to IRNet’s superior predicted pairwise semantic affinities.The comparison uses mIoU on the PASCAL VOC 2012 training set.Source: Table 2 · p. 7, §6.2 · p. 7
6.3. Mask R-CNN for Instance Segmentation
- Our Mask R-CNN outperforms box-supervised SDI by 1.9% and fully supervised SDS by 2.9% in APr, while surpassing image-level PRM.The evaluation uses Mask R-CNN with a ResNet-50-FPN backbone; APr measures mask average precision at different IoU thresholds.Source: Table 3 · p. 7, §6.3 · p. 7
6.4. DeepLab for Semantic Segmentation
- Our semantic segmentation approach outperforms prior methods using the same supervision, is competitive with box-supervised BoxSup, and reaches 88% of the fully supervised counterpart.DeepLab v2-ResNet50 is trained using our pseudo semantic labels.Source: Table 4 · p. 7, §6.4 · p. 8
7. Conclusion
The paper addresses the ill-posed nature of image-level weakly supervised instance segmentation with IRNet, which identifies instances and estimates rough boundaries. IRNet evidence improves class attention for pseudo-label training, yielding state-of-the-art instance and semantic segmentation on Pascal VOC 2012.
7. Conclusion
- Image-level weakly supervised instance segmentation is ill-posed because it lacks instance-specific information.Source: §7 · p. 8
- On Pascal VOC 2012, models trained with the proposed pseudo labels achieve state-of-the-art performance in both instance and semantic segmentation.Source: §7 · p. 8
- IRNet identifies individual instances and estimates their rough boundaries, providing evidence that improves simple class attention.Source: §7 · p. 8
- Improved class attention is used to create pseudo labels for training fully supervised instance segmentation models.Source: §7 · p. 8
A. Appendix
The appendix supplies material omitted from the main sections for brevity, covering the centroid detection algorithm, models used for final evaluation, and additional qualitative results.
A.1
- Section A.1 details the centroid detection algorithm introduced in Section 5.1.Source: A. Appendix · p. 11
A.2
- Section A.2 introduces instance and semantic segmentation models trained with synthetic labels for final evaluation.Source: A. Appendix · p. 11
A.3
- Section A.3 presents additional qualitative results.Source: A. Appendix · p. 11
A.1. Details of the Centroid Detection Algorithm
The algorithm treats small groups of neighboring pixels as centroid candidates when their displacement vectors are below a magnitude threshold, then uses connected-component indices to form a class-agnostic instance map.
A.1. Details of the Centroid Detection Algorithm
- Candidate centroids are pixels whose displacement vectors in D have magnitudes below a threshold, reflecting the use of neighboring pixel groups rather than single-coordinate centroids.The paper defines the candidate set after identifying pixels that meet this displacement-magnitude condition.Source: §A.1 · p. 11
- Each Ĉ_i denotes a connected component of pixels in C, and K is the number of connected components.Source: §A.1 · p. 11
- The class-agnostic instance map I is obtained by assigning each pixel a connected-component index.Source: §A.1 · p. 11
A.2. Details of Our Segmentation Networks … A.3. More Qualitative Results of Our Approach
The appendix details segmentation models trained with synthetic labels and reports that synthetic-label DeepLab reaches 89.4% of its fully supervised counterpart; qualitative examples show instance and semantic segmentation outputs.
A.2. Details of Our Segmentation Networks
- The framework evaluates its synthetic labels by training fully supervised Mask R-CNN for instance segmentation and DeepLab v2 [5] for semantic segmentation, both pretrained on ImageNet.The models are finetuned using synthetic labels instead of ground-truth segmentation masks.Source: §A.2 · p. 11
A.2.1 Mask R-CNN for instance Segmentation
- Mask R-CNN uses Detectron with a ResNet50-FPN [30] backbone and the default training settings, except for adjusted training steps for PASCAL VOC 2012.Source: §A.2.1 · p. 11
A.2.2 DeepLab v2 for Semantic Segmentation
- The manually implemented PyTorch DeepLab v2 [5] uses ResNet-50 [20] followed by atrous spatial pyramid pooling, with the original model’s training setting.Source: §A.2.2 · p. 11
- DeepLab v2 evaluation aggregates eight outputs by pixel-wise average pooling across four image scales, each with an original and horizontally flipped input.The scales are 0.5, 1.0, 1.5, and 2.0.Source: §A.2.2 · p. 11
- DeepLab trained with synthetic labels reaches 89.4% of its fully supervised counterpart on the PASCAL VOC 2012 test set.The measured fully supervised upper bound is below the original paper’s reported performance because dense CRF parameters were not carefully tuned.Source: §A.2.2 · p. 11
A.3. More Qualitative Results of Our Approach
- IRNet uses image-level supervision to find class boundaries and displacement fields to instance centroids, helping synthesize accurate instance masks from CAMs.These signals are not directly available in CAMs; Figure 9 shows qualitative results on the PASCAL VOC 2012 train set.Source: §A.3 · p. 11, Figure 9 · p. 12
- Synthetic labels that differentiate attached instances help the models capture fine object shapes and separate adjacent instances of the same class.Additional instance and semantic segmentation examples are shown in Figures 10 and 11, respectively.Source: §A.3 · p. 11, Figure 10 · p. 13, Figure 11 · p. 14