Source-linked AI summary
Weakly- and Semi-Supervised Panoptic Segmentation
Qizhu Li, Anurag Arnab, Philip H. S. Torr
TL;DR
Pixel-perfect segmentation labels are expensive, so the paper develops a weakly supervised model for joint semantic and instance segmentation. It uses bounding boxes for “things” and image-level tags for “stuff,” avoids overlapping instances, and achieves strong results on Pascal VOC and Cityscapes.
Problem
Pixel-perfect semantic and instance segmentation requires costly per-pixel annotation, motivating effective supervision from bounding boxes and image-level tags.
Method
The model jointly predicts non-overlapping semantic and instance segmentations, using bounding boxes for “thing” classes and image-level tags for “stuff” classes.
Results
The method achieves 95% of fully supervised performance on Pascal VOC, and 88.8% semantic and 85.6% instance performance on Cityscapes relative to fully supervised results.
Takeaways & Limitations
Weak annotations require about 3% of full-labelling time, while semi-supervised results suggest bounding-box annotations can achieve similar performance to more expensive annotation budgets.
Takeaways & Limitations
Image-level tags cannot distinguish multiple instances of the same class, and future work is needed to perform instance segmentation without object detections.
Abstract
from arXiv · showhide
We present a weakly supervised model that jointly performs both semantic- and instance-segmentation -- a particularly relevant problem given the substantial cost of obtaining pixel-perfect annotation for these tasks. In contrast to many popular instance segmentation approaches based on object detectors, our method does not predict any overlapping instances. Moreover, we are able to segment both "thing" and "stuff" classes, and thus explain all the pixels in the image. "Thing" classes are weakly-supervised with bounding boxes, and "stuff" with image-level tags. We obtain state-of-the-art results on Pascal VOC, for both full and weak supervision (which achieves about 95% of fully-supervised performance). Furthermore, we present the first weakly-supervised results on Cityscapes for both semantic- and instance-segmentation. Finally, we use our weakly supervised framework to analyse the relationship between annotation quality and predictive performance, which is of interest to dataset creators.
1 Introduction
The paper targets costly pixel-level segmentation with weak annotations, jointly producing non-overlapping semantic and instance segmentations for both “thing” and “stuff” classes. It reports strong Pascal VOC performance and introduces weakly supervised results on Cityscapes.
- Motivation: Pixel-level Cityscapes annotation takes more than 1.5h per image, motivating bounding boxes and image-level tags as cheaper supervision.Bounding boxes take 7 seconds and image-level tags average 1 second per class, corresponding to a 30× labelling-time reduction.
- Contribution: The model jointly produces semantic and instance segmentations, unlike prior weakly supervised methods that output only instance-agnostic semantic segmentation.The approach is designed to explain every pixel rather than only identify class regions.
- Problem setting: Detector-based approaches can produce overlapping instances, whereas this model assigns each pixel a unique “thing” or “stuff” label.This design addresses the requirement that all image pixels be explained.
- Supervision: Weak supervision uses bounding boxes for “thing” classes and image-level tags for “stuff” classes.The paper also considers semi-supervised training with mixtures of weakly and fully labelled annotations.
- Contribution: The paper reports the first weakly supervised non-overlapping instance segmentation covering both “thing” and “stuff” classes, including jointly trained semantic and instance segmentation.It also presents the first weakly supervised semantic- and instance-segmentation results on Cityscapes.
2 Related Work
The paper extends non-overlapping, semantic-segmentation-based instance segmentation to weaker supervision. It distinguishes its approach from prior weak methods through joint semantic and instance outputs, “stuff” handling, and support for weak-only training.
- Prior approaches: Detector-based instance segmentation processes instances independently, allowing overlaps and limiting its suitability for assigning every pixel a unique label.Semantic-segmentation-based approaches avoid overlapping instances and can naturally handle “stuff” classes.
- Weak supervision: Prior weakly supervised semantic segmentation used image-level labels, bounding boxes, scribbles, and partially labelled images, while weak instance segmentation remained limited.The related work situates this paper at the intersection of weak semantic and instance segmentation.
- Distinction from prior work: Unlike earlier weak instance-segmentation work, the proposed method handles image-level labels, “stuff” classes, non-overlapping instances, and every pixel.It also operates with only weak supervision rather than requiring some fully labelled examples.
3 Proposed Approach
The approach constructs approximate labels from weak annotations, refines them iteratively, and trains a joint semantic- and instance-segmentation network that assigns every pixel a unique instance label while covering both things and stuff.
- Approximate ground truth: Weak annotations are converted into labels for only confident pixels, while ambiguous pixels are marked ignore and excluded from the loss.The method estimates a labeled subset Ω′ of image pixels from bounding boxes or image-level tags rather than trusting all pixels.
- Approximate ground truth: Bounding-box supervision assigns a class only where GrabCut and MCG agree, and overlapping same-class boxes become ignore regions for instance labels.Each bounding box supplies one instance annotation; the MCG mask is selected by highest IoU with the box without using its pixel-supervised ranking stage.
- Approximate ground truth: Image-level tags produce class-specific approximate labels by thresholding weak localisation heatmaps from a multilabel classification network, while unsupported regions remain ignore.This supervision is suitable for stuff classes because image tags do not reveal how many same-class instances are present.
- Iterative refinement: The approximate ground truth is iteratively refined with network outputs, DenseCRF boundary post-processing, and constraints that ignore thing predictions outside their bounding boxes.For Pascal VOC, pixels outside thing boxes can instead be treated as background because background is the only stuff class.
- Network architecture: The architecture first predicts semantic probabilities, then partitions them into instances with a CRF whose variables assign each pixel one detection label.Its energy combines box, global, and pairwise terms; the global term can compensate for mislocalized boxes, while pairwise filtering encourages appearance and spatial consistency.
- Network architecture: Dummy whole-image detections for stuff classes let the same CRF jointly segment things and stuff, with differentiable mean-field inference producing the final labeling.The construction extends the detection-guided instance partitioning to classes for which object detectors are not trained.
4 Experimental Evaluation
Experiments on Pascal VOC and Cityscapes evaluate weak, full, and mixed supervision for semantic and non-overlapping instance segmentation. Weak supervision approaches fully supervised performance on VOC, while Cityscapes benefits substantially from iterative refinement and shows stronger results for thing than stuff classes.
- Datasets and supervision: The experiments use Pascal VOC and Cityscapes, with bounding-box supervision for thing classes and image-level tags for stuff classes.Fully supervised and weakly supervised models use the same images, while Cityscapes also includes coarse and fine annotations for different training stages.
- Pascal VOC results: About 95% of fully supervised performance is obtained by weak supervision for both semantic and instance segmentation on Pascal VOC.The comparison covers the model's fully supervised counterpart under the same evaluation setting.
- Pascal VOC results: Weakly supervised VOC instance segmentation outperforms some recent fully supervised methods, while the fully supervised model outperforms prior work on the dataset.The model uses PSPNet with ResNet-101, whereas one cited comparison uses VGG.
- Semi-supervision: Mixed supervision on Pascal VOC and COCO produces results close to using full supervision from both datasets.Fully supervised COCO plus weakly supervised VOC performs about the same as weak supervision on both datasets, while fully annotated VOC plus weakly labelled COCO obtains similar results to full supervision from both.
- Cityscapes results: On Cityscapes semantic segmentation, weak supervision reaches about 97% of fully supervised performance for things and 83% for stuff.Bounding boxes provide more informative supervision for things than image-level tags provide for stuff.
- Cityscapes results: Iterative refinement greatly improves both tasks on Cityscapes because first-iteration approximate ground truth is insufficient for high accuracy.The dataset's clutter, scale variation, and repeated object co-occurrences make weak localisation cues less effective than on Pascal VOC.
- Evaluation metrics: Changing instance scores can substantially affect AP r without changing segmentation quality, whereas PQ is unaffected by ranking methods.Detection scores work better for things, while mean mask softmax probabilities improve AP r for stuff.
5 Conclusion and Future Work
The paper concludes that weak supervision can jointly support non-overlapping semantic and instance segmentation of thing and stuff classes, while substantially reducing annotation effort. It also identifies image-level-only instance segmentation as future work.
- Conclusion: 88.8% of fully-supervised performance was achieved for semantic segmentation on Cityscapes, while instance segmentation reached 85.6% measured with PQ.On Pascal VOC, the method achieved about 95% of state-of-the-art fully-supervised performance for both tasks.
- Conclusion: Weak annotations required only about 3% of the time needed for full pixel-level labelling.The conclusion frames this reduction against the time-consuming nature of pixel-level annotation.
- Conclusion: The semi-supervised experiment suggests that adding fully labelled data is not an effective use of annotation budgets when similar performance can be obtained from bounding-box annotations alone.
- Future Work: Future work is to perform instance segmentation using only image-level tags and the number of instances of each object as supervision.This would require an architecture that does not use object detections as an additional input.
Appendix
The appendix documents additional dataset-annotation analyses, including comparisons involving Pascal VOC and COCO and an estimate of annotation-time savings from weak supervision.
- Appendix: The appendix presents randomly drawn Pascal VOC and COCO images to illustrate the claim that Pascal VOC annotation quality is better than COCO annotation quality.
- Appendix: It also shows how the overall annotation time is reduced when weak annotations are used.
A Additional Qualitative and Quantitative Results
The appendix provides additional Cityscapes and Pascal VOC results comparing weakly and fully supervised models. The qualitative examples show broadly similar outputs in some cases, but more precise fully supervised segmentations and specific weak-supervision errors.
- Additional Results: Figures 7 and 8 and Tables 7 and 8 provide additional qualitative and quantitative results on Cityscapes and Pascal VOC.
- Cityscapes: On Cityscapes, fully supervised models produce sharper boundaries and more accurate stuff segmentation than weakly supervised models.The weak model uses bounding boxes and image-level tags, while the fully supervised model has denser supervision.
- Pascal VOC: On Pascal VOC, weakly supervised results are typically similar to those of the fully supervised counterpart, although the latter produces more accurate and precise segmentations.
- Pascal VOC: Some Pascal VOC examples show weak supervision failing to segment a green person as well as full supervision, while both models incompletely segment bottles in another example.
- Quantitative Results: Tables 7 and 8 report per-class semantic and instance segmentation results, using IoU for semantic segmentation and AP for instance segmentation.
B.1 Network architecture and training
The network uses a PSPNet reimplementation with a ResNet-101 backbone, followed by staged training of semantic and instance segmentation modules. Training settings are largely shared across weakly and fully supervised experiments.
- Network Architecture: The semantic segmentation network reimplements PSPNet with a ResNet-101 backbone and output stride 8.Predictions are upsampled by a factor of 8 to reach the original image resolution.
- Training: Training uses a single 521×521 image crop per batch, momentum 0.9, and weight decay 5 × 10−4 across experiments.
- Training: The semantic module is trained first, after which the entire instance segmentation network is finetuned.
- Training: Semantic training uses an initial learning rate of 1 × 10−4, reduced to 1 × 10−5 after convergence.The weak Pascal VOC model uses the same schedule without iterative training.
- Training: Instance training fixes the learning rate at 1×10−5, with losses converging after approximately 400k iterations on VOC and Cityscapes.
- Training: Faster-RCNN training uses the default hyperparameters from the publicly available implementation.
B.2 Multi-label classification network
The method obtains weak localization cues by training a multi-label classification network on Cityscapes, adapted from PSPNet and trained with binary cross-entropy over image-level class labels. Full-resolution images exceed available GPU memory, motivating the stated memory considerations.
- Weak localization cues are obtained by first training this multi-label classification network on Cityscapes.
- A PSPNet segmentation architecture is adapted for multi-label classification by adding global average pooling and a 19-output fully connected layer.The 19 outputs correspond to the Cityscapes classes.
- The network is trained with binary cross-entropy for each of the 19 image-level class labels.The labels indicate whether each class is present in the image.
- A full-resolution 2048 × 1024 Cityscapes image would require 48.8 GB with PSPNet at output stride 8 and batch size 1.A standard ResNet-101 would still require 21.7 GB, exceeding the available 12 GB GPU memory.
C Comparison of Pascal VOC and Microsoft COCO annotation quality
Pascal VOC annotations are presented as higher quality than Microsoft COCO annotations, particularly at object boundaries. COCO polygons are often visibly incorrect, whereas Pascal VOC includes void regions where annotators were uncertain.
- COCO object polygons and boundary annotations are often incorrect compared with the higher-quality Pascal VOC annotations.
- Pascal VOC marks uncertain regions as “void” rather than assigning an uncertain label.
D Calculation of reduction factor in annotation time if only weak labels are used
The weak-label annotation estimate combines bounding boxes for thing classes with image-level tags for stuff classes. Under the stated assumptions, this requires 3.0 minutes per image, a 29.6-fold reduction from 90-minute pixel-level annotation.
- Cityscapes contains 11 stuff classes and 8 thing classes, averaging 17.9 thing instances per full-resolution image.
- The estimate assumes 7 seconds per thing bounding box and 1 second per present stuff-class image-level tag.
- 125.3 seconds are required for thing bounding boxes, while 57 seconds are required for stuff tags across 15 fixed crops.The total is based on 17.9 thing instances and an average of 3.8 stuff tags per crop.
- 3.0 minutes per image represents a 29.6-fold reduction from the original 90-minute pixel-level annotation time.
- COCO annotations use polygons whose object boundaries are often incorrect, while Pascal VOC shows uncertain regions as void.