Source-linked AI summary
Semi-Supervised Semantic Segmentation with High- and Low-level Consistency
Sudhanshu Mittal, Maxim Tatarchenko, Thomas Brox
TL;DR
Dense semantic segmentation requires costly pixel-wise annotations, motivating methods that learn from limited labeled samples and annotation-free images. The paper combines a GAN-based segmentation branch with a semi-supervised multi-label classification branch and self-training. Across several benchmarks, it reports best results, including an 11% improvement over the state of the art with 2% labeled data.
Problem
Dense pixel-wise annotation requires substantial manual effort, while semi-supervised learning for dense segmentation has received less attention than image-level classification.
Method
A dual-branch method combines adversarial segmentation with discriminator-based self-training and a Mean Teacher multi-label classifier that filters segmentation outputs.
Results
The method achieves the best reported results across PASCAL VOC 2012, PASCAL-Context, and Cityscapes, with an 11% improvement over the state of the art using 2% labeled data.
Takeaways & Limitations
The complementary branches alleviate low-level and high-level artifacts in low-data semi-supervised semantic segmentation.
Takeaways & Limitations
The method can under-segment protruding object classes, misclassify ambiguous foreground objects, and miss some thin objects.
Abstract
from arXiv · showhide
The ability to understand visual information from limited labeled data is an important aspect of machine learning. While image-level classification has been extensively studied in a semi-supervised setting, dense pixel-level classification with limited data has only drawn attention recently. In this work, we propose an approach for semi-supervised semantic segmentation that learns from limited pixel-wise annotated samples while exploiting additional annotation-free images. It uses two network branches that link semi-supervised classification with semi-supervised segmentation including self-training. The dual-branch approach reduces both the low-level and the high-level artifacts typical when training with few labels. The approach attains significant improvement over existing methods, especially when trained with very few labeled samples. On several standard benchmarks - PASCAL VOC 2012, PASCAL-Context, and Cityscapes - the approach achieves new state-of-the-art in semi-supervised learning.
1 INTRODUCTION
The paper addresses semi-supervised semantic segmentation by using annotation-free images alongside a small set of fully annotated samples. Its complementary branches target low-level and high-level errors, achieving especially strong gains with very few labels.
- Dense pixel-wise annotation is effective for semantic segmentation but requires substantial manual effort.
- The proposed dual-branch method learns from annotation-free images given a very small set of fully annotated samples.
- The GAN-based branch addresses low-level errors by improving object shapes, boundaries, and surface coherence through adversarial training and self-training.
- The multi-label classification branch addresses high-level inconsistencies by identifying present classes and filtering false positives.
- The approach improves over baselines on PASCAL VOC with especially considerable gains below 5% labeled samples, both without and with COCO pre-training.
- 11% performance improvement over the state of the art is reported with 2% labeled data, alongside best results across PASCAL VOC 2012, PASCAL-Context, and Cityscapes.
2 RELATED WORK
Prior work reduces annotation demands through weak supervision, but true semi-supervised segmentation with completely annotation-free images remains comparatively limited. The paper differentiates its GAN, self-training, feature-matching, and classification design from related approaches.
- Weakly supervised methods use image-level labels, bounding boxes, or scribbles, while semi-supervised schemes may additionally use a few pixel-wise labels.
- Only two recent works consider true semi-supervised segmentation using completely annotation-free images alongside limited pixel-wise labels.
- Unlike Hung et al., the paper uses an imagewise discriminator, automatic self-training integration, feature matching loss, and a multi-label classification branch.
- Luc et al. use a GAN to replace CRF post-processing for low-level consistency in fully supervised segmentation, rather than semi-supervised learning.
- Consistency-based semi-supervised classification methods improve over supervised baselines and motivate the classification component used in this segmentation framework.
3 METHOD
The method uses separate segmentation and image-level classification branches to exploit unlabeled images while addressing low-level and high-level prediction errors. Their outputs are combined to filter implausible classes and improve segmentation.
- 3 METHOD: The two-branch method combines s4GAN for pixel-wise segmentation with MLMT for image-level multi-label classification.The branches are trained separately and their outputs are combined for the final result.
- 3 METHOD: s4GAN trains a segmentation generator with supervised and adversarial objectives, while a discriminator distinguishes generated maps from ground-truth maps.The discriminator receives the image concatenated with its predicted or real segmentation map.
- 3.2 Multi-label Semi-supervised Classification: MLMT uses a Mean Teacher classifier whose image-level predictions filter false-positive segmentation maps for classes absent from the input image.The teacher is an exponential moving average of the student, and consistency training uses differently perturbed images.
- 3.1.1 Training S: The segmentation objective combines cross-entropy, feature matching, and self-training losses with positive weights λfm and λst.Cross-entropy is evaluated on labeled samples, while feature matching and self-training incorporate unlabeled data.
- 3.1.1 Training S: Feature matching aligns predicted and ground-truth feature statistics on unlabeled samples, encouraging plausible segmentation maps without dense labels.The loss uses an intermediate discriminator representation and compares feature statistics of predicted and ground-truth maps.
- 3.1.1 Training S: Discriminator scores select high-quality unlabeled predictions for self-training, providing pseudo-label supervision and helping prevent the discriminator from becoming too strong.Predictions above the confidence threshold γ are used with a standard cross-entropy loss.
4 EXPERIMENTS
The approach was evaluated on three semantic segmentation benchmarks: PASCAL VOC 2012, PASCAL-Context, and Cityscapes.
- 4 EXPERIMENTS: The evaluation covers PASCAL VOC 2012, PASCAL-Context, and Cityscapes.These are the three segmentation benchmarks used to assess the proposed approach.
4.1 Setup
Experiments use three segmentation datasets, report mIoU, and compare against supervised and semi-supervised baselines under specified preprocessing and optimization settings.
- 4.1.1 Datasets: PASCAL VOC uses 10,582 training images and 1,449 validation images across 20 foreground classes and one background class.The training set combines original PASCAL images with additional Segmentation Boundary Dataset images.
- 4.1.1 Datasets: PASCAL-Context contains 4,998 training and 5,105 testing images, with semantic labels for 60 frequent classes including background.Its data augmentations match those used for PASCAL VOC.
- 4.1.1 Datasets: Cityscapes provides 2,975 training, 500 validation, and 1,525 testing images across 19 classes.Images are downsampled by a factor of 2, with random 256 × 512 crops and horizontal flipping.
- 4.1.2 Evaluation Metric: Performance is reported using mean Intersection-over-Union (mIoU) for all experiments.The metric is applied across the reported segmentation benchmarks.
- 4.1.1 Datasets: Qualitative PASCAL VOC comparisons include original images, ground truth, a baseline, Hung et al., and the proposed results.The comparison is shown for 5% labeled data without COCO pre-training.
- 4.1.3 Baselines: DeepLabv2 is the fully supervised baseline, while Hung et al. is the main semi-supervised baseline.The supervised baseline trains only on the labeled portion, and Hung et al. uses a two-stage GAN procedure.
4.2 Results
Across PASCAL VOC, PASCAL-Context, and Cityscapes, the approach improves semi-supervised segmentation over labeled-only baselines, particularly with scarce labels. Ablations attribute complementary gains to branches addressing segmentation details and class-level false positives.
- PASCAL VOC: 5%–12% performance gains over the baseline occur across PASCAL VOC labeled-data splits without segmentation pre-training.The method also works with only 2% labeled data and improves over a prior method by 6.1 mIoU points at the 1/50 split with COCO pre-training.
- PASCAL VOC: 50.1 mIoU for baseline DeepLabv2 versus 60.5 for the semi-supervised method on the PASCAL test server.These results use one random split with 5% labeled samples and are consistent with the reported conclusions.
- PASCAL-Context: 3.2% and 2.4% improvements over the baseline are reported for the 1/8 and 1/4 PASCAL-Context splits, respectively.The approach generalizes to this smaller and more difficult whole-scene parsing dataset, although qualitative results are less visually appealing than on PASCAL VOC.
- Cityscapes: 3.1% and 1.7% improvements over the baseline come from the s4GAN branch on Cityscapes at the 1/8 and 1/4 splits.Class imbalance limits the usefulness of eliminating absent-class labels, reducing the MLMT branch’s benefit in this setting.
- Ablation Studies: The s4GAN and MLMT branches complement each other by improving shapes, holes, and boundaries while removing false positives and some s4GAN artifacts.Together, they address low-level segmentation errors and high-level class decisions using unlabeled images.
- Ablation Studies: Adding consistency-based semi-supervised classification increases the improvement over s4GAN alone to 2%.The MLMT classifier also lowers false-positive rates at the same true-positive rate, especially without ImageNet pre-training.
5 CONCLUSION
The paper presents a two-branch approach for semi-supervised semantic segmentation, with branches designed to alleviate low- and high-level artifacts in low-data settings.
- The two branches address both low-level and high-level artifacts that commonly occur when training semantic segmentation models with limited data.The paper demonstrates the design’s effectiveness through extensive experiments.