Source-linked AI summary
Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation
Seunghoon Hong, Hyeonwoo Noh, Bohyung Han
TL;DR
Semantic segmentation needs costly pixel-wise annotations, while existing semi-supervised methods often rely on heuristic iterative procedures. This paper decouples classification and segmentation, using bridging layers to focus binary segmentation on labels identified from image-level data. Experiments report substantial improvements over comparable semi- and weakly-supervised methods with few strong annotations, although performance does not reach the state of the art under near-full supervision.
Problem
DNN segmentation requires costly pixel-wise annotations, while existing heterogeneous-annotation methods rely on heuristic iterative procedures and still need fairly many strong annotations.
Method
A decoupled architecture separately trains classification and segmentation networks, with bridging layers delivering class-specific activation maps from classification to segmentation.
Results
The method outperforms existing semi- and weakly-supervised approaches with substantial margins, including comparable performance advantages with very few strong annotations.
Takeaways & Limitations
Heterogeneous image-level and pixel-wise annotations can support separate classification and segmentation training without iterative intermediate-label inference.
Takeaways & Limitations
The method does not achieve state-of-the-art performance under almost full supervision, probably because classification and segmentation are optimized separately rather than jointly.
Abstract
from arXiv · showhide
We propose a novel deep neural network architecture for semi-supervised semantic segmentation using heterogeneous annotations. Contrary to existing approaches posing semantic segmentation as a single task of region-based classification, our algorithm decouples classification and segmentation, and learns a separate network for each task. In this architecture, labels associated with an image are identified by classification network, and binary segmentation is subsequently performed for each identified label in segmentation network. The decoupled architecture enables us to learn classification and segmentation networks separately based on the training data with image-level and pixel-wise class labels, respectively. It facilitates to reduce search space for segmentation effectively by exploiting class-specific activation maps obtained from bridging layers. Our algorithm shows outstanding performance compared to other semi-supervised approaches even with much less training images with strong annotations in PASCAL VOC dataset.
1 Introduction
Semantic segmentation remains difficult and costly to supervise because pixel-wise annotations are expensive and scarce. The proposed decoupled architecture uses abundant image-level labels alongside a small number of full masks, simplifying training while improving semi-supervised performance.
- Pixel-wise semantic segmentation remains challenging because object appearance varies with pose, scale, occlusion, and background clutter.
- Training DNN-based segmentation requires costly pixel-wise ground truths, which are typically available for only few classes and images.
- Weakly supervised methods exploit cheaper image-level or bounding-box labels but often rely on ad-hoc iterative inference and refinement procedures.
- The proposed architecture decouples classification and segmentation, identifying image labels first and then producing figure-ground segmentation for each identified label.
- Separate training uses image-level annotations for classification and pixel-wise annotations for segmentation, avoiding iterative procedures and enabling reproducibility.
- Experiments report substantially better performance than existing DNN-based semi-supervised techniques with only 5 or 10 strong annotations per class.
2 Related Work
Supervised segmentation achieves strong results but requires extensive pixel-wise annotation. Semi-supervised approaches reduce annotation demands by combining weak and strong labels, though existing methods commonly depend on iterative procedures and substantial strong supervision.
- CNN-based supervised segmentation has advanced the field but requires large collections of segmentation ground truths that are difficult and costly to obtain.
- Weakly supervised methods train with image-level or bounding-box labels and infer latent segmentation masks using MIL or EM frameworks.
- Semi-supervised learning combines many weak annotations with a small number of strong annotations to bridge fully and weakly supervised segmentation.
- Existing heterogeneous-annotation methods iteratively infer and refine pixel-wise labels using guidance from strongly annotated images.
- These approaches can improve accuracy and reduce annotation effort, but their heuristic iterative procedures complicate reproduction and still require fairly many strong annotations.
3 Algorithm Overview
The proposed network performs classification and segmentation as separate successive operations connected by bridging layers. It is trained on heterogeneous annotations without iterative label inference and focuses segmentation on relevant classes.
- Architecture: The architecture contains classification and segmentation networks plus bridging layers, performing label identification followed by class-specific pixel-wise figure-ground segmentation.
- Training: Training first learns classification from abundant image-level annotations, then fixes it while jointly training bridging layers and segmentation with scarce strong annotations.
- Contributions: The architecture decouples the two tasks, allowing a pre-trained classification model while training segmentation and bridging layers with few strongly annotated examples.
- Contributions: Bridging layers construct class-specific activation maps that provide segmentation priors and reduce the search space during training and inference.
- Contributions: Separate network training removes iterative heuristic inference of segmentation labels for weakly annotated images.
- Scope: The framework is closer to few-shot learning with heterogeneous annotations than to standard semi-supervised learning because it does not infer labels iteratively.
- Contributions: The method estimates segmentation only for labels identified by classification, improving scalability with respect to the number of classes.
4 Architecture
The architecture separates classification from binary segmentation through bridging layers that combine class-specific and spatial information. Classification identifies relevant labels, while segmentation produces a foreground/background map for each label.
- Architecture: The proposed network comprises separate classification and segmentation networks connected by bridging layers.The bridging layers connect the two networks within the overall architecture.
- Classification Network: The classification network outputs relevance scores for predefined categories and identifies the labels associated with each input image.The identified labels determine which class-specific regions are subsequently segmented.
- Segmentation Network: The segmentation network performs binary classification for each identified label, using foreground and background output channels.This differs from networks that classify each pixel among all predefined classes.
- Bridging Layers: Bridging layers combine spatial features from pool5 with class-specific saliency information to construct activation maps for segmentation.Class-specific saliency is obtained by back-propagating relevance from the class score to pool5, while pool5 preserves spatial configuration.
- Bridging Layers: Activation patterns for the same class remain similar across substantially different images, reducing variation in the segmentation network’s inputs.The paper states that this property helps the network produce figure-ground maps for individual relevant classes.
- Inference: At inference, the network generates class-specific maps for identified labels and assigns each pixel the foreground label with the maximum score.Figure 3 illustrates separate segmentation maps for individual classes.
5 Training
Training uses heterogeneous annotations by fitting classification first with image-level labels, then fitting bridging and segmentation components with pixel-wise labels. Combinatorial cropping augments the limited strongly annotated examples.
- Separate Training: The classification network is trained on the larger set of images with image-level labels, while bridging layers and segmentation are trained on the smaller pixel-wise labeled set.The classification weights are fixed before the bridging and segmentation components are trained.
- Separate Training: The two networks are optimized separately, reducing training complexity without requiring an iterative procedure.The paper presents separate optimization as a way to simplify training with heterogeneous annotations.
- Data Augmentation: Combinatorial cropping enumerates nonempty combinations of ground-truth labels to construct binary masks with selected labels as foreground.Sub-images enclosing the foreground areas are then generated using region proposals and random sampling.
- Data Augmentation: The augmentation strategy increases the effective number of strongly annotated training examples from Ns to Nt = Ns+Np·.The passage states that Nt is much larger than Ns, although the displayed expression is truncated.
6 Experiments
Experiments on PASCAL VOC 2012 evaluate DecoupledNet under varying amounts of strong pixel-wise supervision and compare it with semi- and fully supervised alternatives. The method performs especially well with very limited strong annotations, while nearly full supervision exposes a performance boundary.
- Dataset and setup: The experiments use 12,031 PASCAL VOC 2012 images with pixel-wise labels, splitting training data into weak-only and strongly annotated subsets.The classification network uses 10,582 images with image-level class labels; strong supervision is varied by class.
- Dataset and setup: DecoupledNet is compared with WSSL, DecoupledNet-Str, and DeconvNet, with segmentation accuracy measured on PASCAL VOC validation and test sets.The comparisons isolate the effects of heterogeneous annotations and image-level weak labels.
- Quantitative results: With the same supervision, DecoupledNet substantially outperforms WSSL, with the largest advantage when strongly annotated images are extremely scarce.The authors attribute this advantage to bridging layers that reduce the segmentation search space.
- Training and inference: Training is straightforward and avoids the heuristic iterative procedures used by common semi-supervised methods.Classification and segmentation networks are trained separately, without an iterative procedure or inference post-processing.
- Quantitative results: With few strong annotations, DecoupledNet outperforms DecoupledNet-Str and DeconvNet by exploiting weakly annotated images.The image-level-only reference reaches 42.0%, whereas five strong annotations per class already produce a higher result; the paper reports that little strong supervision improves performance dramatically.
- Limitations: Under almost full supervision, DecoupledNet does not reach state-of-the-art performance, likely because its classification and segmentation networks are optimized separately.The authors nevertheless identify the strategy as more appropriate for semi-supervised learning.
- Qualitative results: With five strong annotations per class, qualitative results show good generalization; additional strong annotations improve accuracy and reduce label confusions.These observations are reported for PASCAL VOC 2012 validation images.
7 Conclusion
The conclusion presents DecoupledNet as a decoupled architecture for semi-supervised semantic segmentation with heterogeneous annotations. It reports that bridging layers reduce the segmentation search space and that experiments validate strong performance with few segmentation annotations.
- Architecture: DecoupledNet decouples classification and segmentation networks for both training and inference while using heterogeneous annotations.The architecture is designed for image-level class labels and pixel-wise segmentation annotations.
- Training: The decoupled design simplifies training by discarding complex iterative procedures for intermediate label inferences.The conclusion describes this as conceptually appropriate for heterogeneous and unbalanced training data.
- Mechanism: Bridging layers reduce the segmentation output space and facilitate learning with a handful of segmentation annotations.This mechanism is presented as critical to the architecture’s effectiveness.
- Results: Experimental results validate the decoupled network, which outperforms existing semi- and weakly-supervised approaches with substantial margins.The reported conclusion concerns semantic segmentation experiments with limited strong supervision.