Source-linked AI summary
Learning Transferrable Knowledge for Semantic Segmentation with Deep Convolutional Neural Network
Seunghoon Hong, Junhyuk Oh, Bohyung Han, Honglak Lee
TL;DR
Weakly-supervised segmentation lacks pixel-level supervision, while collecting such annotations across many categories is costly. The paper transfers segmentation annotations from different categories through an attention-based encoder-decoder, substantially improving weakly-supervised performance and reducing the gap from stronger supervision.
Problem
Pixel-wise segmentation annotations are costly, while weakly-supervised methods using image-level labels typically provide lower-quality segmentation.
Method
The method uses a decoupled encoder-decoder with attention to transfer segmentation knowledge from annotated source categories and adapt it to weakly annotated target categories.
Results
The method substantially improves over existing weakly-supervised approaches and reduces the gap between strong and weak supervision through transfer learning.
Takeaways & Limitations
Segmentation annotations from different categories can compensate for missing supervision in weakly annotated images.
Takeaways & Limitations
The target and source domains are assumed to contain exclusive category sets, and the attention optimization has a potential overfitting issue because ground-truth labels are input to the attention model.
Abstract
from arXiv · showhide
We propose a novel weakly-supervised semantic segmentation algorithm based on Deep Convolutional Neural Network (DCNN). Contrary to existing weakly-supervised approaches, our algorithm exploits auxiliary segmentation annotations available for different categories to guide segmentations on images with only image-level class labels. To make the segmentation knowledge transferrable across categories, we design a decoupled encoder-decoder architecture with attention model. In this architecture, the model generates spatial highlights of each category presented in an image using an attention model, and subsequently generates foreground segmentation for each highlighted region using decoder. Combining attention model, we show that the decoder trained with segmentation annotations in different categories can boost the performance of weakly-supervised semantic segmentation. The proposed algorithm demonstrates substantially improved performance compared to the state-of-the-art weakly-supervised techniques in challenging PASCAL VOC 2012 dataset when our model is trained with the annotations in 60 exclusive categories in Microsoft COCO dataset.
1. Introduction
Semantic segmentation requires dense pixel labels, but collecting them is costly and weak supervision typically reduces quality. This paper transfers segmentation knowledge from differently categorized annotated data through an attention-based encoder-decoder.
- Semantic segmentation assigns dense class labels to every image pixel, creating a challenging structured prediction problem.
- Pixel-wise annotations are expensive to collect, making semantic segmentation difficult to scale across many categories.
- Weakly-supervised methods reduce annotation effort using image-level labels or boxes but usually produce lower-quality segmentation without direct shape and location supervision.
- The paper uses segmentation annotations from different categories to compensate for missing supervision in weakly annotated images.
- The proposed encoder-decoder with attention transfers category-independent segmentation knowledge while adapting attention to unseen categories using image-level labels.
- The method substantially improves performance over existing weakly-supervised approaches when using exclusive-category segmentation annotations.
2. Related Work
Related work includes strongly, weakly, and semi-supervised segmentation, external-data transfer, and visual attention. Existing weakly-supervised methods infer latent labels but remain inferior to fully supervised approaches.
- CNN segmentation methods learn pixel-wise labels from strong segmentation annotations, while weakly-supervised methods reduce labeling effort through weaker annotations.
- Weakly-supervised approaches commonly use Multiple Instance Learning or Expectation-Maximization to infer latent segmentation labels.
- These methods remain substantially worse than fully supervised approaches, whereas semi-supervised methods use limited strong annotations to narrow the gap.
- Domain adaptation often assumes shared categories, while transfer learning has commonly been limited to simpler tasks such as classification.
- The paper extends visual attention, previously used for recognition, tracking, captioning, and generation, to semantic segmentation by transfer learning.
3. Algorithm Overview
The task transfers segmentation knowledge from a source domain with pixel annotations to an exclusive-category target domain with only image-level labels. An encoder, attention model, and decoder separate localization from foreground segmentation.
- The target domain has only image-level labels, while the source domain provides pixel-wise segmentation annotations for exclusive categories.
- The decoder generates foreground masks from attention outputs, enabling source-trained segmentation knowledge to be applied to target categories.
- The encoder converts an input image into features, and the attention model identifies salient regions for each associated category.
- Source segmentation annotations train the decoder and attention model, while image-level labels from both domains train the attention model for classification.
- The decoupled architecture shares shape-generation information across categories, while attention supplies localization and category-specific adaptation.
4. Architecture
The architecture combines an encoder, category-conditioned attention model, and decoder to transfer segmentation knowledge across categories. Attention identifies category-specific regions, while a shared decoder reconstructs foreground masks using category-independent shape information.
- Architecture: The network comprises an encoder, attention model, classifier, and decoder trained using examples from both domains.The encoder extracts features, while attention and decoding support category-specific segmentation transfer.
- Attention model: The attention model produces positive, category-specific spatial weights from encoded features and associated labels.Softmax normalization converts unnormalized weights into attention maps that focus on image regions relevant to each category.
- Attention model: Multiplicative feature-label interactions model correlations between image features and category labels, with factorization reducing parameter count.The paper reports better results than additive interactions because multiplicative interactions capture higher-order dependencies.
- Attention model: Classification supervision makes attention trainable in target and source domains, while category-specific features are aggregated for decoder input.The attention model uses image-level labels in both domains, and the resulting representation summarizes feature relevance for each category.
- Decoder: Densified attention preserves more object-shape detail than original attention and supplies the decoder with inputs for reconstructing foreground masks.The decoder receives densified attention derived from the intermediate category-specific representation and original feature map.
- Decoder: A shared decoder learns category-independent shape information by generating foreground segmentation for each attention map, enabling application to unseen categories.The decoder is trained with source-domain segmentation annotations using a deconvolution architecture that recovers image-sized masks.
5. Training and Inference
Training jointly optimizes classification and segmentation objectives across source and target domains, while inference applies category-conditioned decoding to predicted image labels. The final segmentation assigns each pixel the label with maximum foreground probability.
- Training: The overall objective combines classification and segmentation losses, with λ controlling their balance during joint optimization.Classification and segmentation are optimized using examples from both domains, while segmentation supervision is available for source-domain images.
- Training: Joint training regularizes attention against noisy target-domain localization that can arise when attention is trained only from image-level classification.After training, the classification layers are removed because they are needed only to learn attention.
- Inference: Inference first uses a separate classifier to identify image labels, then constructs attention and foreground masks for each identified category.The decoder is applied iteratively to the category-specific attention representation for every predicted label.
- Inference: The final segmentation label is selected by taking the maximum foreground probability across the identified category channels.This combines the decoder’s foreground probability maps into the final pixel-level prediction.
6. Experiments
Experiments evaluate transfer from 60 exclusive MS-COCO categories to PASCAL VOC 2012 using weak labels, showing strong performance against weakly supervised methods while exposing domain-specific and attention-related limitations.
- The experiments transfer from 17,443 MS-COCO images across 60 categories to PASCAL VOC 2012, whose 20 categories are excluded from the source domain.
- The method outperforms weakly supervised approaches with substantial margins on PASCAL VOC 2012 validation without using evaluation-category segmentation annotations.Its performance is comparable to semi-supervised methods using some ground-truth segmentations.
- On the PASCAL VOC 2012 test set, the method outperforms weakly supervised approaches but remains substantially behind fully supervised approaches.The results indicate that annotations from different categories cannot replace all domain-specific segmentation knowledge.
- 6.3. Comparison to Baselines: Attention-based TransferNet produces denser and more accurate qualitative segmentations than BaselineNet, while CRF post-processing can recover some category-specific details.Noisy attention and incorrect input labels can still produce inaccurate predictions.
- 6.3. Comparison to Baselines: Attention enables transfer across unseen categories, whereas direct decoder transfer fails to adapt and class-score-map input remains considerably worse than the proposed method.The proposed attention representation provides coherent, richer information for the decoder across domains.
- 6.4. Impact of Number of Annotations in the Source Domain: More source-domain annotations improve target segmentation, yet the method remains better than other weakly supervised methods with a very small annotation fraction.The experiment averages performance across subsets containing 50%, 25%, 10%, 5%, and 1% of source annotations.
7. Conclusion
The paper proposes transferring segmentation knowledge from extra annotations in different categories through an attention-based encoder-decoder. Results suggest this can reduce the gap between strong- and weak-supervision approaches, while scaling to many categories remains future work.
- The approach uses extra segmentation annotations from different categories to improve segmentation where supervision is missing.
- The attention-based encoder-decoder is designed to capture transferable segmentation knowledge across categories.
- Results on a challenging benchmark suggest that transfer learning can reduce the gap between strong and weak supervision.
- Scaling the approach to a large number of categories is identified as an interesting direction for future research.
A. Analysis of Densified Attention
Densified attention combines Gram-matrix rows weighted by sparse attention to propagate activation across correlated feature-map locations. This produces denser object cues and supports more accurate segmentation masks than sparse attention alone.
- Densified attention is obtained by combining rows of the Gram matrix using attention weights.The Gram matrix represents pairwise similarity between feature-map pixels.
- The resulting attention highlights visually correlated areas beyond the initially attended pixels.
- A one-hot attention vector can generate dense activation maps from extremely sparse pixel attention.Figure 6 visualizes this densification by reshaping selected Gram-matrix rows to the feature-map layout.
- Using densified attention as decoder input is more useful for accurate, dense segmentation than using sparse attention weights.
B. Comparisons to Baseline Architectures
The proposed attention-based architecture is compared with transfer-learning baselines that construct decoder inputs from domain-specific representations. Those baselines transfer poorly, whereas attention better supports decoder generalization across domains.
- Architectures: The proposed architecture and two baselines share encoder-decoder components but differ in how they construct decoder input.Figure 7 details the proposed algorithm, BaselineNet, and DecoupledNet†.
- Baseline architectures: BaselineNet converts a selected class’s 4 × 4 score map into decoder input through a fully-connected layer.
- Baseline architectures: DecoupledNet† constructs decoder input by combining feature and gradient maps after identifying image categories.
- Comparison: Feature and gradient representations are domain-specific and change when the classification network is fine-tuned, limiting decoder generalization to unseen categories.
- Comparison: The poor baseline performance indicates that naive extensions of existing architectures struggle to transfer segmentation knowledge across domains.
- Comparison: The attention-based architecture is presented as more suitable for transferring the decoder across domains and obtaining accurate segmentation.
C. Additional Results
On PASCAL VOC 2012 validation images, the proposed method produces more accurate object boundaries than other weakly supervised approaches in qualitative comparisons. It does so without CRF, while competing methods often require CRF refinement and still produce noisy masks.
- Competing methods frequently fail to capture precise object shapes because their CNN predictions are noisy and inaccurate.
- The proposed approach tends to find more accurate object boundaries without CRF by using a decoder trained with segmentation annotations from different categories.
- The qualitative results show distinguishing performance compared with existing weakly supervised approaches.
- The comparison includes EM-Adapt, CCNN, TransferNet, and TransferNet+CRF on PASCAL VOC 2012 validation images.