Source-linked AI summary
L2G: A Simple Local-to-Global Knowledge Transfer Framework for Weakly Supervised Semantic Segmentation
Peng-Tao Jiang, Yuqi Yang, Qibin Hou, Yunchao Wei
TL;DR
Weakly supervised semantic segmentation needs precise class-aware attention maps because image-level labels do not provide pixel-wise annotations. L2G extracts detailed attentions from random local patches and transfers complementary knowledge to a global network, achieving 72.1% mIoU on PASCAL VOC 2012 validation and 44.2% on MS COCO 2014 validation.
Problem
Image-level labels provide object-category presence but not accurate pixel-wise annotations, making high-quality attention maps important for weakly supervised semantic segmentation.
Method
L2G uses a local classification network to extract detailed attentions from multiple cropped views and a global network to learn complementary attention knowledge online.
Results
72.1% mIoU on the PASCAL VOC 2012 validation set and 44.2% mIoU on the MS COCO 2014 validation set set new weakly supervised state-of-the-art records.
Takeaways & Limitations
The global network produces high-quality attention maps that can be used as pseudo annotations for semantic segmentation.
Takeaways & Limitations
Object shape quality remains imperfect, and more advanced classification or saliency models could further improve results.
Abstract
from arXiv · showhide
Mining precise class-aware attention maps, a.k.a, class activation maps, is essential for weakly supervised semantic segmentation. In this paper, we present L2G, a simple online local-to-global knowledge transfer framework for high-quality object attention mining. We observe that classification models can discover object regions with more details when replacing the input image with its local patches. Taking this into account, we first leverage a local classification network to extract attentions from multiple local patches randomly cropped from the input image. Then, we utilize a global network to learn complementary attention knowledge across multiple local attention maps online. Our framework conducts the global network to learn the captured rich object detail knowledge from a global view and thereby produces high-quality attention maps that can be directly used as pseudo annotations for semantic segmentation networks. Experiments show that our method attains 72.1% and 44.2% mIoU scores on the validation set of PASCAL VOC 2012 and MS COCO 2014, respectively, setting new state-of-the-art records. Code is available at https://github.com/PengtaoJiang/L2G.
1. Introduction
L2G addresses weakly supervised semantic segmentation by mining richer class-aware attention from local views and transferring complementary details to a global network. It produces high-quality attention maps and achieves state-of-the-art results on PASCAL VOC 2012 and MS COCO 2014.
- Motivation: Image-level labels are popular weak supervision because they indicate object-category presence while avoiding costly pixel-wise annotations.Pixel-wise labels require substantial human labor and resources.
- Motivation: CAMs provide semantic and location information for target objects and can serve as pseudo pixel-level annotations, making their quality important for segmentation.Prior strategies include adversarial erasing, online attention accumulation, seed-region expansion, and affinity learning.
- Motivation: Local views capture richer object details than the global view, motivating L2G’s use of randomly cropped patches alongside the original image.The local-view attention maps contain complementary details across patches.
- Proposed framework: L2G uses a local network to produce detailed local attentions and a global network to distill discriminative attention knowledge from them online.The transferred knowledge enables the global network to produce more integral object attentions.
- Results: 72.1% mIoU on the PASCAL VOC 2012 validation set and 44.2% mIoU on the MS COCO 2014 validation set set new weakly supervised state-of-the-art records.Using DeepLab-v2, the method also reaches 71.7% mIoU on the PASCAL VOC 2012 test set.
2. Related Work
Prior WSSS methods either train segmentation models directly from image-level labels or refine attention maps generated from global views. L2G differs by transferring complementary attention knowledge from multiple local views to a global network.
- One-stage WSSS: One-stage WSSS methods directly train end-to-end segmentation networks using image-level labels, including through multiple-instance learning and intermediate predictions.These methods provide a distinct alternative to attention-map-based two-stage pipelines.
- Two-stage WSSS: Two-stage WSSS methods generate pseudo segmentation labels from attention maps and then use them to train segmentation networks.Producing high-quality attention maps is the central challenge of this paradigm.
- Attention refinement: Attention refinement methods improve object regions through adversarial erasing, pixel-affinity propagation, boundary learning, or saliency supervision.These approaches target more integral regions and precise boundaries.
- L2G distinction: Unlike prior methods that refine attention maps on the global view, L2G uses both global and multiple local views.It transfers complementary attention knowledge from the local network to the global network.
- L2G distinction: L2G is related to knowledge distillation but transfers attention knowledge from local views rather than focusing only on prediction-distribution imitation.The framework applies this transfer to attention-map generation for WSSS.
3. Method
L2G generates richer object attention maps by transferring complementary detail from randomly cropped local views to a global network, optionally adding shape constraints from saliency maps.
- Overall framework: L2G combines a global network, local network, attention transfer module, and shape transfer module for attention generation.The global and local networks may use CNN classifiers such as VGGNet or ResNet-38.
- Training objective: The objective combines classification loss with a weighted knowledge-transfer loss, using attention transfer unless a shape constraint is added.The loss is L = L_cls + λ·L_kt, with L_kt set to L_at without shape constraints and L_st otherwise.
- Local-to-global attention transfer: Local views are randomly cropped from the input, and the local network extracts attention maps containing rich object details.The global view is processed by the global network, which learns from the local views and generates attentions during inference.
- Local-to-global attention transfer: Attention transfer matches local and corresponding global attention regions using mean squared error across the N local views.During inference, only the global network is needed to generate attention maps.
- Local-to-global shape transfer: Shape transfer binarizes local attentions and combines them with corresponding saliency regions, falling back to ordinary attention transfer when saliency is absent.Element-wise multiplication is used to suppress attention outside salient objects.
4. Experiments
Experiments evaluate L2G across datasets, sampling choices, network designs, ablations, and comparisons with prior WSSS methods. Local-to-global transfer improves attention quality, while final results reach state-of-the-art segmentation performance on PASCAL VOC and MS COCO.
- Experimental Setup: Experiments use PASCAL VOC 2012 and MS COCO 2014, with mIoU as the evaluation metric and augmented training data for VOC.The setup also uses 448×448 global views and 320×320 local patches.
- Ablation Study: 48.5% versus 56.8% mIoU shows that transferring local attention knowledge to the global network is more effective than using local-network attention alone.The local network discovers more object regions, but the transfer process leverages that knowledge more efficiently.
- Ablation Study: Local-to-global attention transfer substantially outperforms baseline CAM, whereas sliding-window inference performs worse than original CAM.The authors attribute the sliding-window weakness to the global-view model’s difficulty responding to non-discriminative regions with different appearances.
- Ablation Study: Adding classification loss to the global network decreases pseudo-label quality from 70.3% to 53.8% because it favors discriminative regions over transferred non-discriminative regions.The attention-transfer loss and classification loss therefore play opposite roles in the global network.
- Ablation Study: Using different local and global backbones improves pseudo-label mIoU by 1.1% and final segmentation mIoU by 1.2% over backbone sharing.With shared backbones, pseudo-label and validation mIoU are 69.2% and 70.9%, respectively.
5. Conclusion
L2G uses local-to-global attention transfer to obtain object attentions, leveraging complementary local-view attention and shape constraints. The method achieves the best results on the evaluated PASCAL VOC 2012 and MS COCO 2014 settings.
- L2G transfers complementary attention from local views to attain object attentions and incorporates a shape constraint in the transfer process.