PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Auxiliary Tasks with Affinity Learning for Weakly Supervised Semantic Segmentation

Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, Ferdous Sohel, Dan Xu

arXiv:2107.11787v2cs.CV

TL;DR

Weakly supervised segmentation lacks reliable pixel supervision because image-level labels yield sparse CAMs and existing saliency cues are often fixed offline. AuxSegNet jointly learns classification, saliency detection, and segmentation with cross-task affinities that iteratively refine pseudo labels, achieving state-of-the-art results on PASCAL VOC 2012 and MS COCO.

  • Problem

    Image-level labels produce inferior segmentation supervision, while coarse off-the-shelf saliency maps are used as fixed cues rather than updated through network training.

  • Method

    AuxSegNet jointly trains classification, saliency detection, and semantic segmentation while learning cross-task pixel affinities to refine predictions and pseudo labels.

  • Results

    The approach achieves state-of-the-art weakly supervised segmentation performance on PASCAL VOC 2012 and MS COCO.

  • Takeaways & Limitations

    Joint auxiliary learning and iterative cross-task pseudo-label updating provide continuous boosts to semantic segmentation performance.

Abstract

from arXiv · show

Semantic segmentation is a challenging task in the absence of densely labelled data. Only relying on class activation maps (CAM) with image-level labels provides deficient segmentation supervision. Prior works thus consider pre-trained models to produce coarse saliency maps to guide the generation of pseudo segmentation labels. However, the commonly used off-line heuristic generation process cannot fully exploit the benefits of these coarse saliency maps. Motivated by the significant inter-task correlation, we propose a novel weakly supervised multi-task framework termed as AuxSegNet, to leverage saliency detection and multi-label image classification as auxiliary tasks to improve the primary task of semantic segmentation using only image-level ground-truth labels. Inspired by their similar structured semantics, we also propose to learn a cross-task global pixel-level affinity map from the saliency and segmentation representations. The learned cross-task affinity can be used to refine saliency predictions and propagate CAM maps to provide improved pseudo labels for both tasks. The mutual boost between pseudo label updating and cross-task affinity learning enables iterative improvements on segmentation performance. Extensive experiments demonstrate the effectiveness of the proposed auxiliary learning network structure and the cross-task affinity learning method. The proposed approach achieves state-of-the-art weakly supervised segmentation performance on the challenging PASCAL VOC 2012 and MS COCO benchmarks.

1. Introduction

The paper targets weakly supervised segmentation, where image-level labels and coarse CAM or saliency cues provide limited pixel supervision. AuxSegNet jointly learns auxiliary classification and saliency tasks with segmentation, using cross-task affinities to iteratively update pseudo labels.

  • Image-level labels reduce annotation cost but provide inferior segmentation supervision compared with dense pixel annotations.
  • CAMs identify discriminative regions but remain sparse with coarse boundaries, while fixed off-line saliency cues are not updated during training.
  • Cross-task global pixel affinities combine saliency and segmentation information to refine predictions and propagate CAM activations into improved pseudo labels.
  • AuxSegNet uses multi-label classification and saliency detection as auxiliary tasks to learn semantic segmentation from image-level ground-truth labels.
  • The method reports state-of-the-art weakly supervised segmentation results on PASCAL VOC 2012 and MS COCO.

2. Related Work

Prior work improves weakly supervised segmentation through CAM refinement, iterative pseudo-label mining, and pixel-affinity learning. AuxSegNet differs by learning global cross-task affinities jointly with auxiliary classification and saliency learning, without requiring strong saliency labels.

  • Existing methods refine CAMs or iteratively mine pseudo labels, but their CAM boundaries and uncovered object regions can remain problematic.
  • Prior affinity methods either use selected samples from sparse CAMs or apply affinities only to enhance object-estimation features.
  • AuxSegNet learns global pixel affinities across tasks and uses them to refine both task-specific predictions and pseudo labels.
  • Unlike a related joint saliency-segmentation framework requiring strong pixel-level saliency labels, AuxSegNet uses image classification and saliency detection as auxiliary tasks.

3. The Proposed Approach

AuxSegNet jointly trains classification, saliency detection, and semantic segmentation, while cross-task affinities refine representations, predictions, and pseudo labels across iterative stages.

  • Multi-Task Auxiliary Learning Framework: AuxSegNet trains classification, saliency detection, and semantic segmentation branches with a shared backbone and three task-specific outputs.The classification branch predicts class probabilities, while the other branches produce dense saliency and semantic segmentation maps.
  • Cross-task Affinity Learning: A self-attention module adaptively weights the two task-specific affinity maps to form one global cross-task affinity map.The learned spatial weight maps aggregate saliency and segmentation affinities into a shared matrix.
  • Cross-task Affinity Learning: Non-local blocks learn task-specific saliency and segmentation affinities from spatial feature correlations to enhance their corresponding task features.Each affinity matrix represents pairwise relationships among spatial positions derived from task-specific feature maps.
  • Training and Inference: The global affinity map refines saliency and segmentation predictions, allowing improved pseudo labels to provide supervision for further affinity learning.This creates an iterative feedback loop between affinity learning and pseudo-label updating.
  • Training and Inference: AuxSegNet optimizes the three tasks with a weighted sum of classification, saliency, and segmentation losses.Saliency and segmentation losses include terms for both predictions and affinity-refined outputs.
  • Training and Inference: The network initializes saliency supervision from coarse pre-trained maps and updates saliency and segmentation pseudo labels after each training stage.Stage-wise optimization first trains classification, then trains the full network while repeatedly refreshing pseudo labels.

4. Experiments

AuxSegNet is evaluated on PASCAL VOC 2012 and MS COCO using auxiliary-task learning, cross-task affinity learning, and iterative pseudo-label refinement. Experiments show strong benchmark performance and progressive gains from the proposed components.

  • Comparison with State-of-the-arts: 69.0% and 68.6% mIoU are achieved on the PASCAL VOC val and test sets, respectively, outperforming recent off-the-shelf-saliency methods on the val set.The reported improvements over four recent methods are 4.7%, 1.2%, 2.8%, and 2.4%.
  • Comparison with State-of-the-arts: 33.9% mIoU is achieved on the MS COCO val set, exceeding the reported state-of-the-art methods.Qualitative results also cover small objects and multiple instances in complex scenes.
  • Ablative Analysis: Jointly learning image classification and saliency detection improves segmentation over the 56.9% single-task baseline, while using both auxiliary tasks gives the strongest variant.The passage reports that either auxiliary task improves segmentation, but does not provide the full table values for every variant.
  • Ablative Analysis: 64.1% mIoU is obtained when cross-task affinity refines both segmentation and saliency predictions, compared with 60.8% without affinity learning.Segmentation-only affinity reaches 61.5%, while integrating segmentation and saliency affinities adds a further 1.1% improvement before the joint refinement setting.
  • Ablative Analysis: Pseudo-label precision, recall, and mIoU rise from 85.6%, 42.9%, and 40.6% initially to 88.2%, 52.2%, and 49.7% after further affinity-learning stages.Pseudo-label quality saturates at Stage 3, while segmentation performance improves consistently across training stages.
  • Ablative Analysis: Refined CAM maps become more complete and boundary-aligned across stages, producing pseudo segmentation labels that progressively approach ground truth.The qualitative progression addresses sparse activations, small objects, multiple instances, and large-object boundaries.
  • Ablative Analysis: Comparable results and consistent baseline improvements are obtained with weakly supervised and fully supervised pre-trained saliency models.This evaluates sensitivity to the choice of pre-trained saliency input.

5. Conclusion

The paper concludes that AuxSegNet combines auxiliary-task learning with cross-task pixel affinity and iterative pseudo-label updating for weakly supervised semantic segmentation. Experiments on PASCAL VOC 2012 and MS COCO establish the method's effectiveness and state-of-the-art results.

  • 5. Conclusion: AuxSegNet uses image classification and saliency detection as auxiliary tasks, while cross-task affinities refine predictions and pseudo labels for semantic segmentation.The framework uses a shared backbone and iterative training to improve learned affinities and segmentation predictions.
  • 5. Conclusion: Experiments on PASCAL VOC 2012 and MS COCO demonstrate the method's effectiveness and establish new state-of-the-art weakly supervised segmentation results.
Loading 2107.11787v2…