PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Label-efficient Deep Image Segmentation: Bridging the Gap between Weak Supervision and Dense Prediction

Wei Shen, Zelin Peng, Xuehui Wang, Huayu Wang, Jiazhong Cen, Dongsheng Jiang, Lingxi Xie, Xiaokang Yang, Qi Tian

arXiv:2207.01223v2cs.CV

TL;DR

Pixel-level labels make segmentation expensive, motivating label-efficient methods. The paper surveys these methods through a taxonomy of weak supervision and segmentation tasks, and unifies them by how heuristic priors bridge weak labels to dense prediction. Reported results show near-parity with fully supervised segmentation in some settings, while the supervision gap remains unresolved and scaling to many object classes is limited.

  • Problem

    Deep segmentation relies on expensive, laborious pixel-level annotations, creating a need to understand methods that reduce dependence on dense labels.

  • Method

    The survey categorizes methods by no, inexact, incomplete, or inaccurate supervision alongside segmentation task, and analyzes heuristic priors that bridge weak labels to dense prediction.

  • Results

    75.9% mIoU for state-of-the-art semi-supervised semantic segmentation is comparable to 76.0% for fully supervised segmentation, while inexact-supervision methods also achieve comparable state-of-the-art results.

  • Takeaways & Limitations

    Label-efficient segmentation has progressed across multiple supervision settings, but methods remain related by the unresolved challenge of converting weak labels into dense predictions.

  • Takeaways & Limitations

    The supervision gap remains unresolved, and existing models have limited ability to scale to large numbers of object classes.

Abstract

from arXiv · show

The rapid development of deep learning has made a great progress in image segmentation, one of the fundamental tasks of computer vision. However, the current segmentation algorithms mostly rely on the availability of pixel-level annotations, which are often expensive, tedious, and laborious. To alleviate this burden, the past years have witnessed an increasing attention in building label-efficient, deep-learning-based image segmentation algorithms. This paper offers a comprehensive review on label-efficient image segmentation methods. To this end, we first develop a taxonomy to organize these methods according to the supervision provided by different types of weak labels (including no supervision, inexact supervision, incomplete supervision and inaccurate supervision) and supplemented by the types of segmentation problems (including semantic segmentation, instance segmentation and panoptic segmentation). Next, we summarize the existing label-efficient image segmentation methods from a unified perspective that discusses an important question: how to bridge the gap between weak supervision and dense prediction -- the current methods are mostly based on heuristic priors, such as cross-pixel similarity, cross-label constraint, cross-view consistency, and cross-image relation. Finally, we share our opinions about the future research directions for label-efficient deep image segmentation.

1 INTRODUCTION

Deep segmentation achieves dense pixel predictions but typically depends on expensive, laborious pixel-level annotations. This survey organizes label-efficient methods by weak-supervision type and segmentation task, then unifies them around heuristic priors for bridging weak labels to dense prediction.

  • Dense labels enable deep segmentation but are expensive and laborious to obtain, motivating methods that reduce dependence on per-pixel annotations.
  • The survey addresses the rapid growth and fragmentation of label-efficient segmentation methods, which earlier surveys covered only for limited tasks or weak-label types.
  • The taxonomy combines four weak-supervision types—no, inexact, incomplete, and inaccurate—with semantic, instance, and panoptic segmentation problems.
  • The taxonomy marks explored and unexplored task–supervision intersections and lists representative works for explored combinations.
  • The survey unifies diverse methods as attempts to bridge weak labels and dense prediction using cross-label, cross-pixel, cross-view, and cross-image priors.

2 FOUNDATIONS

The foundations define segmentation and its weak-supervision variants, then describe common training pipelines, heuristic priors, and evaluation resources. Across these foundations, label-efficient learning replaces complete dense labels with cheaper but incomplete supervision and derived pseudo-label signals.

  • 2.1 Problem Definition: Label-efficient segmentation replaces expensive full dense maps with weak labels that provide incomplete supervision signals but are cheaper and easier to obtain.
  • 2.2 Methodology Overview: Self-training generates dense pseudo-labels from weak labels and trains a segmentation model on them, with regularization used to improve pseudo-label quality.
  • 2.2 Methodology Overview: End-to-end training directly optimizes generation and regularization losses, avoiding expensive iterative inference and sometimes outperforming self-training.
  • 2.3 Functions of the Heuristic Priors: The survey analyzes heuristic priors by their applicability to supervision types and formalizes their use through common loss functions.
  • 2.4 Datasets and Evaluation Metrics: Evaluation generally follows fully supervised segmentation, using datasets such as PASCAL VOC, Cityscapes, COCO-Stuff and metrics including mIoU, accuracy, and panoptic quality.

3 NO SUPERVISION

Unsupervised segmentation generates dense self-supervision without labels using heuristic priors such as cross-pixel similarity, cross-view consistency, and cross-image relations. Methods range from clustering and contrastive or Siamese representation learning to unsupervised instance segmentation, with promising results but unresolved challenges in regional priors.

  • Unsupervised semantic segmentation learns dense feature representations without labels and derives supervision from cross-pixel similarity, cross-view consistency, or cross-image relations.
  • 3.1.1 Cross-pixel Similarity as Dense Self-supervision: SegSort creates pseudo-segments using clustering and contour detection, then pulls pixel features toward their segment prototype while separating them from other prototypes.
  • 3.1.2 Cross-view Consistency as Dense Self-supervision: Contrastive methods generate two image views, align corresponding pixel representations, and distinguish positive from negative pairs using pixel-wise objectives.VADeR extends image-level contrastive learning to dense representation learning; later methods vary pair construction, prototypes, or view generation.
  • 3.1.2 Cross-view Consistency as Dense Self-supervision: Siamese approaches maximize consistency between views without negative samples, including IIC, which maximizes mutual information between adjacent pixels across views.
  • 3.1.3 Cross-image Relation as Dense Self-supervision: Cross-image methods mine semantic correspondences across images from unsupervised pretrained features, while FreeSOLO uses dense correspondences to create coarse masks for unsupervised instance segmentation.
  • 3.3 Discussion: 77.6% mIoU versus 76.0% mIoU was achieved by CP2-pretrained DeepLabV3 versus fully supervised DeepLabV3 on Pascal VOC 2012.The comparison is presented as evidence that unsupervised dense representations can facilitate segmentation.
  • 3.3 Discussion: Unsupervised dense representation learning remains preliminary because accurate regional priors are both required to learn representations and targeted by segmentation itself.The survey identifies this chicken-and-egg problem as a direction for improving or refining regional priors.

4 INEXACT SUPERVISION

Inexact supervision uses image-, box-, or scribble-level annotations instead of dense labels, requiring methods to bridge weak labels and pixel-level prediction. Common pipelines generate pseudo masks from seed areas through propagation or mining, then train segmentation models with them.

  • 4.1 Semantic segmentation with image-level supervision: Image-level semantic segmentation typically generates pseudo masks from classification-derived seed areas before training a segmentation model.CAMs are commonly used to obtain seed areas, followed by pseudo-mask generation and segmentation-model training.
  • 4.1.1 Seed area refinement: CAM-based seed areas are incomplete and may include regions belonging to other classes, motivating refinement through ensemble expansion, erasing, optimization, or decoupled reasoning.These limitations concern both insufficient coverage of target regions and redundant activation on other classes.
  • 4.1 Seed and pseudo-mask refinement: Cross-pixel similarity, cross-view consistency, and cross-image relations refine seeds or pseudo masks by exploiting semantic similarity within and across images.Methods use saliency guidance, augmentation-consistent CAMs, and semantic co-occurrence to improve coverage and discrimination.
  • 4.1.3 Panoptic segmentation with image-level supervision: Panoptic segmentation with image-level supervision remains sparsely explored, with JTSM connecting image-level labels to thing-and-stuff pseudo masks through multiple-instance learning and GrabCut refinement.The unified representations support pixel-level pseudo labels for both things and stuff before self-training.

4.2 Segmentation with Box-level Supervision

Box-level supervision narrows object localization but leaves foreground–background separation inside boxes unresolved. Methods mine pseudo masks from boxes or use box-derived constraints before training semantic, instance, or panoptic segmentation models.

  • 4.2.1 Semantic segmentation with box-level supervision: Semantic segmentation with box supervision mines pseudo masks from annotated boxes using cross-pixel similarity, then trains a segmentation model on those masks.The central challenge is distinguishing foreground objects from background regions within each box.
  • 4.2.1 Semantic segmentation with box-level supervision: MCG-based candidate segments have limited pseudo-mask reliability because they are generated without considering box-level supervision.Later methods instead treat boxes as noisy starting points for foreground-mask mining.
  • 4.2.2 Instance segmentation with box-level supervision: Instance segmentation methods generate box-constrained pseudo masks and self-train, while BBAM and BoxCaSeg add detector or saliency information for more precise foreground masks.SDI uses the box or an initial GrabCut segment and iteratively refines predictions through self-training.
  • 4.2.2 Instance segmentation with box-level supervision: BBTP and BoxInst directly impose projection consistency between predicted masks and annotated boxes through end-to-end training, but projection loss does not constrain mask shape.The missing shape constraint can permit trivial predicted-mask solutions.
  • 4.2.3 Panoptic segmentation with box-level supervision: Panoptic segmentation from box-level supervision remains challenging, with WPS combining image-level background labels, box-level foreground labels, Grad-CAM, and GrabCut.The supplied passage identifies WPS as the only attempted method for this setting.

4.3 Segmentation with Scribble-level Supervision

Scribble supervision provides sparse labeled pixels, so the central task is propagating their semantic information to unlabeled pixels. Existing methods use graph or differentiable propagation models, while others directly connect scribbles to dense predictions through losses.

  • 4.3 Segmentation with Scribble-level Supervision: Scribble-based segmentation treats hand-drawn annotations as seed areas and propagates their semantic information using cross-pixel similarity.The mainstream pipeline is designed to extend sparse scribble labels across the image.
  • 4.3 Segmentation with Scribble-level Supervision: Graph-based methods propagate scribble labels over superpixel nodes connected by appearance or semantic-feature similarities, producing pseudo masks for segmentation training.Alternating optimization updates the graph-based pseudo masks and the segmentation model.
  • 4.3 Segmentation with Scribble-level Supervision: Other approaches directly optimize segmentation with scribble-aware losses, using partial pixel supervision and normalized-cut or CRF consistency.These methods avoid requiring an extra pseudo-mask model in the described formulation.

4.4 Discussion

Inexact supervision can achieve state-of-the-art segmentation results comparable to dense supervision while reducing the need for fully dense training labels. However, CAM-based seeds remain problematic for small or hollow objects, motivating further end-to-end approaches.

  • 4.4 Discussion: State-of-the-art results with inexact supervision are comparable to results from full dense supervision while requiring less fully dense annotation.The section summarizes this pattern across image, instance, and panoptic segmentation tables.
  • 4.4 Discussion: CAM-based seed areas can be significantly inaccurate for small objects and objects with holes.These cases motivate exploring end-to-end pipelines that directly link dense predictions to inexact labels.

5 INCOMPLETE SUPERVISION

Incomplete supervision includes semi-supervised, domain-adaptive, and partially supervised segmentation, where methods bridge missing dense labels using pseudo-labeling, transfer, and auxiliary priors. Performance can approach fully supervised semantic segmentation, but distribution shifts cause larger degradation.

  • 5.1.1 Semi-supervised semantic segmentation: Semi-supervised semantic segmentation incorporates unlabeled images through self-training, while pseudo-label reliability is improved by confidence filtering and iterative refinement.
  • 5.1.1 Semi-supervised semantic segmentation: Siamese structures and contrastive learning enforce cross-view consistency among pseudo-labels for different views of the same image.
  • 5.1.2 Partially supervised instance segmentation: Partially supervised instance segmentation addresses novel categories with box annotations but missing pixel labels, commonly using detection-then-segmentation pipelines and auxiliary information.
  • 5.1.2 Partially supervised instance segmentation: Cross-pixel similarity supplies class-agnostic commonalities for novel categories, while stronger segmentation models can substantially improve performance.
  • 5.2 Domain-adaptive semantic segmentation: Domain-adaptive semantic segmentation adds domain-gap reduction to the semi-supervised pipeline through adversarial learning, domain mixing, or target-domain pseudo-mask improvement.
  • 5.1 Semi-supervised segmentation: 75.9% mIoU for state-of-the-art semi-supervised semantic segmentation was comparable to 76.0% for fully supervised segmentation.

6 INACCURATE SUPERVISION

Inaccurate supervision treats noisy segmentation annotations as a noisy-label problem. Methods seek robustness by preventing memorization of annotation errors and identifying unreliable labels.

  • Pseudo masks inevitably contain noise, making training from them essentially a noisy semantic segmentation problem.
  • Adaptive early stopping, multi-scale cross-view consistency, and uncertainty estimation are used to reduce overfitting to annotation errors or discover noisy labels.

7 CONCLUSION AND DISCUSSION

The survey finds substantial progress across label-efficient segmentation under diverse weak-supervision settings, while emphasizing that performance remains below the fully supervised upper bound. It identifies future challenges for extending these methods.

  • Label-efficient segmentation methods have achieved considerable progress across no, inexact, incomplete, and inaccurate supervision settings.
  • Existing methods remain below the fully supervised upper bound, motivating further discussion of unresolved challenges and future prospects.

1) The supervision gap

The survey identifies the supervision gap between dense predictions and incomplete labels as the main unresolved challenge in label-efficient image segmentation.

  • The main unresolved challenge is bridging the supervision gap between dense predictions and incomplete labels.

1) Zero-shot segmentation with text supervision

Text supervision and large-scale vision-language pretraining enable zero-shot segmentation for unseen categories, while transformer self-attention offers a route to pseudo-dense labels without supervision.

  • Large-scale text supervision enables models such as CLIP to bridge visual representations and text, supporting semantic segmentation of unseen categories.
  • Unsupervised self-attention maps from vision transformers such as DINO, BEiT, MAE, and iBoT may generate reliable pseudo-dense labels without supervision.
  • Existing label-efficient segmentation methods are mainly based on convolutional neural networks, leaving transformer-based approaches as an open direction.

APPENDIX A RESULT SUMMARY

The survey compiles benchmark results across segmentation tasks, weak-supervision types, semi-supervised settings, and domain adaptation scenarios.

  • Benchmark results cover semantic and instance segmentation under image-level, box-level, and scribble-level supervision.
  • Semi-supervised semantic segmentation results report mIoU across labeled-data proportions from 1/2 to 1/16 on PASCAL VOC validation.
  • The benchmark summary includes domain-adaptive semantic segmentation evaluated by mIoU from GTA5 as source to Cityscapes as target.
  • For box-supervised instance segmentation, the summary records training schedules and mask-head depth, noting that heavier mask heads generally improve performance.
Loading 2207.01223v2…