PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple Does It: Weakly Supervised Instance and Semantic Segmentation

Anna Khoreva, Rodrigo Benenson, Jan Hosang, Matthias Hein, Bernt Schiele

arXiv:1603.07485v2cs.CV

TL;DR

Semantic labelling and instance segmentation require costly pixel-wise annotations, motivating learning from cheaper bounding boxes. The paper designs input labels from boxes using recursive denoising and box-driven segmentation while leaving segmentation training unchanged, reaching roughly 95% of fully supervised quality.

  • Problem

    Pixel-wise annotations for semantic labelling and instance segmentation are expensive, whereas bounding box annotations are substantially cheaper.

  • Method

    The approach treats weak supervision as input-label noise and constructs training labels from bounding boxes using recursive training and GrabCut-like segmentation techniques without modifying the segmentation training procedure.

  • Results

    ∼95% of the quality of the fully supervised model is reached for semantic labelling and instance segmentation using bounding box annotations.

  • Takeaways & Limitations

    Carefully designed box-derived labels can support strong weakly supervised pixel-labelling performance while retaining the standard segmentation training procedure.

  • Takeaways & Limitations

    The current approach treats each annotated box individually and leaves exploring cosegmentation and weaker supervision to future work.

Abstract

from arXiv · show

Semantic labelling and instance segmentation are two tasks that require particularly costly annotations. Starting from weak supervision in the form of bounding box detection annotations, we propose a new approach that does not require modification of the segmentation training procedure. We show that when carefully designing the input labels from given bounding boxes, even a single round of training is enough to improve over previously reported weakly supervised results. Overall, our weak supervision approach reaches ~95% of the quality of the fully supervised model, both for semantic labelling and instance segmentation.

1. Introduction

The paper addresses the high annotation cost of semantic labelling and instance segmentation by learning from bounding boxes. Carefully designed input labels and recursive denoising achieve results close to full supervision without changing segmentation training.

  • Pixel-wise annotations for semantic labelling and instance segmentation are expensive, while bounding boxes require roughly 15× less annotation time.The paper investigates how much the performance gap between box-only and pixel-wise supervision can be closed.
  • ∼95% of the accuracy achievable with full supervision is reached using only bounding box annotations on the same training set.The experiments focus on the 20 Pascal classes.
  • The method reports top weakly supervised semantic-labelling results and, to the authors’ knowledge, the first weakly supervised instance-segmentation results.
  • The approach treats weak supervision as input-label noise and uses recursive training to denoise labels with prior-round convnet predictions.It also uses box-guided classic computer-vision segmentation techniques as supervision.
  • GrabCut-like algorithms generate training labels from bounding boxes without modifying the segmentation convnet training procedure.

2. Related work

Prior work established convnets and several weaker annotation regimes for semantic labelling, while instance segmentation commonly relied on proposals or box-driven segmentation. This paper focuses on learning both tasks from bounding boxes.

  • Semantic labelling: DeepLab is used as the reference semantic-labelling network because convnets have shown strong performance on Pascal VOC12.
  • Weakly supervised semantic labelling: Weakly supervised semantic-labelling research has used image labels, points, scribbles, and bounding boxes, with this work focusing on box-level object annotations.
  • Instance segmentation: Instance segmentation groups pixels by object instance rather than merely by object class.
  • Instance segmentation: Object proposals and GrabCut variants can produce instance segments from bounding boxes, while newer systems train convnets for instance segmentation.

3. From boxes to semantic labels

The method converts bounding boxes into noisy but informative semantic labels using object priors, recursive refinement, and box-driven segmentation. It preserves the standard segmentation training procedure while improving label quality through preprocessing and agreement cues.

  • Label cues: Bounding boxes provide background, object-extent, and objectness cues for constructing semantic training labels.Pixels outside exhaustive boxes are background; shape, area, continuity, and boundary priors supply additional information.
  • Box baselines: The naive baseline labels every pixel inside each box with its class and assigns uncovered pixels to background.For overlapping boxes, the smaller box is assumed to be in front.
  • Box baselines: Recursive training feeds convnet predictions from one round into the next, producing labels that capture object shape better than rectangles alone.
  • Box baselines: Between recursive rounds, labels are denoised by resetting outside-box pixels, enforcing a minimum area, and applying DenseCRF boundary filtering.These stages use background, extent, and object-boundary cues.
  • Adding objectness: GrabCut+ replaces RGB differences with HED boundaries, and GrabCut+i trades recall for precision using perturbed outputs and ignore labels.
  • Adding objectness: M∩G+ marks pixels as foreground only when MCG and GrabCut+ agree, using their complementary information to trade recall for precision.
  • Comparison: The approach keeps the training procedure unmodified, unlike BoxSup, and uses ignore regions to control the precision–recall trade-off.

4. Semantic labelling results

The semantic-labelling experiments evaluate box-derived supervision on Pascal VOC12 and show that carefully processed labels substantially narrow the gap to full supervision. M ∩G+ reaches 65.7 versus 69.1 for the fully supervised model, or about 95% of its quality, without changing the training procedure.

  • Experimental setup: The experiments evaluate semantic labelling on Pascal VOC12, using mIoU across 21 classes and, in some settings, additional COCO training images.The validation set guides design choices, while final results are also reported on the test set.
  • Box baselines: Naive recursive training from filled rectangles performs poorly, whereas three post-processing stages substantially improve the Box baseline.Adding ignore regions inside rectangles provides a clear gain and by itself reaches state-of-the-art results.
  • Main results: 65.7 mIoU for M ∩G+ approaches the 69.1 fully supervised DeepLabours result on Pascal VOC12 bounding-box supervision.Table 1 reports Box 61.2, Boxi 62.7, MCG 62.6, GrabCut+ 63.4, GrabCut+i 64.3, M ∩G+ 65.7, and fully supervised DeepLabours 69.1.
  • Main results: M ∩G+ outperforms MCG and GrabCut+ alone, while ignore regions systematically trade recall for precision.The method reaches 95% of fully supervised training quality without modifying the segmentation training procedure.
  • Additional results: Adding 10% of Pascal VOC12 semantic labels changes performance only from 65.7 to 65.8, indicating high quality in the generated M ∩G+ inputs.Using ground-truth Pascal annotations together with COCO bounding boxes yields a 2.5-point gain, from 69.1 to 71.6.
  • Additional results: With DeepLabv2-ResNet101, M ∩G+ reaches 93% or 95% of full supervision using VOC12 or VOC12+COCO training data, respectively.Weak supervision with COCO reaches similar quality to full supervision trained with VOC12 only.

5. From boxes to instance segmentation

The paper extends its box-derived supervision approach to weakly supervised instance segmentation, motivated by the cost of per-instance pixel annotations. It presents this task as a new experimental setting and generates instance masks from individual object boxes.

  • Contribution: The paper reports the first experiments, to its knowledge, on weakly supervised instance segmentation.This task complements the semantic-labelling experiments.
  • Motivation: Weakly supervised instance segmentation addresses the need for richer object outputs than bounding boxes while avoiding expensive per-instance pixel annotations.Such annotations distinguish between object instances and are more detailed and costly than semantic-labelling annotations.
  • Box-to-mask supervision: Each training segment is generated from an individual object bounding box, with GrabCut+ producing foreground-versus-background masks.Each segment represents a different object instance and can directly supervise an instance-segmentation convnet.

6. Instance segmentation results

The instance-segmentation experiments train DeepMask and DeepLabBOX from GrabCut+ masks generated from bounding boxes. Both weakly supervised networks approach fully supervised quality, with DeepLabBOX outperforming the considered non-learned baselines.

  • Experimental setup: The pipeline applies Fast-RCNN detections and estimates a foreground segment for each detection using baselines or instance-segmentation convnets.The experiments use a hyper-columns system and evaluate foreground estimation after detection post-NMS.
  • Models: DeepLabBOX adds a binary bounding-box channel to RGB input so the network segments the instance corresponding to the supplied box.The box guides the network toward the object of interest rather than all objects in the scene.
  • Training and evaluation: DeepMask and DeepLabBOX are trained for one round from GrabCut+ masks using either VOC12 or VOC12+COCO data, without CRF post-processing.Performance is reported using mAPr at IoU thresholds 0.5 and 0.75, together with ABO metrics.
  • Results: GrabCut+ is the strongest training-free baseline and is comparable to DeepMask, while DeepLabBOX outperforms both.The baseline set includes boxes, ellipses, MCG proposals, GrabCut, and GrabCut+.
  • Results: Approximately 95% of fully supervised quality is reached on both mAPr 0.5 and ABO using DeepMask and DeepLabBOX with VOC12 or VOC12+COCO.The result holds across the two evaluated convnets and both training-data regimes.

7. Conclusion

The paper shows that carefully designed labels from bounding boxes can train pixel-labelling convnets effectively without changing the segmentation training procedure. Weakly supervised semantic labelling and instance segmentation reach about 95% of fully supervised quality, while the current approach treats each annotated box individually.

  • ∼95% quality of the same network trained on ground-truth segmentation annotations is reached for semantic labelling.Using extra COCO bounding-box data can match full-supervision results.
  • ∼95% of fully supervised quality is reached for weakly supervised instance segmentation.The paper reports the first results for this weakly supervised task.
  • The approach reaches state-of-the-art performance without modifying the segmentation network training procedure.It generates input labels using box-driven segmentation techniques and balances accuracy with recall in noisy training segments.
  • The current approach treats each annotated box individually and leaves cosegmentation and weaker supervision for future work.These are explicitly identified as directions beyond the current scope.

Supplementary material

The supplementary material adds quantitative, qualitative, and methodological analyses covering recursive training, related training differences, GrabCut variants, and semantic and instance segmentation results.

  • Section B analyzes how post-processing stages contribute during recursive training using Figure S1.
  • Section C discusses training differences between the proposed approach and related work.
  • Section D compares different GrabCut-like methods on Pascal VOC12 boxes.
  • Sections E and G provide weakly supervised semantic-segmentation visualizations and qualitative Pascal VOC12 results.
  • Section F reports detailed per-class semantic-labelling performance, while Section H shows qualitative instance-segmentation results.

B. Recursive training with boxes

Naive recursive training from box-sized foreground labels degrades segmentation quality, but box constraints, outlier handling, and CRF filtering substantially improve the outputs. With these cues, recursive training reaches about 61% mIoU, near more sophisticated methods at about 62% mIoU, while box-guided labels preserve an unmodified training procedure.

  • Recursive training design: Naive recursive training directly using full bounding boxes as foreground labels causes significant degradation in segmentation output quality.
  • Recursive training design: Box enforcing resets pixels outside annotations to background, while outliers reset restores boxes when predicted segments are too small.The outlier condition is illustrated as IoU< 50%, enforcing a minimal segment area.
  • Results: Adding box enforcing and outliers reset produces a dramatic quality improvement after the first recursive-training round.These constraints already improve over previous work using rectangles-only input.
  • Results: Around 61% mIoU is reached after adding CRF filtering, compared with around 62% mIoU for more sophisticated techniques.Performance grows steadily across rounds before stabilizing near 61% mIoU.
  • Results: Recursive training becomes robust to input noise when outputs are denoised between rounds using bounding-box constraints and object priors.
  • Comparison with related work: Unlike BoxSup and WSSL, the approach keeps the segmentation training procedure unmodified and instead carefully generates input labels from bounding boxes.BoxSup modifies training to denoise intermediate outputs, whereas this approach generates labels and uses ignore regions.
  • Box-guided segmentation: The GrabCut-like evaluation covers 3.4k object windows with diverse object sizes, non-salient objects, and occlusions, making it more challenging than usual scenarios.
  • Box-guided segmentation: GrabCut+ replaces RGB colour-difference pairwise terms with HED boundary probabilities and improves GrabCut by approximately 2 points.DenseCut and KGrabCut did not provide significant gains.

F. Detailed test set results for semantic labelling

Detailed Pascal VOC12 test-set results show that weakly supervised semantic labelling achieves about 95% of full-supervision quality across classes, with qualitative examples demonstrating high-quality segmentation from detection boxes alone.

  • Table S2 reports per-class semantic-labelling results on the Pascal VOC12 test set for the methods evaluated in the main paper.
  • ∼95% quality of full supervision is achieved on average across all classes with VOC12-only or VOC12+COCO training.
  • Figure S3 presents qualitative Pascal VOC12 semantic-labelling results using only detection bounding-box annotations.The examples show that high-quality segmentation can be achieved from these annotations alone.

H. Qualitative results for instance segmentations

Supplementary qualitative results compare weakly supervised instance-segmentation outputs across DeepMask, DeepLabBOX, and box-guided methods. The proposed weakly supervised models produce competitive results, with DeepMask yielding higher-quality outputs than box-guided techniques.

  • Qualitative comparisons: Figure S4 provides additional qualitative results for weakly supervised DeepMask and DeepLabBOX instance segmentation.The figure complements Figure 6 from the main paper.
  • Qualitative comparisons: DeepMask achieves competitive performance with fully supervised results on qualitative instance-segmentation examples.The examples are shown on VOC12, with DeepMask trained using Pascal VOC12 and COCO supervision.
  • Qualitative comparisons: DeepMask produces higher-quality outputs than box-guided segmentation techniques in the supplementary examples.The comparison includes DeepMask and DeepLabBOX outputs, with Fast-RCNN proposals used to select segments with the best overlap with ground-truth masks.
  • Qualitative comparisons: DeepLabBOX provides results similar to those reported in Table 4 of the main paper.The supplementary figure presents additional qualitative examples of the model.
Loading 1603.07485v2…