PaperlayerView exact revision on arXiv

Source-linked AI summary

BBAM: Bounding Box Attribution Map for Weakly Supervised Semantic and Instance Segmentation

Jungbeom Lee, Jihun Yi, Chaehun Shin, Sungroh Yoon

arXiv:2103.08907v1cs.CV

TL;DR

Pixel-level segmentation masks are costly to annotate, while existing box-based weak supervision typically relies on low-level, class-agnostic mask generators. BBAM instead uses a trained object detector to find the smallest regions preserving its predictions, producing pseudo ground truth for semantic and instance segmentation. The method achieves state-of-the-art performance on PASCAL VOC and MS COCO weakly supervised segmentation benchmarks and is analyzed in detail.

  • Problem

    Pixel-level masks are laborious to annotate, and existing bounding-box methods typically use class-agnostic generators based on low-level image information.

  • Method

    BBAM uses attribution maps from a trained object detector to find the smallest image regions that preserve nearly the same detection result, then refines and thresholds them into pseudo masks.

  • Results

    The method achieves state-of-the-art performance on PASCAL VOC and MS COCO in weakly supervised semantic and instance segmentation.

  • Takeaways & Limitations

    Detector-derived BBAMs provide pixel-level localization for weakly supervised semantic and instance segmentation while drawing on detector-learned semantics.

Abstract

from arXiv · show

Weakly supervised segmentation methods using bounding box annotations focus on obtaining a pixel-level mask from each box containing an object. Existing methods typically depend on a class-agnostic mask generator, which operates on the low-level information intrinsic to an image. In this work, we utilize higher-level information from the behavior of a trained object detector, by seeking the smallest areas of the image from which the object detector produces almost the same result as it does from the whole image. These areas constitute a bounding-box attribution map (BBAM), which identifies the target object in its bounding box and thus serves as pseudo ground-truth for weakly supervised semantic and instance segmentation. This approach significantly outperforms recent comparable techniques on both the PASCAL VOC and MS COCO benchmarks in weakly supervised semantic and instance segmentation. In addition, we provide a detailed analysis of our method, offering deeper insight into the behavior of the BBAM.

1. Introduction

Pixel-level masks improve segmentation but are laborious to annotate, motivating weaker supervision such as bounding boxes and class labels. BBAM uses a trained detector to localize objects within boxes and generate pseudo ground truth for semantic and instance segmentation.

  • Pixel-level mask annotation takes about 4 minutes for an image containing an average of 2.8 objects.
  • Class labels take around 20 seconds but provide neither object locations nor separation between same-class instances.
  • Bounding boxes take about 38.1 seconds per image and provide information about individual objects and their locations.
  • Existing box-based methods search within boxes using class-agnostic mask generators based mainly on low-level image information.
  • BBAM finds the smallest image areas that preserve an object detector’s result, identifying the object at pixel level for pseudo-ground-truth training.
  • The method significantly outperforms previous weakly supervised segmentation methods on PASCAL VOC 2012 and MS COCO 2017.

2. Related Work

Weakly supervised segmentation with class labels or bounding boxes reduces annotation demands but faces localization limitations. Class-label methods expand incomplete CAM regions, while box-based methods generally rely on low-level mask proposals that ignore box semantics.

  • Weakly supervised approaches replace laborious pixel-level masks with less detailed annotations such as scribbles, bounding boxes, or class labels.
  • Learning with Class Labels: CAMs identify only the most discriminative object regions, so class-label methods expand them using erasure, cross-image information, seed growing, refinement, or multiscale context.
  • Learning with Class Labels: Class labels are inherently unhelpful for instance segmentation because they cannot separate different objects of the same class.
  • Learning with Bounding Boxes: Bounding-box methods usually create pseudo masks with MCG or GrabCut inside each box, sometimes using pixel proportions or iterative refinement.
  • Learning with Bounding Boxes: These box-based methods largely use low-level image information and ignore the semantics associated with bounding boxes.
  • Visual Explanation Methods: Most neural-network explanation studies concern classifiers, with few addressing other tasks and no prior attempts to explain object-detector predictions except one embedded approach.

3. Method

The method uses a trained two-stage object detector to construct BBAMs by finding minimal image regions that preserve detector predictions, then converts them into pseudo-ground-truth masks for segmentation.

  • 3.1. Revisiting Object Detectors: Two-stage detection uses an RPN for class-agnostic proposals, followed by classification and bounding-box regression heads for refinement.RoIAlign converts proposal features to a predefined fixed size before the heads process them.
  • 3.2. Bounding Box Attribution Map: BBAMs identify the smallest image regions that preserve an object detector’s predictions for a proposal.The detector is evaluated on perturbed images, and the mask is optimized with respect to the attribution variables.
  • 3.2. Bounding Box Attribution Map: Adaptive stride matches perturbation-unit size to object size, avoiding artifacts caused by fixed strides that are too coarse for small objects or too fine for large objects.The stride is selected from the ratio of predicted-box area to image area: small objects use small strides and large objects use large strides.
  • 3.3. Generating Pseudo Ground Truth: The pipeline trains an object detector, generates BBAM-based semantic and instance pseudo-ground-truth masks, and uses them to train segmentation networks.Multiple jittered proposals are combined, with positives selected by correct classification and predicted-box IoU greater than 0.8.
  • 3.3. Generating Pseudo Ground Truth: Positive-proposal BBAMs combine classification and box-head localization, are refined with CRFs, and are thresholded to form foreground masks.Pixels above θfg are foreground, pixels below θbg are background, and intermediate pixels are ignored during segmentation training.
  • 3.4. Training Segmentation Networks: For instance segmentation, Mask R-CNN uses seed growing so initially ignored pixels progressively participate in loss computation.The segmentation network is trained from the generated pseudo-labels rather than pixel-level annotations.

4. Experiments

Experiments evaluate BBAM-based weakly supervised segmentation on PASCAL VOC and MS COCO, including comparisons, refinement, head ablations, and parameter sensitivity. BBAM achieves strong instance and semantic segmentation performance across these settings.

  • Experimental setup: Experiments use PASCAL VOC and MS COCO, reporting mean intersection-over-union and average-precision metrics under established weakly supervised segmentation protocols.PASCAL VOC uses 10,582 augmented training images and 20 object classes; MS COCO uses 118K training images and 80 object classes.
  • Instance segmentation: 6.0% higher AP50 and AP70 values than the previous best bounding-box method are achieved on PASCAL VOC instance segmentation.The method reaches 92.2% of fully supervised Mask R-CNN performance in AP50 and 95.7% in ABO.
  • Instance segmentation: 6.7% higher AP75 than the previous best bounding-box method is achieved on MS COCO instance segmentation.The MS COCO setting contains more objects per image and greater instance-size diversity than PASCAL VOC.
  • Semantic segmentation: 73.7 mIoU is obtained on both PASCAL VOC validation and test images, outperforming methods using image-level tags or bounding boxes.This result uses vanilla DeepLab-v2 without network modifications or additional training techniques.
  • Semantic segmentation: BBAM reaches 96.7% of the fully supervised equivalent’s performance, compared with 88.4% for Box2Seg, when accounting for different segmentation backbones.The corresponding values are 73.7 versus 76.2 for BBAM and 76.4 versus 86.4 for Box2Seg.
  • Refinement analysis: MCG mask refinement is particularly effective for medium and large objects, while BBAM without MCG still outperforms Hsu et al. on PASCAL VOC and MS COCO.The experiments separately assess the contribution of MCG proposals to instance segmentation.
  • Ablation study: Combining the detector’s box and classification heads gives the best segmentation performance, indicating complementary contributions from the two heads.Using either head alone also produces competent performance.
  • Parameter sensitivity: Ignoring some pixels and applying seed growing improve AP values, while performance remains similar across a broad range of λ values.The λ analysis covers both semantic and instance segmentation.

5. Detailed Analysis of the BBAM

The analysis shows that BBAMs highlight object boundaries and discriminative parts, while combining box and classification heads improves attribution; BBAMs also remain robust to noisy boxes and adaptive stride choices.

  • Examples of BBAMs: BBAMs assign high values to object boundaries and discriminative parts in PASCAL VOC and MS COCO validation images.These regions are informative for conducting object detection.
  • Complementary operation of the box and cls heads: The box and classification heads focus on complementary regions, and optimizing one head increases the other head’s loss.This indicates that the discriminative area sufficient for one head is not sufficient for the other to maintain its prediction.
  • Label noise in object detection: BBAM is more robust than Hsu et al. to expanded or contracted box annotations with noise extended to ±20%.Hsu et al.’s bounding-box tightness priors are substantially affected by inaccurate coordinates.
  • Complementary operation of the box and cls heads: Using either the box head or classification head alone gives competent performance, but combining both heads achieves the best performance.The combined attribution uses the complementary behavior of the two heads.
  • Effectiveness of an adaptive stride s(a): An adaptive stride performs better across object sizes than fixed strides of 24 or 48.A stride of 24 is ineffective for large objects, whereas a stride of 48 is ineffective for small objects.
  • Comparison with gradient-based methods: Gradient-based attribution maps correlate poorly with ground-truth masks, partly because proposal-limited features can make essential outside-proposal pixel gradients vanish.The reported mean IoU between positive proposals and corresponding predictions is 0.56.

6. Conclusions

BBAM localizes target objects pixelwise by finding the smallest prediction-preserving region inside each bounding box. The method achieves state-of-the-art weakly supervised semantic and instance segmentation on PASCAL VOC and MS COCO.

  • 6. Conclusions: BBAM provides pixel-level localization by finding the smallest region that preserves an object detector’s predictions.The formulation is built on two-stage detectors, but the authors state that adapting it to one-stage detectors with box and classification heads is straightforward.
  • 6. Conclusions: BBAM achieves state-of-the-art performance in weakly supervised semantic and instance segmentation on PASCAL VOC and MS COCO.The authors also analyze BBAMs and compare them with other attribution methods.

A.1. Implementation details

The implementation regularizes the attribution mask with total variation, optimizes it by gradient descent using Adam, and specifies training, post-processing, and dataset-specific settings.

  • TV norm: Total variation regularization suppresses artifacts in the attribution mask, with λTV set to 10^-4 and β set to 3.The resulting mask has little dependency on λTV.
  • Mask optimization: The optimal mask M* is found by gradient descent with respect to M using the update Mt+1 = Mt − ξ∇MtLMt.The update is implemented with Adam, and ξ denotes the learning rate.
  • Optimization details for semantic segmentation: Semantic segmentation training uses batch size 8, 2.4 × 10^4 iterations, and learning rate 2 × 10^-4.These settings differ from the default configuration in the cited baseline.
  • Optimization details for instance segmentation: PASCAL VOC instance segmentation uses image sizes from 512 to 800, FPN anchor sizes [21, 42, 84, 168, 332], and learning rate 8 × 10^-3 for 2 × 10^4 iterations.The passage specifies adjustments for the dataset’s characteristics.
  • Post-processing of semantic and instance segmentation: CRFs are used as post-processing for both semantic and instance segmentation.

A.2. Additional Results

Additional figures provide per-class scores, BBAM examples from both datasets, and semantic and instance mask examples from the method and comparison methods.

  • Comparison of per-class mIoU scores: Table A1 reports per-class mIoU scores for the proposed method and recently produced methods.
  • More examples of BBAMs: Figures A1 and A2 show additional BBAMs for PASCAL VOC and MS COCO validation images obtained with Faster R-CNN.
  • Additional mask examples on semantic segmentation: Figure A3 compares predicted semantic masks from DSRG, Shen et al., FickleNet, Lee et al., and the proposed method.
  • More mask examples on instance segmentation: Figures A4 and A5 show additional predicted instance masks for PASCAL VOC and MS COCO validation images.Figure A4 includes IRNet, Hsu et al., and the proposed method, while Figure A5 shows the proposed method on MS COCO.
Loading 2103.08907v1…