Coarse-to-fine Semantic Segmentation from Image-level Labels

I. INTRODUCTION

• Pixel-wise semantic annotation is time-consuming and expensive, while object-level weak labels can still require laborious accurate boxes or scribbles.The paper motivates image-level labels as cheaper to obtain.Source: §I · p. 1
• Training images need only one category label, yet the framework can generate semantic masks for images containing objects from multiple categories.The paper contrasts this with approaches that use multiple category labels per image.Source: §I · pp. 1–2
• A coarse-to-fine pipeline generates unsupervised foreground masks, enhances them with a graph model, and recursively refines them into pixel-wise semantic masks.The recursive network is trained using enhanced masks, images, and their category labels.Source: §I · p. 2
• The framework reports performance comparable to image-level label-based state-of-the-art methods on PASCAL VOC and to supervised methods on the Internet Object dataset.It is trained on a subset of ImageNet for the PASCAL VOC comparison and can be extended to foreground object segmentation.Source: §I · p. 2

II. RELATED WORK

• The related work classifies segmentation methods as fully supervised pixel-wise, weakly supervised object-level, or weakly supervised image-level approaches.The paper notes that pixel-label methods have the best performance when trained with accurate pixel labels.Source: §II · p. 2
• Object-level approaches use boxes, spots, or scribbles; cited examples include box-trained models and a network trained with scribbles.The paper describes bounding boxes as relatively cheap yet quite accurate annotations.Source: §II · pp. 2–3
• Image-level approaches include transferring classification models, using web videos or GAN-generated images, and generating localization maps or pseudo-annotations.The review also includes approaches using auxiliary annotations, boundary constraints, or stronger supervision.Source: §II · p. 3
• Unlike image-level methods requiring multi-category information per image, the proposed framework uses one category label and reports comparable weakly supervised segmentation performance.It is also extended to foreground segmentation, with comparable performance to supervised methods on the Internet Object dataset.Source: §II · p. 3

A. Overview

• The overview combines coarse-mask generation, graph-based mask enhancement, and recursive training of a fully convolutional semantic segmentation network.Enhanced masks, input images, and category labels are used in the recursive training stage.Source: A. Overview · p. 3

B. Coarse Mask Generation

• An 8-layer CNN student network trained on large-scale video frames generates initial coarse masks through standard feedforward processing.The cited unsupervised segmentation network was trained with millions of unlabeled images.Source: B. Coarse Mask Generation · p. 4
• The initial masks are noisy, often contain holes, and can be misplaced relative to objects, so a graph model enhances them before semantic-network training.The paper says mask quality is essential to segmentation performance.Source: B. Coarse Mask Generation · p. 4

C. Coarse Mask Enhancement

• GrabCut is applied to enhance the initial masks before recursive semantic-segmentation training.It uses foreground and background color distributions in a graph-cut optimization; the resulting masks are described as more accurate and compact.Source: C. Coarse Mask Enhancement · p. 4

D. Recursive Mask Refinement

• The initial semantic training masks assign each image’s category to enhanced-mask foreground pixels and background to the remaining pixels.This relies on the stated assumption that enhanced masks cover most of the main object in ImageNet images.Source: D. Recursive Mask Refinement · p. 4
• DeepLab is trained recursively, using the previous round’s predicted masks as current inputs until the network converges.The process retains pixels matching the image category, sets other pixels to background, and applies GrabCut before continuing training.Source: D. Recursive Mask Refinement · pp. 4–5
• After training, the network segments the interested category, with image-level labels as the only annotation needed for semantic-model training.The student network used to generate initial masks also requires no human-made labels.Source: D. Recursive Mask Refinement · p. 5

E. Extend the Proposed Framework to Foreground Segmentation

• Replacing the recursive-refinement network with a foreground-segmentation network extends the framework to binary foreground object segmentation.The paper uses Feature Pyramid Network for this task because of its multiscale feature-fusion ability.Source: E. Extend the Proposed Framework to… · p. 5

IV. EXPERIMENTS

• Experiments train the framework on an ImageNet subset with only category labels, selecting images from 20 VOC-matched categories excluding background, and evaluate on PASCAL VOC using the same parameters.Source: §IV · p. 5, A. Datasets · p. 5

A. Datasets

• PASCAL VOC 2012 evaluation uses its validation and test splits, with test accuracy obtained through the evaluation server because test masks are unreleased.The benchmark has 1,464 training, 1,449 validation, and 1,456 test images.Source: A. Datasets · p. 5
• The MIT Object Discovery Dataset evaluation covers 2,488 Internet-search images of airplanes, cars, and horses with pixel-level labels.The paper compares performance on the same test set and a subtest set used by other weakly supervised methods.Source: A. Datasets · p. 5

B. Training Details for Recursive Mask Refinement

• Training runs for five rounds, updating masks at each round using recursively generated masks followed by GrabCut.Training uses about 9,000 images from 20 categories, with non-foreground regions treated as background.Source: B. Training Details for Recursive Mask… · p. 5
• Training uses mini-batch SGD with 12 images per batch, a 0.00025 learning rate, 0.9 momentum, and 0.0005 weight decay.Images are resized to 321×321 and randomly cropped to 256×256 for augmentation.Source: B. Training Details for Recursive Mask… · p. 5

C. Evaluation Metrics

• Semantic segmentation is evaluated by mean IoU across 21 VOC categories, while foreground segmentation uses IoU between predicted binary masks and ground truth.Validation results guide experiment design; final semantic-segmentation results are reported on the VOC test split.Source: C. Evaluation Metrics · p. 5

1) Impact of the mask quality.:

• Fig. 4 shows semantic masks for multiple categories in the last three rows, with different colors representing different categories.The figure presents examples from coarse to fine.Source: Figure 4 · p. 6
• 15.2% IoU improvement follows GrabCut enhancement over coarse masks; bounding-box masks are 2.7% higher than coarse masks, and refined masks are 2.7% higher than enhanced-mask training.The reported comparisons support mask quality as important and recursive refinement as beneficial.Source: 1) Impact of the mask quality · p. 6, Table I · p. 6

2) Effectiveness of recursive refinement.:

• Performance improves with each recursive training round and saturates after three or four rounds; GrabCut post-processing performs better than directly reusing masks.The passage attributes the direct-reuse disadvantage to error propagation.Source: 2) Effectiveness of recursive refinement · p. 7, Figure 5 · p. 7

3) Comparison with others.:

• 61.9% validation and 62.8% test performance are reported for the method using only image-level annotations, which outperforms most image-level methods.Fan et al. report 65.6% on the test split using stronger instance-level saliency supervision.Source: 3) Comparison with others · p. 7, Table II · p. 7
• On MIT Object Discovery, the proposed weakly supervised method achieves comparable performance to several state-of-the-art supervised methods.The table caption marks methods using human-labeled pixel-level annotations with an asterisk.Source: Table IV · p. 8
• Bird, Sheep, and Train each exceed 80% pixel accuracy, while Bicycle, Chair, and Plant have more classification errors.The per-class results are reported for PASCAL VOC validation and test sets.Source: 3) Comparison with others · p. 8, Table III · p. 8

• The recursive training process repeatedly refines the training masks with the semantic segmentation network.The paper characterizes recursive training as a mask-refinement process.Source: E. Qualitative Results · p. 8
• Examples show the final masks distinguishing birds from a chair and horses from persons, although each image has only one category label.Source: E. Qualitative Results · p. 8
• Despite noisy coarse masks, the network predicts final pixel-wise semantic masks and distinguishes pixels belonging to multiple categories.The qualitative PASCAL VOC results show this behavior even when only one category label is available per image.Source: E. Qualitative Results · p. 8

F. Foreground Segmentation Results

• The method achieves comparable performance to 14 existing state-of-the-art methods on the MIT Object Discovery test split despite training with noisy masks.The comparison spans joint segmentation, saliency detection, and object proposal methods, including methods trained with accurate human-labeled masks.Source: F. Foreground Segmentation Results · p. 8
• 5.28% higher than UnsupervisedSeg and 69.9%, only 2.06% below PixelObjectness, on MIT Object Discovery.PixelObjectness is trained with accurate human-annotated masks; UnsupervisedSeg uses masks generated by an unsupervised method.Source: F. Foreground Segmentation Results · p. 8

V. CONCLUSIONS

• Trained only on ImageNet, the framework achieves comparable PASCAL VOC performance to other image-level-label semantic segmentation methods.Source: §V · p. 8
• The confusion matrix indicates accurate pixel classification for most categories.Source: Figure 6 · p. 9
• The framework also achieves performance comparable to state-of-the-art supervised methods for foreground object segmentation.Source: §V · p. 9