Source-linked AI summary
BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation
Jifeng Dai, Kaiming He, Jian Sun
TL;DR
Semantic segmentation depends on expensive pixel-level masks, limiting access to the large training sets deep networks benefit from. BoxSup iteratively generates region-proposal masks and trains convolutional networks with them, achieving competitive box-only performance and state-of-the-art benchmark results when leveraging many bounding boxes.
Problem
Pixel-level segmentation masks are expensive to annotate, while deep networks generally benefit from more training data.
Method
BoxSup alternates between selecting or generating candidate masks from bounding boxes and training a convolutional network, using network feedback to improve the masks.
Results
BoxSup shows graceful degradation with boxes only, reaches 66.2 when 9/10 masks are replaced by boxes, and achieves state-of-the-art results with large-scale box annotations.
Takeaways & Limitations
Large-scale bounding boxes improve learned object-recognition features and enable strong semantic-segmentation performance with weaker supervision.
Abstract
from arXiv · showhide
Recent leading approaches to semantic segmentation rely on deep convolutional networks trained with human-annotated, pixel-level segmentation masks. Such pixel-accurate supervision demands expensive labeling effort and limits the performance of deep networks that usually benefit from more training data. In this paper, we propose a method that achieves competitive accuracy but only requires easily obtained bounding box annotations. The basic idea is to iterate between automatically generating region proposals and training convolutional networks. These two steps gradually recover segmentation masks for improving the networks, and vise versa. Our method, called BoxSup, produces competitive results supervised by boxes only, on par with strong baselines fully supervised by masks under the same setting. By leveraging a large amount of bounding boxes, BoxSup further unleashes the power of deep convolutional networks and yields state-of-the-art results on PASCAL VOC 2012 and PASCAL-CONTEXT.
1. Introduction
BoxSup addresses the cost and scarcity of pixel-level masks by using bounding boxes to supervise CNN-based semantic segmentation. It iteratively improves candidate masks and the network, achieving competitive accuracy and benefiting from large-scale box annotations.
- Motivation: Pixel-level mask annotation is over 15 times more labor-intensive than spotting object locations, making bounding boxes a more economical supervision source.Mask labeling also requires specially trained annotators, while many box-level annotations already exist.
- Motivation: BoxSup asks whether coarse bounding-box annotations can retain comparably good segmentation results without pixel-level masks.The motivation is supported by leading methods that do not explicitly exploit fine-resolution mask details.
- Method: BoxSup alternates between generating candidate segmentation masks with region proposals and training a convolutional network on those approximate masks.Network updates provide feedback that improves the estimated masks, and the process is iterated.
- Results: Box-supervised training shows graceful degradation relative to mask-supervised training, while replacing 9/10 mask annotations with boxes yields comparable accuracy.The method therefore reduces reliance on pixel-level masks while retaining competitive performance.
- Results: BoxSup achieves state-of-the-art results on PASCAL VOC 2012 and PASCAL-CONTEXT when using limited masks together with large-scale bounding boxes.The evaluation is conducted on PASCAL segmentation benchmarks.
- Results: Large-scale bounding boxes primarily improve object recognition accuracy inside objects, with boundary improvements playing a secondary role.The analysis attributes this to additional object instances improving learned CNN feature quality.
2. Related Work
Prior work established that adding task-specific data can improve visual recognition, but existing semantic-segmentation studies using boxes had not used box annotations to supervise deep convolutional networks.
- Transfer learning: Deep networks are commonly pretrained on large classification datasets and fine-tuned for tasks with fewer labeled examples, including semantic segmentation.This transfer-learning strategy is used across several visual recognition tasks.
- Training-set augmentation: Augmenting object-detection training data with additional VOC images or video-derived tracking results improves detection accuracy.These examples motivate exploiting additional labeled or weakly labeled data for task-specific training.
- Bounding-box supervision: Earlier semantic-segmentation work investigated using bounding boxes instead of masks, but did not use box-level annotations to supervise deep convolutional networks.BoxSup targets this gap directly.
3. Baseline
The baseline is a fully convolutional network trained by per-pixel regression to segmentation masks, optionally refined with a CRF. Its objective directly requires pixel-level labels, so it cannot be used unchanged with boxes alone.
- Baseline architecture: BoxSup can be applied to CNN-based mask-supervised segmentation methods, and the paper uses an FCN implementation refined by a CRF as its baseline.The baseline represents the mask-supervised comparison system.
- Training objective: FCN formulates semantic segmentation as per-pixel regression from network outputs to ground-truth semantic labels.The per-pixel loss is optimized by back-propagation and stochastic gradient descent, followed by CRF post-processing.
- Training objective: In the objective, p indexes a pixel, l(p) is its ground-truth label, and Xθ(p) is the FCN's per-pixel labeling.The loss e(Xθ(p), l(p)) compares the prediction with the ground-truth label.
- Limitation: The baseline objective demands pixel-level segmentation masks and therefore is not directly applicable when only bounding-box annotations are available.This motivates estimating segmentation masks from boxes before training.
4. Approach
BoxSup alternates between selecting candidate segmentation masks from bounding boxes and training a convolutional network on those estimated masks. Iterative updates jointly improve the candidate labels and network supervision while preserving efficient inference.
- 4.1. Unsupervised Segmentation for Supervised Training: Region proposals provide diverse candidate masks that are used as approximate supervision for convolutional-network training.The proposals offer high recall and variance, supporting candidate selection and data augmentation.
- 4.2. Formulation: At inference, the trained fully convolutional network directly produces pixel-wise predictions without region proposals.Thus, proposal generation is confined to network training rather than test-time prediction.
- 4.2. Formulation: Candidate selection minimizes a combined overlap and regression cost, balancing box agreement with agreement to network predictions.The formulation jointly optimizes network parameters and candidate labels, with λ = 3 weighting the regression term.
- 4.3. Training Algorithm: For each bounding box, the algorithm assigns one candidate segment its object label and assigns all other pixels to background.The selected candidate has the smallest combined cost among candidates under the stated implementation.
- 4.3. Training Algorithm: Randomly sampling among top-ranked candidates increases sample variance and reduces repeated use of similar segments.This strategy is introduced because winner-takes-all selection can become trapped in poor local optima.
- 4.3. Training Algorithm: The method alternates between updating candidate-segment labels and optimizing network parameters with stochastic gradient descent.Each iteration trains for one epoch, then updates segment labels across all images.
5. Experiments
The experiments use a pretrained VGG-16 model and evaluate supervision comparisons on PASCAL VOC 2012 validation. The supplied passages identify the benchmark protocol and comparison table but do not report its results.
- 5. Experiments: Experiments use the publicly released VGG-16 model pretrained on ImageNet, matching the competitors compared in the paper.
- 5. Experiments: Table 1 compares different supervision strategies on PASCAL VOC 2012 validation.
5.1. Experiments on PASCAL VOC 2012
BoxSup is evaluated on PASCAL VOC 2012 using bounding boxes, masks, mixed supervision, and additional region proposals. Across these experiments, iterative mask updates enable competitive accuracy, while large-scale boxes improve recognition and overall segmentation performance.
- Evaluation setup: The VOC 2012 evaluation uses mean IoU on 10,582 augmented training images, with 1,449 validation and 1,456 test images.The benchmark covers 20 object categories and follows the “comp6” evaluation protocol.
- Supervision strategies: 63.8 is the mask-only validation score, while box-only supervision reaches 62.0, a 1.8% degradation.The box-only result uses bounding boxes for all 10,582 training samples.
- Supervision strategies: 63.5 is achieved when 9/10 mask annotations are replaced with bounding boxes, matching the strongly supervised baseline.The semi-supervised setting retains masks for the original 1,464 training images and uses boxes for the remaining 9k images.
- Additional box data: 68.2 with VOC+COCO boxes exceeds the 63.8 VOC-only mask-supervised result by 4.4%, showing the benefit of extra large-scale boxes.COCO contributes 123k images, with tight bounding boxes replacing its segmentation masks in the semi-supervised experiment.
- Error analysis: Error analysis shows that extra COCO boxes improve interior mean IoU considerably more than boundary mean IoU, indicating stronger recognition gains than boundary gains.The authors attribute this pattern to improved feature quality from many additional object instances; boundary improvements are secondary.
- Test-set results: On the VOC 2012 test set, box supervision reaches 64.6, semi-supervision reaches 66.2, and adding COCO boxes raises the result to 71.0.The box-supervised result exceeds WSSL’s 60.4 under the same setting, while semi-supervision replaces 9/10 mask annotations.
5.2. Experiments on PASCAL-CONTEXT
On PASCAL-CONTEXT, BoxSup transfers a model trained with COCO bounding boxes to whole-scene semantic segmentation and improves the FCN+CRF baseline. The evaluation uses 59 frequent categories plus background and reports mean IoU.
- Evaluation: PASCAL-CONTEXT provides semantic labels for the whole scene, including both objects and stuff such as grass, sky, and water.The experiments follow the established protocol for the 59-category setting.
- Evaluation: The task evaluates semantic segmentation across the 59 most frequent PASCAL-CONTEXT categories plus background.Mean IoU is measured on training and validation sets containing 4,998 and 5,105 images, respectively.
- Method: BoxSup is pretrained using COCO boxes from all 80 object categories before fine-tuning on PASCAL-CONTEXT.The model replaces its 81-way output layer with a 60-way layer for the 59 categories plus background, and uses CRF post-processing.
- Results: 40.5 mean IoU with COCO-box pretraining improves the FCN+CRF mask-supervised baseline from 35.7.The reported 4.8% gain is attributed solely to bounding box annotations in COCO improving network training.
6. Conclusion
BoxSup harnesses bounding boxes to train deep networks for semantic segmentation, achieving state-of-the-art results with 133k boxes and 10k masks. The error analysis suggests that object-recognition failures constrain segmentation accuracy and may benefit from large-scale data.
- BoxSup effectively harnesses bounding box annotations to train deep networks for semantic segmentation.
- 133k bounding boxes and 10k masks yield state-of-the-art results with BoxSup.
- Error analysis suggests that failed object recognition hampers semantic segmentation accuracy and that large-scale data may help.