Source-linked AI summary
BoxInst: High-Performance Instance Segmentation with Box Annotations
Zhi Tian, Chunhua Shen, Xinlong Wang, Hao Chen
TL;DR
BoxInst addresses the costly requirement for pixel-level mask annotations by learning instance masks from bounding boxes alone. It redesigns CondInst’s mask loss with projection and pairwise-affinity terms without modifying the segmentation network, and reports strong results on COCO and Pascal VOC, including 33.2% mask AP on COCO with no mask annotations. The method narrows the gap between box-supervised and fully supervised instance segmentation, while its projection assumption may fail for disjointed masks.
Problem
Pixel-level mask annotations are notoriously time-consuming compared with bounding-box annotations, limiting box-supervised instance segmentation.
Method
BoxInst replaces CondInst’s pixel-wise mask loss with projection and pairwise-affinity losses that supervise masks using box annotations and proximal-pixel color similarity.
Results
33.2% mask AP is achieved on COCO with a ResNet-101 backbone and 3× training schedule without using mask annotations.
Takeaways & Limitations
Results on COCO and Pascal VOC significantly improve the state of the art and narrow the performance gap between weakly and fully supervised instance segmentation.
Takeaways & Limitations
The projection correspondence between an instance mask and its box may not hold when the mask consists of multiple disjointed regions.
Abstract
from arXiv · showhide
We present a high-performance method that can achieve mask-level instance segmentation with only bounding-box annotations for training. While this setting has been studied in the literature, here we show significantly stronger performance with a simple design (e.g., dramatically improving previous best reported mask AP of 21.1% in Hsu et al. (2019) to 31.6% on the COCO dataset). Our core idea is to redesign the loss of learning masks in instance segmentation, with no modification to the segmentation network itself. The new loss functions can supervise the mask training without relying on mask annotations. This is made possible with two loss terms, namely, 1) a surrogate term that minimizes the discrepancy between the projections of the ground-truth box and the predicted mask; 2) a pairwise loss that can exploit the prior that proximal pixels with similar colors are very likely to have the same category label. Experiments demonstrate that the redesigned mask loss can yield surprisingly high-quality instance masks with only box annotations. For example, without using any mask annotations, with a ResNet-101 backbone and 3x training schedule, we achieve 33.2% mask AP on COCO test-dev split (vs. 39.1% of the fully supervised counterpart). Our excellent experiment results on COCO and Pascal VOC indicate that our method dramatically narrows the performance gap between weakly and fully supervised instance segmentation. Code is available at: https://git.io/AdelaiDet
1. Introduction
BoxInst targets the annotation cost that limits instance segmentation by learning masks from bounding boxes alone. It replaces CondInst’s pixel-wise mask losses with projection and pairwise-affinity losses, achieving strong results without changing the segmentation network.
- Motivation: Mask annotations are substantially more time-consuming than box annotations, motivating instance segmentation trained with boxes only.The paper notes that accurate masks can benefit downstream tasks such as robotic grasping, but pixel-level annotation is the main obstacle.
- Related work: Prior box-supervised methods often rely on region proposals, slow or difficult-to-parallelize training, iterative refinement, and extra hyperparameters.Examples include BoxSup and Box2Seg, which use MCG or GrabCut-generated proposals.
- Method: BoxInst replaces CondInst’s original pixel-wise mask losses with projection and pairwise-affinity terms while leaving the network model unchanged.The projection term matches predicted-mask and ground-truth-box projections; the pairwise term uses label similarity among proximal pixels.
- Caveat: The projection-box relationship may fail when an instance mask contains multiple disjointed regions.This is stated as a qualification to the claim that ground-truth masks and boxes share the same projections.
- Results: 33.2% mask AP is achieved on COCO with a ResNet-101 backbone and 3× schedule using no mask annotations.This exceeds YOLACT’s 31.2% AP and PolarMask’s 32.1% AP under the cited comparable settings.
2. Approach
BoxInst builds on CondInst by replacing pixel-wise mask supervision with projection and pairwise affinity losses, enabling mask learning from box annotations. The projection term aligns predicted-mask projections with the ground-truth box, while color-based pairwise supervision exploits likely label agreement between similar neighboring pixels.
- CondInst: CondInst uses instance-aware dynamically generated mask heads to produce full-image instance masks without RoI operations.The mask head predicts class-agnostic masks, while the detector’s classification branch determines each instance’s category.
- Projection loss: BoxInst represents each ground-truth box as a binary mask and supervises horizontal and vertical projections of predicted and box masks.The projections are one-dimensional segmentation masks obtained along the x- and y-axes, and the operation can be implemented with axis-wise maxima.
- Projection loss: The projection loss uses Dice loss to compare predicted-mask projections with ground-truth box projections and applies the loss across training instances.The operations are differentiable, and the authors report decent instance segmentation results from this term without mask annotations.
- Pairwise affinity loss: Pairwise affinity loss labels neighboring pixel pairs by whether their ground-truth categories match, then trains the network to predict agreement between endpoint mask probabilities.Edges connect each pixel to K × K − 1 neighbors, with an optional dilation trick.
- Learning without mask annotations: Without mask annotations, BoxInst treats sufficiently similar-color pixel pairs as confidently positive and discards edges whose labels remain ambiguous.Color similarity is computed in LAB space using a hyperparameter θ equal to 2 in this work; only edges with Se ≥ τ contribute to the pairwise loss.
- Learning without mask annotations: At τ = 0.1, more than 98% of selected edges are positive, while increasing τ improves this proportion but reduces the number of supervised edges.The method therefore trades off label confidence against supervision coverage; edges outside the box also help eliminate undesirable solutions.
3. Experiments
Experiments evaluate BoxInst’s loss design across COCO, Pascal VOC, fully supervised, box-supervised, and semi-supervised settings. Results show competitive mask quality using only box annotations, with performance affected by pairwise-loss hyperparameters and loss-term choices.
- Experimental Setup: BoxInst uses COCO train2017 and Pascal VOC 2012 training data, generally with only box annotations, and reports COCO results on test-dev.COCO ablations use val2017; Pascal VOC evaluation uses the 2012 validation split.
- Hyperparameter Ablations: 30.7 mask AP is achieved on COCO val2017 with box annotations at τ = 0.1, compared with 35.4% fully-supervised mask AP and 10.6% box-mask AP.The ablation varies the color-similarity threshold and pixel-neighborhood configuration.
- Fully Supervised Setting: 35.4% versus 35.6% mask AP shows that the proposed mask loss performs similarly to CondInst’s original pixelwise loss under full supervision.The comparison keeps other training settings unchanged.
- Hyperparameter Ablations: Increasing the color threshold to 0.2 lowers performance to 30.6% mask AP, potentially because fewer positive edges receive pairwise supervision.The reported explanation is tied to the reduced number of supervised positive edges.
- Hyperparameter Ablations: 30.5% mask AP from 5 × 5 neighborhoods improves over 29.7% from 3 × 3, while dilation on 3 × 3 reaches 30.7% without added computational overhead.Further dilation of 5 × 5 patches does not improve performance because similar colors may not imply the same label at larger distances.
- Benchmark Results: 33.2% mask AP is achieved on COCO test-dev with ResNet-101 and a 3× schedule using no mask annotations, while BoxInst reaches 36.5% mask AP on Pascal VOC.On COCO, BoxInst improves the previous best reported result from 21.1% to 31.6% mask AP; on Pascal VOC it exceeds GrabCut’s 19.0% mask AP.
- Semi-supervised Instance Segmentation: 30.9% mask AP on unseen COCO classes is obtained with projection and pairwise losses plus partial mask annotations, improving over 19.9% without the proposed losses.Using only the projection term reaches 29.7%, while the box-only setting reaches 29.6%.
4. Conclusions
BoxInst achieves high-quality instance segmentation using only box annotations by replacing pixelwise mask loss with projection and pairwise affinity losses, with strong results on COCO and Pascal VOC.
- BoxInst replaces the original pixelwise mask loss with projection and pairwise affinity losses for box-supervised instance segmentation.