Source-linked AI summary
Learning to Segment Every Thing
Ronghang Hu, Piotr Dollár, Kaiming He, Trevor Darrell, Ross Girshick
TL;DR
Instance segmentation is difficult to scale because existing methods require mask annotations for every category, while masks are costly and available for relatively few classes. The paper introduces partially supervised training and a learned transfer function that predicts mask parameters from box-detection parameters, improving mask generalization and producing a qualitative 3000-category Visual Genome model. The authors report up to a 40% relative increase in mask AP on COCO categories without training masks.
Problem
Instance segmentation methods typically require masks for all categories, restricting practical systems to around 100 well-annotated classes because new mask annotations are expensive.
Method
The method combines mask supervision for categories in A with box supervision for categories in B and learns a transfer function that predicts segmentation parameters from box-detection parameters.
Results
Up to a 40% relative increase in mask AP is reported on COCO categories without training masks, alongside a qualitative model covering 3000 Visual Genome categories.
Takeaways & Limitations
The approach demonstrates a path toward instance segmentation models spanning thousands of categories using abundant box annotations alongside limited mask annotations.
Takeaways & Limitations
The large-scale Visual Genome experiment is qualitative because its categories overlap semantically and its annotations are not exhaustive, while mask supervision remains limited to set A.
Abstract
from arXiv · showhide
Most methods for object instance segmentation require all training examples to be labeled with segmentation masks. This requirement makes it expensive to annotate new categories and has restricted instance segmentation models to ~100 well-annotated classes. The goal of this paper is to propose a new partially supervised training paradigm, together with a novel weight transfer function, that enables training instance segmentation models on a large set of categories all of which have box annotations, but only a small fraction of which have mask annotations. These contributions allow us to train Mask R-CNN to detect and segment 3000 visual concepts using box annotations from the Visual Genome dataset and mask annotations from the 80 classes in the COCO dataset. We evaluate our approach in a controlled study on the COCO dataset. This work is a first step towards instance segmentation models that have broad comprehension of the visual world.
1. Introduction
The paper introduces partially supervised instance segmentation, combining mask annotations for some categories with bounding boxes for the rest. A learned transfer function extends mask prediction to categories without training masks, achieving up to a 40% relative mask AP increase and qualitative scaling to 3000 Visual Genome categories.
- Motivation: Typical instance segmentation systems cover only around 100 categories because mask annotations are expensive and require strong supervision.Bounding boxes are more abundant and less expensive to collect for new categories.
- Task formulation: Partially supervised instance segmentation uses mask annotations for a small subset of categories and bounding boxes for the remaining categories.The task combines strongly annotated and weakly annotated examples to segment all categories of interest.
- Method: The proposed transfer function predicts mask parameters from bounding-box detection parameters, using classes with mask annotations during training and all categories at inference.The approach is built on Mask R-CNN, whose detection and mask heads are trained jointly.
- Quantitative evaluation: Up to a 40% relative increase in mask AP is achieved on categories without training masks compared with a strong baseline on COCO.COCO provides reliable quantitative evaluation because its 80 categories are semantically separated and well annotated.
- Large-scale demonstration: The model is trained for 3000 Visual Genome categories using box annotations from Visual Genome and mask annotations from 80 COCO classes.Visual Genome supports qualitative rather than reliable quantitative evaluation because its categories overlap semantically and its annotations are not exhaustive.
2. Related Work
Prior instance segmentation methods generally assume masks for every category, limiting scale, while related work explores weak supervision, weight prediction, task transfer, and visual embeddings. This paper transfers category information from bounding-box detection to instance segmentation through a learned function.
- Instance segmentation: Fully supervised instance segmentation assumes mask annotations for all categories, making expansion to thousands of categories difficult.This paper relaxes that assumption by requiring masks for only a small subset.
- Weight prediction: Prior weight-prediction methods construct model parameters from natural-language descriptions, few-shot examples, or compositions of concepts.These approaches motivate predicting parameters rather than learning every parameter directly.
- Task transfer learning: LSDA transfers knowledge from image classification to object detection, whereas this work transfers knowledge from bounding-box detection to instance segmentation.The two methods differ in the source and target tasks connected by transfer learning.
- Weak supervision: Weakly supervised segmentation methods use image-level labels, boxes, points, or objectness constraints, but most address semantic rather than instance segmentation.A related weakly supervised instance segmentation method trains on GrabCut results derived from bounding boxes.
- Visual embeddings: Mask R-CNN box-head parameters provide class-specific visual embeddings that enable sharing appearance information between visually related categories.The paper also compares these embeddings with NLP-based GloVe embeddings.
3. Learning to Segment Every Thing
The paper relaxes full mask-supervision requirements by transferring category information from bounding-box detectors to mask predictors. It combines strong mask labels for some categories with box-only labels for others, using Mask R-CNN and training strategies designed to generalize across categories.
- Partially supervised instance segmentation: Partially supervised learning trains on categories with mask annotations and categories with only bounding-box annotations.The category set is partitioned into A, with masks, and B, with boxes; the model is trained to segment instances across both sets.
- Weight transfer: MaskX R-CNN transfers category-specific information from bounding-box detectors to instance mask predictors.The method builds on Mask R-CNN, whose detection and mask components contain category-specific parameters.
- Weight transfer: A generic transfer function predicts each category’s mask parameters from its bounding-box detection parameters.The transfer function is trained end-to-end using categories with mask annotations, then applied to every category at inference time.
- Training: Box detection losses train the bounding-box head on all categories, while mask losses train the mask head and transfer function only on categories with masks.The paper considers both stage-wise training and end-to-end joint training; stage-wise training fixes detection weights during mask training, whereas joint training updates both heads.
- Class-agnostic baseline: A class-agnostic mask-prediction baseline can generalize from masks on 20 COCO categories to the other 60 categories at inference time.The baseline uses a class-agnostic FCN mask head, and the cited prior evidence motivates it as a strong comparison.
- Fused mask heads: Fusing class-agnostic MLP predictions with FCN or transferred predictions is proposed to combine object gist with boundary detail.For K classes, class-agnostic scores are tiled across classes, added to class-specific scores, passed through sigmoid units, and resized to the bounding box.
4. Experiments on COCO
COCO experiments simulate partial supervision by training on mask annotations for set A and box annotations for set B, then evaluate mask AP across controlled splits and ablations. The transfer approach improves mask prediction on weakly supervised classes while preserving performance on strongly supervised classes.
- Evaluation Protocol and Baselines: COCO provides exhaustive mask annotations for 80 categories, enabling rigorous quantitative evaluation with average precision.The experiments use COCO train2017 for training and val2017 for testing.
- Ablation Experiments: The class-agnostic MLP mask branch consistently improves both the baseline and transfer approach on classes in set B.This supports combining FCN-based and MLP-based mask heads as complementary predictors.
- Ablation Experiments: End-to-end training improves results, but stopping gradient propagation from the transfer function to detection weights is crucial.The ablation compares stage-wise and end-to-end training with and without stopping this gradient.
- Evaluation Protocol and Baselines: A class-agnostic mask baseline and a fully supervised Mask R-CNN oracle provide comparison points for the proposed transfer method.The oracle has mask annotations for all classes and serves as a performance upper bound for the partially supervised task.
- Ablation Experiments: Using cls+box detection weights as transfer input outperforms random and GloVe inputs, while a 2-layer LeakyReLU transfer function gives the best mask AP on set B.The selected configuration is cls+box with a 2-layer LeakyReLU network.
- Ablation Experiments: Random COCO splits show up to over 40% relative increase in mask AP, with transfer performance associated with the average number of mask samples per class in set A.The results suggest collecting more mask samples per annotated category may be more effective for transfer to set B.
- Results and Comparison of Our Full Method: MaskX R-CNN exceeds the compared approaches by over 20% relative mask AP, and the trend generalizes from ResNet-50-FPN to ResNet-101-FPN.The reported comparison concerns the full method against the other evaluated approaches.
5. Large-Scale Instance Segmentation
The paper takes a step toward large-scale instance segmentation by training MaskX R-CNN on 3000 Visual Genome classes using transferred mask parameters from 80 COCO classes. Qualitative results show reasonable masks for many categories without mask training data, while also exposing challenging cases.
- Large-scale training: MaskX R-CNN is trained on 3000 Visual Genome classes, using box annotations from VG and mask annotations from 80 COCO classes.The model uses a stage-wise procedure: detection on all 3000 VG classes, followed by mask-head training on the 80 COCO classes.
- Evaluation: Visual Genome lacks instance masks and has overlapping, non-exhaustive annotations, so quantitative mask evaluation is difficult.The paper therefore uses VG primarily for qualitative analysis of the large-scale model.
- Qualitative results: The model segments abstract concepts such as shadows and paths, as well as whole objects and object parts.Examples include trolley-car windows and refrigerator handles.
- Qualitative results: Figure 5 shows reasonable mask predictions for many of the 2920 Visual Genome classes without COCO mask training data.The red boxes identify set B, whose classes have box annotations but no mask annotations during training.
- Qualitative results: Segmentation is reasonable for isolated trees but tends to fail when a detected tree is more like a forest.The qualitative results therefore illustrate both the potential and the difficulty of scaling instance segmentation to thousands of concepts.
6. Conclusion
The paper formulates large-scale instance segmentation as partially supervised learning, transferring information from box-trained detection parameters to mask prediction. COCO experiments show improved generalization to categories without mask training data, while the 3000-class Visual Genome model provides encouraging but challenging qualitative evidence.
- Problem: The task trains on categories with mixed supervision: only a subset has instance masks, while the remaining categories have bounding boxes.The goal is to segment instances across all categories in the set of interest.
- Method: The proposed transfer function predicts each class’s segmentation parameters from parameters learned for bounding-box detection.This transfers category information from detection to mask prediction for classes lacking mask annotations.
- Results: COCO experiments demonstrate greatly improved mask generalization to categories without mask training data.The paper also builds a 3000-class Visual Genome model with encouraging qualitative results.
- Takeaway: The approach establishes a direction for large-scale instance segmentation without full supervision.The conclusion identifies scaling to thousands of categories as an exciting research direction.
- Limitation: Scaling instance segmentation to thousands of categories without full supervision remains extremely challenging and offers substantial opportunity for improved methods.This scope boundary is stated explicitly in the conclusion.