Source-linked AI summary
FreeSOLO: Learning to Segment Objects without Annotations
Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, Jose M. Alvarez
TL;DR
FreeSOLO addresses the costly annotation requirements of instance segmentation by learning class-agnostic masks from unlabeled images. It combines Free Mask with a SOLO-based self-supervised segmenter and reports strong unsupervised segmentation, object discovery, and limited-mask pre-training results.
Problem
Instance segmentation requires costly annotations, while existing weakly supervised methods still rely on localization annotations such as boxes or points.
Method
FreeSOLO uses Free Mask to generate coarse masks and semantic embeddings, then trains and self-trains a SOLO-based instance segmenter with unlabeled images.
Results
FreeSOLO successfully demonstrates unsupervised class-agnostic instance segmentation, outperforms some manually annotated proposal methods, and exceeds DenseCL by +9.8% AP with 5% labeled COCO masks.
Takeaways & Limitations
FreeSOLO serves as both an unsupervised class-agnostic segmenter and a strong self-supervised pre-training method for instance segmentation.
Takeaways & Limitations
Without category labels, FreeSOLO produces class-agnostic object masks rather than object categories.
Abstract
from arXiv · showhide
Instance segmentation is a fundamental vision task that aims to recognize and segment each object in an image. However, it requires costly annotations such as bounding boxes and segmentation masks for learning. In this work, we propose a fully unsupervised learning method that learns class-agnostic instance segmentation without any annotations. We present FreeSOLO, a self-supervised instance segmentation framework built on top of the simple instance segmentation method SOLO. Our method also presents a novel localization-aware pre-training framework, where objects can be discovered from complicated scenes in an unsupervised manner. FreeSOLO achieves 9.8% AP_{50} on the challenging COCO dataset, which even outperforms several segmentation proposal methods that use manual annotations. For the first time, we demonstrate unsupervised class-agnostic instance segmentation successfully. FreeSOLO's box localization significantly outperforms state-of-the-art unsupervised object detection/discovery methods, with about 100% relative improvements in COCO AP. FreeSOLO further demonstrates superiority as a strong pre-training method, outperforming state-of-the-art self-supervised pre-training methods by +9.8% AP when fine-tuning instance segmentation with only 5% COCO masks. Code is available at: github.com/NVlabs/FreeSOLO
1 The University of Adelaide 2 NVIDIA 3 Caltech 4 Zhejiang University
The paper identifies affiliations involving NVIDIA and the University of Adelaide, provides qualitative results, and makes its code publicly available.
- Figure 1 presents qualitative FreeSOLO results for class-agnostic instance segmentation.
- The work includes an affiliation note for NVIDIA and the University of Adelaide.
- FreeSOLO code is available at github.com/NVlabs/FreeSOLO.
1. Introduction
The paper targets class-agnostic instance segmentation without annotations, combining Free Mask with Self-Supervised SOLO and using the resulting representations for object discovery and pre-training.
- FreeSOLO addresses class-agnostic instance segmentation without any annotations.The approach is motivated by the cost of bounding-box and mask supervision and builds on SOLO and DenseCL.
- FreeSOLO unifies pixel grouping, object localization, and feature pre-training through a top-down-meets-bottom-up design.The design also learns object-level semantic representations without supervision for later supervised instance segmentation initialization.
- Free Mask generates coarse object masks and semantic embeddings, which Self-Supervised SOLO uses to train an instance segmentation model with weak supervision.The framework uses self-training to improve segmentation from noisy coarse masks.
- FreeSOLO outperforms some manually annotated proposal methods and improves unsupervised object discovery by about 100% relative COCO AP.The introduction also reports +9.8% AP over DenseCL when fine-tuning with 5% labeled COCO masks.
2. Related Work
Prior work addresses instance segmentation, self-supervised representation learning, object discovery, and unsupervised segmentation, while FreeSOLO removes annotation requirements for the harder instance-level task.
- Instance segmentation: Instance segmentation methods include top-down detection-then-segmentation, bottom-up embedding-then-clustering, and combined designs.
- Instance segmentation: BoxInst achieves strong results with box annotations, whereas FreeSOLO reports instance segmentation without annotations.
- Self-supervised learning: Self-supervised learning has expanded from image classification pretexts to dense prediction tasks such as detection and semantic segmentation.FreeSOLO differs by learning intermediate representations for instance segmentation.
- Unsupervised object discovery: Unsupervised object discovery has been formulated through statistical, graph-based, clustering, matching, optimization, and ranking approaches, but challenging scenes remain difficult.
- Unsupervised segmentation: Unsupervised segmentation methods often assume repeated objects, simple scenarios, salient-object estimators, or boundary annotations, while FreeSOLO segments each object individually.
3. Method
FreeSOLO combines Free Mask, which extracts coarse object masks from self-supervised features, with Self-Supervised SOLO, which learns instance segmentation from these noisy masks and embeddings.
- 3.1. Overview of FreeSOLO: FreeSOLO trains an instance segmentation model from unlabeled images, transferring all parameters except the last classification layer for downstream supervised fine-tuning.The method uses Free Mask outputs and semantic embeddings to train a SOLO-based segmenter.
- 3.2. Free Mask: Free Mask constructs queries from downsampled backbone features and keys from the original features, using cosine similarities to produce one score map per query.Each normalized query can be viewed as a 1 × 1 convolutional kernel applied to the normalized key features.
- 3.2. Free Mask: The score maps become soft masks, are thresholded into binary masks, ranked by maskness confidence, and filtered with mask NMS to produce object masks.Maskness emphasizes masks with confident foreground pixels, while NMS removes redundant outputs.
- 3.2. Free Mask: Dense contrastive pre-training is particularly effective for Free Mask because its local feature correspondence objective aligns with query-to-key cosine similarity.This alignment provides a self-supervised basis for extracting reasonable object masks.
- 3.3. Self-Supervised SOLO: Self-Supervised SOLO learns from noisy coarse masks using projection and pairwise losses, then improves them through one round of confidence-filtered self-training.The mask loss combines average-projection, max-projection, and pairwise terms; further self-training iterations add no gains.
4. Experiments
FreeSOLO is evaluated for unsupervised class-agnostic instance segmentation, object discovery, and transfer to supervised instance segmentation. Experiments also study pre-training, self-training, semantic embedding, qualitative behavior, and failure cases.
- Main Results: FreeSOLO achieves better class-agnostic instance-segmentation results than annotated proposal methods, while reaching 4.8% versus 6.8% AP against fully supervised Mask R-CNN on UVO.The comparison includes methods using manually annotated datasets, whereas FreeSOLO uses no annotations.
- Main Results: FreeSOLO pre-training outperforms ImageNet supervised pre-training by 4.0% AP with 5% COCO training images and exceeds DenseCL by 2.0% AP.These results evaluate fine-tuning the self-supervised instance segmenter with annotations.
- Qualitative Results: Free Mask extracts coarse masks of common objects in unlabeled images, but FreeSOLO can fail on truncated, crowded, or small objects.The qualitative results show the coarse-mask behavior and identify localization failure cases.
- Main Results: With only 5% segmentation masks, FreeSOLO gains 9.8% AP over supervised pre-training and outperforms supervised and state-of-the-art self-supervised pre-training.The experiments use limited-mask fine-tuning to compare pre-training methods.
- Ablation Study: One self-training iteration clearly improves performance from coarse-mask training, whereas additional iterations provide no further gains.The ablation compares the initial coarse masks, zero self-training iterations, and subsequent iterations.
- Ablation Study: Semantic embedding loss improves instance-segmentation fine-tuning with annotations, while omitting Lavg proj causes the model to segment contours when trained longer.The semantic-embedding comparison uses 10% fully annotated COCO images; the projection-loss observation comes from qualitative results.
5. Discussion and Conclusion
FreeSOLO is presented as a self-supervised framework for class-agnostic instance segmentation without annotations. The authors identify remaining limitations, including absent category prediction, a gap to supervised models, and failures in some scenarios.
- 5. Discussion and Conclusion: FreeSOLO learns class-agnostic instance segmentation without pixel-level or image-level labels.The framework is intended to reduce annotation effort for instance-level recognition tasks.
- 5. Discussion and Conclusion: Without category labels, the model generates class-agnostic object masks rather than predicting detected-object categories.
- 5. Discussion and Conclusion: The method remains behind supervised models trained with rich annotations and can fail in some scenarios.
A.1. Evaluation protocol
The evaluation protocol reports class-agnostic instance-segmentation performance using standard mask AP and an easier protocol focused on medium and large objects.
- A.1. Evaluation protocol: AP* evaluates mask AP at IoU 0.5 using only objects larger than 642 pixels.
- A.1. Evaluation protocol: AP*_M and AP*_L separately cover objects with areas in (642, 1922) and greater than 1922, respectively.
- A.1. Evaluation protocol: MCG and COB results are computed using their official segmentation masks.
A.2. Supervised fine-tuning
The supervised fine-tuning protocol evaluates pretrained instance-segmentation models on COCO using limited manually annotated data settings.
- A.2. Supervised fine-tuning: Fine-tuning uses a dynamic SOLO model on COCO train2017 and evaluates it on COCO val2017.Synchronized batch normalization and an FPN are used in the backbone during training.
- A.2. Supervised fine-tuning: The experiments provide limited-data settings with 5% and 10% of COCO images.
A.3. Training details
Training uses unlabeled COCO data for FreeSOLO and standardized optimization settings, with copy-paste augmentation applied between image pairs.
- A.3. Training details: FreeSOLO uses COCO train2017 and unlabeled2017, totaling approximately 241k unlabeled images.ResNet-50 is used for fine-tuning and ablations, while ResNet-101 is used for other results and visualizations.
- A.3. Training details: Copy-paste augmentation transfers randomly selected objects between image pairs unless their overlap with existing objects reaches IoU 0.5.
B. Additional results
Additional results show that FreeSOLO performs favorably on medium and large objects and can produce precise qualitative masks without annotations.
- B. Additional results: The easier AP* protocol shows larger gains over MCG and COB, especially for large objects.AP* evaluates medium and large objects on COCO val2017; the comparison is reported in Table S1.
- C. Additional visualizations: FreeSOLO can produce more precise segmentation than manual annotations at some object boundaries, suggesting potential for auto-labeling.The qualitative comparison is presented in Figure S3.
- C. Additional visualizations: FreeSOLO performs class-agnostic instance segmentation without manual annotations and can infer at 16 FPS on a V100 GPU.Additional qualitative results are shown in Figure S1.
- C. Additional visualizations: Without Lavg proj, longer training tends to produce contour-only masks when learning from coarse masks.Figure S2 compares FreeSOLO with and without Lavg proj.