PaperlayerView exact revision on arXiv

Source-linked AI summary

Cut and Learn for Unsupervised Object Detection and Instance Segmentation

Xudong Wang, Rohit Girdhar, Stella X. Yu, Ishan Misra

arXiv:2301.11320v1cs.CVcs.AIcs.LG

TL;DR

Human annotations for object localization are costly, while prior unsupervised methods have limited multi-object, architectural, or transfer capabilities. CutLER uses MaskCut, robust detector training, and self-training to learn from self-supervised features alone, and as a zero-shot detector outperforms prior work by over 2.7× across 11 diverse benchmarks.

  • Problem

    Object detection and instance segmentation require costly human localization annotations, motivating models that can discover and localize objects without labels.

  • Method

    CutLER generates multiple coarse masks from self-supervised ViT features, trains detectors with DropLoss, and refines predictions through multiple rounds of self-training.

  • Results

    Over 2.7× higher detection performance was achieved across 11 benchmarks spanning diverse domains for CutLER’s zero-shot unsupervised detector.

  • Takeaways & Limitations

    CutLER trained solely on ImageNet provides a multi-object, architecture-flexible zero-shot detector and can also improve label-efficient supervised detection and segmentation.

  • Takeaways & Limitations

    Using additional eigenvectors to obtain multiple TokenCut instances significantly degrades multi-object discovery performance.

Abstract

from arXiv · show

We propose Cut-and-LEaRn (CutLER), a simple approach for training unsupervised object detection and segmentation models. We leverage the property of self-supervised models to 'discover' objects without supervision and amplify it to train a state-of-the-art localization model without any human labels. CutLER first uses our proposed MaskCut approach to generate coarse masks for multiple objects in an image and then learns a detector on these masks using our robust loss function. We further improve the performance by self-training the model on its predictions. Compared to prior work, CutLER is simpler, compatible with different detection architectures, and detects multiple objects. CutLER is also a zero-shot unsupervised detector and improves detection performance AP50 by over 2.7 times on 11 benchmarks across domains like video frames, paintings, sketches, etc. With finetuning, CutLER serves as a low-shot detector surpassing MoCo-v2 by 7.3% APbox and 6.6% APmask on COCO when training with 5% labels.

1. Introduction

CutLER addresses the cost of human localization annotations by training unsupervised detection and instance-segmentation models from self-supervised features. Its MaskCut, robust loss, and self-training mechanisms support multiple-object discovery, architecture flexibility, zero-shot transfer, and supervised pretraining.

  • Motivation: ∼164K COCO images with masks for 80 classes required more than 28K human annotation hours.This motivates learning localization without human labels.
  • Method: CutLER uses MaskCut to generate multiple coarse masks from self-supervised ViT features, then trains a detector with a loss robust to missed objects.Self-training further refines the model’s masks and boxes.
  • Comparison: CutLER detects multiple objects, unlike prior salient-object methods that generally locate only one prominent object per image.The paper positions multi-object discovery as necessary for real-world images containing multiple objects.
  • Method: CutLER supports various detection architectures and trains solely on unlabeled ImageNet without requiring in-domain unlabeled data.It is designed to transfer across domains without target-dataset training images.
  • Results: CutLER doubles APbox50 on 10 of 11 zero-shot benchmarks and outperforms supervised detectors on the UVO video instance-segmentation benchmark.The benchmarks span domains including video frames, sketches, paintings, and clip arts.
  • Results: CutLER also serves as a pretrained model that improves fully supervised detection and instance segmentation, including few-shot object detection.This extends its use beyond unsupervised zero-shot evaluation.

2. Related Work

Prior self-supervised and unsupervised methods either discover too few objects, depend on particular architectures, or use low-quality initial masks. CutLER combines multi-object MaskCut masks with detector learning and refinement, achieving substantially higher performance across detection and segmentation benchmarks.

  • Self-supervised representation learning: Self-supervised representation learning derives useful structure from unlabeled data through contrastive, similarity-based, clustering-based, or masked-autoencoder objectives.These approaches provide the representation-learning context for unsupervised object discovery.
  • Single-object discovery: DINO, LOST, and TokenCut use self-supervised ViT features to segment a single salient object, limiting their use for multi-object images.Their graph-based procedures operate on DINO patch features.
  • CutLER: CutLER’s MaskCut generates multiple initial masks, while DropLoss and self-training improve detector outputs beyond the coarse supervision.The pipeline is summarized as mask generation, detector learning, and iterative refinement.
  • Results: 2×–4× higher APbox and APmask than FreeSOLO and MaskDistill was achieved on almost all experimented detection and segmentation benchmarks.This comparison holds even when the competing methods are trained and tested on the same domain.

3. Method

CutLER is a cut-and-learn pipeline that generates coarse multi-object masks from self-supervised features, trains detectors with a loss tolerant of missing annotations, and improves them through self-training.

  • MaskCut for Discovering Multiple Objects: MaskCut applies Normalized Cuts to DINO patch similarities, then masks previously found foreground nodes and repeats the process to discover multiple objects.The default pipeline repeats the process three times.
  • MaskCut for Discovering Multiple Objects: MaskCut selects foreground patches using eigenvector prominence and an object-centric corner prior.The foreground should include the patch with the maximum absolute second-eigenvector value and fewer than two image corners.
  • DropLoss for Exploring Image Regions: DropLoss ignores penalties for predicted regions with small overlap with MaskCut ground-truth, allowing detectors to explore regions containing missed objects.The reported threshold is τ_IoU = 0.01.
  • Self-Training: Self-training uses confident predictions as additional pseudo-annotations, and three rounds were sufficient to obtain good performance.The confidence threshold is 0.75−0.5t in round t.
  • Implementation Details: CutLER uses only unlabeled ImageNet images for training and evaluates zero-shot multi-object detection across 11 datasets and diverse domains.The implementation uses 1.3 million ImageNet images and reports class-agnostic multi-object detection performance averaged across 11 datasets.

4. Experiments

CutLER is evaluated as a zero-shot unsupervised detector and segmenter across diverse datasets, then as a pretrained model under labeled finetuning. It consistently outperforms prior methods, including in low-shot COCO settings.

  • 4.1. Unsupervised Zero-shot Evaluations: CutLER is evaluated without labels or downstream-data finetuning on eleven datasets spanning diverse object categories, styles, resolutions, and camera angles.Because unsupervised detectors lack semantic classes and datasets may be partially labeled, evaluation uses class-agnostic detection and Average Recall.
  • 4.1. Unsupervised Zero-shot Evaluations: Qualitatively, CutLER discriminates instances, discovers additional objects, and produces higher-quality masks, including for small objects and instances missed by human annotators.Examples include person versus skis, apple and raisins, a kite, and novel streetlight and clock-tower instances.
  • 4.1. Unsupervised Zero-shot Evaluations: More than 2× performance gains over FreeSOLO are reported on 10 of 11 benchmarks, despite using ResNet50 rather than ResNet101.The benchmarks span video frames, paintings, clip arts, and complex scenes.
  • 4.1. Unsupervised Zero-shot Evaluations: CutLER often achieves 2∼3× higher AP on COCO detection and segmentation tasks than prior works, despite not training on COCO images.It also surpasses existing methods trained on COCO by more than 10% in APmask.
  • 4.1. Unsupervised Zero-shot Evaluations: CutLER outperforms all prior works on every reported COCO 20K and COCO val2017 evaluation metric, despite using an older detector.Table 3 distinguishes methods trained with extra downstream images from zero-shot methods trained only on ImageNet.
  • 4.1. Unsupervised Zero-shot Evaluations: CutLER more than quadruples prior-SOTA AP on UVO and nearly triples APbox 50, with APmask 50 exceeding fully supervised SOLOv2 trained on LVIS.On VOC, it obtains over 3× higher AP than previous SOTA, with the largest improvement in APL.
  • 4.2. Label-Efficient and Fully-Supervised Learning: With 5% labeled COCO data, CutLER improves APbox by 7.3% over MoCo-v2, while full supervision still yields more than 2% improvements for detection and segmentation.The finetuned model uses Cascade Mask R-CNN with a ResNet50 backbone.

5. Ablations

Ablations show that MaskCut, DropLoss, and self-training each contribute to localization quality, while architecture compatibility and matched pretraining distributions affect performance. CutLER also improves over TokenCut in recall and precision.

  • Component contributions: MaskCut improves APmask 50 by 4.7% on UVO and 2.7% on COCO, with the larger gain occurring on the densely annotated UVO dataset.The results support MaskCut’s role in segmenting multiple instances per image.
  • Self-training: Multiple self-training rounds increase pseudo-mask quantity and quality and produce a 1.3% improvement in APmask 50.Three rounds are sufficient by default, and gains are larger on densely labeled UVO.
  • Comparison with TokenCut: CutLER improves average recall ARbox 100 by 4× over TokenCut and surpasses TokenCut’s APbox by 4.8×.Using more eigenvectors increases recall but significantly reduces precision, whereas CutLER improves recall without that precision reduction.
  • Architecture and data choices: CutLER works with various detection architectures, and stronger architectures further improve its performance.The architecture study includes ViTDet with a ViT-B backbone.
  • Architecture and data choices: Detection performance remains similar when DINO and CutLER use the same ImageNet or YFCC dataset, but worsens when their training distributions differ.This indicates that matching the pretraining and CutLER-training distributions matters in the tested settings.

6. Summary

CutLER achieves strong unsupervised object detection and instance segmentation without human annotations. Trained solely on ImageNet, it outperforms prior work across diverse domains.

  • CutLER performs object detection and instance segmentation without training on human annotations.
  • Over 2.7× performance improvement is reported across 11 benchmarks for CutLER as a zero-shot unsupervised detector.

A.1. Training details

The implementation trains detector architectures on ImageNet using initial masks and bounding boxes, then applies multi-round self-training. Training uses specified SGD schedules and omits exploration loss during self-training.

  • CutLER trains Mask R-CNN and Cascade Mask R-CNN models on ImageNet, with Cascade Mask R-CNN used by default.
  • Initial detector training runs for 160K iterations with batch size 16, using self-supervised DINO initialization for ResNet-50 models.
  • Copy-paste augmentation randomly downsamples masks using a scalar uniformly sampled between 0.3 and 1.0.
  • Each self-training stage initializes from the previous stage and runs SGD for 80K iterations without exploration loss.

A.2. Datasets used for zero-shot evaluation

Zero-shot evaluation covers diverse object detection and instance segmentation datasets, including COCO, Pascal VOC, UVO, LVIS, and CrossDomain styles. The evaluations use class-agnostic COCO-style metrics without COCO fine-tuning.

  • COCO evaluation uses class-agnostic detection and segmentation metrics on val2017 and COCO 20K without fine-tuning on COCO images.
  • The evaluation includes Pascal VOC, UVO video objects, and LVIS with class-agnostic object detection and instance segmentation results.
  • CrossDomain evaluates watercolor, clipart, and comics subsets containing watercolor, sketch, and painting styles.
  • A summary of the zero-shot evaluation datasets is provided in Table 12.

A.3. Additional results for zero-shot detection & segmentation

Additional zero-shot results use standard COCO metrics across IoU thresholds and object scales. The reported metrics include AP, AP50, AP75, APS, APM, and APL.

  • AP averages precision over IoU thresholds from 0.5 to 0.95 in steps of 0.05.
  • AP50 and AP75 measure precision at IoU thresholds of 0.5 and 0.75, respectively.
  • APS, APM, and APL report performance for small, medium, and large objects.
  • Detailed results across the listed benchmarks are reported in Table 13.

A.4. CutLER vs. Selective Search

CutLER substantially improves both precision and recall over Selective Search for unsupervised region proposal generation, while also providing real-time inference speed.

  • CutLER significantly improves both precision and recall compared with Selective Search.The comparison uses a precision-recall curve on VOC07 trainval.
  • Selective Search generates possible object locations with sliding windows, which reduces inference speed.
  • CutLER is a better-performing unsupervised method for region proposal generation with real-time inference speed.
Loading 2301.11320v1…