PaperlayerView exact revision on arXiv

Source-linked AI summary

Detecting Twenty-thousand Classes using Image-level Supervision

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, Ishan Misra

arXiv:2201.02605v3cs.CV

TL;DR

Detection datasets have smaller vocabularies and fewer annotations than image-classification datasets, limiting detectors’ coverage. Detic decouples localization and classification by training detector classifiers with image-level supervision, improving open-vocabulary and long-tail detection and transferring to new datasets without finetuning.

  • Problem

    Detection datasets are smaller in size and vocabulary than classification datasets, and many detection classes lack sufficient annotations for robust detectors.

  • Method

    Detic decouples localization and classification by using image-level supervision with fixed label-to-region assignment instead of prediction-based assignment.

  • Results

    Detic improves novel-class open-vocabulary LVIS detection by 8.3 points over the baseline and matches performance using full class annotations.

  • Takeaways & Limitations

    A detector trained with more than twenty-thousand ImageNet-21K classes generalizes to new datasets with disjoint label spaces without seeing their training images.

  • Takeaways & Limitations

    Detic supervises all image labels to the same region, does not use overall dataset statistics, and has no guarantees for open-vocabulary generalization in extreme domains.

Abstract

from arXiv · show

Current object detectors are limited in vocabulary size due to the small scale of detection datasets. Image classifiers, on the other hand, reason about much larger vocabularies, as their datasets are larger and easier to collect. We propose Detic, which simply trains the classifiers of a detector on image classification data and thus expands the vocabulary of detectors to tens of thousands of concepts. Unlike prior work, Detic does not need complex assignment schemes to assign image labels to boxes based on model predictions, making it much easier to implement and compatible with a range of detection architectures and backbones. Our results show that Detic yields excellent detectors even for classes without box annotations. It outperforms prior work on both open-vocabulary and long-tail detection benchmarks. Detic provides a gain of 2.4 mAP for all classes and 8.3 mAP for novel classes on the open-vocabulary LVIS benchmark. On the standard LVIS benchmark, Detic obtains 41.7 mAP when evaluated on all classes, or only rare classes, hence closing the gap in performance for object categories with few samples. For the first time, we train a detector with all the twenty-one-thousand classes of the ImageNet dataset and show that it generalizes to new datasets without finetuning. Code is available at \url{https://github.com/facebookresearch/Detic}.

1 Introduction

Detection datasets have smaller vocabularies and fewer images than classification datasets, limiting detectors’ coverage. Detic adds image-level supervision, using a simple fixed proposal assignment to expand detection vocabulary and improve open-vocabulary performance.

  • Motivation: Detection datasets are smaller and have fewer classes than classification datasets, limiting supervision for robust detectors.LVIS has 1000+ classes and 120K images, while ImageNet has 21K classes and 14M images.
  • Approach: Detic decouples localization and classification by training the detector’s classifier with image-level labels while retaining detection supervision for localization.The method focuses image-level supervision on classification because modern proposal networks already localize many new objects.
  • Approach: Detic avoids prediction-based label-to-box assignment by applying image labels to a fixed largest-size proposal.This fixed assignment is intended to avoid error-prone assignments that depend on weak initial recognition of rare or novel classes.
  • Results: 2.4 mAP and 8.3 mAPnovel improvements over a strong box-supervised baseline demonstrate gains on open-vocabulary LVIS.The reported improvements cover all classes and novel classes, respectively.
  • Results: 5 points over OVR-CNN with the same detector and data establishes a further comparison against prior state of the art.The paper also reports transfer to new datasets and vocabularies without finetuning.
  • Contributions: The paper identifies simpler weakly supervised losses as a source of improved detection performance and releases ready-to-use code.The contribution claims cover both the loss design and practical availability of the implementation.

2 Related Work

Related work spans weakly supervised and open-vocabulary detection, with methods differing in their use of box supervision, proposal assignment, clustering, self-training, and language embeddings. Detic belongs to semi-supervised weakly supervised detection and uses image-labeled data without distillation.

  • Weakly Supervised Detection: Weakly supervised object detection trains detectors with image-level labels, sometimes without any box supervision.Examples include WSDDN, OIRC, PCL, and CASD, which aggregate or assign proposal-level evidence from image labels.
  • Semi-Supervised Detection: Semi-supervised weakly supervised detection combines bounding-box supervision with image labels.YOLO9000, DLWL, and MosaicOS combine detection data with classification data using assignment, self-training, clustering, or augmentation.
  • Detic’s Position: Detic belongs to semi-supervised weakly supervised detection but uses a simple image-supervised loss.The paper distinguishes this from complementary point- or scribble-based localization supervision.
  • Open-Vocabulary Detection: Open-vocabulary detectors replace or enhance classifier weights with language embeddings, including GloVe and CLIP.Prior methods also use image-text pretraining, visual-language distillation, or external text information.
  • Detic’s Position: Detic uses a CLIP classifier and additional image-labeled data for co-training, without distillation.The experiments report that proposals generalize well enough, while further RPN improvements may improve results.

3 Preliminaries

The paper formulates detection as localization plus classification using detection data with box labels and classification data with image-level labels. Its approach trains both data sources jointly while applying image-level supervision only to classification features from a fixed proposal.

  • Problem Setup: Object detection finds object boxes and assigns class labels from a test-time vocabulary.The formulation separates localization of boxes from classification of detected objects.
  • Problem Setup: Detection datasets provide image, box, and class labels, whereas classification datasets provide images and image-level class labels.Their vocabularies may overlap or differ from the test vocabulary.
  • Training Overview: Detic mixes detection and classification images, applying localization losses only to box-labeled images and classification loss to a fixed region for image-labeled images.This training design uses detection data for standard two-stage training while restricting image-only supervision to classification.
  • Two-Stage Detection: A two-stage detector uses an RPN to produce proposals and region features, followed by classification and box regression.The classifier uses W and the regression branch uses B to predict class scores and refined boxes.
  • Two-Stage Detection: Detic focuses on second-stage classification because the paper reports that proposal generation and box regression are not the current bottleneck.The detector uses many test-time proposals relative to the number of objects per image.
  • Open-Vocabulary Detection: Open-vocabulary detection permits test classes outside the detection-training vocabulary and can use fixed language embeddings such as CLIP.The paper notes that this approach can theoretically detect any class but performs unsatisfactorily in practice without additional supervision.

4 Detic: Detector with Image Classes

Detic combines detection and image-level supervision while decoupling localization from classification. It uses fixed region assignments for image labels and shared classification parameters to expand detector training beyond box-labeled data.

  • Training with image-level supervision: Detic composes mini-batches from detection and classification datasets, using standard detector training for box-labeled images and classification training on fixed proposals for image-labeled images.Image-labeled images train classification features without localization losses.
  • Decoupling localization and classification: Detic focuses image-level supervision on classification rather than localization, using fixed assignments to regions instead of prediction-based label-to-box assignment.This avoids requiring accurate initial detections for assignment.
  • Non-prediction-based losses: The image-box loss applies classification loss to features from an injected whole-image proposal, using the same proposal to solve classification across labeled classes.The whole-image box is b′ = (0, 0, w, h).
  • Non-prediction-based losses: Alternative fixed strategies use the proposal with the maximum object score or the proposal with the maximum size.These alternatives replace the whole-image proposal with smaller boxes.
  • Loss selection and shared parameters: The evaluated losses all leverage image-level supervision, but the max-size loss performs best and is used by default for image-supervised data.Classification parameters are shared across detection and classification data, which greatly improves detection performance.
  • Relation to prediction-based assignments: Prediction-based assignments depend on classification scores, whereas Detic uses a fixed supervision criterion because rare or novel classes have weak initial recognition.The fixed criterion sidesteps the prediction-and-assignment process entirely.

5 Experiments

Detic is evaluated across open-vocabulary LVIS and COCO, standard LVIS, and cross-dataset detection settings using image-level supervision and non-prediction-based losses. It consistently improves over strong baselines and prior methods, including for novel and rare classes, while transferring across datasets without finetuning.

  • Experimental setup: Detic experiments use LVIS, with open-vocabulary evaluation removing 337 rare classes from training, plus ImageNet-21K, IN-L, and Conceptual Captions image supervision.LVIS contains 1203 classes and 100K images; IN-21K contains 14M images for 21K classes, while IN-L overlaps with 997 LVIS classes.
  • Prediction-based vs non-prediction-based methods: All proposed non-prediction-based losses outperform prediction-based counterparts, with approximately 4.2-point novel-class gains using ImageNet and 2.6-point gains using Conceptual Captions.The max-size loss performs best consistently and becomes Detic’s default for subsequent experiments.
  • Prediction-based vs non-prediction-based methods: 92.8% of target objects are covered by Detic’s max-size assignment, compared with 69.0% for prediction-based assignment on an annotated IN-L subset.The max-size assignment is more likely to contain objects and remains more consistent during training, although it can miss some, especially small, objects.
  • Open-vocabulary LVIS: 2.4 mAP overall and 8.3 mAPnovel are Detic’s gains over the strong Box-Supervised baseline on open-vocabulary LVIS.Detic performs favorably against a fully supervised detector despite lacking novel-class box labels, indicating that box annotations may not be required for new classes.

6 Limitations and Conclusions

Detic combines image-level supervision with detection supervision through simple classification losses and supports multiple image-labeled data types, including captions. The paper reports broad gains but notes limits in label-statistics modeling and extreme-domain generalization.

  • Limitations: Detic does not incorporate overall dataset statistics and provides no guarantees for open-vocabulary generalization in extreme domains.The authors leave dataset-statistics integration for future work and identify extreme-domain generalization as an unsupported setting.
  • Caption supervision: The framework can incorporate free-form caption text by converting captions into embeddings with a CLIP language encoder.Caption features are stacked into a dynamic classification layer, with each image paired positively with its own caption.
  • Method: The max-size loss applies supervision to the largest proposal, while other image-labeled outputs receive no supervision.The caption formulation uses image text as the positive label and other batch captions as negatives.
  • Conclusions: Detic uses image-level labels and captions to improve both overall and novel-class mAP on open-vocabulary LVIS and COCO.Combining max-size image-label loss with caption loss produces a more significant improvement than either loss alone.

C LVIS baseline details

The LVIS baseline evolves from a standard Detectron2 configuration into a stronger open-vocabulary detector through class-agnostic heads, federated loss, CenterNet2, longer training, CLIP, and ImageNet-21K pretraining. These changes improve performance but require additional training time or larger computational settings.

  • Baseline evolution: Class-agnostic box and mask heads prepare the model for new classes, while federated loss and CenterNet2 further improve mAP.CenterNet2 replaces RPN and multiplies proposal scores by classification scores, with a slight training-time increase.
  • Training configuration: Large-scale jittering, a 4× schedule, 640 × 640 training images, and batch size 64 produce a slightly better result than default multi-scale training.The resulting configuration uses 30% more training time, while testing remains at 800 × 1333.
  • Classifier and pretraining: A CLIP classifier improves rare-class mAP, and ImageNet-21K pretraining is included in the final baseline configuration.The ImageNet-21K pretrained ResNet-50 model requires the Adam optimizer with learning rate 2e-4.
  • Baseline: The Detectron2 LVIS baseline uses a ResNet-50 FPN backbone and a 2× training schedule, with 35.3 mAPbox and 31.5 mAPmask.The baseline trains for 180k iterations with batch size 16 and serves as the main-paper reference model.
  • Compute trade-offs: Increasing training resolution or backbone size can further increase performance, but at the cost of longer training time.The paper uses large models only for comparisons with state-of-the-art systems.

D Resolution change for classification data

Resolution changes affect classification-data training differently depending on the weak dataset. Smaller input resolution improves performance with ImageNet but has little effect with Conceptual Captions.

  • Resolution ablation: Using a smaller input resolution improves approximately 1 point for both mAP and mAPnovel when ImageNet supplies image-labeled data.With Conceptual Captions, the resolution change has little impact.
  • Resolution ablation: Using more batches for weak datasets is slightly better than maintaining a 1:1 batch ratio.

E Prediction-based losses implementation details

The implementation compares prediction-based weakly supervised losses by aggregating or selecting proposal scores, with DLWL* additionally clustering proposals before selecting peaks. The reproduced DLWL implementation omits its self-training and bootstrapping assignment step.

  • WSDDN: WSDDN learns proposal weights and sums weighted proposal classification scores into one image classification score.Its aggregation uses a learnable network parameter W′.
  • Predicted: Predicted selects the proposal with the highest predicted score for the labeled class.
  • DLWL*: DLWL* clusters proposals at IoU threshold 0.5 and selects the top Nc = 3 cluster peaks by class score.Each peak is the proposal within a cluster having the maximum predicted score for class c.
  • DLWL*: The implementation omits DLWL’s IoU-based assignment matrix from self-training and bootstrapping to compare only the training losses.

F More comparison between prediction-based and non-prediction-based methods

Non-prediction-based losses outperform prediction-based alternatives, with max-size loss performing best in the current performance regime. Its assigned regions are safer and more stable during training, although cover and consistency metrics do not alone determine final performance.

  • Non-prediction-based losses perform significantly better than prediction-based losses, and max-size loss performs best in the current performance regime.The predicted loss depends on good initial predictions, which are inherently flawed in open-vocabulary detection.
  • Max-size loss usually covers the target object despite coarse boundaries and assigns consistent regions across training iterations.Predicted-loss regions often change across iterations, especially early in training when predictions are unstable.
  • Cover rate measures how often an assigned region sufficiently overlaps the ground-truth box, while consistency measures assigned-region IoU across training schedules.The cover-rate threshold is greater than 0.5 intersection-over-area, and consistency compares half-schedule with final-schedule regions.
  • Max-size loss is more favorable than predicted loss on cover rate and consistency, but these metrics do not always correlate with final performance.Image-box loss is perfect on both metrics yet underperforms max-size loss.

H Open-vocabulary COCO benchmark details

The open-vocabulary COCO evaluation uses manually selected base and novel classes with generalized zero-shot testing. Under a shared OVR-CNN setup, Detic’s non-prediction-based image supervision is consistently stronger, while the max-size loss performs best among its variants.

  • Open-vocabulary COCO defines 48 base classes and 17 novel classes from the 80 COCO categories.Training uses images containing at least one base class, while testing uses all COCO validation images.
  • The benchmark follows an OVR-CNN recipe with ResNet50-C4, a 1× schedule, fixed 800×1333 resolution, and box mAP at IoU threshold 0.5.Training uses horizontal flipping, SGD, learning-rate drops at 60k and 80k iterations, and batch size 16.
  • Detic’s non-prediction-based methods outperform prediction-based counterparts on open-vocabulary COCO, with max-size loss best among its variants.The observations are consistent with the LVIS experiments.
  • Detic outperforms MosaicOS in mAP and mAPr without multi-stage training or mosaic augmentation, using a simpler framework and less additional training data.The comparison follows the MosaicOS baseline setup.

J Generalization to Deformable-DETR.

Detic generalizes beyond its original detector architecture: applied to Deformable-DETR, it improves full-LVIS performance and demonstrates compatibility with Transformer-based detectors. Improvements also extend to classes without image labels when using the CLIP classifier.

  • Detic applied to Deformable-DETR improves full-LVIS performance by +0.8 mAP and +4.8 mAPr over the baseline.The experiment uses the detector’s default recipe, Federated Loss, a 4× schedule, and image supervision on the maximum-predicted-size encoder query.
  • The Deformable-DETR result shows that Detic generalizes to Transformer-based detectors.
  • Most improvements arise from classes with image-level labels, while ImageNet training also improves classes without image labels through the CLIP classifier’s interclass relations.
  • Detic’s improvements remain consistent under the newer mAPFixed evaluation metric.
Loading 2201.02605v3…