PaperlayerView exact revision on arXiv

Source-linked AI summary

Open-Vocabulary One-Stage Detection with Hierarchical Visual-Language Knowledge Distillation

Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, Weiming Hu

arXiv:2203.10593v1cs.CV

TL;DR

One-stage open-vocabulary detectors lack class-agnostic proposals that support distillation on unseen objects. HierKD combines global-level caption-to-image distillation with instance-level distillation, and achieves reported AP50 gains over the previous best one-stage detector on MS-COCO. The paper notes that its ability to mimic CLIP remains short of an upper bound.

  • Problem

    One-stage open-vocabulary detection lacks class-agnostic proposals, limiting knowledge distillation on unseen objects.

    Source: §1 · pp. 1–2

  • Method

    HierKD combines global-level distillation aligning image features with caption semantics and instance-level distillation to transfer PVLM knowledge in one-stage detection.

    Source: §1 · p. 2

  • Results

    11.9% and 6.7% AP50 gains over the previous best one-stage detector under zero-shot and generalized zero-shot detection, respectively, on MS-COCO.

    Source: §1 · p. 2

  • Takeaways & Limitations

    HierKD transfers novel-category semantic knowledge from CLIP through captions and can identify novel objects more accurately and confidently, according to the authors.

    Source: §5 · p. 8

    HierKD's total AP50 and size-specific AP remain far from the upper bound, indicating room to improve its ability to mimic CLIP.

    Source: §4.6 · p. 8

Abstract

from arXiv · show

Open-vocabulary object detection aims to detect novel object categories beyond the training set. The advanced open-vocabulary two-stage detectors employ instance-level visual-to-visual knowledge distillation to align the visual space of the detector with the semantic space of the Pre-trained Visual-Language Model (PVLM). However, in the more efficient one-stage detector, the absence of class-agnostic object proposals hinders the knowledge distillation on unseen objects, leading to severe performance degradation. In this paper, we propose a hierarchical visual-language knowledge distillation method, i.e., HierKD, for open-vocabulary one-stage detection. Specifically, a global-level knowledge distillation is explored to transfer the knowledge of unseen categories from the PVLM to the detector. Moreover, we combine the proposed global-level knowledge distillation and the common instance-level knowledge distillation to learn the knowledge of seen and unseen categories simultaneously. Extensive experiments on MS-COCO show that our method significantly surpasses the previous best one-stage detector with 11.9\% and 6.7\% $AP_{50}$ gains under the zero-shot detection and generalized zero-shot detection settings, and reduces the $AP_{50}$ performance gap from 14\% to 7.3\% compared to the best two-stage detector.

1. Introduction

Open-vocabulary detection targets novel categories without requiring instance-level annotations for every category in training. HierKD combines global-level and instance-level distillation to address limitations of one-stage distillation, and outperforms the previous best one-stage detector on MS-COCO.

1. Introduction

  • Open-vocabulary detection recognizes novel categories while training uses annotations from seen categories; unlike zero-shot detection, it may encounter novel objects without instance-level annotations during training.The paper presents open-vocabulary detection as an alternative to labeling more categories, which may be unaffordable and exacerbate long-tail imbalance.Source: §1 · p. 1
  • One-stage methods distill from sparse positive sample points that cover base-category objects, while class-agnostic proposals in two-stage methods can cover novel objects.The authors argue that this limits one-stage semantic supervision from the PVLM and contributes to the performance gap.Source: §1 · p. 2
  • GKD aligns caption representations from the PVLM with global image representations, allowing sample points associated with novel categories to learn related semantics.Captions may contain semantic knowledge of novel categories, and GKD implicitly aligns all image sample points with caption semantics.Source: Figure 1 · p. 1, §1 · p. 2
  • HierKD combines global-level distillation with commonly used instance-level distillation in an end-to-end one-stage detector.The contributions also identify weakly supervised global-level language-to-visual distillation as a way to learn novel-category knowledge beyond training labels.Source: §1 · p. 2
  • 11.9% and 6.7% AP50 gains over the previous best one-stage detector under zero-shot and generalized zero-shot detection, respectively, on MS-COCO.The paper also frames narrowing the gap with two-stage methods as a goal.Source: §1 · p. 2

2. Related Work

Prior work spans zero-shot recognition and open-vocabulary detection, including two-stage and one-stage approaches. HierKD's framework draws on CLIP and hierarchical distillation for one-stage detection.

2. Related Work

  • Zero-shot learning studies recognition of novel categories beyond training sets, with recent PVLMs such as CLIP learning visual concepts from natural language.This paper focuses instead on transferring PVLM zero-shot recognition ability to open-vocabulary instance-level detection through distillation.Source: §2 · p. 2
  • HierKD uses a pre-trained CLIP teacher and a student detector, with instance-level distillation from positive sample points and global-level distillation aligning multi-layer features with captions.The framework initializes the classification branch with CLIP textual embeddings of base categories during training; at inference, distillation modules are removed and embeddings are initialized with novel categories.Source: Figure 2 · p. 3
  • Prior open-vocabulary detectors include two-stage approaches and one-stage methods using modified losses, transductive learning, or synthesized features for unseen objects.The cited work reports a large performance gap between an advanced two-stage method and prior one-stage methods.Source: §2 · p. 3

3. Approach

HierKD trains a student detector to learn from a pre-trained visual-language model through instance-level and global-level distillation. Its approach combines a modified ATSS detector, image-caption alignment, and a negative-sample strategy.

3.1. Choosing and Modifying a Base Detector

  • ATSS is adapted with CLIP text-embedded category classifiers, a learned background embedding, softmax focal loss, and an IoU branch replacing centerness.ATSS is chosen for adaptive sample selection and one anchor per feature-map location.Source: §3.1 · pp. 3–4

3.2. Instance-level Knowledge Distillation

  • Instance-level distillation transfers CLIP image-encoder features from predicted-box crops to positive detector samples filtered by an IoU threshold.Crops are resized to 224 × 224 using long-side resizing with padding.Source: §3.2 · p. 4

3.3. Global-level Knowledge Distillation

  • GKD matches image-caption pairs using pooled patches from multiple FPN layers and whole-caption features, with cross-attention and a symmetric contrastive loss.The caption serves as the query over patch features; paired images and captions are positives, while other batch pairs are negatives.Source: §3.3 · pp. 4–5, Figure 3 · p. 4

3.4. Sampling the Negative Samples

  • Sampling 10% of negative samples trades off improved novel-category recall against maintaining base-category performance.The paper contrasts this strategy with using all negatives or sampling negatives 1:1 with positives.Source: §3.4 · p. 5

3.5. Direct Inference Alternative with CLIP

  • CLIP direct inference compares detector and CLIP classification results for the same sample points to measure their difference.The alternative takes CLIP’s image and text encoders, novel categories, the trained model, and test images as inputs, and outputs detection boxes.Source: §3.5 · p. 5

4. Experiments and Results

Experiments evaluate HierKD on MS-COCO under ZSD and GZSD, examining proposal generalization, distillation choices, and comparisons with prior detectors. HierKD improves on the best prior one-stage detector across both reported base/novel splits, while upper-bound analysis identifies remaining room to improve its mimicking ability.

4.1. Dataset and Evaluation Protocol

  • The evaluation uses MS-COCO 2017, both 48/17 and 65/15 base/novel splits, and mAP and Recall@100 at IoU=0.5; implementation uses a frozen CLIP teacher.The reported focus is novel-category performance; the implementation uses ResNet-50 and divides FPN feature maps into 3 × 3 patches.Source: §4.1 · p. 5, §4.2 · pp. 5–6

4.3. Test on generalization ability of RPN

  • AR reaches 37.4 with 100 proposals at IoU=0.75 when an RPN trained on base categories is transferred to novel categories.The authors report that these category-agnostic proposals cover novel objects and contribute to feature learning during distillation.Source: §4.3 · p. 6, Table 1 · p. 6

4.4. Ablation Study

  • Replacing L1 with L2 in IKD drops AP50 by 1.8%, while 1.5× expanded crops improve performance by adding contextual information.The crop factor is not sensitive to using predicted or ground-truth boxes.Source: §4.4 · p. 6, Table 2 · p. 6
  • Adding IKD and GKD yields 4.4% and 10.5% AP50 gains, respectively; combining them brings further AR50 gains of 7.7% and 9.0% over the separate modules.HierKD improves APS and APM by 1.4% and 1.7% versus GKD; the IOU branch improves medium and large objects more than small ones.Source: §4.4 · p. 6, Table 4 · p. 6
  • HierKD recognizes novel-category examples missed or assigned low confidence by IKD, while GKD and HierKD identify more novel objects than IKD in qualitative detections.The visualizations also report improved classification of an elephant and suppression of multiple partial airplane detections by HierKD versus GKD.Source: §4.4 · p. 7, Figure 4 · p. 7

4.5. Comparison with the Start-of-the-Art

  • On the 48/17 base/novel split, HierKD scored 25.3% novel-category AP50 in ZSD, 11.9% above ZSD-YOLO; under GZSD, it gained 6.7% over ZSD-YOLO and narrowed ViLD’s AP50 gap from 14% to 7.3%.Source: §4.5 · p. 8
  • 10.1% and 2.5% AP50 gains over ZSD-YOLO on novel categories under ZSD and GZSD, respectively, on the 65/15 split.Source: §4.5 · p. 8, Table 7 · p. 8

4.6. Upper Bound Analysis

  • CLIP classification of ground-truth boxes provides an upper bound, and HierKD’s recall is relatively high while its total AP50 and size-specific AP remain below that bound.The authors identify room to improve HierKD’s mimicking ability and point to prompt learning as a way to improve CLIP’s zero-shot recognition ability.Source: §4.6 · p. 8, Table 9 · p. 8

5. Conclusion

HierKD uses image captions for language-to-visual distillation, transferring novel-category semantics from CLIP. The authors report more accurate and confident novel-object detection and performance that surpasses previous methods.

5. Conclusion

  • HierKD identifies novel objects more accurately and confidently and significantly surpasses previous methods.The authors plan to explore more efficient and advanced distillation methods to transfer teacher models’ zero-shot recognition ability.Source: §5 · p. 8
  • HierKD uses image captions for language-to-visual knowledge distillation in one-stage open-vocabulary detection.The rich vocabulary in captions enables transfer of novel-category semantic knowledge from CLIP during training.Source: §5 · p. 8

Related papers

Loading 2203.10593v1…