Object-Aware Distillation Pyramid for Open-Vocabulary Object Detection

1. Introduction

• Prior KD-based OVD may crop away object parts or add distractors, while proposal-only transfer misses relations among objects and scene context.Source: §1 · pp. 1–2
• OAKE adaptively resizes proposals and uses an object token attending only to covered patch tokens to extract precise, complete object knowledge.Source: §1 · p. 2
• DP combines object distillation with global scene and block-level distillation to transfer contextual and more diversified knowledge from CLIP.Source: §1 · p. 2
• On MS-COCO, OADP improves the state-of-the-art mAPN50 result from 32.3 to 35.6, demonstrating stronger performance on this dataset under the reported evaluation.Source: §1 · p. 2
• On LVIS, OADP reaches 21.9 APr for detection and 21.7 APr for instance segmentation, exceeding prior methods by more than 1.1 and 1.9 APr, respectively.Source: §1 · p. 2

Knowledge Distillation for Object Detection

• Knowledge distillation trains compact students under powerful teachers; Chen et al. use feature- and response-based losses for Faster R-CNN.Source: §2 · p. 2
• Detection KD methods target proposal features, foreground regions, foreground and background, disagreements, detector-wide frameworks, or attention maps; FGD combines focal and global distillation.Source: §2 · p. 2

Open-Vocabulary Detection

• PVLM-era OVD includes CLIP distillation in ViLD, prompt optimization in DetPro, and region-text alignment in RegionCLIP; OADP targets comprehensive knowledge extraction and transfer.Source: §2 · p. 2

3. OVD Benchmarks

• The benchmarks are V-OVD, C-OVD, G-OVD, and WS-OVD; base categories appear in instance-level annotations, while novel categories do not.Source: §3 · p. 2
• V-OVD trains on a fixed-category object-detection dataset without novel-category information, while allowing unannotated data and image-text pairs for learning open-vocabulary knowledge.Source: §3 · pp. 2–3
• C-OVD adds in-domain image captions to V-OVD; these captions enrich instance annotations and can indicate potential novel categories.Source: §3 · p. 3
• G-OVD adds human priors about novel categories, often using pseudo annotations; because category names are known during training, its results may not be fairly comparable with V-OVD or C-OVD.Source: §3 · p. 3
• WS-OVD adds image-level labels indicating base- and novel-category presence, at a higher annotation cost than the other described settings.Source: §3 · p. 3

4.1. Preliminaries

• OVD trains detectors on annotated base categories while evaluating their ability to detect novel categories described by text.The category space is divided into base categories in training data and novel categories appearing only in validation data.Source: §4.1 · p. 3

4.2. Overview of OADP

• OADP combines object, block, and global distillation losses with the Faster R-CNN detection loss during training.The block and global modules are not used at inference, leaving inference computation the same as ViLD-ensemble.Source: §4.2 · p. 4

4.3. Object Distillation

• OAKE squares proposals with an adaptive transformation, then uses masked attention so an [OBJ] token attends to patches covered by the original proposal.The transformed regions retain object structure, while the token's restricted attention suppresses surrounding contextual distractors.Source: §4.3 · pp. 4–5

4.4. Global and Block Distillation

• Global distillation transfers whole-image CLIP knowledge, while block distillation gives image regions equal-weight supervision to complement information CLIP may ignore.The block pathway partitions the image into fixed-size regions and extracts their embeddings using block pooling and a block head.Source: §4.4 · p. 5

4.5. Pseudo Label Generation

• For pseudo labels, OADP scores proposals using category probability and objectness to reflect localization quality, then applies class-wise NMS to novel categories.The probability is computed over all categories, including base categories, to suppress false positives even though pseudo labels exclude base-category instances.Source: §4.5 · p. 5
• Pseudo-label proposals come from an RPN pretrained on base-category annotations, whose generalization ability is reported as sufficient to recall most novel-category objects.The accompanying tables compare methods on OV-COCO and OV-LVIS, respectively.Source: §4.5 · pp. 5–6, Table 2 · p. 6, Table 3 · p. 6

5.1. Datasets

• The experiments use OV-COCO with 48 base and 17 novel categories, and OV-LVIS with 866 base and 337 rare categories treated as novel.OV-COCO uses mAPN50 as its main metric; OV-LVIS reports APr, APc, APf, and AP for detection and instance segmentation.Source: §5.1 · p. 6

5.2. Implementation Details

• Training uses a ResNet-50 student and ViT-B/32 CLIP teacher, with 40,000 OV-COCO iterations and a 2× (24-epoch) OV-LVIS schedule.The setup uses eight V-100 GPUs and SGD; learning rates are divided by 10 at iteration 32,000 for OV-COCO and epochs 16 and 22 for OV-LVIS.Source: §5.2 · p. 6

5.3. Main Results

• On G-OVD, training OADP on mixed data—novel-category pseudo labels merged with base-category instance annotations—yields 35.6 mAPN50, surpassing VL-PLM by 3.3 mAPN50.Source: §5.3 · pp. 6–7
• 21.9 APr for OV-LVIS object detection surpasses DetPro by 1.1 APr, and 21.7 APr for instance segmentation exceeds the previous state of the art by 1.9 APr.The reported OV-LVIS comparisons use the V-OVD benchmark.Source: §5.3 · p. 7

5.4. Ablation Study

• In the ablation against the re-implemented ViLD-ensemble baseline, adding MG, MB, and MO individually yields gains of 0.73, 2.26, and 2.24 mAPN50, respectively; all three together reach 29.95 mAPN50.Source: Table 4 · p. 7, §5.4 · p. 7
• Adaptive and fixed enlarged crops underperform other crop strategies without mask attention, while the object-aware encoder improves both strategies.The authors attribute the gains to mask attention focusing the CLIP visual encoder on the proposal object despite surrounding distractors.Source: §5.4 · p. 7, Table 5 · p. 7
• The pseudo-label ablation evaluates novel-category mAP and average labels per image, and reports the highest mAP when label quantity is sufficient.The baseline’s classification-score-only ranking does not accurately localize objects without an objectness score.Source: §5.4 · p. 8, Table 6 · p. 8

5.5. Visualization

• OADP’s activation map highlights more of the novel object “cup” than the baseline in the illustrated example.The figure encodes pseudo labels in green, ground-truth annotations in blue, and feature-response intensity from blue to red.Source: §5.5 · p. 8, Figure 3 · p. 8

• OADP outperforms previous methods on two popular open-vocabulary object detection benchmarks.Source: §6 · p. 8
• Object-Aware Knowledge Extraction adaptively transforms proposals to preserve complete object representations and extract precise object knowledge.Source: §6 · p. 8
• The Distillation Pyramid transfers contextual knowledge about relations among objects to support better scene understanding.Source: §6 · p. 8