Exploiting Unlabeled Data with Vision and Language Models for Object Detection

1 Introduction

• The approach addresses annotation cost and long-tailed category coverage by using unlabeled data for detection tasks that differ in their label regimes.SSOD pairs a small labeled set with unlabeled images; OVD trains on annotated base categories and evaluates on novel categories.Source: §1 · pp. 1–2
• The method uses a two-stage class-agnostic proposal generator, then CLIP assigns category probabilities to each region for task-specific pseudo labels.The proposal generator is trained with limited ground truth: known base categories for OVD and labeled images for SSOD.Source: §1 · pp. 2–3
• VL-PLM improves localization by combining CLIP and proposal-generator objectness scores and repeatedly applying the proposal generator's localization head to remove redundant proposals.The resulting pseudo labels are combined with the original ground truth to train the detector.Source: §1 · pp. 2–3
• VL-PLM outperforms ViLD on COCO novel categories by +6.8 AP and improves known-category semi-supervised detection over STAC by a clear margin.For SSOD, the comparison replaces STAC's pseudo labels with VL-PLM's.Source: §1 · p. 3
• The paper presents vision-and-language-guided pseudo-label mining as applicable to both OVD and SSOD, with state-of-the-art OVD results and benefits for SSOD.The contribution summary also identifies improved localization quality as a design contribution.Source: §1 · p. 3

2 Related Work

• Pre-trained models such as CLIP and ALIGN learn generic representations from large-scale image-text pairs, but have limited object-localization capability.The paper links that limitation likely to the lack of region-word alignment in their training pairs.Source: §2 · p. 3
• ViLD and LSeg map object or pixel features into CLIP’s joint vision-language space without explicit pseudo labels, whereas VL-PLM generates pseudo labels, enabling semi-supervised object detection [46].Source: §2 · p. 4
• Unlike, VL-PLM needs no image captions and outperforms it on COCO OVD; unlike RegionCLIP, it trains detectors directly with pseudo labels.The paper describes RegionCLIP's contribution as orthogonal and potentially incorporable into VL-PLM.Source: §2 · p. 4
• VL-PLM mines unlabeled images for both novel OVD categories and known SSOD categories, while its pseudo-label framework may extend to semantic segmentation.The paper leaves that extension to future work.Source: §2 · p. 4
• For incomplete annotations, prior methods use an initial model to infer missing information; VL-PLM uses vision-and-language models to improve pseudo labels across OVD and SSOD.The paper focuses on these two settings and leaves broader applications for future work.Source: §2 · pp. 4–5

3.1 Training object detectors with unlabeled data

• Detector training combines supervised loss on labeled images with weighted unsupervised loss on unlabeled images, using high-confidence pseudo labels as supervision.The supervised loss includes classification and box-regression terms.Source: §3.1 · p. 6

3.2 VL-PLM: Pseudo labels from vision & language models

• VL-PLM scores class-agnostic proposals against task-selected CLIP text categories, then combines V&L and proposal scores to produce pseudo labels.The selected category names can be adjusted for the downstream task.Source: §3.2 · p. 7, Figure 2 · p. 7
• RPN scores correlate positively with IoU (0.51); combining RPN and CLIP scores and repeatedly applying the RoI head improves pseudo-label localization.The proposal generator's thresholding and NMS are removed before repeated RoI-head refinement, which pushes redundant boxes closer together.Source: Figure 3 · p. 8, §3.2 · pp. 8–9
• A class-agnostic two-stage detector trained on available annotations proposes regions intended to cover both seen and novel categories.For open-vocabulary detection, the available annotations are for base categories; for semi-supervised detection, they come from labeled images.Source: §3.2 · p. 8

3.3 Using our pseudo labels for downstream tasks

• For open-vocabulary detection, novel-category names guide CLIP and novel pseudo labels train alongside base ground truth; semi-supervised detection merges VL-PLM and teacher pseudo labels.The resulting student model is trained with available ground truth and pseudo labels from both sources.Source: §3.3 · p. 9

4.1 Open-vocabulary object detection

• VL-PLM outperforms prior state-of-the-art ViLD by nearly +7% in Novel AP on COCO open-vocabulary detection.The comparison is reported in Table 1.Source: Table 1 · p. 10, §4.1 · pp. 10–11
• On VOC 2007, Object365, and LVIS, VL-PLM significantly outperforms with similar iterations and smaller batch sizes.Unlike, VL-PLM generates pseudo labels for any given category without requiring additional image captions.Source: Table 2 · p. 11, §4.1 · p. 11

4.2 Semi-supervised object detection

• Models using VL-PLM pseudo labels outperform Supervised+PLs and STAC, with larger gains reported when less labeled data is available.The authors attribute the comparison to adding VL-PLM pseudo labels; Table 3 evaluates pseudo labels for SSOD on COCO.Source: Table 3 · p. 12, §4.2 · p. 12

4.3 Analysis of pseudo label generation

• RPN score fusion improves pseudo-label quality, while repeated RoI refinement improves it until convergence at about 10 steps; Selective Search and RoI-only proposals have reported quality drawbacks.Table 5 compares proposal sources at similar pseudo-label counts; the text attributes Selective Search's low AP@PL to missing objectness scores and excess noisy boxes, and RoI-only's poor quality to novel objects being classified as background.Source: Figure 4 · p. 13, §4.3 · pp. 13–14, Table 5 · p. 14
• A good balance between pseudo-label AP@PL and count matters: high thresholds reduce label count but also degrade AP@PL and final-detector performance, with τ = 0.8 reported as a good trade-off.Table 4 relates pseudo-label quality to final open-vocabulary detector performance.Source: Table 4 · p. 13, §4.3 · p. 13
• VL-PLM takes 0.54 seconds per image with ResNet50; Fast VL-PLM cuts runtime 5× with a slight performance drop, while Multiscale Fast VL-PLM cuts it 3× without that drop.Multithreading on 8 GPUs can generate pseudo labels for COCO's training set within 6 hours.Source: §4.3 · p. 14

5 Conclusion

• VL-PLM mines unlabeled images with pre-trained vision-and-language models to generate task-specific pseudo labels for object detection.The paper demonstrates the approach for open-vocabulary and semi-supervised object detection.Source: §5 · p. 14
• Training a standard detector with VL-PLM pseudo labels sets a new state-of-the-art for open-vocabulary detection on COCO.Source: §5 · p. 14
• VL-PLM pseudo labels benefit semi-supervised models, especially when ground-truth labels are limited.The paper identifies better vision-and-language models as a possible direction for further improvement.Source: §5 · p. 14

Supplemental Materials

• Supplemental experiments cover LVIS open-vocabulary detection, faster VL-PLM variants, and semi-supervised detection.Source: Supplemental Materials · p. 19
• Supplemental analyses examine semi-supervised pseudo-label fusion, pseudo-label quality, background modeling, and proposal-generator generalization.Source: Supplemental Materials · p. 19
• The supplement also presents qualitative results for pseudo labels and the final open-vocabulary detector.Source: Supplemental Materials · p. 19

A.1 Open-vocabulary detection results on LVIS

• LVIS is a 1,203-category dataset sharing images with COCO; the evaluation reports box mAP and follows ViLD’s setup.The supervised baseline uses repeat factor sampling; ViLD was re-evaluated for a fair comparison.Source: §A.1 · p. 19
• VL-PLM uses frequent and common LVIS categories as base classes to train its proposal generator, then generates pseudo labels for rare novel categories.A standard detector is trained with base-category ground truth and the generated pseudo labels.Source: §A.1 · p. 19
• +0.6 APr for rare categories, +2.6 APc for common categories, and +3.5 APf for frequent categories; VL-PLM outperforms ViLD on all LVIS splits.Source: §A.1 · p. 19, Table 6 · p. 19
• VL-PLM also outperforms the supervised baseline by a large margin in APr, and may exceed it in mAP because its noisy pseudo labels add annotations beyond LVIS’s federated annotations.The authors suggest these additional annotations can provide useful information about rare-category objects, such as their texture.Source: §A.1 · p. 20

A.2 Fast VL-PLM and Multi-scale Fast VL-PLM

• Fast VL-PLM reduces runtime by 5× with a slight pseudo-label quality drop; Multi-scale Fast VL-PLM reduces runtime by 3× while almost entirely removing that drop.Source: §A.2 · p. 20, Table 7 · p. 20
• Fast VL-PLM computes shared ResNet50 feature maps, crops proposal features, and uses CLIP’s last attention layer to produce region embeddings.Unlike ROI pooling, crop dimensions remain proportional to each proposal; the authors report worse performance with ROI pooling.Source: §A.2 · p. 20
• Multi-scale Fast VL-PLM assigns proposal crops to image-pyramid feature maps by area, using larger scales for smaller proposals.The smallest input dimension is resized to 224, 672, or 1120 pixels, and proposal-area thresholds determine the selected scale.Source: §A.2 · pp. 20–21

A.3 Scaling up unlabeled images for SSOD

• With 5% labeled COCO images, detection performance rises as unlabeled images increase, but with diminishing returns.Source: §A.3 · p. 21, Table 8 · p. 21

B.1 Fusing Pseudo Labels from SSOD teacher with VL-PLM

• For semi-supervised detection, VL-PLM pseudo labels are merged with teacher labels, then thresholded and passed through NMS.Source: §B.1 · p. 21
• Fusing VL-PLM and teacher pseudo labels outperforms either source alone on all tested COCO splits.VL-PLM alone beats teacher labels at 1% and 2% labeled data but trails them at 5% and 10%; the authors describe the two sources as complementary.Source: §B.1 · p. 22, Table 9 · p. 22

B.2 Analysis on the Quality of PLs

• PL v4 yields much better novel-category detector results than PL v1 despite similar AP@PL, while PL v1 produces far more annotations.The analysis concludes that neither AP@PL nor #@PL alone determines pseudo-label quality; both matter.Source: §B.2 · p. 23, Table 10 · p. 23
• An extremely high threshold harms novel-category predictions but slightly improves base-category performance; the authors select τ = 0.8 within a reasonable range of [0.6, 0.95].Source: §B.2 · p. 23
• RoI-head refinement significantly improves pseudo labels: PL v4 gives similar base-category but much better novel-category detector results than PL v2.Source: §B.2 · p. 23

B.3 Modeling Background in PL Generation

• Novel+BG slightly improves AP@PL and lowers #@PL versus Novel; Novel+Base clearly improves PL quality, while Novel+OV decreases it versus Novel+Base.The authors suggest Novel+Base helps V&L models identify base-category objects that are removed as background.Source: Table 11 · p. 23, §B.3 · p. 24

• On COCO-ZS, an RPN trained without novel categories has lower novel-category recall but still reaches reasonable AR@1000; on LVIS, recall is nearly the same with base-only or base-plus-novel training.The authors suggest LVIS's larger number of base categories and instances may help the proposal generator learn a better concept of objects.Source: §B.4 · pp. 24–25, Table 12 · p. 25

C.1 Visualization of Our Pseudo Labels

• VL-PLM locates multiple novel instances in good cases, but errors include part domination, redundant, missing, and grouped boxes; close or partial instances can drive missing and grouped cases.Part domination and redundant boxes are attributed mainly to CLIP's poor localization; incorrect region proposals may also degrade pseudo-label quality.Source: §C.1 · pp. 25–26, Figure 7 · p. 27

C.2 Visualization of Our OVD Detector

• The final detector detects novel-category objects, with missing, redundant, and grouped boxes among its main failures; part domination may be reduced during training with ground truth and pseudo labels.Source: §C.2 · p. 26, Figure 8 · p. 28