PaperlayerView exact revision on arXiv

Source-linked AI summary

F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models

Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, Anelia Angelova

arXiv:2209.15639v2cs.CV

TL;DR

Open-vocabulary detection must localize and recognize objects beyond the limited vocabularies supported by costly manual annotation. F-VLM freezes the VLM backbone, trains only a detector head, and combines detector and VLM region predictions at inference. It achieves strong LVIS results, competitive transfer detection, and substantial training efficiency gains, while the paper notes unresolved bias risks and open questions about more sophisticated finetuning.

  • Problem

    Costly bounding-box and mask annotation limits detection vocabularies, motivating detection beyond a fixed set of categories.

  • Method

    F-VLM trains only a detector head on frozen VLM features and combines detection scores with VLM region predictions at test time.

  • Results

    F-VLM surpasses the previous LVIS state of the art by 6.5 mask APr on novel categories and achieves competitive transfer detection with substantial compute savings.

  • Takeaways & Limitations

    Frozen VLMs can support open-vocabulary detection without knowledge distillation, detection-tailored pretraining, or weakly supervised learning.

  • Takeaways & Limitations

    The pretrained VLM knowledge may reinforce biases from Internet data, so the models require more rigorous probing for bias and fairness before other uses.

Abstract

from arXiv · show

We present F-VLM, a simple open-vocabulary object detection method built upon Frozen Vision and Language Models. F-VLM simplifies the current multi-stage training pipeline by eliminating the need for knowledge distillation or detection-tailored pretraining. Surprisingly, we observe that a frozen VLM: 1) retains the locality-sensitive features necessary for detection, and 2) is a strong region classifier. We finetune only the detector head and combine the detector and VLM outputs for each region at inference time. F-VLM shows compelling scaling behavior and achieves +6.5 mask AP improvement over the previous state of the art on novel categories of LVIS open-vocabulary detection benchmark. In addition, we demonstrate very competitive results on COCO open-vocabulary detection benchmark and cross-dataset transfer detection, in addition to significant training speed-up and compute savings. Code will be released at the https://sites.google.com/view/f-vlm/home

1 INTRODUCTION

Open-vocabulary detection addresses the limited vocabulary and costly annotation of conventional detectors. F-VLM uses frozen VLM features and a trained detector head to reduce training complexity while achieving strong benchmark performance.

  • Motivation: Manual bounding-box and mask annotation is costly, limiting modern detection vocabularies to roughly 10^3 categories.Open-vocabulary detection seeks to detect beyond a fixed category set.
  • Motivation: Frozen VLMs retain locality-sensitive features for object shapes and discriminative features for region classification.These observations motivate using frozen VLM features for accurate localization and open-vocabulary recognition.
  • F-VLM: F-VLM attaches a detector head to frozen VLM features, trains only that head, and combines detector scores with VLM predictions at test time.The approach avoids knowledge distillation, detection-tailored pretraining, and weakly supervised learning.
  • Evaluation: F-VLM is evaluated on LVIS, COCO, and Objects365 to assess open-vocabulary detection and transfer performance.The paper presents these benchmarks as evidence of the method's efficacy.
  • Results: F-VLM surpasses the previous LVIS open-vocabulary detection state of the art by 6.5 mask APr on novel categories and provides up to 200× training compute savings over ViLD.The method also shows consistent gains when increasing backbone capacity.
  • Implications: The paper argues that frozen VLMs warrant further exploration for a broader range of computer vision tasks.This is presented as a direction motivated by the reported findings.

2 RELATED WORK

Prior work develops open-vocabulary recognition and detection through image-text representations, region alignment, distillation, pretraining, prompt optimization, and weak supervision. F-VLM instead studies direct use of frozen VLMs within an open-vocabulary object detector.

  • Zero-Shot/Open-Vocabulary Recognition: Earlier zero-shot recognition methods represent categories with visual attributes or learn joint image-text embedding spaces.These approaches primarily address image-level recognition.
  • Object-Level Understanding: The paper distinguishes its object-level focus from prior image-level recognition and from frozen classification models used for closed-vocabulary detection.It also contrasts direct frozen VLM use with frozen VLMs serving as teachers for segmentation.
  • Zero-Shot/Open-Vocabulary Object Detection: Open-vocabulary detection methods align region features with word embeddings or synthesize visual features to detect categories absent from training data.These methods address the cost of scaling detection data and annotation.
  • Zero-Shot/Open-Vocabulary Object Detection: ViLD distills pretrained VLM representations, DetPro optimizes prompts, RegionCLIP performs region-text pretraining, and Detic uses weak supervision.These approaches represent distinct strategies for adapting VLM knowledge to detection.

3 METHOD

F-VLM builds open-vocabulary detection on a frozen VLM by training only a detector head, then combining detector and VLM region scores at inference. Its design uses text embeddings for expandable classification, region-level VLM pooling, and class-agnostic localization heads for novel objects.

  • 3 METHOD: F-VLM assumes detection labels for base categories during training and detects base and novel categories at test time using a pretrained VLM.The VLM is assumed to have learned from large-scale Internet image-text pairs.
  • 3.2 PRETRAINING FROM VISION AND LANGUAGE MODELS: Only the detector head is trained on frozen VLM backbone features, preserving the pretrained model’s open-vocabulary classification ability.The detector head uses a Mask R-CNN head and feature pyramid network and is randomly initialized.
  • 3.3 TEXT-EMBEDDING REGION CLASSIFIER: The detector classifier replaces a fixed K-way output layer with base-category text embeddings, which can be expanded to include novel categories at inference.Region embeddings are compared with category and background text embeddings using cosine similarity and a learnable temperature.
  • 3.4 OPEN-VOCABULARY RECOGNITION: At test time, F-VLM applies the frozen VLM pooling layer to ROI-aligned region features to obtain open-vocabulary region scores.The text embeddings include both base and novel categories, and whole-image regions recover zero-shot image-classification scores.
  • 3.4 OPEN-VOCABULARY RECOGNITION: F-VLM combines detector and VLM scores with a geometric mean because cropped VLM features are strong for recognition but insufficiently sensitive to localization quality.Separate weights control VLM contributions for base and novel categories, while the background score comes directly from the detector.
  • 3.5 OPEN-VOCABULARY LOCALIZATION: Class-agnostic box regression and mask prediction produce one box and mask per proposal across categories, enabling localization of novel objects.This design addresses the limitation of standard class-specific localization heads in open-vocabulary detection.

4 EXPERIMENTS

F-VLM evaluates open-vocabulary detection, transfer detection, training efficiency, and design choices using a frozen VLM backbone with a trainable detector head. Across benchmarks, it reports strong accuracy, scaling, transfer, and compute-efficiency results.

  • LVIS Benchmark: 32.8 APr is F-VLM's best LVIS system-level result, improving 14.2 APr over the R50 baseline.The model uses a fixed amount of trainable parameters while scaling frozen model capacity.
  • LVIS Benchmark: +6.5 mask APr over ViLD-EN-B7 is F-VLM's improvement on LVIS novel categories.The comparison is at system level and also yields +5.6 overall mask AP.
  • COCO Benchmark: F-VLM is competitive on COCO without detection-tailored pretraining, weak supervision, or knowledge distillation.It directly uses a frozen CLIP backbone and significantly surpasses RegionCLIP's CLIP-R50 pretrained version.
  • Training Resource Benchmark: 226× more compute efficiency and 57× faster wall-clock training are achieved versus ViLD-EN-B7 at system level.F-VLM reaches better performance with only 7.4 training epochs; pretrained VLM costs are excluded from the comparison.
  • Transfer Detection Benchmark: F-VLM transfers from LVIS to COCO and Objects365 by swapping vocabularies without finetuning.In transfer settings, β = 0 is optimal on COCO, while β = 0.3 to 0.4 is optimal on Objects365.
  • Transfer Detection Benchmark: +3.2/+5.9 and +4.9/+5.6 are F-VLM's gains over ViLD and DetPro on Objects365/COCO transfer detection.The best F-VLM closes the gap with a supervised model to -33% on COCO and -40% on Objects365.
  • Analyses and Visualization: Finetuning improves base-category detection but slightly hurts novel-category open-vocabulary detection.Whether more sophisticated finetuning strategies can help open-vocabulary detection remains open.
  • Analyses and Visualization: Geometric-mean score fusion is significantly better than arithmetic-mean fusion, with a +8 APr improvement.The study also finds β = 0.65 and α = 0.35 most beneficial, while larger feature pyramids improve base detection without compromising novel detection.

5 CONCLUSION

F-VLM is a simple open-vocabulary detector built on frozen VLMs, avoiding knowledge distillation, detection-tailored pretraining, and weak supervision. It achieves strong LVIS performance, efficient training, and competitive transfer detection.

  • F-VLM builds open-vocabulary detection on frozen VLMs without knowledge distillation, detection-tailored pretraining, or weakly supervised learning.
  • F-VLM offers significant training speedup and compute savings while achieving the new state-of-the-art on LVIS at system level.
  • F-VLM shows very competitive transfer detection and is intended to encourage exploration of frozen VLMs for broader vision tasks.

7 ETHICS STATEMENT

The paper demonstrates detection of previously unseen object categories on challenging benchmarks and transfer settings, while warning that internet-trained VLMs may propagate harmful biases or stereotypes.

  • The models demonstrate detection capabilities for previously unseen object categories on challenging benchmarks and transfer settings.
  • Because the models use information embedded in VLMs, they may reinforce deficiencies and biases in internet data.
  • The trained models are presented for evaluation and benchmarking and require more rigorous probing for bias and fairness before other uses.

A.1 FINETUNING VERSUS FROZEN BACKBONE

The analyses compare frozen and finetuned backbones, score-fusion choices, feature-pyramid capacity, and background weighting. Frozen backbones preserve novel-category performance, while geometric fusion and larger pyramids improve key outcomes.

  • Finetuning versus frozen backbone: Finetuning improves base-category detection but slightly compromises novel-category detection while increasing memory and compute requirements.
  • Score fusion: Geometric-mean fusion is more than 8 points better than arithmetic-mean fusion for combining VLM and detection scores.
  • Score fusion: β is the main tunable score-fusion parameter, whereas performance is relatively robust to α; T = 0.01 is recommended for most practical uses.
  • Feature pyramid capacity: Larger feature-pyramid capacity significantly improves base-category metrics without compromising novel-category APr.
  • Feature pyramid capacity: The enlarged feature pyramid produces improvements across all categories, including a slight 0.1 improvement on APr.
  • Background weight: A background weight of 0.9 is slightly better than the default 1.0, so it is used as the default setting.

B COMPUTATION-FRIENDLY TRAINING

F-VLM is evaluated under shorter, computation-friendly training recipes and shows robustness to schedule, batch size, and augmentation changes. The analyses also probe feature structure and background weighting.

  • Computation-friendly training: F-VLM is validated in computation-friendly 1× and 3× training settings using smaller batch sizes and no LSJ augmentation.
  • Computation-friendly training: F-VLM remains robust to training epochs, batch size, and LSJ choices for both the smallest and largest backbones.
  • Feature visualization: The study probes frozen VLM feature structure with k-means clustering using a CLIP R50x4 backbone and LVIS data.

D ANALYSIS OF TRANSFER DETECTION BENCHMARK

The transfer-detection benchmark has substantial vocabulary overlap with LVIS base categories, limiting how cleanly it measures transfer to novel categories. F-VLM is memory efficient and its frozen-backbone recipe leaves exploration of other VLM architectures open.

  • Benchmark limitations: Substantial vocabulary overlap exists between COCO and Objects365 categories and LVIS base categories, weakening the benchmark’s novelty.Simple name matching shows overlap, and removing near duplicates such as synonyms reveals even more overlap.
  • Benchmark limitations: COCO has more vocabulary overlap with LVIS base categories than Objects365 because COCO has a smaller vocabulary.
  • Efficiency: F-VLM’s memory consumption is almost the same as Mask R-CNN because its class-specific heads are replaced with class-agnostic heads.It also uses the same batch size as Gu et al. (2022) by default, while working with smaller batch sizes and shorter epoch lengths.
  • Efficiency: F-VLM can save substantial memory versus methods that fine-tune the backbone, especially when using large backbones.The frozen backbone avoids storing forward activations, gradients, and gradient moments during training.
  • Other VLMs: Extending F-VLM beyond CLIP to other VLMs remains open and may require substantial architectural modifications.ViT-based VLMs require single-scale ViT-based detectors, while multimodal encoders require efficient all-pair region-text similarity computation.

G IMPLEMENTATION DETAILS

The implementation uses benchmark-specific hyper-parameters, with shorter LVIS scheduling enabled by the frozen backbone and COCO settings adjusted to reduce overfitting. Visualizations use R50x4 and show detections across LVIS novel categories and Objects365 transfer scenes.

  • Hyper-parameters: LVIS uses Gu et al. (2022)’s hyper-parameters except for a shorter schedule and a background weight from prior work.The shorter schedule is attributed to the frozen backbone.
  • Hyper-parameters: COCO hyper-parameters are adjusted to mitigate overfitting to the 48-category ZSD-COCO split.This split is much smaller than the 800 LVIS base categories.
  • Hyper-parameters: R50x64 improves by 1.0 APr with T = 0.02 instead of T = 0.01, while R50 improves by 0.5 APr with gradient clipping at maximum norm 1.0.Reported performance uses the optimal settings for these two backbones.
  • Visualization: Figure 6 visualizes F-VLM detections on LVIS novel categories and Objects365 transfer detection using the R50x4 backbone.The examples include rare LVIS categories and complex Objects365 scenes.

I APPLICATION ON EGO-CENTRIC DATA

F-VLM is evaluated on Ego4D without Ego4D training, using user-provided categories after training on LVIS, Objects365, and COCO. It detects many novel objects despite domain shift and challenging viewing conditions.

  • Setup: F-VLM is applied to Ego4D after training on a mixture of full LVIS, Objects365, and COCO to expand its vocabulary.The model uses the R50x16 backbone for this application.
  • Setup: The Ego4D model is not trained on Ego4D, and users provide categories based on visual inspection of the video.
  • Setup: Ego4D indoor and grocery-store evaluations use user-specified category lists covering household objects, foods, signs, and store items.
  • Results: Figure 7 reports visualizations of F-VLM transfer detection on Ego4D.
  • Results: F-VLM detects many objects in ego-centric videos despite large domain shift and challenging viewing conditions.Figure 7 includes novel categories such as light switch, door lock, canned food, and burrito.
Loading 2209.15639v2…