PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-modal Queried Object Detection in the Wild

Yifan Xu, Mengdan Zhang, Chaoyou Fu, Peixian Chen, Xiaoshan Yang, Ke Li, Changsheng Xu

arXiv:2305.18980v2cs.CV

TL;DR

Language queries support open-set object detection but can underdescribe ambiguous and fine-grained categories. MQ-Det fuses visual exemplars with text queries and reports gains in finetuning-free and few-shot detection, including improved LVIS performance without downstream fine-tuning.

  • Problem

    Language-queried detection supports open-set concepts, but simple prompts can underdescribe ambiguous or fine-grained categories.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    MQ-Det fuses class-wise visual cues into category text with a gated perceiver and uses vision-conditioned masked language prediction to modulate frozen detectors.

    Source: §2.2 · p. 4, §2.3 · pp. 4–5

  • Results

    MQ-Det improves finetuning-free GLIP detection by +7.8% AP on LVIS without downstream finetuning and also reports gains in few-shot settings.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results suggest that using the complementary nature of language and vision is an important step toward open-world object detection.

    Source: §5 · p. 10

    Multi-modal queries contribute less when each category has sufficient training data, and applications to segmentation remain unexplored.

    Source: C Further discussion · p. 15

Abstract

from arXiv · show

We introduce MQ-Det, an efficient architecture and pre-training strategy design to utilize both textual description with open-set generalization and visual exemplars with rich description granularity as category queries, namely, Multi-modal Queried object Detection, for real-world detection with both open-vocabulary categories and various granularity. MQ-Det incorporates vision queries into existing well-established language-queried-only detectors. A plug-and-play gated class-scalable perceiver module upon the frozen detector is proposed to augment category text with class-wise visual information. To address the learning inertia problem brought by the frozen detector, a vision conditioned masked language prediction strategy is proposed. MQ-Det's simple yet effective architecture and training strategy design is compatible with most language-queried object detectors, thus yielding versatile applications. Experimental results demonstrate that multi-modal queries largely boost open-world detection. For instance, MQ-Det significantly improves the state-of-the-art open-set detector GLIP by +7.8% AP on the LVIS benchmark via multi-modal queries without any downstream finetuning, and averagely +6.3% AP on 13 few-shot downstream tasks, with merely additional 3% modulating time required by GLIP. Code is available at https://github.com/YifanXu74/MQ-Det.

1 Introduction

MQ-Det combines language queries with visual exemplars to address the limited granularity of text prompts while retaining their open-set generalization. Its plug-in architecture and training strategy improve finetuning-free and few-shot detection.

1 Introduction

  • Text queries represent broad concepts but can lack granularity for homonyms and fine-grained categories [4] [9] [32].Adding longer textual descriptions is laborious, may make model understanding harder, and adds computational overhead.Source: §1 · p. 1
  • Visual exemplars offer richer object clues, while human-generated text has greater information density and stronger generalization capability [32] [8].Directly fine-tuning with limited visual exemplars risks catastrophic forgetting.Source: §1 · p. 2
  • MQ-Det interleaves a gated class-scalable perceiver into the text encoder to fuse class-wise visual cues with text, without class-specific parameters.The module conditionally gates visual cues and adds them to text representations; vision-conditioned masked language prediction addresses learning inertia while the foundation detector stays frozen and only GCP modules are trained.Source: §1 · p. 2
  • MQ-Det improves GLIP by +7.8% AP on finetuning-free LVIS detection and averages +6.3% AP across 13 downstream few-shot tasks.The LVIS result uses five visual exemplars with textual category descriptions; modulation takes 3% of GLIP's total pre-training time.Source: §1 · p. 2
  • The approach is designed as a plug-in for existing language-queried detectors and combines visual detail with generalizable text cues.The reported modulation uses one epoch on Objects365 and 3% of GLIP's total pre-training time.Source: §1 · p. 2

2 Methodology

MQ-Det adds class-wise visual queries to language-queried detection while retaining the detector’s text generalization. Its plug-in perceivers and modulated pre-training are designed to fuse visual details into frozen detectors.

2.1 Preliminaries

  • Language-queried detectors such as GLIP align image regions with category content in text prompts, but prompts can limit performance for ambiguous or fine-grained names.The method section frames MQ-Det as augmenting these models with visual queries.Source: §2.1 · p. 3

2.2 Architecture design

  • MQ-Det interleaves class-wise visual cues with text through a plug-in design, unlike prompt tuning or heavily coupled two-branch few-shot architectures.The design targets class scalability, complementary visual detail, and retention of generalization.Source: §2.2 · pp. 3–4, Figure 2 · p. 5
  • GCP modules cross-attend each category token to its corresponding vision queries and gate the visual residual according to exemplar quality.The gate is initialized to zero, so the module initially matches the pre-trained text encoder; the class-wise scheme has no class-specific parameters and supports variable numbers of vision queries at inference.Source: Figure 1 · p. 3, §2.2 · p. 4

2.3 Modulated pre-training

  • Modulated pre-training freezes the detector and trains only the newly added GCP modules using a small subset of pre-training data.This strategy is intended to enable rapid learning while retaining the detector’s general semantic representations.Source: §2.3 · pp. 4–5
  • +4.4% AP over GLIP-T on finetuning-free LVIS is reported when language masking prompts visual-query use during modulated pre-training.A ground-truth text token for an instance in the image is randomly masked with 40% probability, addressing learning inertia in the frozen residual architecture.Source: §2.3 · p. 5

3 Experiments

MQ-Det evaluates multi-modal queries on LVIS and ODinW in finetuning-free, few-shot, and full-shot settings. Results show strong finetuning-free and few-shot gains, while ablations examine query quality, masking, gating, and detector freezing.

3.1.1 Datasets and benchmarks

  • LVIS covers 1,203 long-tail categories, while ODinW-13 and ODinW-35 test transfer across real-world datasets, including fine-grained fish detection and pothole detection.Objects365, with 0.66 million images and 365 categories, is used for modulated pre-training.Source: §3.1.1 · pp. 5–6

3.1.2 Implementation details

  • The study applies MQ-Det to GLIP and GroundingDINO, inserts GCP in the last six layers of the frozen text encoder, and modulates for one epoch on Objects365.For finetuning-free evaluation, five instances per category are extracted as vision queries without finetuning.Source: §3.1.2 · p. 6

3.2.1 Multi-modal queried detection without finetuning

  • 34.7% AP on LVIS Val v1.0 gives MQ-GLIP-L a +7.8% gain over GLIP-L with five visual exemplars and no downstream finetuning.MQ-GLIP-T uses 12% data and 2% additional training time when modulated upon GLIP-T; MQ-GLIP-T-Img, using vision queries alone, reaches 17.6% AP.Source: §3.2.1 · p. 6, Table 1 · p. 6

3.2.2 MQ-Det as a strong few-shot learner

  • On ODinW-13, MQ-GLIP-T improves over GLIP-T by +3.7% AP finetuning-free and +6.3% AP few-shot, but only +0.6% AP full-shot.MQ-GLIP-T also reaches 43.0% AP on ODinW-35 few-shot; the few-shot evaluation uses 3-shot data.Source: Table 2 · p. 7, §3.2.2 · p. 8

3.3 Ablation studies

  • At a 0% mask rate, MQ-GLIP-T shows a -3.5% AP degradation on LVIS and improves GLIP-T by only +0.9% AP; an 80% rate may overfit vision queries.Source: §3.3 · p. 8
  • Positive vision queries improve detection, hard positives with partial information or noise can still help, and negative queries do not benefit detection.The experiments vary query quality while always supplying language queries for all 1,203 LVIS categories.Source: §3.3 · p. 8, Figure 4 · p. 8
  • On LVIS, rough visual-cue addition and a shared gate reduce AP by -3.3% and -1.1%, respectively, relative to MQ-GLIP-T.A content-aware gate adjusts to visual-query quality; a simple linear gate performs slightly worse than the nonlinear MLP design.Source: §3.3 · pp. 8–9
  • Unfreezing the detector or modulating only its text encoder reduces LVIS AP by -6.0% and -3.8%, respectively, and training without freezing nearly doubles time.The plug-and-play design is described as compatible with language-queried detectors that share GLIP- or GroundingDINO-like parallel architectures.Source: §3.2.1 · p. 7, §3.3 · p. 9

4 Related work

Related work contrasts dataset-specific open-vocabulary detection with large-scale fundamental detectors, then motivates MQ-Det as a way to combine language’s open-set generalization with vision queries’ fine-grained cues.

4 Related work

  • Open-vocabulary detection trains on base classes and evaluates on novel classes, but its generalization is limited to specific datasets.The passage contrasts this pipeline with large-scale fundamental detection models targeting arbitrary categories.Source: §4 · p. 9
  • Large-scale fundamental detectors use extensive region-level image-text data to generalize to arbitrary categories, with GroundingDINO achieving state-of-the-art open-set detection performance.The passage describes MDETR, GLIP, and GroundingDINO as examples of this pipeline.Source: §4 · p. 9
  • Two-branch few-shot methods use vision queries but lack language instruction; prior joint-query methods do not support joint inputs or perform well with vision queries.The passage reports no more than 10% AP in 3-shot COCO for current few-shot state-of-the-art methods.Source: §4 · p. 9
  • MQ-Det combines language and vision queries to seek both open-set generalization and fine-grained recognition.The passage characterizes the two query types as complementary.Source: §4 · pp. 9–10

5 Conclusion

MQ-Det supports text interleaved with images as queries for vision-language detection foundation models. The appendix describes downstream transfer and alternative vision-query acquisition strategies, including test-time updating.

5 Conclusion

  • MQ-Det is an efficient pre-training architecture and strategy design that lets vision-language detection foundation models accept text interleaved with images as queries.The conclusion identifies the complementary nature of language and vision as an important step toward open-world object detection.Source: §5 · p. 10

Appendix

  • The appendix covers transfer to finetuning-free, few-shot, and full-shot settings, customized tuning, additional experiments, and limitations and broader impacts.It also lists results on 35 ODinW downstream tasks and an open-vocabulary evaluation.Source: Appendix · p. 13

A.1 Different ways to acquire vision queries

  • Finetuning-free evaluation uses five training-set instances per category as vision queries, with retrieval and test-time online updating offered as alternatives.Retrieval uses five ImageNet-21K exemplars per category; online updating stores high-confidence detections during evaluation.Source: §A.1 · pp. 13–14
  • Across three ODinW datasets, the three vision-query acquisition approaches show similar performance; retrieved ImageNet exemplars improve performance, while online updating shows a slight drop.The passage attributes the online-update drop to lower-quality queries without manual annotations and retrieval gains to object-centric, low-noise exemplars.Source: §A.1 · p. 14

A.2 Test-time online update

  • Online updating begins with language-only detection, stores high-confidence instances as category vision queries, then uses both query types to find more exemplars.The passage points to Table I as verification of the approach.Source: §A.2 · p. 14

A.3 Partial tuning rivals full-model tuning

Partial tuning updates only the added GCP modules while freezing the rest of the detector, and performs comparably to full-model tuning.

A.3 Partial tuning rivals full-model tuning

  • -0.4 % AP on ODinW-13 separates partial tuning from full-model tuning.The passage characterizes the partial-tuning strategy as achieving comparable performance.Source: §A.3 · p. 14
  • Partial tuning updates only the newly added GCP modules and freezes all other parameters.Source: §A.3 · p. 14
  • Partial tuning uses fewer learnable parameters and is described as friendlier to training time and memory costs.The main-text experiments all use partial tuning.Source: §A.3 · p. 14

B.1 All results

The results section reports per-dataset performance for ODinW-13 and ODinW-35, with additional visualizations; the appendix also identifies a highlighted fine-tuning strategy table.

B.1 All results

  • Table II covers MQ-Det fine-tuning strategies under the 5-shot setting, with the evaluation implementation highlighted in color.Source: Table II · p. 14
  • Per-dataset performance is reported for ODinW-13 and ODinW-35 in Tables III and VII, respectively.Source: §B.1 · p. 14
  • Visualized MQ-Det results are provided in Figure I.Source: §B.1 · p. 14

B.2 Explicit evaluation on an open-vocabulary detection setting

The finetuning-free open-vocabulary evaluation finds higher AP on novel LVIS categories for all three MQ-Det detectors than for their baselines.

B.2 Explicit evaluation on an open-vocabulary detection setting

  • +4.1%, +5.7%, and +6.3% AP on novel LVIS classes for MQ-GroundingDINO-T, MQ-GLIP-T, and MQ-GLIP-L, respectively, over their baselines.The comparison uses five vision queries and categories absent during modulated pre-training.Source: §B.2 · p. 14
  • The evaluation separates 986 novel LVIS categories from 217 base categories among 1,203 categories.Categories present in Objects365 were removed from the novel set.Source: §B.2 · p. 14
  • Table IV presents finetuning-free detection results under explicit open-vocabulary category separation on LVIS.The evaluation uses five vision queries.Source: §B.2 · p. 14, Table IV · p. 15

B.3 Training hyper-parameters

The modulated pre-training hyperparameters are reported in Table VI, while other settings match the corresponding language-queried detectors.

B.3 Training hyper-parameters

  • The modulated pre-training hyperparameter settings are reported in Table VI.Source: §B.3 · p. 15
  • Settings other than the reported hyperparameters are the same as those of the corresponding language-queried detectors.Source: §B.3 · p. 15
  • The section specifies hyperparameters for modulated pre-training rather than changing the corresponding detectors’ other settings.Source: §B.3 · p. 15

C Further discussion

The discussion identifies limits in settings with ample category training data and in untested segmentation applications, and notes potential misuse of open-set detection.

C Further discussion

  • MQ-Det contributes less when each category has sufficient training data, which the authors suggest may reflect already accurate foundation-model classification boundaries.Source: C Further discussion · p. 15
  • Applications to dense prediction tasks such as segmentation remain unexplored.Source: C Further discussion · p. 15
  • Large-vocabulary MQ-Det models could be misused for illegal video surveillance, and specialized cues could promote targeted detections and unfair predictions.Source: C Further discussion · p. 16
  • Table VII reports per-dataset AP on ODinW-35 for zero- and few-shot detection, with zero-shot denoting finetuning-free inference using five vision queries.Source: Table VII · p. 17
Loading 2305.18980v2…