PaperlayerView exact revision on arXiv

Source-linked AI summary

LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models

Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, Wei-Shi Zheng

arXiv:2501.18954v1cs.CV

TL;DR

Open-vocabulary detectors have largely relied on abundant region-level annotations and short captions, leaving the value of detailed whole-image captions underexplored. LLMDet addresses this gap by pairing grounding supervision with LLM-generated image- and region-level captions through GroundingCap-1M. It reports improved open-vocabulary detection and shows that the resulting detector can help build a stronger large multimodal model.

  • Problem

    Prior methods mainly use coarse, independent region captions, while the benefits of detailed image-level captions for open-vocabulary detectors remain to be explored.

  • Method

    LLMDet trains an open-vocabulary detector with grounding and caption-generation objectives on GroundingCap-1M, using an LLM to generate image-level long captions and region-level short phrases.

  • Results

    LLMDet outperforms the baseline by 3.3%/3.8%/14.3% AP and 3.1%/3.3%/17.0% APr on LVIS minival with Swin-T/B/L backbones and achieves state-of-the-art performance across a wide range of benchmarks.

  • Takeaways & Limitations

    The improved LLMDet can serve as a vision foundation model for building a stronger large multimodal model, producing mutual benefits.

  • Takeaways & Limitations

    The co-trained LLM tends to produce relatively short whole-image descriptions, and region-level descriptions remain simple grounding phrases.

Abstract

from arXiv · show

Recent open-vocabulary detectors achieve promising performance with abundant region-level annotated data. In this work, we show that an open-vocabulary detector co-training with a large language model by generating image-level detailed captions for each image can further improve performance. To achieve the goal, we first collect a dataset, GroundingCap-1M, wherein each image is accompanied by associated grounding labels and an image-level detailed caption. With this dataset, we finetune an open-vocabulary detector with training objectives including a standard grounding loss and a caption generation loss. We take advantage of a large language model to generate both region-level short captions for each region of interest and image-level long captions for the whole image. Under the supervision of the large language model, the resulting detector, LLMDet, outperforms the baseline by a clear margin, enjoying superior open-vocabulary ability. Further, we show that the improved LLMDet can in turn build a stronger large multi-modal model, achieving mutual benefits. The code, model, and dataset is available at https://github.com/iSEE-Laboratory/LLMDet.

1. Introduction

LLMDet addresses the limits of short region-level supervision by adding detailed image-level captioning to open-vocabulary detection. It introduces GroundingCap-1M and reports stronger detection and multimodal-model performance under LLM supervision.

  • Open-vocabulary detection targets arbitrary user-provided text classes, making it more general than closed-set detection.
  • Prior language-task extensions mainly use coarse, independent region captions, motivating detailed image-level captions that capture richer image information.These captions can describe object attributes, actions, locations, texts, backgrounds, and relations among objects.
  • LLMDet jointly trains standard grounding and caption-generation objectives, with an LLM producing image-level long captions and region-level short phrases.
  • GroundingCap-1M pairs grounding annotations with detailed whole-image captions to provide the supervision needed for LLMDet.
  • 3.3%/3.8%/14.3% AP and 3.1%/3.3%/17.0% APr improvements over the baseline are reported on LVIS minival with Swin-T/B/L backbones.
  • The improved LLMDet can be integrated with an LLM to build a stronger large multimodal model, yielding mutual benefits.

2. Related Work

Related work establishes open-vocabulary detection as a vision-language task and shows that language supervision can improve visual representations. LLMDet is motivated by the remaining lack of detailed, image-level captions in prior detector training.

  • Open-vocabulary detectors aim to recognize arbitrary test-time classes from text labels despite limited training classes.
  • Existing approaches build object-aware vision-language spaces from large mixtures of classification, detection, grounding, and image-text datasets.
  • Large vision-language models combine a vision foundation model, a projector, and a large language model to process visual and textual inputs.
  • High-quality captions are presented as important for training both large vision-language models and open-vocabulary detectors under LLM supervision.

3. GroundingCap-1M Dataset

GroundingCap-1M is designed as a multimodal supervision dataset pairing grounding annotations with detailed factual image captions. It aggregates several source datasets and contains 1120k training samples.

  • Each GroundingCap-1M sample is a quadruple containing an image, short grounding text, mapped bounding boxes, and a detailed whole-image caption.
  • Detailed captions are intended to maximize factual information about object types, textures, colors, parts, actions, locations, and text.The collection principles also exclude imaginary or reasoning-based content to preserve information density.
  • The dataset combines object detection, grounding, and image-text sources, including COCO, V3Det, GoldG, and LCS-558k.
  • 168k captions come from ShareGPT4V and 42k from ASv2 for COCO, while V3Det captions are generated with Qwen2-VL-72b.
  • GoldG grounding texts are merged when their bounding boxes do not conflict, reducing the dataset from 769k to 437k samples.
  • 1120k samples comprise the final GroundingCap-1M dataset.

4. Training LLMDet under the Supervision of Large Language Models

LLMDet trains an open-vocabulary detector jointly with a language model using grounding and caption-generation objectives. Image-level and region-level captioning provide complementary supervision, with region queries linked to specific entities.

  • Joint training objectives: LLMDet combines the detector’s grounding task with image-level and region-level caption generation tasks during end-to-end finetuning.The grounding objectives include word-region alignment and box regression, while the added language tasks generate captions at two scales.
  • Training implementation: The framework separately forwards the language model for image-level and region-level generation because their token counts differ substantially.This separation is intended to reduce memory and computation during training.
  • Image-level caption generation: The language model consumes detector feature maps to generate long, detailed captions describing the whole image.The image-level task uses the detector’s feature maps as visual inputs and applies language modeling loss to the annotated captions.
  • Region-level caption generation: Image-level generation alone makes it difficult to associate caption entities with specific regions in the image.Whole-image features do not directly identify which region corresponds to an entity such as “dishes.”
  • Region-level caption generation: LLMDet therefore feeds positive detector object queries to the language model to generate grounding phrases for their corresponding regions.Cross-attention layers let object queries gather additional information from detector feature maps before producing phrases such as “young man,” “mother,” and “dishes.”

5. Experiment

Experiments evaluate LLMDet across zero-shot detection, transfer, domain shift, referring expression comprehension, and ablations. LLMDet consistently improves open-vocabulary performance, with gains linked to detailed captions, factual data, cross-attention, and projector alignment.

  • Zero-Shot Detection Transfer Ability: LLMDet outperforms MM-GDINO on LVIS minival by 3.3%/3.8%/14.3% AP and 3.1%/3.3%/17.0% APr across Swin-T/B/L backbones.With Swin-L, LLMDet reaches 50.6% AP despite using much less training data.
  • Zero-Shot Detection Transfer Ability: LLMDet achieves the highest AP on ODinW35, demonstrating transfer across 35 datasets spanning varied domains and vocabularies.ODinW reports average AP on selected 13 datasets and all 35 datasets.
  • Zero-Shot Detection Transfer Ability: LLMDet improves MM-GDINO by 2.1% AP on COCO-O, indicating stronger robustness to domain shifts across sketches, weather, cartoons, paintings, tattoos, and handmade images.COCO-O shares COCO’s 80 classes but changes the image domains.
  • Zero-Shot Detection Transfer Ability: LLMDet outperforms MM-GDINO on multiple referring expression comprehension datasets through detailed-caption co-training and enriched vision-language alignment.These tasks require language understanding and fine-grained vision-language alignment.
  • Ablation Study: Combining image-level and region-level generation fully exploits LLM supervision, while grounding-only finetuning raises AP from 41.4% to 43.8%.Region-level generation alone does not improve performance; image-level generation provides a slight gain, and combined generation improves rare-class performance by 3.9% APr.
  • Ablation Study: Caption quality, dataset scale, factual filtering, cross-attention, and projector pretraining materially affect performance, especially for rare classes.Removing LCS data reduces AP to 42.8%; retaining hallucinations yields 44.2% AP and 35.0% APr; removing projector pretraining decreases rare-class AP by 3.5% APr.

6. Conclusion

LLMDet uses detailed image-level captions and region-level grounding phrases to strengthen open-vocabulary detectors, achieving strong benchmark performance and mutual benefits with large multi-modal models.

  • LLMDet combines image-level detailed captions with region-level coarse grounding phrases to build richer vision-language representations.
  • The resulting detector achieves state-of-the-art performance across a wide range of benchmarks.
  • The improved LLMDet can build a strong large multi-modal model, producing mutual benefits between the detector and the multi-modal model.

A. LLMDet Builds a Stronger Large Vision-Language Model

LLMDet can serve as a vision foundation model for large multi-modal models, and stronger pre-aligned detectors improve multi-modal performance across evaluated benchmarks.

  • LLMDet is evaluated as a vision foundation model for large multi-modal models based on pretrained language and vision models.
  • The evaluation uses MME, POPE, and GQA to assess comprehensive understanding, hallucination, and academic visual question answering.
  • Replacing the vision encoder with stronger LLMDet achieves the highest performance on MME, supporting mutual benefits between the detector and large multi-modal model.

B. Limitations

The method has two stated limitations: image-level outputs remain relatively short, and region-level descriptions are too simple to provide rich information.

  • The co-trained LLM tends to produce relatively short whole-image descriptions despite prompts requesting detailed captions.
  • The authors attribute the short image-level outputs to the greater quantity of region-level data than image-level data.
  • Region-level descriptions are limited to simple grounding phrases, and the authors suggest collecting more informative regional data.

C. Implement Details of Zero-Shot Test on Referring Expression Comprehension Datasets

The zero-shot referring-expression evaluation transfers a phrase-grounding detector to a task requiring only the object referred to by each sentence, with a subject-based selection adjustment.

  • Phrase grounding detects every phrase in a grounding text, such as both “the man” and “umbrella” in “the man with an umbrella”.
  • Referring expression comprehension instead detects only the single object referred to by the sentence, creating a discrepancy with phrase grounding.
  • The transfer procedure uses NLTK to identify the sentence subject and selects the highest-confidence box corresponding to that subject.
  • GPT-4o provides comprehensive scores for caption-image pairs using prompts that separately assess hallucination and detailedness.
  • Hallucination scoring ranges from 0 to 5 according to descriptive errors, while detailedness scoring rewards factual content across visual attributes and relationships.

E. Detailed Zero-Shot Results

This section reports detailed zero-shot evaluations on ODinW35 and COCO-O, including domain-shifted settings, while GroundingCap-1M supplies detailed image-level captions for training.

  • ODinW35 results are detailed per dataset for Grounding-DINO-T, MM-GDINO-T, and LLMDet.
  • COCO-O evaluates models across cartoon, handmake, painting, sketch, tattoo, and weather domains.COCO-O shares COCO’s 80 classes but changes the visual domains.
  • GroundingCap-1M augments grounding data with detailed image-level captions describing object types, textures, colors, parts, actions, locations, and text.The captions are intended to contain many details without imaginary contents.

F.2. Visualizations of the Captions Generated by LLMDet

The visualizations show LLMDet generating image-level and region-level captions, while revealing that its image-level captions are coarser than GroundingCap-1M captions.

  • LLMDet generates precise COCO class names for detected objects during caption generation.The visualization uses COCO class names as grounding text for deep fusion, so only COCO objects are detected for caption generation.
  • LLMDet’s image-level captions are relatively coarse-grained compared with GroundingCap-1M captions.The authors suppose this reflects much more region-level data and overly simplistic region-level annotations.
  • GroundingCap-1M visualizations mark detailed caption content in green and imaginary content with underlining.
  • LLMDet visualizations place image-level captions below images and region-level captions beside bounding boxes.Only object queries scoring above 0.3 are visualized.
Loading 2501.18954v1…