PaperlayerView exact revision on arXiv

Source-linked AI summary

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang

arXiv:2303.05499v5cs.CV

TL;DR

Open-set object detection aims to identify arbitrary objects specified by human language, but existing detectors have limited cross-modal integration. Grounding DINO extends DINO with tight language–vision fusion and grounded pre-training, achieving strong results across detection and referring-object settings, including 52.5 AP on COCO without COCO training data.

  • Problem

    Open-set object detection seeks to detect arbitrary objects specified by human language, supporting its potential use as a generic object detector.

  • Method

    Grounding DINO extends DINO with tight language–vision fusion and grounded pre-training through feature enhancement, language-guided query selection, and cross-modality decoding.

  • Results

    52.5 AP on COCO without COCO training data and 26.1 mean AP on the ODinW zero-shot benchmark accompany strong performance across closed-set, open-set, and referring-object detection.

  • Takeaways & Limitations

    Grounding DINO enables open-set object detection from text queries and extends evaluation to referring-expression comprehension, where existing detectors perform poorly without fine-tuning.

  • Takeaways & Limitations

    Grounding DINO cannot be used for segmentation and can produce false positives, while its smaller training set may limit performance.

Abstract

from arXiv · show

In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean $26.1$ AP. Code will be available at \url{https://github.com/IDEA-Research/GroundingDINO}.

3 The Hong Kong University of Science and Technology

Grounding DINO is evaluated across open-set object detection and referring object detection benchmarks using object detection, grounding, and caption data. It performs strongly across COCO, LVIS, ODinW, and RefCOCO/+/g, including zero-shot results on COCO and ODinW.

  • Evaluation: Grounding DINO is evaluated on both open-set object detection and referring object detection benchmarks.The evaluation uses object detection data, grounding data, and caption data.
  • Evaluation: Grounding DINO performs remarkably well across COCO, LVIS, ODinW, and RefCOCO/+/g benchmarks.These benchmarks cover the paper’s reported open-set and referring-object evaluation settings.
  • Zero-shot results: 52.5 AP is achieved on the COCO zeroshot1 detection benchmark.This result uses no COCO training data, as specified in the paper context.
  • Zero-shot results: 26.1 mean AP sets a new record on the ODinW zero-shot benchmark.The reported metric is the benchmark mean AP.

1 Introduction

Grounding DINO targets open-set object detection by detecting arbitrary objects specified through human language. It combines tight DINO-based modality fusion with large-scale grounded pre-training and demonstrates strong results across closed-set, open-set, and referring object detection.

  • Introduction: Grounding DINO detects arbitrary objects specified by human language inputs, targeting open-set object detection for generic object detection applications.The paper also highlights cooperation with generative models for image editing.
  • Introduction: Its design follows two principles: tight modality fusion based on DINO and large-scale grounded pre-training for concept generalization.DINO’s consistent Transformer structure enables language interaction across the detector pipeline’s feature-fusion phases.
  • Introduction: Sub-sentence-level text features reduce attention between unrelated categories during grounded training, mitigating interference from concatenated category names.This refinement is intended to improve model performance during grounded training.
  • Introduction: The paper evaluates closed-set detection, open-set detection, and referring object detection to comprehensively assess open-set detection performance.It advocates a fully zero-shot approach in which the test dataset’s training split is not used during training.
  • Introduction: 52.5 AP on COCO minival is achieved without any COCO training data, while 26.1 mean AP establishes a new state of the art on the ODinW zero-shot benchmark.The paper states that Grounding DINO outperforms competitors by a large margin.

2 Related Work

Grounding DINO builds on DETR-like Transformer detectors and addresses limitations in prior multimodal open-set methods. Earlier approaches fuse modalities only in selected phases and commonly overlook referring expression comprehension evaluation.

  • Detection Transformers: Grounding DINO is built upon DINO, an end-to-end Transformer-based detector in the DETR family.DETR introduced the architecture, while DAB-DETR and DN-DETR added anchor-box queries and query denoising, respectively.
  • Open-Set Detection: DetCLIP uses large-scale image-captioning datasets and generated pseudo labels to expand its knowledge database and generalization ability.Its formulation operates at both phrase and region levels and can achieve stronger performance on fully supervised detection benchmarks.
  • Open-Set Detection: Previous methods fuse multimodal information only in partial phases, potentially limiting language generalization.GLIP fuses during feature enhancement, whereas OV-DETR injects language only at decoder inputs.
  • Open-Set Detection: Referring expression comprehension is commonly overlooked in open-set detection evaluation despite being an important scenario.The paper compares Grounding DINO with other open-set methods in Table 1.

3 Grounding DINO

Grounding DINO combines image and text encoders with feature enhancement, language-guided query selection, and a cross-modality decoder to detect objects specified by text. Its design supports both object detection and referring expression comprehension while preserving fine-grained word-level information through sub-sentence attention masks.

  • Task Alignment: The pipeline aligns object detection and referring expression comprehension by pairing predicted boxes with noun phrases and selecting the highest-scoring output for REC.For object detection, category names are concatenated as input text; REC requires a bounding box for each text input.
  • Architecture: Grounding DINO uses a dual-encoder-single-decoder architecture with image and text backbones, a feature enhancer, language-guided query selection, and a cross-modality decoder.The feature enhancer fuses image and text features before query initialization, and the decoder refines boxes.
  • Language-Guided Query Selection: Language-guided query selection chooses image features most relevant to the input text as decoder queries.It selects Nq indices using image-text feature alignment and initializes decoder queries from the selected features with mixed query selection.
  • Cross-Modality Decoder: Each cross-modality decoder layer applies self-attention, image cross-attention, text cross-attention, and an FFN to improve modality alignment.The decoder adds text cross-attention compared with the DINO decoder to inject text information into queries.
  • Text Representation: Sub-sentence attention masks block interactions among unrelated category names while retaining per-word features for fine-grained understanding.This addresses unwanted word interactions in word-level text representations.
  • Training Objectives: Training uses L1 and GIOU losses for box regression and focal loss on contrastive query-token logits for classification.Box regression and classification costs are used for bipartite matching between predictions and ground truths.

4 Experiments

The experiments evaluate Grounding DINO across closed-set, open-set, referring-expression, ablation, transfer, and efficiency settings. Results show strong zero-shot generalization, scalability, compactness, and benefits from multimodal design choices, alongside limitations on rare LVIS categories.

  • 4 Experiments: Experiments cover closed-set COCO detection, zero-shot COCO/LVIS/ODinW, referring detection on RefCOCO/+/g, ablations, DINO transfer, and efficiency.The evaluation includes both direct zero-shot transfer and selected fine-tuned comparisons.
  • COCO Benchmark: 52.5 AP is achieved on COCO without seeing COCO training images, while COCO minival reaches 62.6 AP versus DINO’s 62.5 AP.Grounding DINO outperforms prior models in zero-shot transfer and slightly exceeds DINO on COCO minival.
  • ODinW Benchmark: 26.1 AP establishes a new ODinW zero-shot record, and Grounding DINO’s APmedian is 11.9 versus GLIPv2-T’s 8.9 under the zero-shot setting.Grounding DINO also uses 172M parameters compared with GLIPv2’s 232M, indicating a more compact model with more consistent performance.
  • Referring Detection: Grounding DINO outperforms GLIP on RefCOCO/+/g under the same setting, but both models perform poorly without referring-expression-comprehension data.Injecting RefCOCO/+/g data produces significant gains, whereas caption data or larger models provide only minor improvements without that data.
  • Ablation Study: +3.0 AP, +1.8 AP, and +0.5 AP gains come from language-guided query selection, text cross-attention, and sub-sentence text prompts, respectively, while encoder fusion significantly improves COCO and LVIS.These ablations support the effectiveness of the proposed multimodal fusion design.

5 Conclusion

Grounding DINO extends DINO to open-set object detection through tighter cross-modal fusion and sub-sentence representations, and extends evaluation to referring expression comprehension. Despite strong open-set detection performance, it cannot be used for segmentation tasks like GLIPv2 and may be limited by its smaller training set.

  • Contributions: Grounding DINO extends DINO to open-set object detection, detecting arbitrary objects from text queries.The model uses a tight fusion approach to better fuse cross-modality information.
  • Contributions: Grounding DINO proposes sub-sentence-level representations to use detection data for text prompts more reasonably.
  • Referring expression comprehension: Grounding DINO extends open-set object detection to referring expression comprehension and evaluates performance accordingly.Existing open-set detectors do not work well for REC data without fine-tuning, motivating attention to REC zero-shot performance.
  • Limitations: Grounding DINO cannot be used for segmentation tasks like GLIPv2 despite its strong open-set object detection performance.Its training data is smaller than that of the largest GLIP model, which may limit performance.

A More Implementation Details · A.1 Hyperparameters · A.2 Pseudo Code Language-Guided Query Selection

The implementation details document the hyperparameters used in the main experiments and pre-trained models, and specify the pseudo-code and variable conventions for Language-Guided Query Selection. The implementation sets the decoder query count to 900.

  • A.1 Hyperparameters: Table 8 lists the hyperparameters used in the main experiments.The table is titled “Hyper-parameters used in our pre-trained models.”
  • A.1 Hyperparameters: The implementation presents Table 8 as the reference for hyperparameters in the pre-trained models.
  • A.2 Pseudo Code Language-Guided Query Selection: Algorithm 1 provides pseudo-code for the Language-Guided Query Selection module.
  • A.2 Pseudo Code Language-Guided Query Selection: 900 is the decoder’s number of queries in the implementation.The pseudo-code denotes this quantity as num_query.
  • A.2 Pseudo Code Language-Guided Query Selection: image_feat and text_feat denote image and text features, respectively.
  • A.2 Pseudo Code Language-Guided Query Selection: bs and ndim denote batch size and feature dimension, while num_img_tokens and num_text_tokens count image and text tokens.

B Data Usage · C More Experiment Results · C.1 Transfer from DINO to Grounding DINO

Grounding DINO is pre-trained with detection and grounding data, while transferring pre-trained DINO weights reduces training cost and accelerates convergence. The transferred model achieves similar performance to standard Grounding DINO and outperforms it on LVIS under the same setting.

  • B Data Usage: Grounding DINO pre-training uses detection data from COCO, O365, and OpenImage, with category names concatenated into text prompts and sampled randomly during training.Grounding data comes from GoldG and RefC, both preprocessed by MDETR.
  • B Data Usage: O365v1 contains about 600K images, whereas O365v2 contains about 1.7M images; Grounding DINO T uses O365v1 for fair comparison and Grounding DINO L uses O365v2 for better results.
  • C.1 Transfer from DINO to Grounding DINO: Pre-training Grounding DINO from scratch is computationally expensive, so the transfer experiments initialize it with pre-trained DINO weights.Shared DINO and Grounding DINO modules are frozen, and only the remaining parameters are fine-tuned.
  • C.1 Transfer from DINO to Grounding DINO: Using a pre-trained DINO, Grounding DINO achieves similar performance while training only the text and fusion blocks, and outperforms standard Grounding DINO on LVIS under the same setting.
  • C.1 Transfer from DINO to Grounding DINO: With pre-trained DINO initialization, Grounding DINO converges faster than training from scratch.The comparison uses curves without exponential moving average, and only early epochs are shown for the slower scratch-trained model.
  • C.1 Transfer from DINO to Grounding DINO: The paper compares DINO and Grounding DINO by distinguishing shared DINO blocks from newly proposed modules in the model illustration.

C.2 Detailed Results on COCO Detection Benchmarks · C.3 Detailed Results on ODinW Benchmarks · C.4 Model Efficiency

The appendix reports Grounding DINO’s COCO and ODinW benchmark results and compares its efficiency with GLIP-T. On COCO, it achieves 48.1 AP in the research setting but remains below the original DINO, while the efficiency comparison favors Grounding DINO over GLIP.

  • C.2 Detailed Results on COCO Detection Benchmarks: All COCO models use a ResNet-50 backbone and are trained for 12 epochs in the 1× setting.The benchmark uses the COCO val2017 split.
  • C.2 Detailed Results on COCO Detection Benchmarks: 48.1 AP is achieved by Grounding DINO on the standard COCO detection benchmark under the research setting.This result indicates strong closed-set detection performance.
  • C.2 Detailed Results on COCO Detection Benchmarks: Grounding DINO is inferior to the original DINO on the COCO benchmark.The authors suspect the newly introduced components make optimization harder than in DINO.
  • C.3 Detailed Results on ODinW Benchmarks: Detailed Grounding DINO results on ODinW35 are presented across Tables 12, 13, and 14.The supplied passage identifies the benchmark and result tables but reports no metric values.
  • C.4 Model Efficiency: Table 20 compares model size and efficiency between Grounding DINO T and GLIP-T.The comparison focuses on the two models’ parameters and efficiency.
  • C.4 Model Efficiency: Grounding DINO has a smaller parameter size and better efficiency than GLIP.The reported comparison is specifically between Grounding DINO T and GLIP-T.

C.5 Ablations for More Decoder Queries … D.2 Physical meaning of language-guided query

The ablations show that additional decoder queries offer only marginal gains, while language-encoder scaling does not resolve REC limitations rooted in detection. Further experiments demonstrate competitive REC training from scratch, dataset-dependent ODinW performance, and dynamic, prompt-dependent query selection.

  • C.5 Ablations for More Decoder Queries: 1200 and 1500 decoder queries slightly outperform 900 queries on LVIS rare classes, but gains remain marginal because 900 queries already cover COCO and LVIS objects.More queries also exacerbate training data imbalance, potentially offsetting their benefits.
  • C.6 Results with Different Language Encoder for REC: BERT-B outperforms or matches BERT-L on most REC metrics, with neither encoder showing significant late-training improvements.Both variants were trained on a leakage-filtered combination of RefCOCO, RefCOCO+, and RefCOCOg.
  • C.6 Results with Different Language Encoder for REC: REC improvement appears primarily limited by the detection branch rather than language processing, motivating a dedicated model for REC data.
  • C.7 Detic Pseudo-labeled Data for LVIS: The oracle LVIS experiment created IN22K-LVIS-1M from about 1M pseudo-labeled ImageNet images generated by Detic after filtering LVIS-related images.Because Detic was trained with LVIS data, this setting may not constitute true zero-shot evaluation.
  • C.8 Comparison between Grounding DINO and GLIP on ODinW: On PlantDoc, Grounding DINO scores 0.36 versus GLIP’s 1.1, illustrating weak performance on uncommon categories that are poorly represented in training data.The findings emphasize the need to improve data quality for overall performance.
  • C.9 Training Grounding DINO on RefCOCO/+/g from scratch: Despite only 9 training epochs, Grounding DINO trained from scratch on RefCOCO/+/g achieves performance comparable to state-of-the-art REC models.The result supports the approach’s effectiveness for REC tasks.
  • D.1 Detection Visualizations: Visualizations show generalization across different scenes and text inputs, including accurate localization of “man in blue” and “child in red.”
  • D.2 Physical meaning of language-guided query: The top 900 language-guided queries vary across prompts during inference, demonstrating dynamic query selection.

D.3 Comparison of RefCOCO and Grounding Data · D.4 Marry Grounding DINO with Stable Diffusion for Object Detection and Inpainting · D.5 Marry Grounding DINO with Stable Diffusion for Object Detection and Grounded Generation

The paper contrasts RefCOCO’s one-box-per-prompt formulation with grounding data and demonstrates applications combining Grounding DINO with Stable Diffusion or GLIGEN for object editing and grounded generation.

  • D.3 Comparison of RefCOCO and Grounding Data: RefCOCO’s grounded-training formulation creates a substantial performance gap without RefCOCO data, partly because each text prompt maps to one box while the model often predicts multiple objects.Figure 8 compares the model’s predictions with RefCOCO ground truths.
  • D.4 Marry Grounding DINO with Stable Diffusion for Object Detection and Inpainting: For image inpainting, Grounding DINO detects objects and produces masks, after which Stable Diffusion renders new images from the originals, masks, and generation prompts.The authors use released Stable Diffusion checkpoints and provide additional results in Figure 9.
  • D.4 Marry Grounding DINO with Stable Diffusion for Object Detection and Inpainting: The detection prompt supplies language to Grounding DINO, whereas the generation prompt supplies language to the inpainting model.The two prompts therefore control detection and image generation in separate stages.
  • D.5 Marry Grounding DINO with Stable Diffusion for Object Detection and Grounded Generation: For fine-grained editing, Grounding DINO is combined with GLIGEN, using a phrase prompt as the language input associated with each detected bounding box.This assigns object-specific language descriptions to spatial regions.
  • D.5 Marry Grounding DINO with Stable Diffusion for Object Detection and Grounded Generation: GLIGEN uses grounding results to generate assigned objects at specific positions and can fill each bounding box, ensuring objects occupy the detected regions.The paper reports improved visualization when GLIGEN fulfills each bounding box, including with the same generative prompt used in Figure 9(b).
  • D.5 Marry Grounding DINO with Stable Diffusion for Object Detection and Grounded Generation: The supplementary material further includes detailed ODinW results, an LVIS oracle analysis, and training Grounding DINO on RefCOCO from scratch.The LVIS table cautions that its setting may not be truly zero-shot because the Detic pseudo-labeler was trained on LVIS.
Loading 2303.05499v5…