Source-linked AI summary
HDINO: A Concise and Efficient Open-Vocabulary Detector
Hao Zhang, Yiqun Wang, Qinran Lin, Runze Fan, Yong Li
TL;DR
Open-vocabulary detectors face dependence on curated training data and costly layer-wise cross-modal extraction. HDINO uses a two-stage DINO-based strategy with noisy positive samples, difficulty-weighted classification, and lightweight fusion, achieving 49.2 mAP on COCO while using simpler training resources. Its performance is suboptimal on long-tailed datasets without grounding supervision or prompt templates.
Problem
Existing open-vocabulary detectors often depend on curated or heterogeneous data and resource-intensive cross-modal feature extraction.
Method
HDINO uses two-stage training: noisy positives form one-to-many semantic alignment with difficulty-weighted classification, followed by lightweight feature fusion for textual awareness.
Results
49.2 mAP on COCO enables HDINO-T to exceed Grounding DINO and T-Rex2 by 0.8 mAP and 2.8 mAP, respectively.
Takeaways & Limitations
HDINO provides a concise and efficient open-vocabulary detection framework with strong modality alignment and minimal reliance on additional modules.
Takeaways & Limitations
HDINO exhibits suboptimal performance on long-tailed datasets because training omits grounding data and prompt templates.
Abstract
from arXiv · showhide
Despite the growing interest in open-vocabulary object detection in recent years, most existing methods rely heavily on manually curated fine-grained training datasets as well as resource-intensive layer-wise cross-modal feature extraction. In this paper, we propose HDINO, a concise yet efficient open-vocabulary object detector that eliminates the dependence on these components. Specifically, we propose a two-stage training strategy built upon the transformer-based DINO model. In the first stage, noisy samples are treated as additional positive object instances to construct a One-to-Many Semantic Alignment Mechanism(O2M) between the visual and textual modalities, thereby facilitating semantic alignment. A Difficulty Weighted Classification Loss (DWCL) is also designed based on initial detection difficulty to mine hard examples and further improve model performance. In the second stage, a lightweight feature fusion module is applied to the aligned representations to enhance sensitivity to linguistic semantics. Under the Swin Transformer-T setting, HDINO-T achieves \textbf{49.2} mAP on COCO using 2.2M training images from two publicly available detection datasets, without any manual data curation and the use of grounding data, surpassing Grounding DINO-T and T-Rex2 by \textbf{0.8} mAP and \textbf{2.8} mAP, respectively, which are trained on 5.4M and 6.5M images. After fine-tuning on COCO, HDINO-T and HDINO-L further achieve \textbf{56.4} mAP and \textbf{59.2} mAP, highlighting the effectiveness and scalability of our approach. Code and models are available at https://github.com/HaoZ416/HDINO.
1 Introduction
Open-vocabulary detection addresses the fixed-category limits of closed-set detectors, but existing approaches often require restrictive data or costly cross-modal processing. HDINO combines one-to-many semantic alignment, difficulty-weighted learning, and lightweight fusion to improve efficiency and detection performance.
- Open-vocabulary detection targets objects from categories unseen during training, addressing the limited applicability of fixed-category detectors.
- Existing methods either use text features only as classifier parameters or repeatedly integrate visual and textual features, limiting semantic modeling or increasing computational overhead.
- HDINO treats multiple noisy samples as positive instances and matches them with auxiliary queries to form a One-to-Many Semantic Alignment Mechanism.
- Difficulty Weighted Classification Loss emphasizes initially challenging prior samples, while second-stage lightweight feature fusion improves textual semantic awareness.
- 49.2 mAP on COCO lets HDINO-T surpass Grounding DINO and T-Rex2 by 0.8 mAP and 2.8 mAP, respectively, with fewer training resources.The model is trained on roughly one-third of the publicly available data and avoids grounding data.
2 Related Work
Related work spans transformer-based detection, one-to-many matching, and open-vocabulary detection. HDINO builds on these directions while using its own noisy-positive sampling, lightweight architecture, and language-guided components.
- DETR introduced transformers into object detection but required long training schedules because of slow convergence.
- Earlier one-to-many DETR approaches increase supervision through multiple queries, auxiliary processing, or alternative matching strategies.
- HDINO differs by injecting noisy samples into the original image, treating them all as positive, and weighting their contributions differently.
- HDINO’s overview includes a text-projection Feat Map and Language-Guided Query Selection for initial anchors and inference-time removal of auxiliary queries.
- Open-vocabulary detection extends detection beyond fixed category sets by leveraging language and vision-language models.
3 Method
HDINO uses positive noisy samples and difficulty-aware classification to align visual and textual representations, then adds lightweight feature fusion to strengthen cross-modal semantics. Its training combines auxiliary-query supervision with standard object-query detection losses.
- 3.1 One-to-Many Semantic Alignment Mechanism: HDINO generates positive noisy boxes by perturbing ground-truth coordinates, using multiple overlaps to provide additional semantic-alignment instances.The perturbation scale is set to λ=0.4 by default to keep generated samples above 0.5 IoU with their corresponding ground-truth boxes.
- 3.1 One-to-Many Semantic Alignment Mechanism: The one-to-many semantic alignment mechanism matches noisy samples with auxiliary queries while retaining the original object-query inference architecture.Auxiliary-query interactions with object queries are blocked during decoder self-attention, and auxiliary queries are removed at inference.
- 3.1 One-to-Many Semantic Alignment Mechanism: DWCL weights positive noisy samples by initial localization difficulty, emphasizing samples with lower IoU rather than relying only on final prediction confidence.Its difficulty factor, 1−IoU, adjusts both the focusing and weighting factors; negative samples retain the standard focal-loss formulation.
- 3.2 Feature Fusion: The feature-fusion module projects textual features into a visual semantic space, applies lightweight cross-attention, and adds the resulting features to visual representations.The fused representations are passed to the encoder, while previously trained modules are fine-tuned and no explicit textual feature decoding is required.
- 3.3 Training Objective: HDINO optimizes object and auxiliary queries jointly, using focal loss for object-query classification and DWCL for auxiliary-query classification alongside L1 and GIoU box losses.The total objective is the sum of object-query and auxiliary-query losses.
4 Experiment
HDINO is evaluated as an efficient open-vocabulary detector using DINO with a frozen CLIP text encoder, detection-only pretraining, and lightweight cross-modal components. Experiments show strong zero-shot and fine-tuned COCO performance, while ablations identify one-to-many alignment as the main contributor and long-tailed performance as a limitation.
- 4 Experiment: HDINO uses a Swin Transformer vision backbone, deformable attention encoder and decoder layers, and a frozen CLIP-B text encoder projected to dimension 256.The model discards DINO’s Contrastive DeNoising Training module while otherwise following DINO’s visual design.
- 4.2 Zero-Shot Detection Results: 49.2 mAP on COCO is achieved by HDINO under Swin-T zero-shot evaluation, surpassing Grounding DINO and T-Rex2 by 0.8 mAP and 2.8 mAP.HDINO uses only detection data and avoids grounding data and computationally intensive layer-wise cross-modal extraction.
- 4.3 Ablation Study: 2.0 mAP is contributed by the one-to-many semantic alignment mechanism over the DINO+CLIP baseline, making it the primary source of HDINO’s ablation gains.The result indicates that standard one-to-one matching limits semantic knowledge internalization from text embeddings.
- 4.3 Ablation Study: DWCL emphasizes difficult positive samples, while lightweight feature fusion on aligned representations increases sensitivity to textual semantics.The DWCL hyperparameter study adopts β1 = 1 and β2 = 2, and Figure 4 compares it with focal loss under IoU = 0.5.
- 4.4 Downstream Transferring: 59.2 mAP is reported after COCO fine-tuning for the larger HDINO configuration, and the model is described as transferable with minimal tuning.The supplied downstream passage reports the 59.2 mAP result and characterizes HDINO as a strongly generalizing pretrained model.
- 4.5 Limitations: HDINO exhibits suboptimal performance on long-tailed datasets because grounding data and prompt templates are deliberately excluded during training.Integrating grounding supervision or more effective prompting is left for future work.
5 Conclusion
HDINO is an open-vocabulary detector trained with a two-stage pre-training scheme. Its alignment, hard-sample weighting, and feature-fusion designs form a concise and efficient framework.
- 5 Conclusion: HDINO combines noisy positive samples with DINO object queries for one-to-many semantic alignment and uses DWCL to emphasize initially difficult samples.A lightweight feature fusion module is then added to enhance textual semantic awareness.