DINO-X: A Unified Vision Model for Open-World Object Detection and Understanding

1 Introduction

• Open-set detectors identify objects corresponding to user-provided prompts, with applications including robotics, autonomous vehicles, and multimodal language models.The passage also describes potential roles in reducing MLLM hallucinations and improving response reliability.Source: §1 · p. 2
• DINO-X uses the Grounding DINO 1.5 Transformer encoder-decoder architecture and adopts open-set detection as its core training task..Source: §1 · p. 2
• DINO-X supports text, visual, and customized prompts, including a universal object prompt for detecting objects without user-provided prompts.Customized prompts can be pre-defined or user-tuned for domain- or function-specific needs.Source: §1 · p. 2
• Grounding-100M comprises over 100 million high-quality grounding samples, and pre-training on it supports multiple perception heads for object-level tasks.The heads include segmentation masks, keypoints, and fine-grained object captions, alongside detection.Source: §1 · p. 3
• 63.3 AP and 56.5 AP on the rare classes of LVIS-minival and LVIS-val improve over Grounding DINO 1.6 Pro by 5.8 AP and 5.0 AP, respectively.DINO-X Pro also achieves 56.0 AP on COCO, 59.8 AP on LVIS-minival, and 52.4 AP on LVIS-val zero-shot transfer benchmarks.Source: §1 · p. 3

2 Approach

• DINO-X Pro achieves new state-of-the-art zero-shot performance on COCO, LVIS-minival, and LVIS-val, with larger margins on rare LVIS classes.The figure compares it with Grounding DINO 1.5 Pro and Grounding DINO 1.6 Pro.Source: Figure 2 · p. 3

2.1.1 DINO-X Pro

• DINO-X Pro supports text, visual, and customized prompts, then applies task-specific heads to produce boxes, masks, keypoints, or captions.Visual prompts address cases not well described by text; customized prompts can be prompt-tuned for long-tailed, domain-specific, or function-specific detection.Source: §2.1.1 · pp. 4–5, Figure 3 · p. 4
• A universal object prompt enables prompt-free open-world detection, while CLIP supplies DINO-X Pro’s text prompt embeddings.Visual prompts are encoded from user-provided boxes or points, and the box head selects prompt-relevant features as object queries.Source: §2.1.1 · pp. 4–5
• The language head combines detected-region representations with task tokens to generate autoregressive outputs for captioning, recognition, text reading, and region-based VQA.The keypoint head decodes person or hand detections into 17 or 21 predefined keypoints, respectively.Source: §2.1.1 · p. 5

2.1.2 DINO-X Edge

• DINO-X Edge uses an EfficientViT backbone and Transformer encoder-decoder, with Pro-to-Edge feature and response distillation to strengthen zero-shot capability.It uses the Pro model’s CLIP text encoder; the described FP16 inference reaches 20.1 FPS, compared with 15.1 FPS for Grounding DINO 1.6 Edge and 10.7 FPS for Grounding DINO 1.5 Edge.Source: §2.1.2 · pp. 5–6

3 Dataset Construction and Model Training

• Grounding-100M contains over 100 million web-collected images, supplemented by visual-prompt grounding data and over 10 million region-understanding examples.The collection includes pseudo-mask annotations and box annotations used for mask-head and prompt-free detection training, respectively.Source: §3 · p. 6
• Training first jointly covers text-prompt detection, visual-prompt detection, and segmentation without COCO, LVIS, or V3Det data, then freezes the backbone while separately training keypoint and language heads.A subsequent universal-object-prompt tuning stage supports prompt-free any-object detection while preserving other capabilities.Source: §3 · p. 6

4.1.1 Open-World Detection and Segmentation

• 63.3 AP on LVIS-minival rare classes and 56.5 AP on LVIS-val rare classes exceed Grounding DINO 1.6 Pro by 5.8 AP and 5.0 AP, respectively.DINO-X Pro also reaches 59.8 box AP on LVIS-minival and 52.4 box AP on LVIS-val; its mask AP is 37.9 on COCO, 43.8 on LVIS-minival, and 38.5 on LVIS-val, while Grounded SAM performs best on LVIS instance segmentation.Source: §4.1.1 · p. 7, Table 1 · p. 8
• DINO-X achieves state-of-the-art performance on few-shot visual-prompt counting across FSC147 and FSCD-LVIS.The task provides three exemplar boxes; FSC147 uses MAE for single-target scenes and FSCD-LVIS uses AP for multi-target scenes.Source: §4.1.1 · p. 7, Table 2 · p. 8

4.1.2 Keypoint Detection

• DINO-X’s pose AP is 1.6 below ED-Pose, but exceeds existing models on CrowdPose and Human-Art by 3.4 AP and 1.8 AP, respectively.The pose head is jointly trained on COCO, CrowdPose, and Human-Art; only the pose head is trained while the backbone is frozen.Source: §4.1.2 · pp. 7–8, Table 3 · p. 9
• DINO-X achieves the best PCK@0.05 performance on HInt for hand-pose estimation.Training combines HInt, COCO, and OneHand10K, and evaluation uses the HInt test set.Source: §4.1.2 · p. 8, Table 4 · p. 9

4.1.3 Object-Level Vision-Language Understanding

• On LVIS-val object recognition, DINO-X scores 71.25% in SS and 41.15% in S-IoU, exceeding Osprey by 6.01% and 2.06%, respectively.The language head is evaluated zero-shot; on PACO, DINO-X is inferior to Osprey.Source: §4.1.3 · p. 9, Table 5 · p. 9
• Region captioning reaches 142.1 CIDEr on Visual Genome zero-shot and 201.8 CIDEr after Visual Genome fine-tuning.The zero-shot result uses frozen DINO-X backbone features without Visual Genome training data; the fine-tuned result uses a lightweight language head.Source: §4.1.3 · p. 10, Table 6 · p. 10

4.2 DINO-X Edge

• DINO-X Edge achieves 48.3 AP on LVIS-minival and 42.0 AP on LVIS-val after Grounding-100M pre-training, and outperforms existing real-time open-set detectors on COCO.These are zero-shot object-detection evaluations.Source: §4.2 · p. 10, Table 7 · p. 10
• At 640×640 input size, DINO-X Edge reaches 20.1 FPS, a 33% improvement over Grounding DINO 1.6 Edge’s 15.1 FPS.The speed evaluation includes FP32 and FP16 TensorRT models on NVIDIA Orin NX.Source: §4.2 · p. 10

5.1 Open-World Object Detection

• Text-prompt detection spans common categories, long-tailed classes, and dense-object scenes.The section presents these cases as evidence of DINO-X's open-world detection capability.Source: Figure 5 · p. 11, §5.1 · p. 11

5.2 Long Caption Phrase Grounding

• DINO-X maps noun phrases in long captions to corresponding image regions.The paper identifies this capability as useful for more accurate and reliable MLLM responses.Source: Figure 6 · p. 12, §5.2 · p. 12

5.3 Open-World Object Segmentation and Visual Prompt Counting

• Text-prompt detection can return a segmentation mask for each object, while box or point visual prompts support object counting.The paper contrasts the segmentation output with Grounding DINO 1.5 [47].Source: Figure 7 · p. 13, §5.3 · p. 13

5.4 Prompt-Free Object Detection and Recognition

• A universal prompt enables detection and identification of objects without user-provided prompts.This prompt-free capability is paired with DINO-X's language head.Source: Figure 8 · p. 14, §5.4 · p. 14

5.5 Dense Region Caption

• DINO-X generates fine-grained captions for specified image regions, while region-based QA and other region-understanding tasks remain in development.The paper says these features are planned for release in a next version.Source: Figure 9 · p. 15, §5.5 · p. 15

5.6 Human Body and Hand Pose Estimation

• Text-prompted keypoint heads predict human body and hand keypoints, trained on COCO, CrowdHuman, and Human-Art.Source: Figure 10 · p. 16, §5.6 · p. 16

5.7 Side-by-side comparison with Grounding DINO 1.5 Pro

• A side-by-side comparison reports enhanced language comprehension and strong performance in dense-object detection for DINO-X relative to Grounding DINO 1.5 Pro and 1.6 Pro.Source: Figure 11 · p. 17, §5.7 · p. 17

• The conclusion presents DINO-X Pro as setting new records on COCO and LVIS zero-shot benchmarks, while DINO-X Edge targets faster inference and edge-device deployment.DINO-X supports text, visual, and customized prompts and extends beyond detection to segmentation, pose estimation, and object-level understanding.Source: §6 · p. 18

• The acknowledgments identify contributors for DINO-X Pro, its mask, visual-prompt, pose, language, and Edge components, and Grounding-100M.They also name project leads, including the overall project lead.Source: §7 · p. 18
• The acknowledgments recognize the playground and API support team, including application, product, development, UX, operations, and testing roles.They also thank a contributor for suggestions and feedback on customized scenarios.Source: §7 · p. 18