RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection

1 INTRODUCTION

• Reusable attributes, actions, states, and relations learned from seen data may help match instances to unseen concepts, echoing compositional zero-shot learning.Examples include cues associated with chairs and wheels for recognizing wheelchairs, or wooden material for recognizing wooden spoons.Source: §1 · p. 1, Figure 1 · p. 2
• Compact real-time OVD methods emphasize vocabulary coverage and matching efficiency, while limited representation capacity can hinder absorption of instance semantics and scene context.The paper frames transferring rich description semantics without compromising inference efficiency as underexplored.Source: §1 · p. 2
• RT-DETR-World uses a frozen DINOv3 encoder, lightweight projector, DETR decoder, and compact MiniLM for category-oriented query–text matching at inference.GroundingCapv2 adds category names, instance-level object descriptions, and image descriptions of relations, scene context, and global semantics as training-time supervision.Source: §1 · p. 2, Figure 2 · p. 3
• DDA aligns MiniLM category queries and object descriptions with matched queries, while offline LLM features supervise object-level queries and global image representations.Teacher features are precomputed, and auxiliary alignment modules are removed after training.Source: §1 · p. 2
• RNR uses teacher-derived semantic similarities to relax related negatives while preserving exact positives across MiniLM object-description and teacher-based object- and image-level alignments.It addresses contrastive objectives that otherwise treat unmatched pairs as equally strong negatives.Source: §1 · pp. 2–3

2.1 TRI-LEVEL DATA FORMULATION

• GroundingCapv2 combines category names for standard OVD with object descriptions of instance details and image descriptions of relations, scene context, and global semantics.Regions may have different valid category and object-description annotations, so each receives only reliable available supervision.Source: §2.1 · p. 3
• The standardized corpus contains 1,115,690 samples and 8,050,813 regions, with 95.8% category coverage, 98.5% object-description coverage, and image descriptions for 1,115,656 samples.COCO and V3Det retain source categories and receive generated instance descriptions; GQA, Flickr30K Entities, and LLaVA-Cap retain grounded phrases, with categories added when supported by concrete visual concepts.Source: §2.1 · p. 3

2.2 CONFIDENCE-ROUTED DATA CURATION

• GroundingAgent-Opt uses Qwen3-VL-8B proposal and verification, routing ambiguous cases to Qwen3-VL-32B for multimodal adjudication.For detection-origin data, unresolved descriptions fall back to source categories; for region–phrase data, original phrases remain object-description supervision when no reliable category is identified.Source: §2.2 · pp. 3–4, Figure 3 · p. 4
• Generated descriptions are accepted for 85.5% of COCO regions and 87.6% of V3Det regions, with the remainder using category-name fallback.Original phrases are retained for GQA, Flickr30K Entities, and LLaVA-Cap, and mappings, region indices, token spans, and reconstructed text are checked.Source: §2.2 · p. 4

3.1 OVERALL ARCHITECTURE

• The detector uses frozen DINOv3 features, a lightweight multi-scale projector, MiniLM-guided query selection and classification, and a DETR decoder for query refinement and box regression.The LLM teacher and auxiliary alignment modules are absent at inference; the category-oriented MiniLM interface remains.Source: §3.1 · pp. 4–5

3.2 DUAL-PATH DESCRIPTION ALIGNMENT (DDA)

• DDA pairs MiniLM object-description alignment with training-only LLM targets for matched queries and global visual representations, transferring instance and scene semantics without inference overhead.Teacher embeddings are precomputed, and teacher-alignment projection heads are discarded after training.Source: §3.2 · p. 5

3.3 RELATION-AWARE NEGATIVE RELAXATION (RNR)

• RNR uses teacher-derived semantic similarities to discount related negatives while preserving exact positives across the description-alignment objectives.Related unmatched pairs are relaxed, never treated as positives; setting α = 0 recovers standard bidirectional contrastive learning.Source: §3.3 · p. 6

3.4 TRAINING OBJECTIVE AND IMPLEMENTATION DETAILS

• Training combines Hungarian-matched classification and box regression with three DDA losses; models train for 150k iterations and retain one 300-query group at inference.Offline teacher targets and training-only alignment modules are discarded at inference, while the MiniLM query–text pathway remains unchanged.Source: §3.4 · pp. 6–7

4.1 SETUP

• Evaluation uses LVIS category queries without LVIS annotation training, excludes overlapping evaluation images from GroundingCapv2, and removes teacher modules at inference.LVIS evaluation reports standard and Fixed AP; efficiency is measured on a T4 GPU using TensorRT.Source: §4.1 · p. 7, §4.2 · p. 8

4.2 TRANSFER AND ROBUSTNESS EVALUATION

• 40.0 Fixed AP at 32 FPS lets RT-DETR-World-B exceed the strongest prior real-time LVIS result by 4.1 points; RT-DETR-World-S+ reaches 36.6 Fixed AP at 58 FPS.OV-DEIM-L remains a compact operating point at 35.9 Fixed AP and 91 FPS.Source: §4.2 · p. 8
• RT-DETR-World-B reaches 43.8 AP on ODinW13 and 20.2 AP on ODinW35, surpassing OV-DEIM-L by 2.2 and 1.0 points, respectively.The comparison covers zero-shot cross-domain transfer on ODinW.Source: §4.2 · p. 8, Table 3 · p. 8
• RT-DETR-World-B achieves 46.2 AP on COCO-O and +24.9 effective robustness, exceeding OV-DEIM-L by 2.9 and 2.2 points, respectively.COCO-O evaluates the 80 COCO categories across six shifted domains and is treated as robustness evaluation, not unseen-category recognition.Source: §4.2 · pp. 8–9, Table 4 · p. 8

4.3 ABLATION STUDIES

• A frozen DINOv3-S/16 backbone and identical training settings are used for the ablations, with AP denoting Fixed AP.The supervision ablation reports complementary gains from image- and object-level targets, while RNR preserves exact positives and relaxes related negatives.Source: §4.3 · p. 9
• On GroundingCapv2, adding MiniLM object-description alignment raises AP from 33.1 to 35.2 and APr from 25.7 to 32.0; combining both teacher levels reaches 35.2 AP.The supervision ablation compares category-only, MiniLM description, and object- and image-level teacher supervision.Source: §4.3 · p. 9, Table 5 · p. 9
• RNR achieves 35.2 AP, beating hard contrastive alignment by 1.6 points and SRCL-style regulation by 1.3 points, with a 4.4-point APr gain over hard alignment.RNR preserves exact-positive targets and discounts related negatives using frozen LLM-teacher relations.Source: §4.3 · p. 9

5 CONCLUSION

• RT-DETR-World uses category-, object-, and image-level supervision, with DDA transferring instance semantics and scene context and RNR relaxing semantically related negatives.Teacher-side components are removed after training, adding no inference overhead.Source: §5 · pp. 9–10

AI USE STATEMENT

• Generative AI tools helped construct and curate GroundingCapv2 annotations and translate or polish author-written text; authors reviewed the assisted writing and verified generated annotations.The authors state these tools were not used to develop the methodology, design experiments, implement the method, or interpret results.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

• The work uses publicly available datasets and processed derivatives, with GroundingCapv2 augmenting existing data using model-generated annotations and quality control.The authors report no new human-subject data or personally identifiable information.Source: ETHICS STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The authors state they will make code and processed dataset annotations publicly available to facilitate reproducibility.The paper describes model architecture, training settings, evaluation protocols, data construction, and ablations.Source: REPRODUCIBILITY STATEMENT · p. 10

A RELATED WORK

• OVD work spans visual-semantic transfer and grounded pretraining, while efficient methods such as Grounding DINO 1.5-Edge (Ren et al., 2024), YOLO-World (Cheng et al., 2024), YOLOE (Wang et al., 2025), and OV-DEIM (Wang et al., 2026) target real-time detection.Earlier approaches include ZSD (Bansal et al., 2018), OVR-CNN (Zareian et al., 2021), and large-scale VLM methods including CLIP (Radford et al., 2021).Source: A RELATED WORK · p. 14
• Relational alignment methods address unmatched-pair negatives with soft targets or similarity-based reweighting, but often derive relations from model predictions or external signals rather than sample-specific descriptions.Examples include CLIP-PSD (Andonian et al., 2022), SoftCLIP (Gao et al., 2024), CyCLIP (Goel et al., 2022), and SRCL.Source: A RELATED WORK · p. 14
• In contrast to prior OVD methods, RT-DETR-World uses sample-specific object and image descriptions encoded by a frozen LLM2CLIP teacher to supervise matched queries and global visual representations.Teacher-derived semantic similarities relax related negatives while preserving exact positives, and precomputed teacher features leave inference unchanged.Source: A RELATED WORK · p. 14

B HYPERPARAMETER SENSITIVITY

• λaux = 0.1 gives the highest overall AP of 35.2, while increasing it to 0.2 raises APr to 32.5 but lowers overall AP to 34.3.The study varies the teacher weight individually while keeping other hyperparameters at defaults.Source: B HYPERPARAMETER SENSITIVITY · p. 14, Table 7 · p. 14
• Increasing RNR α from 0 to 0.25 raises AP from 33.6 to 35.2 and APr from 27.6 to 32.0, but excessive relaxation at α = 0.5 degrades performance.The selected default is α = 0.25.Source: B HYPERPARAMETER SENSITIVITY · pp. 14–15, Table 7 · p. 14
• Raising the MiniLM description weight from 0 to 0.25 improves AP from 33.1 to 35.2 and APr from 25.7 to 32.0, while weights of 0.75 and 1.0 reduce AP to 33.3.The selected default is λdesc = 0.25, although λdesc = 0.5 yields the highest APc of 34.1.Source: B HYPERPARAMETER SENSITIVITY · pp. 14–15, Table 7 · p. 14

• Using all 1,203 LVIS category names as queries, RT-DETR-World-B localizes objects across indoor, street, and natural scenes.Source: C VISUALIZATION AND ANALYSIS · p. 15
• Detections include small or fine-grained objects and multiple instances in crowded scenes.Examples include wall sockets, bottle caps, wristbands, and tabletop objects.Source: C VISUALIZATION AND ANALYSIS · p. 15
• Low-confidence and overlapping predictions remain in dense scenes, leaving fine-grained discrimination and confidence calibration challenging with a large vocabulary.Source: C VISUALIZATION AND ANALYSIS · p. 15