PaperlayerView exact revision on arXiv

Source-linked AI summary

OPUS: A Simple yet Effective Unified Framework for Open-Vocabulary Detection

Xiaoyan Wei, Zhimin Yao, Ruilin Yang, Wei Zhang, Yong Dai, Yi Zhang, Wei Ge

arXiv:2608.30247v1cs.CV

TL;DR

Unified open-vocabulary detection has often depended on complex fusion, staged training, and iterative annotation, raising whether stronger foundation models can support a simpler approach. OPUS combines semantic-rich visual features, unified prompt reasoning, one-stage text-visual training, and single-pass grounding, achieving strong cross-task performance and complementary mixed prompting. Its main boundary is that generic visual prompting remains sensitive to reference quality and prototype noise.

  • Problem

    Unified OVD has relied on heavy fusion, staged training, and iterative annotation, motivating investigation of whether this complexity is necessary.

  • Method

    OPUS uses a semantic-rich DINOv3 encoder with a prompt-aware decoder, one-stage ICA training, and a SAM3-based single-pass grounding pipeline.

  • Results

    OPUS achieves state-of-the-art Visual-I performance while maintaining strong Text and Visual-G accuracy, and mixed prompting improves over individual prompts.

  • Takeaways & Limitations

    Unified OVD can combine simple design with strong prompting capability and complementary text-visual interaction.

  • Takeaways & Limitations

    Generic visual prompting remains sensitive to reference quality and prototype noise because training and inference use mismatched visual prompts.

Abstract

from arXiv · show

Recent unified open-vocabulary detection (OVD) supports heterogeneous prompts, including text queries, visual exemplars, and their combinations, but often rely on increasingly complex designs such as heavy cross-modal fusion, staged training, and iterative annotation pipelines. We revisit whether such complexity is necessary in the era of stronger foundation models. Our finding is that unified OVD can be made substantially simpler with semantic-rich visual representations and scalable grounding supervision. We present OPUS (\textbf{O}pen-vocabulary, \textbf{P}rompt-\textbf{U}nified, \textbf{S}imple), a unified detector supporting text, interactive visual, generic visual, and mixed prompting within one framework. OPUS adopts a simple three-part design. Its model architecture combines a semantic-rich visual encoder, built on a DINOv3-ConvNeXt-B backbone with efficient hybrid encoding, with a prompt-aware decoder that avoids prompt-specific branches for unified prompt reasoning. OPUS is trained with a one-stage text-visual training strategy with Instance-level Contrastive Alignment (ICA), and is supported by a SAM3-based single-pass data engine for heterogeneous grounding supervision. Experiments on COCO, LVIS-minival, and ODinW35 show that OPUS achieves state-of-the-art Visual-I performance, reaching 68.1/69.2/54.7 AP, while maintaining balanced Text and Visual-G accuracy. OPUS also turns mixed prompting from interference into complementarity, improving over text or visual prompt alone. These results show that simplicity and strong unified prompting capability can be achieved together.

1 Introduction

OPUS revisits whether unified open-vocabulary detection requires complex fusion, staged training, and iterative annotation, proposing a simpler foundation-model-based alternative. Its unified design achieves strong Visual-I performance and makes mixed text-visual prompting complementary.

  • Unified OVD methods often use heavy cross-modal fusion, staged training, and iterative re-annotation, motivating a simpler design.
  • Stronger DINOv3 visual representations and SAM3 grounding supervision motivate replacing complex OVD pipelines with single-pass supervision.
  • OPUS supports text, interactive visual, generic visual, and mixed prompting within one unified detector.
  • OPUS combines a semantic-rich DINOv3 encoder, prompt-aware decoder, one-stage ICA training, and a SAM3-based single-pass data pipeline.
  • OPUS achieves state-of-the-art Visual-I performance across COCO, LVIS-minival, and ODinW35 while maintaining competitive Text and Visual-G accuracy.On ODinW35 Visual-I, OPUS exceeds PET-DINO-T by +6.4 AP and DETR-ViP-T by +7.9 AP.
  • 49.6 to 49.9 on COCO and 43.0 to 45.2 on LVIS-minival show OPUS gains from mixed prompting, unlike T-Rex2’s 45.8 to 42.4 COCO decline.The analysis also reports a shift in the relative contributions of text and visual prompts as grounding supervision expands.

2 Related Work

Related work progresses from text-prompted detection toward unified text-visual prompting, while increasing architectural, training, and data-construction complexity. OPUS is positioned as a simpler route enabled by modern foundation models and single-pass grounding.

  • From Text-Prompted Detection to Visual Prompting: Early OVD methods primarily use category names or noun phrases as text queries through vision-language pretraining and region-text alignment.
  • Unified Prompting and System Complexity: Unified detectors extend OVD to visual prompts, using staged optimization, prompt-enriched training, or additional cross-modal interaction modules.
  • Unified Prompting and System Complexity: Grounding supervision is expanded through pseudo boxes, generative annotation engines, large multi-source corpora, and iterative relabeling pipelines.
  • Unified Prompting and System Complexity: These approaches broaden open-vocabulary coverage, but collecting and cleaning their data remain costly and difficult to reproduce.
  • Foundation Models for Simpler Unified OVD: DINOv3 and SAM3 enable semantically rich representations and scalable single-pass grounding, motivating simpler unified OVD designs.

3 Method

OPUS combines a semantic-rich image encoder, unified prompt generation, prompt-aware decoding, one-stage text-visual training, and single-pass grounding supervision. Its design supports heterogeneous prompts without prompt-specific decoder branches.

  • 3.1 Model Architecture: OPUS uses a semantic-rich image encoder, prompt-aware decoder, one-stage text-visual training, and single-pass grounding data engine.The framework maps heterogeneous prompts into a shared prompt space and conditions object queries on image and prompt features.
  • 3.1 Model Architecture: The visual encoder applies efficient hybrid encoding to DINOv3 multi-scale features without encoder-side image-text fusion.The hybrid encoder combines attention-based intra-scale interaction with CNN-based cross-scale fusion, while DINOv3 provides semantic transferability.
  • 3.1 Model Architecture: Text and visual prompts are converted into a unified representation that may use textual prompts, visual prompts, or their combination.Visual prompts are encoded from user-provided boxes or points into instance-level and class-level features before prompt construction.
  • 3.1 Model Architecture: The prompt-aware decoder initializes object queries through prompt-guided proposal selection and iteratively refines them using query self-attention, prompt cross-attention, and image cross-attention.The resulting decoding process supports text, interactive visual, generic visual, and mixed prompting without prompt-specific architectural branches.
  • 3.2 Training Strategy and Objective: OPUS alternates text-prompt and visual-prompt batches in one-stage training, using DETR-style detection and global prompt alignment losses.Visual-prompt batches additionally align class-level visual prompts with matched text prompts.
  • 3.2 Training Strategy and Objective: ICA adds instance-level coordination by treating matched query-content prompt pairs as positives and other content prompts in the minibatch as negatives.ICA is applied only to visual-prompt batches and is intended to preserve fine-grained instance information.
  • 3.2 Training Strategy and Objective: The SAM3-based data engine creates heterogeneous grounding supervision from text-image, classification, and segmentation sources in a single pass.The pipeline converts generated regions or existing masks into box annotations, applies unified semantic labels, and filters noisy annotations.

4 Experiments

Across zero-shot benchmarks and prompting modes, OPUS delivers strong Visual-I accuracy, balanced unified performance, and improved efficiency through semantic-rich encoding, prompt-aware decoding, ICA, and scalable grounding supervision.

  • Main Results: OPUS achieves the best Visual-I performance among the compared baselines on COCO, LVIS-minival, and ODinW35.It exceeds DETR-ViP-T by 2.7 AP on COCO and 3.1 AP on LVIS-minival, and surpasses PET-DINO-T and DETR-ViP-T by 6.4 and 7.9 AP on ODinW35.
  • Main Results: OPUS remains more balanced across Visual-I, Visual-G, and Text prompting than methods that specialize in a single prompt mode.DETR-ViP-T, for example, has strong Visual-G but weak Text performance.
  • Main Results: OPUS (Swin-T) improves Visual-I LVIS-minival performance by 11.3 AP over T-Rex-T, while the final ConvNeXt-B setting reaches the highest FPSgen of 21.3.The final setting uses hybrid encoding and 640×640 input, recovering Visual-G relative to OPUS (Swin-T) while remaining competitive in accuracy.
  • Main Results: OPUS stays within about 3 AP of DETR-ViP-L on COCO and LVIS-minival and leads ODinW35 by 3.5 AP despite a smaller backbone and lower resolution.This comparison is against larger Swin-L models.
  • Mixed Prompting Complementarity: Mixed prompting improves OPUS from 49.6 to 49.9 on COCO and from 43.0 to 45.2 on LVIS-minival, unlike T-Rex2-T's drop from 45.8 to 42.4 on COCO.The results indicate that OPUS combines text semantics and visual appearance cues more effectively.
  • Design Analysis: The semantic-rich encoder raises FPSgen from 15.4 to 22.0 while improving all prompting modes.The change replaces Swin-T with DINOv3-based ConvNeXt-B, reduces input resolution, and adopts a hybrid encoder.
  • Design Analysis: The prompt-aware decoder improves Visual-I from 57.5 to 63.0 and Text from 29.5 to 32.7, but reduces Visual-G from 35.3 to 33.3.The reported trade-off is associated with differences between same-image Visual-I training prompts and cross-image Visual-G inference prototypes.
  • Design Analysis: ICA raises rare-category Visual-G APr from 26.1 to 30.0 and Mixed AP from 36.5 to 38.0, while overall prompting AP changes by less than 0.5.The gains occur under O365+GoldG after Step C, with Text APr remaining 20.3.

5 Conclusion

OPUS presents a simple unified open-vocabulary detector built around semantic-rich visual representations, unified prompt reasoning, one-stage training, and scalable grounding supervision. Across benchmarks, it achieves strong Visual-I performance while maintaining Text and Visual-G accuracy and making mixed prompting complementary.

  • OPUS simplifies unified open-vocabulary detection through a semantic-rich DINOv3 backbone, prompt-aware decoder, one-stage text-visual training, and SAM3-based single-pass supervision.The decoder avoids prompt-specific branches, while training uses Instance-level Contrastive Alignment.
  • Mixed prompting becomes complementary rather than interfering, with Visual-G increasingly relying on textual semantics as grounding supervision expands.The analysis identifies noisy cross-image category prototypes as a limitation for generic visual prompting.

Limitations

OPUS’s main limitations concern generic visual prompting and the scope of its grounding data engine. Visual-G is sensitive to reference quality and prototype noise, while the SAM3-based pipeline remains validated only on standard natural-image benchmarks.

  • Generic visual prompting improves less than text and interactive visual prompting because training uses ground-truth instances whereas inference aggregates cross-image category prototypes.This mismatch makes Visual-G especially sensitive to reference quality and prototype noise for rare categories.
  • Current generic visual prompting requires more targeted modeling, including better reference selection, prototype aggregation, and prototype denoising.
  • The SAM3-based data engine depends on the quality and domain coverage of the underlying promptable model.
  • The annotation pipeline is validated on COCO, LVIS-minival, and ODinW35 but not on specialized medical, industrial, or remote-sensing imagery.Extending and validating it in those domains remains future work.

A.2 Training and Optimization

OPUS jointly learns text and visual prompting by alternating text and visual batches within one detector, with visual batches adding prompt interaction and auxiliary visual-prompt losses. Inference supports four prompt modes through shared prompt-space representations.

  • All model parameters are optimized end-to-end with AdamW during alternating training.
  • The detector supports text, interactive visual, generic visual, and mixed inference by dynamically constructing or retrieving the relevant prompt representations.Visual prompts are merged into the text embedding space through the same visual-to-text interaction used during training.

A.4 Data Engine

The SAM3-based data engine converts heterogeneous sources into grounding supervision through filtering, shared post-processing, and human verification. The resulting pseudo-labels achieve box accuracies near 90%, while residual errors vary by source.

  • The pipeline converts text-image, classification, and segmentation sources into Bamboo-CLS, CC3M, and annotated SA-1B supervision in a single pass.
  • Human verification reports box accuracies near 90% across Bamboo-CLS, CC3M, and SA-1B after filtering.A box is incorrect when its localization or assigned label is wrong.
  • Source-specific filtering grounds CC3M noun phrases, mitigates coarse or partial Bamboo-CLS labels, and removes unreliable SA-1B tags with category-dependent thresholds.
  • Shared post-processing applies confidence thresholding, category-wise and image-wise NMS, and filters problematic full-image or oversized boxes.
  • Residual failures include mislabeled, nested, and redundant background boxes, with nested or redundant boxes more common in Bamboo-CLS and CC3M and mislabels more common in SA-1B.

B.2.1 Visual-G Reference Sampling

The section evaluates how reference-image count affects Visual-G and Mixed prompting, establishing N=16 as a stable default and showing that text cues reduce sensitivity to reference count.

  • Reference-count sensitivity: Visual-G rises from 25.0 AP at N=1 to 38.6 AP at N=16 on LVIS-minival, then saturates beyond N=16.Rare categories may have fewer usable training images than the requested reference count.
  • Reference-count sensitivity: Mixed prompting already exceeds Text-only from N=4 on LVIS-minival and from N=16 on COCO.The text prompt provides an additional anchor, making performance vary less with N.
  • Training sensitivity: A 1:8 visual:text alternating ratio is adopted as a balanced default across Visual-I, Text, Visual-G, and Mixed prompting.A larger visual share favors Visual-I, whereas a larger text share favors Text, Visual-G, and Mixed.
  • Training sensitivity: Changing ICA temperature across 0.05, 0.07, and 0.10 alters results only slightly, with Visual-G varying within 0.3 AP.The default is τ=0.07.

C.1 Breakdown of Component-wise Ablation

The component ablation shows broad gains from the full OPUS progression, while exposing a Visual-G weakness caused by prototype mismatch that ICA selectively repairs, especially for rare categories.

  • Overall progression: The baseline-to-full-OPUS progression improves performance across most category groups and prompting modes, not only frequent categories.The breakdown covers APf, APc, and APr across the four prompting modes.
  • Decoder effects: Adding the prompt-aware decoder improves Text and Visual-I but temporarily reduces Visual-G, with the largest drop on rare categories.The reported interpretation attributes this to mismatch between Visual-I-style training prompts and cross-image Visual-G prototypes.
  • ICA effects: ICA raises Visual-G APr from 26.1 to 30.0, while APf gains slightly and APc remains below its predecoder level.The selective recovery suggests greater benefit when category-level visual prototypes are unreliable.
  • Prototype alignment: Class-level alignment pulls aggregated visual prototypes toward text features, which can weaken fine-grained visual details and transfer text-modality limitations to rare-category prototypes.The limitation is strongest when rare categories have less reliable text representations and cross-image visual prototypes.
  • Prototype alignment: ICA instead aligns instance-level visual prompt features with matched decoder queries grounded in image regions, providing a more localized and appearance-sensitive target.This supplies stronger supervision for rare categories than alignment based only on class-level text anchors.
  • Training strategy comparison: With only O365 visual-prompt training, the one-stage reimplementation reaches Visual-I 57.2/56.3 on COCO/LVIS-minival, compared with official staged-training results of 41.1/40.6 before SA-1B and 56.6/59.3 after.These are comparisons against reported results rather than controlled retraining under the same implementation.
  • Training strategy comparison: Under Visual-G, adding SA-1B lowers official T-Rex2-T results but leaves the one-stage reimplementation flat on COCO and higher on LVIS-minival.Official results change from 41.1 to 38.8 on COCO and 38.1 to 37.4 on LVIS-minival; the reimplementation changes from 41.3 to 41.4 and 33.6 to 40.5.

C.4 Visualizations

The visualizations illustrate OPUS across interactive visual, generic visual, and text prompting, emphasizing accurate localization, cross-image exemplar transfer, and broad prompt understanding.

  • Qualitative coverage: Qualitative examples cover interactive visual prompting, generic visual prompting, and text prompting across dense scenes, exemplar transfer, and zero-shot detection.These examples complement the quantitative evaluation by showing behavior across the three prompting settings.
  • Interactive visual prompting: OPUS shows stronger visual prompt understanding than T-Rex2 and PET-DINO for single-category and multi-category targets in diverse and dense scenes.This comparison is presented in Figure 6.
  • Interactive visual prompting: Compared with T-Rex2, OPUS produces fewer background false positives and more accurate instance localization in challenging interactive cases.This comparison is presented in Figure 7.
  • Generic visual prompting: OPUS transfers exemplar prompts from reference images to different test images while preserving category-level visual semantics across scene layout, object scale, and instance appearance changes.Figure 8 shows exemplars in the top row and corresponding detection results below.
  • Overall qualitative result: Together, the visualizations show robust prompt understanding across text, interactive visual, and generic visual prompting settings.Text-prompted zero-shot examples are shown separately in Figure 9.
Loading 2608.30247v1…