PaperlayerView exact revision on arXiv

Source-linked AI summary

Pro-Bench: Prompt-Robust Open-Vocabulary Visual Grounding Across Real-World Heterogeneous Environments

Linus Nwankwo, Muslim Alaran, Christian Rauch, Stanley Chukwuebuka Obilikpa, Elmar Rueckert

arXiv:2609.27076v1cs.CVcs.RO

TL;DR

Existing benchmarks largely use short category labels and web-sourced imagery, leaving open-vocabulary grounding of varied queries in real-world robotic conditions unclear. Pro-Bench combines heterogeneous robotic data, fine-grained annotations, and query reformulations to evaluate grounding and prompt robustness. Its results show architecture-dependent prompt sensitivity, with most evaluated configurations performing better with categorical prompts than with free-form queries.

  • Problem

    Existing benchmarks largely rely on short category labels and web-sourced or synthetic imagery, leaving open-vocabulary models’ grounding of diverse queries and real-world visual conditions unclear.

    Source: §I · p. 1

  • Method

    Pro-Bench combines heterogeneous real-world robotic images, annotated target queries across eight semantic types, prompt reformulations, and a zero-shot evaluation of 16 model configurations.

    Source: §I · p. 1, §III · p. 2

  • Results

    10/16 models show lower mAP.5 with original free-form queries than with categorical prompts, and prompt-formulation effects are strongly architecture-dependent.

    Source: C. Quantitative Results · p. 5

  • Takeaways & Limitations

    Prompt robustness warrants evaluation alongside grounding accuracy because target recovery and localisation stability are distinct properties across prompt reformulations.

    Source: §I · p. 1, C. Quantitative Results · p. 6

Abstract

from arXiv · show

Open-vocabulary visual grounding enables robots to localise task-relevant entities from natural-language queries without dependence on predefined perceptual taxonomies. However, existing benchmarks largely rely on short category labels and web-scraped imagery, leaving it unclear whether open-vocabulary models can robustly ground diverse queries and visual conditions under real deployments. We introduce \textbf{Pro-Bench}, a prompt-conditioned benchmark for open-vocabulary visual grounding in heterogeneous, real-world environments. Pro-Bench includes $13k+$ RGB frames from independent robotic domains (subterranean, industrial, indoor, outdoor, urban), with $74.5k$ manual instance annotations and $515$ target queries covering categorical, attributive, relational, affordance, state, part-whole, negative, and compositional semantics. We benchmarked $16$ open-vocabulary model configurations in strict zero-shot inference, measuring localisation accuracy across IoU thresholds, end-to-end inference latency, prompt-induced performance variation, and target recovery consistency. Our results show that prompt-robustness is strongly architecture-dependent. Most model configurations ($10/16$) perform best with short category labels, whereas free-form queries yield the highest accuracy for only one. Moreover, similar aggregate mAP can conceal substantial differences in consistent target recovery across reformulations. Pro-Bench enables systematic evaluation of these gaps and supports prompt-robust visual grounding. Pro-Bench: https://pro-bench.github.io/.

I. INTRODUCTION

Open-vocabulary models let robots identify task-relevant objects through natural-language prompts, but existing benchmarks leave their performance on richer queries, real-world imagery, and prompt reformulations unclear. Pro-Bench addresses these gaps with heterogeneous robotic data, diverse target prompts, and a protocol for assessing prompt robustness.

I. INTRODUCTION

  • Existing benchmarks emphasize short category labels and web-sourced or synthetic imagery, leaving richer semantics and robustness under robotic visual conditions unclear.The missing query types include attributes, relations, compositions, affordances, states, and constituent parts; relevant conditions include low illumination, occlusion, unusual viewpoints, and sensor artefacts.Source: §I · p. 1
  • Pro-Bench combines heterogeneous robotic images, fine-grained target annotations, and 515 prompt-defined queries spanning eight semantic types.The benchmark includes more than 13k RGB frames and 74.5k annotated instances, including compositional multi-constraint queries.Source: §I · p. 1
  • Its prompt-robustness protocol measures both performance changes under query reformulation and consistency of target recovery and localisation.Prompt-consistent recall and localisation consistency assess whether the same ground-truth target is recovered across reformulations and whether its localisation remains spatially consistent.Source: §I · p. 1
  • The benchmark evaluates 16 open-vocabulary model configurations without Pro-Bench-specific fine-tuning and is intended to support reproducible benchmarking and future extensions.The planned release includes the dataset, annotations, utilities, and per-model configurations.Source: §I · pp. 1–2

II. RELATED WORK

Prior benchmarks cover broad categories, varied visual domains, or fine-grained language, but the reviewed work does not combine heterogeneous robotic deployments with evaluation of target-preserving prompt reformulations. Pro-Bench is constructed from real-world robotic data and target-grounded queries.

II. RELATED WORK

  • Large-scale categorical benchmarks provide broad class coverage but do not assess richer constraints such as attributes, spatial relations, and physical state.The reviewed examples include COCO, LVIS, Objects365, V3Det, and Open Images V4.Source: §II · p. 2
  • Domain-diverse benchmarks extend beyond everyday web imagery, while retaining mostly category-level supervision or, for RefDrone, focusing on aerial viewpoints.The reviewed examples are ODinW, R.Flow 100 [33], LAE-80C, and RefDrone.Source: §II · p. 2
  • Fine-grained and referential benchmarks include expressions, attributes, subcategories, and sometimes negative queries, but do not evaluate target-preserving reformulation across heterogeneous robotic domains.The reviewed work includes, –, gRefCOCO, OmniLabel, D3, and OVDEval.Source: §II · p. 2

III. THE PRO-BENCH

  • Pro-Bench is constructed from public and self-curated real-world robotic datasets, with fine-grained instance annotations and target-grounded natural-language queries.Its eight prompt types are designed to reflect how human operators may describe, identify, and refer to objects in robotic missions.Source: §III · p. 2

A. Pro-Bench Datasets and Domains

Pro-Bench draws on in-situ robot-sensed RGB frames from heterogeneous operating conditions, excluding non-robot, web-scraped, and synthetic imagery. Its data span subterranean and industrial, outdoor and urban, and indoor and laboratory environments.

A. Pro-Bench Datasets and Domains

  • The dataset construction selects real robot-sensed RGB frames to retain deployment-relevant variation in illumination, viewpoint, occlusion, scene structure, and object appearance.Non-robot, web-scraped, and synthetic image modalities are excluded.Source: A. Pro-Bench Datasets and Domains · p. 2
  • Subterranean and industrial scenes cover tunnels, quarry operations, and mining, including low illumination, constrained geometry, and domain-specific infrastructure.Sources include DARPA, EnvoDat, Hilti-SLAM, and GOOSE-Ex.Source: A. Pro-Bench Datasets and Domains · p. 2
  • Outdoor and urban scenes include forest trails, unstructured terrain, and streets, with dynamic entities and strong lighting or viewpoint changes.The passage names EnvoDat, GOOSE-Ex, KITTI, and MCD multi-campus data, along with self-collected campus data.Source: A. Pro-Bench Datasets and Domains · pp. 2–3
  • The construction pipeline culminates in more than 13k images, 74.5k instance annotations, and 515 prompts; the comparison table defines eight prompt taxonomies and a sensitivity indicator.The table caption says sensitivity indicates whether robustness to alternative prompt formulations is evaluated; it does not provide comparative outcomes in the supplied passage.Source: Figure 2 · p. 3, Table I · p. 3
  • Indoor and laboratory data include office, laboratory, and tabletop scenes, where structured layouts coexist with visual clutter and similar-looking objects.Sources include TUM RGB-D, EnvoDat, and in-house collected scenes.Source: A. Pro-Bench Datasets and Domains · p. 3

B. Prompt Taxonomy and Construction

Pro-Bench associates annotated targets with free-form queries and semantic prompt types, then derives categorical and template-wrapped variants to test prompt sensitivity. Its annotations use manually created, fine-grained polygon boundaries.

B. Prompt Taxonomy and Construction

  • Pro-Bench uses free-form natural-language expressions so targets can be queried beyond canonical class names.Each target has an original query and at least one assigned prompt type.Source: B. Prompt Taxonomy and Construction · pp. 3–4
  • The taxonomy includes categorical, attributive, relational, affordance, state, part-whole, negative, and compositional prompts.Compositional queries require two or more jointly necessary constraints, with no single constituent constraint sufficient to identify the target.Source: B. Prompt Taxonomy and Construction · p. 4

C. Target Annotation and Prompt Variants

  • Seven independent annotators create instance-level polygon boundaries, and an additional reviewer checks each annotation against its intended target.The annotations cover common and uncommon objects and support both box- and mask-based evaluation.Source: C. Target Annotation and Prompt Variants · p. 4
  • Ground-truth annotations are supplied in multiple formats, including COCO, for visual grounding applications.Fine polygon boundaries are intended to minimise overlap between neighbouring instances.Source: C. Target Annotation and Prompt Variants · p. 4
  • Each target’s original query is paired with a manually reviewed categorical form that preserves the intended target while abstracting additional semantic constraints.The abbreviated form is a concise class-level noun phrase; the abstracted constraints can include attributes, relations, state, or affordance.Source: C. Target Annotation and Prompt Variants · p. 4
  • The template-wrapped query is derived from the original query to test robustness to prompt specification.The provided template text begins “a photo of” before the original prompt.Source: C. Target Annotation and Prompt Variants · p. 4

D. Prompt-Robustness and Spatial Stability

The section compares target-preserving prompt formulations and defines measures of prompt-induced performance variation, target recovery, and spatial consistency.

D. Prompt-Robustness and Spatial Stability

  • The evaluation asks how free-form and template-wrapped prompts affect mAP.5, whether a prompting strategy generalises across architectures, and whether reformulations change target recovery or localisation.It compares categorical, original, and template-wrapped formulations.Source: D. Prompt-Robustness and Spatial… · p. 4
  • Prompt sensitivity is measured through mAP.5 differences between original and categorical prompts and between template-wrapped and original prompts, with the maximum variation indicating sensitivity.The section also records each model’s best-performing formulation.Source: D. Prompt-Robustness and Spatial… · p. 4
  • Prompt-consistent recall measures whether the same target is recovered across target-preserving reformulations, using only reformulations verified to preserve the intended referent.Experiments set the recovery IoU threshold to 0.5.Source: D. Prompt-Robustness and Spatial… · p. 4
  • Prompt localisation consistency is the mean pairwise IoU among prompt-conditioned predictions for targets recovered under all corresponding formulations; higher values indicate stronger spatial consistency.The spatial-extent analysis considers targets recovered under all verified corresponding formulations.Source: D. Prompt-Robustness and Spatial… · p. 4

A. Models and Experimental Setup

The study evaluates 16 open-vocabulary configurations in strict zero-shot conditions, using frozen pretrained weights and each model’s native inference protocol.

A. Models and Experimental Setup

  • The evaluation covers 16 representative open-vocabulary configurations spanning transformer-based detectors, real-time single-stage models, and segmentation-based models.The configurations use publicly hosted pretrained weights.Source: A. Models and Experimental Setup · pp. 4–5
  • The experiments use each model’s published prompt-conditioning interface and run on the same NVIDIA RTX 4090 GPU with 24 GB VRAM.The selected open-source weights are available through Hugging Face, Ultralytics, or GitHub.Source: A. Models and Experimental Setup · pp. 4–5
  • All models are tested in strict zero-shot inference with frozen pretrained weights and no Pro-Bench-specific fine-tuning.Predictions are cached for reproducible metric computation.Source: A. Models and Experimental Setup · p. 5

B. Evaluation Protocol

The protocol evaluates localisation with bounding-box average precision and measures end-to-end latency alongside prompt robustness.

B. Evaluation Protocol

  • Detection predictions are ranked by confidence within each model-query pair and matched one-to-one to ground-truth instances at IoU thresholds.The protocol reports mAP.5, mAP.75, and mAP.5:.95, with the latter averaging AP from IoU 0.5 to 0.95 in 0.05 increments; it also extends the IoU sweep to 0.1–0.9 and 0.95.Source: B. Evaluation Protocol · p. 5
  • Bounding-box AP is the primary localisation metric because it enables consistent comparison across detection- and segmentation-based models.Polygon annotations are retained for mask-based evaluation of segmentation-capable models and future extensions.Source: B. Evaluation Protocol · p. 5
  • End-to-end latency includes preprocessing, the forward pass, and post-processing, measured over ten runs on 50 images after three warm-ups on an NVIDIA RTX 4090.CUDA synchronisation is used before and after inference, while each model follows its native prompt-processing protocol under a constant query workload.Source: B. Evaluation Protocol · p. 5

C. Quantitative Results

Across tested configurations, localisation, prompt-formulation sensitivity, latency, and target-recovery consistency differ substantially by model. Short categorical prompts are best for most configurations, while high spatial consistency can coexist with poor recovery.

C. Quantitative Results

  • 85.1% of SAM3's mAP.5 is retained under mAP.5:.95, versus 80.7% for SAM3-Lite and 78.1% for OWLv2-L; several configurations decline sharply beyond IoU 0.5.Fig. 3 plots mAP against IoU threshold across the model configurations.Source: Figure 3 · p. 5, C. Quantitative Results · p. 5
  • 10/16 configurations favour categorical prompts, while free-form queries improve mAP.5 for 6/16 and decrease it for 10/16; only YOLOE-X favours free-form.Template wrapping improves 5/16 configurations and degrades 11/16; SAM3 falls from 0.417 to 0.044.Source: C. Quantitative Results · p. 5
  • SAM3-Lite has the strongest recovery consistency, while SAM3 pairs PLC 0.984 with PCR 0.024, showing stable localisation among recovered targets can coexist with very low recovery.Aggregate mAP alone often conceals differences in whether the same physical target is recovered under reformulation.Source: C. Quantitative Results · pp. 5–6
  • SAM3-Lite achieves the strongest overall localisation across mAP.5, mAP.75, and mAP.5:.95.Table II summarises the zero-shot detection, localisation, efficiency, and prompt-robustness results.Source: C. Quantitative Results · p. 5, Table II · p. 6
  • At the 33.3 ms (30 Hz) reference, OmDet-T provides the strongest accuracy; beyond it, OWLv2-B/L offer better accuracy at increased cost, while SAM3-Lite is strongest but slowest.YOLO-W-S is fastest at 7.3 ms per image, with mAP.5 = 0.14.Source: Figure 5 · p. 6, C. Quantitative Results · p. 6

D. Qualitative Results

Qualitative examples show that semantic recovery does not guarantee precise localization, and that model behavior diverges across architectures and semantically constrained queries.

D. Qualitative Results

  • Similar aggregate mAP can conceal different target-recovery and localization consistency; OWLv2-L localizes the tunnel exit door compactly, while YOLO-W-X predicts a region spanning much of the tunnel.The examples also show that a model may recover the correct semantic region but localize it too coarsely.Source: D. Qualitative Results · pp. 6–7
  • Figure 6 presents top-confidence predictions per prompt from four representative model configurations, with inference on original images and outputs resized only for visualization.Source: Figure 6 · p. 7
  • All four representative models produce no valid visible prediction for “an object suitable for sensor calibration,” despite the target being clearly visible.The examples show that recognizing a plausible category does not ensure that a model resolves the query’s semantic constraints.Source: D. Qualitative Results · p. 7

V. CONCLUSION

Pro-Bench evaluates open-vocabulary grounding with linguistically diverse queries in heterogeneous real-world robotic environments. Results show architecture-dependent prompt effects and distinguish consistent target recovery from localization stability.

V. CONCLUSION

  • Most evaluated model configurations perform best with short categorical labels rather than free-form operator-style queries, and prompt reformulation effects vary strongly by architecture.Template wrapping severely degrades some models’ behavior.Source: §V · p. 7
  • Pro-Bench evaluates grounding under linguistically diverse operator-style queries across heterogeneous real-world robotic environments.Source: §V · p. 7
  • PCR and PLC distinguish target-recovery consistency from localization stability: models can localize reliably when recovery succeeds yet recover only a limited fraction of targets across reformulations.Source: §V · p. 7
  • Future work will extend Pro-Bench to temporal video, interaction-driven grounding, and broader characterization of prompt robustness in embodied settings.Source: §V · p. 7
Loading 2609.27076v1…