Vague2Detect: Handling Ambiguous Prompts in Knowledge-Based Open-World Detection

Ibrohimjon Muminov; Jihie Kim

Source-linked AI summary

Vague2Detect: Handling Ambiguous Prompts in Knowledge-Based Open-World Detection

Ibrohimjon Muminov, Jihie Kim

arXiv:2609.09949v1Sep 9, 2026cs.CVcs.CLcs.LG

TL;DR

Functional and ambiguous prompts expose a gap in detectors that rely on explicit class labels or misalign vague language with objects. Vague2Detect addresses this with fine-tuned SBERT retrieval from a structured household Knowledge Base, YOLO-World visual verification, and GPT fallback for unseen concepts. On household-scene benchmarks, the system reaches 85% VPSR and 83% detection accuracy, while its scope remains constrained by limited KB coverage and errors on fine-grained or visually similar classes.

Problem
Functional prompts are insufficiently handled because conventional benchmarks emphasize explicit labels and open-vocabulary detectors can misalign vague language with intended objects.
Method
Vague2Detect retrieves Knowledge Base candidates with fine-tuned SBERT, verifies them using YOLO-World, and uses GPT fallback to expand coverage for unseen categories.
Results
85% VPSR and 83% detection accuracy were achieved on a household-scene benchmark constructed from Open Images and the authors’ photographs.
Takeaways & Limitations
The modular design supports affordance-driven and task-oriented perception by linking functional language to visually grounded object detections.
Takeaways & Limitations
The system is constrained by a Knowledge Base of approximately 100 objects and occasional errors on fine-grained or visually similar classes.

Abstract

from arXiv · show

Real-world detectors must often interpret functional or ambiguous prompts, yet conventional models such as YOLO remain restricted to fixed class lists. Even open-vocabulary models like YOLO-World frequently misalign vague language with the intended objects. Building on our prior work Commonsense-Guided Open-World Object Detection Using LLMs and Visual-Semantic Matching, we address YOLO-World's limitations in grounding task-driven queries. We propose Vague2Detect, a hybrid pipeline in which a fine-tuned Sentence-BERT retrieves candidates from a structured household Knowledge Base (KB), and YOLO-World verifies their presence in the image. For prompts outside the KB, a large language model (GPT-3.5-turbo) generates candidate descriptions, dynamically expanding the KB to cover novel concepts. On a benchmark of household scenes using custom images and an Open Images V7 subset, YOLO-World alone achieves only 32% Vague Prompt Success Rate (VPSR), the ability to map ambiguous queries to correct detections. In contrast, Vague2Detect improves performance to 61% VPSR with high precision, and up to 85% when augmented with GPT fallback.

1 Introduction

Vague2Detect addresses the gap between functional language and concrete object detection by combining a knowledge base, fine-tuned semantic retrieval, visual verification, and fallback generation. It introduces a benchmark and VPSR metric for evaluating this setting.

Motivation: Functional prompts remain under-evaluated because conventional benchmarks emphasize explicit category labels, despite open-vocabulary detectors still misaligning indirect queries.COCO-style evaluation can miss failures on affordance-driven language.
Contributions: Vague2Detect combines an LLM-bootstrapped household Knowledge Base, fine-tuned SBERT, YOLO-World verification, and GPT fallback for vague-prompt detection.The proposed system targets functional grounding and out-of-KB queries through complementary symbolic, neural, and generative components.
Novelty: Vague2Detect extends prior work by addressing affordance-driven prompts that previously required ambiguous queries to map to objects already present in the Knowledge Base.The paper also adds semantic grounding through fine-tuned SBERT and a dedicated benchmark and metric.
Contributions: The method’s stated contributions include functional SBERT grounding, GPT-based expansion for out-of-KB queries, and visual verification through YOLO-World.These modules are presented as complementary benefits of the hybrid design.
Contributions: The paper introduces a household benchmark built from Open Images and custom data, together with the Vague Prompt Success Rate metric.The benchmark is designed to measure whether vague queries map to correct detections.

2 Related Work

The paper positions Vague2Detect as a more adaptive and streamlined alternative to prior knowledge- and language-based detection systems. Its design combines fine-tuned retrieval, selective LLM fallback, and visual validation to handle functional and unseen-object queries.

Efficiency and adaptability: Compared with the earlier framework, the proposed design avoids running multiple heavyweight models in parallel and uses one LLM for KB bootstrapping and fallback.The paper characterizes the resulting system as lighter and more adaptive for real-time or resource-constrained settings.
Open-vocabulary detection: Prior open-vocabulary methods generally assume explicit labels or near-synonyms, leaving functional prompts insufficiently addressed.Vague2Detect instead combines persistent knowledge with fine-tuned retrieval for reusable affordance grounding.
Knowledge resources: The paper describes a middle ground between static knowledge resources and direct LLM querying by reserving GPT fallback for rare or unseen prompts.The intended trade-off is controlled vocabulary expansion with efficiency, transparency, and reduced reliance on costly queries.
Semantic retrieval: Fine-tuning SBERT on 500 varied prompt–object pairs improves household-affordance grounding over general-purpose BGE embeddings.The diversified prompts are designed to reduce semantic drift between functional language and intended object classes.
Knowledge-based grounding: Vague2Detect dynamically expands its Knowledge Base at inference time with GPT suggestions that YOLO-World verifies before acceptance.This combines adaptability to unseen concepts with grounding in detected visual evidence.

3 Methodology

Vague2Detect maps an affordance-driven query to a Knowledge Base object with fine-tuned SBERT, checks visual presence using YOLO-World, and invokes GPT when retrieval lacks a suitable entry. Verified fallback candidates are added to the Knowledge Base for future queries.

Pipeline: Vague2Detect first retrieves a semantically relevant object with SBERT, then passes qualifying candidates to YOLO-World for visual verification.The pipeline therefore checks both linguistic alignment and image presence before returning a detection.
GPT fallback: When no Knowledge Base entry exceeds the similarity threshold, GPT generates visual and usage descriptions that are added to the KB and validated by YOLO-World.Only candidates detected in the current image are retained, limiting hallucinated knowledge expansion.
Knowledge Base construction: The Knowledge Base contains more than 100 household objects with complementary visual and usage descriptions, enabling retrieval from both perceptual and affordance-driven queries.The prompt corpus contains approximately 500 varied prompt–object pairs for SBERT fine-tuning.
Semantic retrieval: SBERT accepts the top retrieved object when its maximum similarity score reaches the threshold τ=0.6; otherwise, the fallback module is triggered.At lower confidence, the system can check multiple top-k candidates before invoking GPT.
Visual verification: YOLO-World rejects semantically plausible candidates absent from the image, while its real-time operation keeps verification cost low relative to GPT calls.The detector is used directly as an open-vocabulary verifier rather than fine-tuned for this pipeline.

4 Benchmark Setup

The benchmark combines household images with in-KB and out-of-KB objects to test vague-prompt grounding, visual detection, and dynamic KB expansion. Results show that semantic grounding and GPT fallback substantially improve over direct YOLO-World detection.

Dataset Construction: The benchmark combines approximately 1000 household images with target labels and bounding boxes from Open Images and indoor photographs.The dataset is designed to reflect everyday household settings.
Dataset Construction: In-KB and out-of-KB object groups test semantic retrieval for known categories and GPT-based dynamic expansion for novel concepts.Examples include sink, microwave, and toothbrush in the KB, versus speaker among out-of-KB objects.
Qualitative Results: Qualitative cases show correct grounding for sink and knife prompts, speaker expansion through GPT fallback, and remaining fine-grained ambiguity errors.The examples include a baseline failure on “I need something to cut with,” where fine-tuning correctly maps the prompt to knife.

5 Experimental Setup

The evaluation uses a household benchmark containing common and rare objects, with vague prompts mapped to classes and localized using bounding-box overlap. Three configurations isolate direct detection, SBERT grounding, and GPT-enabled KB expansion.

Dataset Construction: The benchmark contains approximately 1000 images spanning 60 household-object classes, combining Open Images with photographs from real indoor environments.Each sample includes an image, ground-truth object label, and bounding-box annotation.
Evaluation Protocol: The task maps an affordance-driven prompt and image to the most appropriate object and its location in the scene.An example prompt is “something to cut fruits.”
Evaluation Protocol: VPSR measures successful grounding to the correct object class, while Det.Acc additionally requires IoU ≥0.5 with the ground-truth box.Together, the metrics assess linguistic grounding and visual localization.
Compared Configurations: The compared configurations are direct YOLO-World detection, fine-tuned SBERT plus YOLO-World, and a full pipeline with GPT fallback for out-of-KB queries.The full pipeline supports continual KB expansion and broader coverage.
Knowledge-Base Adaptation: The dynamic pipeline uses GPT to generate and validate rare or unconventional KB entries while resolving most queries within the existing KB.This design keeps GPT usage minimal for ordinary queries.

6 Conclusion

Vague2Detect combines a curated Knowledge Base, fine-tuned SBERT, YOLO-World verification, and GPT fallback for unseen categories. It reaches strong benchmark performance, while remaining limited by KB coverage, fine-grained recognition errors, and GPT dependence.

Conclusion: Vague2Detect combines a curated Knowledge Base, fine-tuned SBERT, YOLO-World visual verification, and GPT-based KB expansion.The components jointly address functional language and unseen categories.
Conclusion: 85% VPSR and 83% detection accuracy were achieved on household scenes, substantially outperforming the reported baselines.The benchmark combines Open Images with the authors’ household photographs.
Conclusion: Ablations associate SBERT with improved functional grounding, YOLO-World with visual verification, and GPT fallback with dynamic knowledge expansion.Each module contributes a distinct role in the pipeline.
Limitations: The system is constrained by a roughly 100-object KB, occasional confusion among visually similar classes, and dependence on GPT fallback.The authors identify examples such as salt versus pepper and note the fallback’s cost.
Conclusion: The modular design extends detection from identifying objects toward reasoning about why they are needed in task-oriented perception.The stated direction concerns affordance-driven perception systems.

Additional Results

Additional system logs illustrate KB retrieval, GPT-based expansion, and YOLO-World verification for functional prompts. The examples cover both unseen concepts and objects already represented in the KB.

Fallback Example: The prompt “something to listen to music with” triggers GPT fallback, adds speaker to the KB, and yields a speaker detection with confidence 0.98.No suitable KB match was found before fallback activation.
In-KB Retrieval: The prompt “I need something sharp to cut the fish” retrieves knife directly from the KB and detects it with confidence 0.84.Fallback is unnecessary when the relevant object is already represented.
In-KB Retrieval: The prompt “Where can I wash my dishes?” maps to sink through the KB and produces a sink detection with confidence 0.81.The KB description links sink with washing hands, dishes, or food.