Source-linked AI summary
DetGPT: Detect What You Need via Reasoning
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, Tong Zhang
TL;DR
The paper addresses the gap between natural-language visual reasoning and precise object localization. DetGPT introduces reasoning-based object detection by combining a multimodal model with an open-vocabulary detector, and demonstrates localization of relevant objects, including objects absent from task-tuning data. The authors also identify bottlenecks in both stages of the two-stage pipeline.
Problem
Existing multimodal models lack fine-grained visual understanding and precise localization, while detection methods rely on specific category phrases that may not describe the intended objects.
Method
DetGPT uses a multimodal model to reason from images and natural-language instructions, then passes relevant object names to an open-vocabulary detector for localization.
Results
DetGPT interprets human instructions and localizes relevant objects, including objects missing from the task-tuning set.
Takeaways & Limitations
Reasoning-based object detection enables abstract natural-language queries to guide object localization rather than requiring explicit object names.
Takeaways & Limitations
The two-stage design is bottlenecked by failures of the open-vocabulary detector and the multimodal model.
Abstract
from arXiv · showhide
In recent years, the field of computer vision has seen significant advancements thanks to the development of large language models (LLMs). These models have enabled more effective and sophisticated interactions between humans and machines, paving the way for novel techniques that blur the lines between human and machine intelligence. In this paper, we introduce a new paradigm for object detection that we call reasoning-based object detection. Unlike conventional object detection methods that rely on specific object names, our approach enables users to interact with the system using natural language instructions, allowing for a higher level of interactivity. Our proposed method, called DetGPT, leverages state-of-the-art multi-modal models and open-vocabulary object detectors to perform reasoning within the context of the user's instructions and the visual scene. This enables DetGPT to automatically locate the object of interest based on the user's expressed desires, even if the object is not explicitly mentioned. For instance, if a user expresses a desire for a cold beverage, DetGPT can analyze the image, identify a fridge, and use its knowledge of typical fridge contents to locate the beverage. This flexibility makes our system applicable across a wide range of fields, from robotics and automation to autonomous driving. Overall, our proposed paradigm and DetGPT demonstrate the potential for more sophisticated and intuitive interactions between humans and machines. We hope that our proposed paradigm and approach will provide inspiration to the community and open the door to more interative and versatile object detection systems. Our project page is launched at detgpt.github.io.
1 Introduction
Multi-modal models can interpret instructions and reason over images, but robots still need accurate object localization for physical actions. The paper proposes reasoning-based object detection, combining language-driven reasoning with open-vocabulary detection so users can specify goals rather than exact object names.
- 1 Introduction: Robots require accurate object localization to perform physical actions, which multi-modal conversation alone does not provide.Object detection is presented as a prerequisite for actions such as moving and grasping objects.
- 1 Introduction: Open-vocabulary detectors expand beyond fixed trained classes but still require users to provide accurate object categories.This requirement is difficult when users lack the knowledge needed to name relevant objects.
- 1 Introduction: Reasoning-based object detection lets users provide natural-language queries while the model reasons which image objects satisfy the requested goal.The proposed system uses a multimodal model to infer relevant objects and an open-vocabulary detector to predict their locations.
- 1 Introduction: DetGPT is supported by a fine-tuning dataset containing 5000 images and around 30000 query-answer pairs.The dataset is open-sourced to support instruction-following research.
- 1 Introduction: The task is illustrated as a pipeline in which an LLM interprets instructions and visual context while a detector outputs the objects of interest.The LLM supplies reasoning and the detector supplies visual localization.
2 Related Work
Related work covers the progression from large language models to multimodal models and object detectors. It highlights that traditional detectors are limited to trained categories, motivating open-vocabulary alternatives.
- 2 Related Work: Large language models have advanced language understanding and generation, motivating subsequent research on vision-language interaction.Multimodal models extended this trajectory toward cross-modal understanding tasks.
- 2 Related Work: Traditional object detectors use fixed classification heads, limiting predictions to classes represented during training.This limitation restricts their practicality for broader object categories.
- 2 Related Work: Open-vocabulary object detection emerged to address the restricted category coverage of traditional detectors.The supplied passage introduces this direction after describing fixed-category detection.
3 Problem Statement
Existing multimodal models reason over visual scenes but lack fine-grained understanding and precise localization, while detectors depend on specific category phrases. Reasoning-based object detection addresses this mismatch by mapping abstract natural-language goals to localized objects.
- 3 Problem Statement: Existing multimodal models support visual reasoning but lack fine-grained understanding and precise localization for embodied applications.The passage specifically identifies robotics and autonomous driving as affected scenarios.
- 3 Problem Statement: Reasoning-based object detection asks models to interpret abstract natural-language queries, infer which image objects fulfill the user’s goal, and detect their locations.This task is proposed to address the limitations of both multimodal reasoning and category-based detection.
- 3 Problem Statement: For a request for a cold beverage, the model can infer that a refrigerator may contain one even when the beverage is not directly identified.The inference uses common-sense knowledge associated with refrigerators.
- 3 Problem Statement: The task supports applications such as finding missing cooking ingredients, lost keys, items needing cleaning, or specific books.These examples connect abstract queries with practical domestic-robot capabilities.
4 Query-Answer Instruction Data Generation
The instruction-data generation process uses existing image captions and object categories to provide visual context for ChatGPT. It produces detailed descriptions and query-answer pairs containing reasoning and matching object names.
- 4 Query-Answer Instruction Data Generation: Automatic instruction-data generation addresses the labor required for manual labeling, while text-only LLMs cannot directly interpret image inputs.The method therefore relies on existing visual annotations as an intermediate representation.
- 4 Query-Answer Instruction Data Generation: The pipeline combines image captions and object categories to bridge visual content and textual instruction annotations.Captions describe image content from different viewpoints, while categories identify present objects.
- 4 Query-Answer Instruction Data Generation: For each image, ChatGPT generates one detailed description followed by multiple instruction-answer pairs containing reasoning and matching object names.The annotations are reorganized so each image remains associated with its corresponding instruction-answer pairs.
5 Method
DetGPT performs reasoning-based object detection by interpreting natural-language instructions with a multimodal model, then localizing the inferred object names through an open-vocabulary detector. Its method combines staged alignment and instruction tuning with prompt-constrained inference outputs.
- 5.1 Model Architecture: DetGPT first interprets the image and user instruction to identify relevant object names or phrases, then passes them to an open-vocabulary detector for localization.The framework uses a vision encoder and language model for reasoning, followed by detector-based localization.
- 5.2 Training: Image-text pretraining aligns visual and textual features using SBU, LAION, and Conceptual Caption while training only the projection linear layer.The authors report that tuning this layer suffices to align BLIP-2 visual features with a new language model such as Vicuna.
- 5.2 Training: Instruction tuning trains the model on curated query-answer pairs to interpret user goals and generate matching object categories after reasoning.The model receives an image and instruction and is trained with a language-modeling loss; the authors report improved fine-grained image understanding and structured object summarization.
- 5.2 Training: Prompting stabilizes object extraction by requiring names or phrases to follow a specific output format during training and inference.At inference, DetGPT extracts names following “Therefore the answer is:” and sends them with the image to the detector.
6 Demonstration
DetGPT demonstrates reasoning from natural-language queries and visual scenes to identify and localize relevant objects, including objects implied rather than explicitly named.
- DetGPT reasons from a user query and image to identify an object that satisfies the user’s goal, even when it is not directly visible.For a request for a cold beverage in a kitchen, it infers that a refrigerator may contain one and localizes the fridge.
- DetGPT uses knowledge stored in LLMs to reason beyond ordinary common sense, such as linking potassium-containing foods with relief of high blood pressure.The example identifies bananas and apples as relevant foods.
- DetGPT handles abstract, non-exhaustive instructions by identifying relevant objects in the image, such as detecting a cigarette among items inappropriate for children.
- DetGPT generalizes to objects absent from its task-tuning set, including game props in a videogame screenshot.Because task tuning uses COCO’s 80 daily object categories, the authors attribute this broader generalization to pretraining.
7 Limitation
DetGPT’s two-stage design inherits bottlenecks from both its open-vocabulary detector and multi-modal model.
- The detector may fail to localize objects that the multi-modal model correctly identifies when its training data lacks the relevant visual concepts.
- The multi-modal model may miss relevant objects because it lacks fine-grained visual recognition ability.
- These limitations motivate more advanced solutions for reasoning-based object detection.
8 Conclusion
The paper introduces reasoning-based object detection and DetGPT, a two-stage system that derives relevant objects from instructions and images before localizing them.
- The proposed task requires interpreting human instructions, reasoning over a visual scene, and localizing the objects of interest.
- DetGPT first uses a multi-modal model to derive objects matching the query, then uses an off-the-shelf open-vocabulary detector to localize them.
- A ChatGPT-assisted task-tuning dataset supports instruction interpretation and localization, including objects absent from the task-tuning dataset.
- Figures 3 and 4 demonstrate DetGPT’s reasoning process and generated bounding boxes.
- Figure 5 demonstrates failure cases involving the multi-modal model and open-vocabulary detector.
A In-context Examples
Table 6 presents an in-context example.
- Table 6 is labeled as an in-context example.
- The section includes a table presenting an example within the model’s context.
- The supplied passage identifies Table 6 but does not report its contents or outcome.