Source-linked AI summary
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language
Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Zeqiang Lai, Yang Yang, Qingyun Li, Jiashuo Yu, Kunchang Li, Zhe Chen, Xue Yang, Xizhou Zhu, Yali Wang, Limin Wang, Ping Luo, Jifeng Dai, Yu Qiao
TL;DR
Pure-language interaction can make it inefficient to identify visual instances in complicated scenes. InternGPT combines pointing with language and coordinates tools through an LLM controller; the paper reports improved performance over pure-language systems and a strong Husky dialogue-model result.
Problem
Pure-language interactive systems make users describe visual targets in detail, which is inefficient in complicated scenes with multiple instances.
Source: §1 · p. 2Method
InternGPT combines pointing and language instructions with a perception unit, an LLM controller, an auxiliary control mechanism, and an open-world toolkit.
Source: §1 · p. 2Results
iGPT reports higher accuracy and efficiency than pure-language systems on complicated visual tasks, while Husky achieves 93.89 % GPT-4 Quality.
Source: §1 · p. 3Takeaways & Limitations
User surveys report that combining pointing and language can improve efficiency, particularly in complicated scenes with more than two objects.
Source: §1 · p. 3Some users may have difficulty using combined pointing and language instructions, potentially affecting their experience.
Source: §4.4 · p. 10
Abstract
from arXiv · showhide
We present an interactive visual framework named InternGPT, or iGPT for short. The framework integrates chatbots that have planning and reasoning capabilities, such as ChatGPT, with non-verbal instructions like pointing movements that enable users to directly manipulate images or videos on the screen. Pointing (including gestures, cursors, etc.) movements can provide more flexibility and precision in performing vision-centric tasks that require fine-grained control, editing, and generation of visual content. The name InternGPT stands for \textbf{inter}action, \textbf{n}onverbal, and \textbf{chat}bots. Different from existing interactive systems that rely on pure language, by incorporating pointing instructions, the proposed iGPT significantly improves the efficiency of communication between users and chatbots, as well as the accuracy of chatbots in vision-centric tasks, especially in complicated visual scenarios where the number of objects is greater than 2. Additionally, in iGPT, an auxiliary control mechanism is used to improve the control capability of LLM, and a large vision-language model termed Husky is fine-tuned for high-quality multi-modal dialogue (impressing ChatGPT-3.5-turbo with 93.89\% GPT-4 Quality). We hope this work can spark new ideas and directions for future interactive visual systems. Welcome to watch the code at https://github.com/OpenGVLab/InternGPT.
1 Introduction
Pure-language visual systems can be inefficient in complicated scenes with multiple instances because users must describe the target in detail. InternGPT combines pointing and language instructions, and the paper reports stronger interaction performance and a Husky dialogue model result.
1 Introduction
- Existing interactive systems rely mainly on text, making target selection inefficient when users must distinguish among multiple visual instances.Pointing movements such as gestures and cursors offer a brief, precise non-verbal instruction channel.Source: §1 · p. 2
- InternGPT combines pointing and language instructions through a perception unit, an LLM controller with auxiliary control, and an open-world toolkit.The toolkit includes online and user-trained models as well as applications such as calculators and search engines.Source: §1 · p. 2
- 93.89 % GPT-4 Quality is reported for Husky, a 7B-parameter model for multimodal dialogue, captioning, and visual question answering.The paper says Husky works better than MiniGPT-4 and LLaVA in complex reasoning on a casually shot photo.Source: §1 · p. 3, Figure 2 · p. 3
- iGPT performs intricate visual tasks that pure-language systems fail, with higher accuracy and efficiency across editing, understanding, question-answering, and content creation.User surveys report efficiency gains particularly when scenes contain more than two objects.Source: Figure 1 · p. 2, §1 · p. 3
2 Related Work
Related work describes LLMs as task planners that can coordinate perception models and APIs, alongside developments in language and vision models. Existing LLM-based interactive systems nevertheless depend on pure-language instructions, which iGPT addresses by adding pointing.
Large language model
- LLMs offer instruction following, reasoning, and task splitting, enabling them to schedule APIs and models across modalities.The passage also reviews instruction tuning and chain-of-thought prompting as relevant developments.Source: §2 · p. 3
Perception model
- Vision-model development spans CNNs, transformers, and hybrid architectures, supporting recognition tasks such as detection, segmentation, and video understanding.The passage presents these models as perception components that can serve LLMs.Source: §2 · p. 4
LLM-based interactive system
- LLM-based systems combine perception models and language models to reason and act, but Visual ChatGPT, MM-REACT, and HuggingGPT remain limited by pure-language instructions.TaskMatrix.AI similarly connects foundation models with APIs for digital and physical tasks.Source: §2 · p. 4
3 InternGPT
InternGPT combines pointing, language, and off-the-shelf tools in an interactive system for vision-centric tasks. Its components interpret user input, select and execute tools, and support interactions ranging from basic commands to precise visual manipulation.
3 InternGPT
- iGPT supports three interaction levels: predefined basic tasks, language-guided tasks that resolve ambiguity through dialogue, and pointing-language tasks for precise visual specifications.Pointing cues can select, move, or draw objects when words alone are insufficient.Source: §3 · pp. 4–5
3.1 Interacting with User
- The perception unit interprets pointing input on images and video, while the LLM controller interprets language, decomposes tasks, and selects tools during continuous user interaction.Conversation history lets users revisit prior tasks and results.Source: §3.1 · p. 5
3.2 Perception Unit
- The perception unit uses click, stroke, drag, and draw gestures for selection, movement, and content creation, routing typical objects to SAM, scene text to OCR, and generation strokes to AIGC tools.Its operations are built on community projects including SAM and OCR.Source: §3.2 · pp. 5–6
3.3 LLM Controller
- The LLM controller decomposes language requests, plans task order and dependencies, and allocates tasks to APIs; auxiliary control extracts and validates arguments when API invocation fails.The mechanism can retrieve arguments from dialogue history and apply rule-based corrections before invoking an API.Source: §3.3 · p. 6
3.4 Toolkit
- The open-world toolkit accepts pointing-device input alongside API descriptions and includes vision, vision-language, and general-purpose applications.Examples include masked-object question answering, conditional image generation, and video interpretation; Husky is built following BLIP-2, with LLaMA-7B and three-stage finetuning.Source: §3.4 · pp. 6–7
4 Experiment
The experiments report greater efficiency and user preference for iGPT, demonstrate image- and video-focused interactions, and evaluate Husky’s capabilities. The limitations section identifies risks related to model quality, complex or unfamiliar scenarios, usability, compatibility, and privacy.
4.1 User Studies
- User-study findings report greater efficiency and user-friendliness for iGPT, with higher user preference than Visual ChatGPT.The study compares systems on image-removal and object-replacement tasks.Source: §4.1 · p. 7, Table 2 · p. 8, Table 3 · p. 9
4.2 Demonstrations
- Demonstrations use clicks to target image edits or question-answering regions, and dragging to assemble image parts for prompt-conditioned generation.The paper describes pleasing image-editing performance with concise, precise instructions.Source: §4.2 · pp. 7–8
- Video Highlight Interpretation locates a highlight’s beginning and end from its temporal position, then provides a humorous voice-over.The feature is intended to help users grasp a video’s essence without watching the entire video.Source: §4.2 · p. 8
4.3 Demonstrations of Husky
- 93.89% GPT-4 Quality: Husky’s responses reached this score in ChatGPT-3.5-turbo evaluation; its reasoning was described as comparable to GPT-4, while conversation and detailed descriptions lagged.The evaluation used 90 questions about 30 COCO validation images, with GPT-4 predictions as reference.Source: Table 4 · p. 9, §4.3 · p. 9
4.4 Limitations
- iGPT depends on underlying open-source model quality, and limitations or biases in those models could adversely affect system performance.Source: §4.4 · pp. 9–10
- Complex interactions may challenge accuracy and response time; untuned vision-language cooperation may limit model capacity, and unfamiliar scenarios may compromise performance.Source: §4.4 · p. 10
- Other stated risks include user difficulty with mixed instructions, device and platform integration challenges, and privacy or security concerns when sensitive data is processed or shared.Source: §4.4 · pp. 10–11
5 Conclusion
InternGPT combines pointing and text commands through a perception unit, LLM controller, and open-world toolkit. The conclusion reports successful intricate tasks beyond pure-language systems and survey-reported efficiency gains in challenging situations.
5 Conclusion
- InternGPT combines pointing and text commands with a perception unit, LLM controller, and open-world toolkit to execute applications.The framework is presented as user-friendly and effective for complex visual scenarios involving multiple instances.Source: §5 · p. 11
- iGPT performed intricate interactive tasks beyond pure-language systems, while user surveys reported efficiency gains in challenging situations.The conclusion describes these situations as involving many objects.Source: §5 · p. 11
- Figure 8 presents demonstrations of Husky across various scenarios.The conclusion includes these demonstrations alongside iGPT’s broader interactive framework.Source: Figure 8 · p. 12