Source-linked AI summary
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, Nan Duan
TL;DR
Visual ChatGPT addresses ChatGPT’s inability to process visual information and the fixed task interfaces of Visual Foundation Models. It combines ChatGPT with multiple VFMs through prompt management for multi-step visual interaction, and experiments report broad understanding and generation capabilities. The system remains dependent on the accuracy and effectiveness of ChatGPT and the VFMs.
Problem
ChatGPT cannot process visual information, while existing vision-language approaches support specific tasks and require labeled training data.
Method
Visual ChatGPT combines ChatGPT with multiple VFMs and uses prompt management to inject visual information and coordinate iterative model interactions.
Results
Massive zero-shot experiments and selected cases demonstrate Visual ChatGPT’s understanding and generation abilities across different tasks.
Takeaways & Limitations
Visual ChatGPT enables users to interact with ChatGPT beyond language format and solve complex visual questions step-by-step.
Takeaways & Limitations
Visual ChatGPT relies heavily on ChatGPT for task assignment and VFMs for execution, so performance is heavily influenced by their accuracy and effectiveness.
Abstract
from arXiv · showhide
ChatGPT is attracting a cross-field interest as it provides a language interface with remarkable conversational competency and reasoning capabilities across many domains. However, since ChatGPT is trained with languages, it is currently not capable of processing or generating images from the visual world. At the same time, Visual Foundation Models, such as Visual Transformers or Stable Diffusion, although showing great visual understanding and generation capabilities, they are only experts on specific tasks with one-round fixed inputs and outputs. To this end, We build a system called \textbf{Visual ChatGPT}, incorporating different Visual Foundation Models, to enable the user to interact with ChatGPT by 1) sending and receiving not only languages but also images 2) providing complex visual questions or visual editing instructions that require the collaboration of multiple AI models with multi-steps. 3) providing feedback and asking for corrected results. We design a series of prompts to inject the visual model information into ChatGPT, considering models of multiple inputs/outputs and models that require visual feedback. Experiments show that Visual ChatGPT opens the door to investigating the visual roles of ChatGPT with the help of Visual Foundation Models. Our system is publicly available at \url{https://github.com/microsoft/visual-chatgpt}.
1. Introduction
Visual ChatGPT addresses ChatGPT’s inability to process visual information by combining it with Visual Foundation Models through a prompt-managed, multi-step interaction system.
- ChatGPT supports conversational language interaction but cannot process visual information, while VFMs are constrained by task-specific fixed input-output formats.
- Visual ChatGPT incorporates multiple VFMs into ChatGPT instead of training a new multimodal model from scratch.
- The Prompt Manager describes VFM capabilities and formats, converts visual signals into language, and manages model histories, priorities, and conflicts.
- A complex instruction can trigger a sequence of depth estimation, depth-to-image, and style-transfer models to produce the requested visual result.
- The system combines ChatGPT with 22 VFMs and supports complex visual tasks through their defined correlations and interactions.
- Massive zero-shot experiments and selected cases examine Visual ChatGPT’s visual understanding and generation abilities.
2. Related Works
Related work frames Visual ChatGPT as a response to the difficulty of connecting language models and visual processing across flexible, multimodal interactions.
- 2.1. Natural Language and Vision: Language and vision jointly support many information-processing tasks, but LLMs cannot process visual information directly.
- 2.1. Natural Language and Vision: Connecting language and vision requires difficult model training, designed instructions, and conversions between modalities.
- 2.1. Natural Language and Vision: Earlier vision-language methods typically support specific tasks and require labeled training data.
- 2.3. Guidance of Pre-trained LLMs for VL tasks: Chain-of-Thought elicits intermediate answers to support multi-step reasoning in language models.
- 2.3. Guidance of Pre-trained LLMs for VL tasks: Visual ChatGPT’s overview depicts multi-round dialogue and iterative invocation of Visual Foundation Models for answering visual questions.
3. Visual ChatGPT
Visual ChatGPT represents multimodal dialogue as iterative reasoning over user queries, dialogue history, and outputs from specialized Visual Foundation Models managed by prompts.
- System principles require sensitivity to image filenames and use of VFMs for image handling rather than imagined results from chat history.
- The system combines a set of VFMs, each with a determined function and explicit inputs and outputs.
- Visual ChatGPT represents each conversation round using the user query, prior dialogue history, and histories of intermediate VFM reasoning.
- Complex queries are solved step-by-step by invoking different VFMs and producing multiple intermediate answers.
- The Prompt Manager converts visual signals into language, adapts intermediate answers to each model’s input format, and coordinates model execution.
- Tool use is explicitly prompted through a thought-and-action format, including an image-question tool that receives an image path and question.
BLIP VQA
Visual ChatGPT uses Prompt Manager prompts to connect ChatGPT with visual foundation models for language-and-image tasks. The manager specifies model capabilities and formats, translates visual signals into language, and supports iterative multi-model execution.
- Prompt Manager: Prompt Manager converts images, depth information, and mask matrices into language that ChatGPT can process.
- System Principles: Visual ChatGPT supports text and visual tasks including visual question answering, image generation, and editing.
- Prompt Manager: Prompt Manager integrates ChatGPT with visual foundation models by describing each model’s capabilities and input-output formats.It also distinguishes among models with similar visual-language functions.
- User Query Management: Force VFM Thinking instructs Visual ChatGPT to invoke tools for visual observation instead of relying on imagination.It also encourages outputs generated by foundation models rather than generic responses.
- Foundation Model Outputs: Chained filenames encode an intermediate image’s operation, input image, and original image, helping ChatGPT track generated-file dependencies.
- Multi-step Execution: Visual ChatGPT can repeatedly call additional visual foundation models, summarize intermediate outputs, and continue until an ending condition or user feedback is reached.
4. Experiments
Experiments examine Visual ChatGPT through multi-round multimodal dialogue and qualitative prompt-removal case studies. The studies show that prompt design affects tool selection, reasoning, file handling, and ambiguity resolution.
- Multiple-Round Dialogue: A 16-round multimodal dialogue combines text and image questions, multiple images, multiple foundation models, and multi-step processing.
- Prompt Manager Case Studies: Removing system-principle prompt components produces different degrees of capacity degradation.
- Foundation Model Prompts: Clearly specifying a foundation model’s name, usage, and input-output format supports correct tool selection and parameter formation.Examples show ambiguous names can cause tool errors, unclear usage can trigger incorrect tools, and missing formats can produce wrong parameters.
- Query and Output Prompts: Unique filenames reduce overwriting and reference errors, while chained filenames help Visual ChatGPT recognize file types and dependencies.
- Query and Output Prompts: Forcing explicit VFM-oriented thinking encourages tool invocation and strict reasoning formats instead of imagination based on textual context.
- Query and Output Prompts: Asking for more details enables Visual ChatGPT to address ambiguous references in user requests.
5. Limitations
Visual ChatGPT has limitations stemming from its dependence on component models, prompt engineering, multi-model execution, model-context length, and security requirements.
- Performance depends heavily on ChatGPT's task assignment and the VFMs' accuracy and effectiveness.
- Substantial prompt engineering is required to translate VFMs into language and distinguish their model descriptions.The process is time-consuming and requires expertise in computer vision and natural language processing.
- Automatically decomposing specific tasks across multiple VFMs can limit real-time capabilities compared with specialized expert models.
- ChatGPT's maximum token length may restrict the number of foundation models that can be supplied to the system.A pre-filter module may be needed when thousands or millions of foundation models are available.
- Plugging and unplugging foundation models, especially remote API models, raises security and privacy concerns.Sensitive data requires careful consideration and automatic checks to avoid exposure or compromise.
6. Conclusion
Visual ChatGPT is presented as an open system that combines ChatGPT with different VFMs through prompts to support complex visual questions step by step. The authors report broad potential while identifying self-correction and inference time as future concerns.
- Prompt design injects visual information into ChatGPT so it can solve complex visual questions step by step.
- Experiments and selected cases demonstrate Visual ChatGPT's potential and competence across different tasks.
- Self-correction is needed to check execution results against human intentions and make corresponding edits when VFM outputs or prompts are unsatisfactory.This behavior can significantly increase inference time and is left for future work.
A. Tool Details
The tool details describe a collection of visual analysis, editing, detection, and generation functions, each mapped to foundation models and prompt-defined inputs.
- Image question-answering and captioning tools provide answers about image content and generate photo descriptions.The question-answering tool accepts an image path and an image-based question.
- Text-to-image generation creates images from user descriptions and saves the generated output to a file.
- Edge, line, HED-boundary, sketch, and segmentation tools detect structural image information for downstream processing.Their outputs include edge, line, HED, or segmentation representations derived from an image path.
- ControlNet-based generation conditions new images on Canny edges, line images, soft HED boundaries, or segmentations together with textual descriptions.
- Depth and normal-map tools predict geometric representations and support conditioned image generation from those representations and text.