Source-linked AI summary
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, Lijuan Wang
TL;DR
Existing vision systems require specialized models for different problems, while jointly finetuned multimodal approaches require substantial data and computation. MM-REACT combines ChatGPT with vision experts through textual prompts and file-path representations, and zero-shot experiments demonstrate capabilities across challenging visual-understanding tasks. The system is flexible enough to incorporate upgraded language models and additional tools, subject to limitations in evaluation, experts, context, representation, and prompt engineering.
Problem
Different vision problems require separate models and manual composition, while joint vision-language approaches require substantial computing resources and annotated data for specific capabilities.
Method
MM-REACT uses textual prompts to let ChatGPT select and invoke existing vision experts, representing images and videos with file paths and serializing expert outputs as text.
Results
Zero-shot experiments demonstrate MM-REACT capabilities across multi-image reasoning, multi-hop document understanding, open-world concept understanding, video summarization, and other challenging visual-understanding tasks.
Takeaways & Limitations
MM-REACT provides a simple, flexible way to empower language models with vision experts and can incorporate new tools or stronger language models without training.
Takeaways & Limitations
The system lacks systematic accuracy evaluation, depends on the correctness and availability of integrated experts, and is constrained by context length, text serialization, and manual prompt engineering.
Abstract
from arXiv · showhide
We propose MM-REACT, a system paradigm that integrates ChatGPT with a pool of vision experts to achieve multimodal reasoning and action. In this paper, we define and explore a comprehensive list of advanced vision tasks that are intriguing to solve, but may exceed the capabilities of existing vision and vision-language models. To achieve such advanced visual intelligence, MM-REACT introduces a textual prompt design that can represent text descriptions, textualized spatial coordinates, and aligned file names for dense visual signals such as images and videos. MM-REACT's prompt design allows language models to accept, associate, and process multimodal information, thereby facilitating the synergetic combination of ChatGPT and various vision experts. Zero-shot experiments demonstrate MM-REACT's effectiveness in addressing the specified capabilities of interests and its wide application in different scenarios that require advanced visual understanding. Furthermore, we discuss and compare MM-REACT's system paradigm with an alternative approach that extends language models for multimodal scenarios through joint finetuning. Code, demo, video, and visualization are available at https://multimodal-react.github.io/
1. Introduction
MM-REACT addresses the difficulty of composing specialized vision models for complex visual tasks and the resource demands of jointly finetuned multimodal models. It instead combines ChatGPT with existing vision experts through prompting, supporting broad visual understanding capabilities.
- Motivation: Different vision problems typically require separate models and manual selection and composition for each use case.The paper illustrates this with image tagging followed by celebrity recognition.
- Motivation: Joint vision-language models provide dialogue around images but require substantial computing resources and annotated data for specific capabilities.
- Approach: MM-REACT composes numerous vision experts with ChatGPT for multimodal reasoning and action through a textual prompt interface.Images and videos are represented by file paths, while expert capabilities, input types, and outputs are specified in prompts.
- Capabilities: MM-REACT demonstrates multi-image reasoning, open-world concept understanding, video summarization, and event localization through prompting rather than additional multimodal training.
- Capabilities: The system may provide flexibility in module upgrades and use specialized experts for tasks such as celebrity recognition and dense captioning.
2. Related Work
Prior work combines language models with reasoning, external tools, or visual representations, but MM-REACT emphasizes language-model planning to allocate vision experts in a multi-round dialogue system.
- LLM Prompting Methods: Chain-of-thought and tool-using language models separately support intermediate reasoning or action planning, while REACT merges these phases for complicated NLP tasks.
- Vision+LLMs: Vision-language systems project visual inputs into discrete text tokens or continuous features that language models can process.
- Vision+LLMs: Multimodal chain-of-thought methods include finetuning on annotated reasoning chains and zero-shot capabilities enabled by large-scale training.
- Multimodal Reasoning and Action: MM-REACT uses high-level language-model planning to allocate various vision experts rather than only generating text conditioned on visual inputs.
- Multimodal Reasoning and Action: Unlike Visual ChatGPT’s primary focus on image generation and editing, MM-REACT mainly targets visual understanding.
- Multimodal Reasoning and Action: Unlike ViperGPT’s one-round Python-code approach, MM-REACT is multi-round, dialogue-based, and can integrate a strong question-answering model as a vision expert.
3. MM-REACT Prompting
MM-REACT prompts ChatGPT to represent non-text inputs with file paths, select vision experts through structured requests, and use serialized observations in iterative reasoning. Its prompt and text-standardization design supports multiple experts, modalities, and tool upgrades.
- 3. MM-REACT Prompting: A vision expert is a computer-vision model that interprets image content from a particular perspective, such as captioning, OCR, celebrity recognition, or object detection.
- 3. MM-REACT Prompting: MM-REACT aims to automate expert selection and composition from the requirements expressed in a user’s natural-language query.
- 3. MM-REACT Prompting: Because ChatGPT accepts text but lacks visual understanding, the system instructs it to determine which expert to invoke and which image to process.
- 3. MM-REACT Prompting: The flowchart distinguishes thought from action request and observation: ChatGPT reasons or requests an expert, then receives the expert’s response.
- 3.1. User Input: File paths serve as placeholders for multiple images and videos, allowing ChatGPT to identify inputs for vision experts even though paths contain no visual information themselves.
- 3.2. ChatGPT Response: Watchwords in ChatGPT’s action requests signal when an expert is needed, and regular-expression matching parses the expert name and file path for execution.
- 3.3. Vision Experts: Expert outputs are standardized as text, combined with chat history, and used by ChatGPT to invoke additional experts or return a final answer.
- 3.3. Vision Experts: Prompt prefixes describe each expert’s capability, input format, and output information, while examples help ChatGPT select one or multiple experts; the scheme also extends to other modalities and tools without training.
4. Experiments
MM-REACT demonstrates multimodal reasoning and action through ChatGPT calls, vision-expert executions, and zero-shot evaluations across diverse visual tasks. Experiments also compare its training-free design with PaLM-E and examine extensibility and limitations.
- MM-REACT’s Full Execution Flow: MM-REACT’s execution flow alternates ChatGPT reasoning with selected vision-expert calls, producing text observations that support later actions or final answers.The system uses prompts, expert executions, and intermediate outputs to gather information behind the scene.
- Representative Capabilities: MM-REACT evaluates visual math, memes, spatial understanding, multi-image reasoning, document understanding, and other application scenarios.Figures 4–14 cover capabilities including bar charts, floorplans, flowcharts, and tables.
- Capability Comparison with PaLM-E: MM-REACT achieves competitive results to PaLM-E while composing existing vision experts with ChatGPT without training.PaLM-E instead trains a vision-language model with dedicated datasets.
- MM-REACT Extensibility: Upgrading from ChatGPT to language-only GPT-4 improves some examples, including correctly answering a physics question that ChatGPT fails.The system also incorporates an image-editing tool for multi-round dialogue-based editing without training.
- Limitations: The experiments identify limitations involving difficult evaluation, expert errors or absence, context-window constraints, text conversion, and manual prompt engineering.The authors note that systematic accuracy evaluation is difficult because annotated benchmarks are lacking.
5. Conclusion
The conclusion presents MM-REACT as a flexible system paradigm that combines multimodal reasoning and action with a pool of vision experts. Zero-shot experiments demonstrate capabilities across challenging visual understanding tasks.
- Conclusion: MM-REACT combines multimodal reasoning and action to address complicated visual understanding problems.The system empowers language models with a pool of vision experts.
- Conclusion: Zero-shot experiments demonstrate capabilities in multi-image reasoning, multi-hop document understanding, open-world concept understanding, video summarization, and related tasks.The conclusion describes these as a wide range of challenging understanding tasks.
- Conclusion: MM-REACT offers a simple and flexible way to empower LLMs with vision experts.This flexibility is presented as part of the system paradigm’s contribution.
Ours
The examples illustrate MM-REACT’s ability to combine visual inputs with reasoning, specialized recognition, document understanding, and multimodal action. Demonstrations span image, video, multi-image, chart, flowchart, physics, and image-editing scenarios.
- Image Understanding: MM-REACT answers image questions involving object descriptions, celebrity identification, and visual-conditioned factual reasoning.Examples identify a dog, Kobe Bryant, and his five championship rings.
- Visual Math: The system performs visual arithmetic by deriving totals and averages from information contained in images.Two pizzas at $9.99 each produce a total of $19.98.
- Multi-Image Reasoning: MM-REACT links multiple images and timestamps to answer a question about what was eaten for lunch and when.The example identifies a sandwich at 12:45pm.
- Document Understanding: The system reads charts and computes cross-category cost comparisons from their extracted values.The average charging cost is $82.41 cheaper than the average fueling cost.
- Reasoning and Planning: A flowchart example triggers additional reasoning, while a physics example derives a linear temperature profile from boundary conditions.The physics response gives T(x) = (Tb - T0) * (x / L) + T0.
- Extensibility: Extensibility demonstrations include GPT-4 upgrades, an image-editing expert, and unfolded multimodal reasoning procedures.The image-editing example supports multi-round, dialogue-based editing.