Source-linked AI summary
LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang, Jianfeng Gao, Chunyuan Li
TL;DR
Multimodal assistants need better ways to handle diverse vision-language tasks and reliably use external tools. LLaVA-Plus learns tool use through end-to-end visual instruction tuning over a skill repository, keeping visual signals grounded throughout interaction. It consistently outperforms LLaVA across benchmarks, achieves a new state of the art on VisiT-Bench, and exhibits emergent multimodal interaction capabilities.
Problem
Existing approaches do not yet provide a unified, robust way to incorporate diverse multimodal skills and select and compose tools for real-world tasks.
Method
LLaVA-Plus uses end-to-end visual instruction tuning to teach an LMM to predict skill use, answers, and stopping decisions through a repository of vision and vision-language tools.
Results
LLaVA-Plus consistently outperforms LLaVA across many benchmarks, achieves a new state of the art, and shows emergent multimodal interaction capabilities.
Takeaways & Limitations
Grounding raw visual signals throughout human-AI interaction enables tool use for planning, reasoning, and new multimodal scenarios.
Abstract
from arXiv · showhide
LLaVA-Plus is a general-purpose multimodal assistant that expands the capabilities of large multimodal models. It maintains a skill repository of pre-trained vision and vision-language models and can activate relevant tools based on users' inputs to fulfill real-world tasks. LLaVA-Plus is trained on multimodal instruction-following data to acquire the ability to use tools, covering visual understanding, generation, external knowledge retrieval, and compositions. Empirical results show that LLaVA-Plus outperforms LLaVA in existing capabilities and exhibits new ones. It is distinct in that the image query is directly grounded and actively engaged throughout the entire human-AI interaction sessions, significantly improving tool use performance and enabling new scenarios.
1 INTRODUCTION
LLaVA-Plus addresses the challenge of building multimodal assistants that combine broad visual skills with reliable tool selection and composition. It learns to use a diverse skill repository through end-to-end visual instruction tuning and demonstrates expanded capabilities across benchmarks and real-world tasks.
- Multimodal assistants still lack recipes that effectively support broad computer-vision and vision-language tasks.
- Tool chaining broadens visual task coverage but prompting is not adaptable or robust enough for reliable tool selection and composition.
- LLaVA-Plus combines end-to-end training with tool chaining through a skill repository of vision and vision-language tools.
- Its visual instruction tuning teaches the assistant to select, activate, and compose tools for multimodal tasks unseen during training.
- The system can incorporate new tools through additional instruction tuning, while raw visual signals remain active throughout interaction sessions for planning and reasoning.
- The paper introduces multimodal tool-use data and reports consistently improved benchmark results, including new state of the art on VisiT-Bench.
2 LEARNING TO USE TOOLS WITH VISUAL INSTRUCTION TUNING
LLaVA-Plus extends multimodal assistants with a planner that selects and composes specialist tools through visual instruction tuning. Its pipeline represents tool use as skill-oriented dialogue and curates data across visual understanding, extended skills, and training settings.
- 2 LEARNING TO USE TOOLS WITH VISUAL INSTRUCTION TUNING: Existing LMMs handle language-image inputs and language outputs, but incorporating skills such as segmentation and generation into one architecture remains challenging.The paper contrasts end-to-end LMM training with prompt-based tool chaining.
- 2.1 LLAVA-PLUS: LLaVA-Plus uses an LMM as planner with a skill repository of vision and vision-language models that can be selected for different tasks.The modular architecture is designed to expand capabilities by adding specialist tools.
- 2.1 LLAVA-PLUS: The four-step interaction takes an image and instruction, predicts tool use and its prompt, receives the tool result, then aggregates it into the final answer.The sequence explicitly connects human input, tool invocation, tool output, and assistant response.
- 2.1 LLAVA-PLUS: Training adds skill-oriented dialogue to LLaVA’s sequence and applies loss to learn skill use, answers, and stopping decisions.Only designated subsequences in the training sequence contribute to the autoregressive loss.
- 2.1 LLAVA-PLUS: The unified prediction format contains Thought, Action, and Value fields for deciding whether to invoke tools, specifying function calls, and producing responses.An empty Action list represents dialogues that do not invoke the skill repository.
- 2.2 SKILL REPOSITORY: MULTIMODAL TOOL USE INSTRUCT DATA GENERATION: The repository data-generation process uses GPT-4 to curate multimodal instruction-following examples for image-only and argument-conditioned skills.The examples include tool outputs, repeated questions, image context, and generated responses.
- 2.2.1 CORE SKILLS: UNDERSTANDING: Core visual-understanding skills expand beyond existing LMM abilities to include detection, grounding, segmentation, tagging, captioning, OCR, and compositions.These skills use visual inputs and outputs, with separate procedures for image-only and additional-argument cases.
- 2.2.2 EXTENDED SKILLS: Extended skills cover external knowledge retrieval, image generation and editing, and visual prompts, while training combines tool-use data with LLaVA-158K.The recipe is presented as applicable to different tools and instantiated in All Tools and other training settings.
3 RELATED WORKS
Related multimodal systems use either tool chaining or instruction tuning, while unified multimodal models pursue broad capabilities in a single architecture. LLaVA-Plus is positioned as an instruction-tuned multimodal planner that keeps image inputs active throughout tool-use interactions.
- AI Agents with Multimodal Tool Use: Multimodal tool-use agents generally rely on prompt engineering and in-context learning or instruction tuning to determine which tools an LLM should invoke.The comparison includes systems such as Visual ChatGPT, MM-ReAct, GPT4Tools, and Gorilla.
- AI Agents with Multimodal Tool Use: LLaVA-Plus is described as the first work using an LMM as the planner while considering image inputs throughout the entire interaction session.This distinguishes its interaction design from the related tool-use approaches discussed.
- Unified Multimodal Models with Versatile Capabilities: Unified multimodal models aim to provide versatile capabilities in one architecture, with proprietary and open-source systems demonstrating multimodal task transfer.The paper places LLaVA-Plus among these broader efforts while separately comparing tool allocation and training requirements.
4 EXPERIMENTS
LLaVA-Plus improves existing multimodal capabilities and enables tool-based capabilities across benchmark evaluations and visual interaction scenarios. Its results include stronger benchmark performance, new state-of-the-art performance on VisIT-Bench, and demonstrations of grounding, segmentation, editing, and multimodal posting.
- 4.1 THE EFFECTIVENESS OF LEARNING TO USE SKILLS: LLaVA-Plus variants improve LLaVA on two LMM benchmarks.
- 4.1 THE EFFECTIVENESS OF LEARNING TO USE SKILLS: LLaVA-Plus significantly outperforms commercial visual chat systems and tool-chaining baselines on LLaVA-Bench (Tools).The benchmark evaluates grounding, tagging, caption, and OCR capabilities.
- 4.2 COMPARISONS WITH SOTA LMM SYSTEMS: LLaVA-Plus consistently outperforms LLaVA at both 7B and 13B sizes on MMVet, with the largest gains in OCR and spatial categories.
- 4.2 COMPARISONS WITH SOTA LMM SYSTEMS: 100+ ELO points separate LLaVA-Plus from LLaVA on VisIT-Bench, giving LLaVA-Plus a new state-of-the-art leaderboard result.
- 4.3 VISUAL EXAMPLES OF NEW CAPABILITIES: Visual examples show object grounding, semantic segmentation, conditional image generation, and multimodal social-media post creation.The examples include detecting a frisbee’s coordinates, generating an underwater scene with a preserved layout, and revising images with caption suggestions.
5 CONCLUSION
LLaVA-Plus combines a multimodal tool-use dataset with an LMM that learns to use skills, improving over LLaVA across benchmarks and showing new interaction capabilities. The authors identify hallucinations and tool-use conflicts as practical limitations.
- LLaVA-Plus is a general-purpose multimodal assistant that plugs into and learns to use skills for vision-language tasks.
- The paper introduces a visual instruction dataset specifically designed for multimodal tool use.
- LLaVA-Plus consistently outperforms LLaVA across benchmarks, creates a new state-of-the-art result, and shows emergent multimodal interaction capabilities.
- Practical limitations include hallucinations and conflicts between tool uses.
A DATA
The data pipeline augments LLaVA instruction data and generates multimodal tool-use conversations through specialized question-generation procedures. It covers visual prompts and multi-turn examples for detection and other skills.
- The original LLaVA questions and answers are converted into a unified format with values, empty actions, and ChatGPT-generated thoughts.The generated thoughts indicate when the model can answer without invoking tools.
- Question-generation pipelines target visual prompts and image-related parameters, with GPT-4 used to rewrite questions and self-instruction used for multi-turn detection conversations.
- The appendix provides self-instruct examples for generating multi-turn conversations with in-context learning.
B EXTENDED SKILLS
Extended skills add external knowledge retrieval, image generation and editing, visual prompts, and compositions of multiple skills. These data-creation procedures support multimodal interaction beyond single-skill tasks.
- External-knowledge data uses CLIP search over LIAON to retrieve top-K image-to-text items for InfoSeek image queries.
- Generation data uses Stable Diffusion and Instruct-Pix2Pix, with ChatGPT producing human-like instructions for ambiguous or short image-generation requests.
- Visual-prompt data represents user points as text inputs and uses Semantic-SAM to specify multi-granularity segmentation functionality.
- Compositional instruction data teaches LLaVA-Plus to combine several skills during multi-turn human-AI interactions.
C RESULTS
LLaVA-Plus improves captioning quality on COCO Caption and reduces false positives from Grounding DINO. These results illustrate both enhancement of existing tools and correction of their erroneous outputs.
- COCO Caption: LLaVA-Plus outperforms BLIP2 and LLaVA on the COCO Caption benchmark in CLIP score.However, both language models have lower CIDEr scores than BLIP2 and perform poorly on language-language metrics.
- False Positives of Grounding DINO: Grounding DINO occasionally hallucinates false-positive object instances.The false-positive evaluation uses negative category prompts on the first 100 images of the COCO validation set.
- False Positives of Grounding DINO: LLaVA-Plus can significantly reduce Grounding DINO’s false-positive rate by analyzing both model outputs and image content.The model was evaluated by prompting with all negative categories associated with each image.
D EXAMPLE SCENARIOS
LLaVA-Plus uses specialized tools to extend visual chat beyond language-only responses, supporting localization, segmentation, retrieval, image generation, editing, and multimodal social-media content. Its examples show tool outputs being incorporated into interactive responses and creative workflows.
- Object Detection and Counting: LLaVA-Plus localizes objects and uses their spatial information to describe scenes, activities, and object counts.The examples contrast this capability with language-output-only systems and commercial visual chat systems that cannot provide object locations.
- Detection and Segmentation: Combining segmentation with language descriptions provides fine-grained, instance-level visual understanding.Figure 8 shows segmentation results enriched with language descriptions, while Figure 11 uses semantic segmentation to provide scene layout.
- External Knowledge: External knowledge retrieval introduces relevant facts and entities into LLaVA-Plus responses.The example uses a CLIP retriever to search for information about food in an image before answering.
- Image Generation: LLaVA-Plus generates detailed Stable Diffusion-favored prompts from brief image-generation requests.The paper states that these prompts can help improve image-generation quality.
- Interactive Image Editing: Interactive editing combines OpenSEED segmentation with InstructPix2Pix and ControlNet to preserve scene layout while changing other image aspects.The workflow supports multi-turn requests for segmentation and subsequent image editing.
- Multimodal Social Media Post: LLaVA-Plus can produce edited images and attractive accompanying text for multiple seasonal versions of the same scene.The examples also include a fireworks scenario.
- Visual Prompt for Interactions: Visual pointing and referring segmentation let users specify regions or segmentation intents when language is insufficient.The examples use semantic SAM for visual pointing and selected regions from another image for visual referring segmentation.