MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learning

1. Introduction

• Text-only tool-use systems can misinterpret ambiguous intent, whereas visual or auditory cues can clarify which tool fits the instruction.The example contrasts a prompt without an image with inputs depicting a human pose or a scribble.Source: §1 · p. 1
• MLLM-Tool integrates multimodal encoders with open-source LLMs to identify external tools from multimodal instructions.Its illustrated workflow receives an instruction and image, selects an API, calls it, and returns a quality-improved image.Source: §1 · p. 2, Figure 2 · p. 2
• ToolMMBench provides multimodal tool-use examples with multiple potential API choices for a single query.The alternatives reflect identical or synonymous functions and are intended to represent cases where several APIs may address the same challenge.Source: §1 · p. 2
• 88.19% tool-selection accuracy is reported after fine-tuning MLLM-Tool on ToolMMBench.The paper describes this as a promising result from experiments across multiple popular LLMs.Source: §1 · p. 2

2. Related Works

• MLLM-Tool targets tool-function awareness and selection rather than improving multimodal understanding and reasoning generally.The paper distinguishes its goal from work aligning LLMs to vision, video, and audio spaces.Source: §2 · p. 2
• Earlier tool-agent work spans external-tool bridging, domain applications, API retrieval, complex-task execution, and visual-model integration.Examples include Toolformer [24], Gorilla [19], ToolLLM [21], GPT4Tools [39], and Visual ChatGPT [34].Source: §2 · p. 2
• Unlike HuggingGPT [25], which selects models after text-based planning while receiving non-text data by URL, MLLM-Tool directly processes multimodal inputs for API selection.The comparison concerns when and how non-text data enters the tool-selection process.Source: §2 · p. 2

3. Dataset Construction

• The construction process comprises API collection, instruction-answer generation, one-to-many instruction matching, and benchmark statistics.The benchmark is built from HuggingFace models and their API information.Source: §3 · p. 3

3.1. API Collection

• From 263,945 crawled models, filtering for quality retains 932; model cards document API details, and at most five downloaded APIs are kept per functionality category.The authors also replace coarse platform labels with hierarchical coarse- and fine-functionality categories and add an “Unknown” option when no collected API fits.Source: §3.1 · p. 3

3.2. Instruction-Answer Pairs Generation

• GPT-4 is used to understand each API call’s functionality and generate 20 queries per call; after manual review and refinement, 10 well-crafted queries are selected for each API call.For multimodal cases, manual intervention searches for prompt-matching inputs, using relevant datasets or online searches.Source: §3.2 · pp. 3–4

3.3. Instruction Matching

• Instructions are matched to one or several APIs according to functional overlap and specificity, retaining multiple valid answers when a query permits them.A style-specific request narrows API choices, while a less specific image-generation request can match several APIs.Source: §3.3 · p. 4

3.4. Dataset Statistics

• ToolMMBench has a long-tailed task and modality distribution; its split contains 44,533 training instructions and 2,397 test instructions.Training uses an 8:2 split strategy by API-generated queries, splitting one-to-many queries into separate training inputs while each appears once in testing.Source: §3.4 · pp. 4–5
• Four input combinations are supported, with modality-specific frozen encoders, a trainable projection into language-feature space, and LoRA-adapted LLMs predicting API names.The architecture depicts the pipeline from encoded inputs and prompt through API-name prediction.Source: Figure 4 · p. 5

4. MLLM-Tool

• ImageBind serves as the primary multimodal encoder for the image, video, and audio data in the benchmark.The paper uses its unified embedding space for six modalities.Source: §4 · p. 5
• The system uses Vicuna, Llama, Llama2, and Llama2-Chat as its open-source LLM components.The paper selects these models for inference, knowledge retention, and comprehension capabilities.Source: §4 · p. 5
• ImageBind extracts fixed-weight features from image, video, or audio inputs, a linear layer aligns them with LLM features, and LoRA fine-tunes the LLM.The model receives text, text-plus-image, text-plus-video, or text-plus-audio inputs and outputs API recommendations with model-card information.Source: §4 · p. 5

• Because no prior metrics address text-to-any-modality prompt ambiguity, the paper proposes measures tailored to ambiguity types, multiple options, and diverse modality inputs.Source: §5 · p. 5
• Testing subsets separate five ambiguity scenarios, instructions with one versus multiple valid APIs, and fixed modality combinations.Source: §5 · pp. 5–6
• Accuracy evaluates API predictions against suitable API lists, while hallucination rate tracks outputs that name APIs outside the corpus.Source: §5 · p. 6
• Repeated-inference recall addresses one-to-many API mappings, while format accuracy checks whether outputs follow the required JSON key-value structure.The required format is {api name: [Recommended API]}; deviations such as missing symbols or changed ordering count as formatting errors.Source: §5 · p. 6

6.1. Experimental Setup

• Llama, Llama2, Llama2-Chat, and Vicuna 7B and 13B variants were fine-tuned; Llama2-7B and Llama2-Chat-7B achieved recall rates of 62.13% and 61.47%, respectively, at K=10.The study reports accuracy, hallucination rate, and format accuracy for the full test set, but recall for only those two models due to resource limits.Source: §6.2 · p. 6, Table 3 · p. 6

6.2. Evaluation of Fine-Tuned LLMs and Comparative Performance

• 88.19% accuracy was achieved by Vicuna-13B; MLLM-Tool also showed a significant accuracy advantage over closed-source MLLMs and a slight advantage over open-source models at the same scale.The paper reports that larger models improve accuracy and generally reduce hallucination; format accuracy is 100% across models.Source: Table 4 · p. 7, §6.2 · p. 7

6. Experiments

• Overall, the experiments support MLLM-Tool’s ability to recommend tools for multimodal instructions.The authors describe this as bringing out tool-use potential in LLMs and improving their handling of multimodal inputs.Source: §6.2 · p. 7

6.3. Specific Conditions’ Performance Analysis

• Ambiguity types I (Domains) and II (Categories) were hardest, while types III, IV, and V achieved stronger results, with type V reaching 100% accuracy across models.The authors attribute difficulty in types I and II to varying dataset granularity; non-text inputs without ambiguity were compared against ambiguity cases.Source: §6.3 · p. 7, Table 6 · p. 8
• One-to-one queries performed worse than one-to-many queries, with average gaps of 21.90% for 7B models and 16.14% for 13B models, and had higher hallucination rates.The authors suggest the smaller number of one-to-one examples and their splitting during training may contribute to poorer performance and overfitting risk.Source: §6.3 · p. 7, Table 7 · p. 8
• Audio inputs reached 93.29% accuracy for both Llama2-7B and Vicuna-7B; pure text performed less well, and video performed worst.The authors suggest video results may reflect limited data and task complexity.Source: §6.3 · pp. 7–8

6.4. Ablation Study

• Using a multimodal encoder significantly improved tool-selection accuracy compared with text instructions containing modality placeholders and no non-text data.The authors conclude that text descriptions alone are insufficient for optimal results.Source: Table 8 · p. 8, §6.4 · p. 8

7. Conclusion

• ToolMMBench contains 932 machine-learning-related APIs, and fine-tuning popular LLMs on it yielded promising tool-recommendation results for multimodal instructions.The paper presents MLLM-Tool as combining multimodal perception with tool-augmented LLMs.Source: §7 · p. 8
• Because of computation limits, MLLM-Tool supports single-turn and independent multi-turn conversations, not continuous multi-turn interaction.The authors cite per-turn modality encoding, insufficient window size, and low encoding efficiency as challenges to continuous multi-turn support.Source: §7 · p. 8
• The work focuses on API-type prediction, which the authors describe as complex and still suboptimal, while the dataset may also offer information for community use.Source: §7 · p. 8

8. Dataset Details

• Dataset construction involved over 2,000 person-hours and covers API collection, instruction-answer generation, one-to-many matching, and training and testing distributions.The process also includes API selection, function-boundary criteria, prompt construction, annotation guidance, and sourcing matched multimodal inputs.Source: §8 · p. 9

8.1. Visualization of Dataset Construction Process

• Construction combined API crawling and task re-categorization, instruction generation and ambiguity checks, multimodal content matching, and one-to-many instruction-answer pairing.The authors emphasize precise task labels and matching APIs with identical or similar functions to instructions that fit their functional boundaries.Source: Figure 5 · p. 9, §8.1 · p. 9

8.2. Filtering Rules

• Filtering excluded APIs with poor documentation or maintenance, NSFW-generation risks, restrictive licenses, and redundant identical functions.For duplicate structures and datasets, the dataset retains the most downloaded version; for distinct architectures with the same function and training data, it keeps up to five by download statistics.Source: §8.2 · pp. 9–10

8.3. API Function Boundaries

• Function boundaries account for task scope and dataset context, including text-task domains and image-task contexts; GPT-4-generated instructions target APIs with the same function for one-to-many matches.For multi-task APIs, the dataset focuses on text generation and adjusts descriptions to match query functionality.Source: §8.3 · p. 10

8.4. Prompt Construction

• GPT-4 prompts use API information, expert-written examples, and requirements; for ambiguity-prone multimodal APIs, the ten selected instructions balance ambiguous and unambiguous cases equally.Query ambiguity pairs share identical text but differ in multimodal inputs.Source: §8.4 · p. 10

8.5. Visualization of Data Distributions

• The dataset contains 2,533 ambiguity pairs, predominantly conditional image cases (1,530); its 932 APIs have a long-tail distribution across modalities and tasks, including 10 video-input APIs.Testing subsets are divided by ambiguity type, number of API options, and modality.Source: §8.5 · pp. 10–11, Figure 7 · p. 11

8.6. Crowdsourcing Details

• Eight annotators with NLP or CV backgrounds received training and consent materials, and annotation used cross-checking and additional manual review.Annotators were instructed to avoid private information and license-risky data from unknown or sensitive sources.Source: §8.6 · p. 11

8.7. Collection of text-matched multimodal input

• Non-text inputs came from API datasets or Google Images; the team prioritized open-source datasets and manually checked content against generated instructions.For image-search results, the team paid attention to licensing.Source: §8.7 · p. 11

9. Experiment Setup and Data Processing

• Video is divided into five clips, with two frames randomly sampled from each clip to cover its full length.Source: §9 · p. 12
• Audio is sampled at 16KHz and converted into a log mel spectrogram with 128 frequency bins, treated as a 2D image for ViT processing.Source: §9 · p. 12
• A modality-specific linear projection head aligns each encoder’s features to a shared 1024-dimensional representation.The paper states that ViT is used for the three multimodal encoders.Source: §9 · p. 12

10.1. Results of Multiple Options Categorized by Input Modality

• Performance is consistently better with multiple API options than with one option, regardless of input modality.The analysis subdivides the testing subsets jointly by API-option availability and input modality.Source: §10.1 · p. 12
• The authors say multiple options reduce problem complexity and increase the likelihood of selecting the correct API.Source: §10.1 · p. 12
• During training, multiple-option data can be split into one-to-one strategies, which the authors say further improves performance.Source: §10.1 · p. 12

11.1. Model Card Visualization

• Six model-card samples cover the information described in the main text and span different modalities and functions.Source: §11.1 · p. 12

11.2. Instruction-Answer Pairs Visualization

• Seven testing-set instruction-answer examples are shown; conversation format is used for training, while other attributes divide examples into sub-testing sets.Table 10 is described as reporting testing subsets jointly categorized by multiple-option availability and input modality; its caption supplies no result values.Source: §11.2 · p. 12, Table 10 · p. 12

11.3. Agent Output Visualization

• Four MLLM-Tool cases illustrate support for four input modalities, predicted-API execution, and customer-expected results.Source: §11.3 · p. 12

12. Categorization System

• The categorization system uses a two-level hierarchy: coarse tasks mainly follow HuggingFace categories, while fine-grained subtasks follow task descriptions.Source: §12 · p. 12
• The unambiguous template applies when text reveals multimodal information, while the ambiguous template applies when it does not.The unambiguous case includes pure-text inputs; both templates’ masked fields are populated with API names, descriptions, and prohibited words in practice.Source: Figure 8 · p. 14, Figure 9 · p. 16
• Tables 11–13 list two category levels for each task and provide example candidate models in a third column.Source: Table 11 · p. 25, Table 12 · p. 26, Table 13 · p. 27