Source-linked AI summary
M^3-Bench: Multi-Modal, Multi-Hop, Multi-Threaded Tool-Using MLLM Agent Benchmark
Yang Zhou, Mingyu Zhao, Zhenting Wang, Difei Gu, Bangwei Guo, Ruosong Ye, Ligong Han, Can Jin, Dimitris N. Metaxas
TL;DR
M3-Bench addresses the lack of systematic evaluation for multimodal MCP workflows, where images and text jointly condition tool calls and results. It builds standardized multi-hop, multi-threaded trajectories and aligns calls with similarity-bucketed matching; evaluations find persistent gaps in argument fidelity and structural consistency.
Problem
Existing MCP benchmarks are mostly text-only or focus on linear API planning and database queries, while systematic evaluation of workflows jointly conditioned on images and text is missing.
Source: §1 · p. 1Method
M3-Bench constructs realistic multi-hop, multi-threaded workflows and aligns serialized tool calls through sentence-encoder similarity and tool-name-bucketed Hungarian matching.
Source: §1 · p. 2Results
Evaluations of representative MLLMs reveal persistent gaps in multimodal MCP tool use, particularly in argument fidelity and structural consistency.
Source: §6 · p. 9Takeaways & Limitations
The results underscore the need for agents that jointly reason over images, text, and tool graphs.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We present M^3-Bench, the first benchmark for evaluating multimodal tool use under the Model Context Protocol. The benchmark targets realistic, multi-hop and multi-threaded workflows that require visual grounding and textual reasoning, cross-tool dependencies, and persistence of intermediate resources across steps. We introduce a similarity-driven alignment that serializes each tool call, embeds signatures with a sentence encoder, and performs similarity-bucketed Hungarian matching to obtain auditable one-to-one correspondences. On top of this alignment, we report interpretable metrics that decouple semantic fidelity from workflow consistency. The benchmark spans 28 servers with 231 tools, and provides standardized trajectories curated through an Executor & Judge pipeline with human verification; an auxiliary four large language models (LLMs) judge ensemble reports end-task Task Completion and information grounding. Evaluations of representative state-of-the-art Multimodal LLMs (MLLMs) reveal persistent gaps in multimodal MCP tool use, particularly in argument fidelity and structure consistency, underscoring the need for methods that jointly reason over images, text, and tool graphs. Our Benchmark's anonymous repository is at https://github.com/EtaYang10th/Open-M3-Bench
1. Introduction
M3-Bench targets a gap in evaluating multimodal MCP workflows, where images and text jointly condition tool calls and results. It combines realistic workflow challenges with similarity-aware alignment and interpretable diagnostics.
1. Introduction
- Existing MCP benchmarks are mostly text-only and focus on linear API planning or database queries, leaving multimodal MCP workflows systematically unevaluated.The missing setting jointly conditions tool calls and results on images and text.Source: §1 · pp. 1–2
- Realistic trajectories require visual grounding, cross-step dependencies, and parallel order-independent calls within a step.Later actions may consume earlier artifacts, so simple string matching or linear-sequence scoring is inadequate.Source: §1 · p. 2
- M3-Bench is presented as the first benchmark explicitly targeting multimodal MCP workflows and combines multimodality, multi-hop/threaded causality, and similarity-aware alignment.The introduction states that existing benchmarks do not pair these three properties.Source: §1 · p. 2
- Predicted and reference calls are aligned through tool-name-bucketed Hungarian matching over serialized, sentence-encoder embeddings, yielding auditable one-to-one correspondences.The call-level alignment does not rely on an LLM judge.Source: §1 · p. 2
- The benchmark also provides an Executor–Judge pipeline for standardized reference trajectories and evaluates mainstream MLLMs across tasks, budgets, and error types.The pipeline allows an optional lightweight human-verification pass.Source: §1 · p. 2
2. Related Work
Prior work develops tool-use capabilities and MCP standardizes interfaces for heterogeneous tools. Recent MCP benchmarks broaden realistic tool use, while M3-Bench focuses on multimodal tasks and similarity-aware, structure-sensitive evaluation.
2.1. LLM Tool Use and Protocols
- Tool-use research spans API-aware finetuning, retrieval, instruction and data scaling, hierarchical retrieval, self-reflection, and tool-selection benchmarks; MCP standardizes interfaces across heterogeneous tools.The passage cites ToolBench and AnyTool/AnyToolBench among the approaches.Source: §2.1 · pp. 2–3
- M3-Bench’s examples include image-conditioned tasks and a workflow that detects hazards, processes an image, then creates and populates a slide deck.The workflow crosses vision, image-processing, and office/productivity MCP servers.Source: Figure 2 · p. 3
2.2. Benchmarks for Tool Use and MCP
- Recent MCP-native benchmarks use live servers, long-horizon tasks, marketplace-sourced servers, or human-authored multitool workflows; M3-Bench instead targets multimodal tasks with similarity-aware, structure-sensitive diagnostics.The cited efforts include MCP-Bench, MCP-Universe, MCPToolBench++, and MCP-Atlas.Source: §2.2 · p. 3
3. M3-Bench
M3-Bench builds standardized multimodal MCP trajectories through an Executor–Judge pipeline, then organizes tasks across a curated server ecosystem. Its references capture stepwise execution, persistent resources, and verified workflow structure.
3.1. Benchmark Construction
- Executor outputs use valid JSON calls, allow multi-threaded interactions within a step, and preserve file-image paths in a temporary workspace for later steps.After each step, a binary decision prompt checks whether the evidence suffices to conclude.Source: §3.1 · pp. 3–4
- The Executor plans and executes stepwise tool calls, while the Judge rejects causal misordering or factual errors and consolidates valid calls into an audited reference trajectory.The Judge removes failed or redundant calls and merges operations that can run concurrently; human verifiers audit scope and logical consistency.Source: §3.1 · pp. 3–4, Figure 3 · p. 3
3.2. Task Distribution
- The evaluation set contains 211 trajectories with 644 reference steps and 1,337 MCP tool calls across 28 multimodal tasks.Per-task goals and input-output constraints are provided in the appendix.Source: §3.2 · p. 4
3.3. MCP Ecosystem
- The curated ecosystem spans 27 MCP servers and 232 tools across seven functional areas, selected for orthogonality, documentation, and stability.The areas include knowledge, science, geography and travel, finance, office automation, weather and air quality, and health.Source: §3.3 · p. 4, Figure 4 · p. 4
4. Metrics
The metrics treat tool-use trajectories as structured sequences of dependent steps and parallel calls, then evaluate call detection, trajectory alignment, and end-task quality.
4.1. Mathematical foundations
- Similarity-bucketed Hungarian matching pairs serialized calls within tool-name buckets, producing deterministic, auditable one-to-one matches while allowing step-agnostic recovery of correct calls.Similarity thresholds screen out weak matches; unmatched or missing calls remain unmatched rather than receiving credit.Source: §4.1 · pp. 4–5
4.2. Detection Metrics
- Recall measures matched ground-truth call coverage, while Precision measures the validity of predicted calls and controls false positives.The metrics address omitted or incorrect tools and unnecessary calls, respectively.Source: §4.2 · p. 5
4.3. Trajectory Alignment Metrics
- Argument Similarity measures semantic faithfulness of matched arguments, and Step Coherence penalizes splitting matches from one ground-truth step across multiple predicted steps.Step Coherence captures unnecessary splits of operations that could run within one multi-threaded step.Source: §4.3 · pp. 5–6
- Merge Purity assesses whether causally distinct ground-truth steps are merged, while Order Consistency measures inversions in the order of causally related steps.Higher normalized purity indicates less over-merging, and higher Order Consistency indicates fewer inversions.Source: §4.3 · p. 6
4.4. LLMs Judges Metrics
- A four-LLM judge ensemble scores Task Completion and step-level Information Grounding as complementary measures of trajectory quality.Task Completion uses a trimmed mean that drops the highest and lowest judge scores; judges assess Information Grounding using both reference and predicted trajectories.Source: §4.4 · p. 7
5. Experiment
M3-Bench evaluations show that leading models differ in coverage, structural consistency, and task-specific strengths. Longer trajectories do not necessarily mean better performance, while interpreter outcomes expose tool-use hygiene as another source of error.
5.1. Evaluation Results
- GPT-5 leads with Average Score (0.482), Recall (0.627), Argument Similarity (0.583), Step Coherence (0.502), and Order Consistency / Merge Purity (0.290 / 0.453).Gemini 2.5 Pro (0.423) and Grok 4 (0709) (0.411) have similarly balanced profiles; Claude 4.5 Haiku pairs Precision (0.480) with much lower Recall and modest structure metrics.Source: §5.1 · p. 7
5.2. Per-Task Analysis
- Task profiles cross: Grok-4 is weak on Fruit Cal but strong on QR-code/Attraction, GPT-5 trails on Vision Research but dominates Biology Cut, and Gemini 2.5 Pro excels on Travel Comparison.Figure 7 presents average scores for selected tasks.Source: §5.2 · pp. 7–8, Figure 7 · p. 8
5.3. Budgets across models.
- GPT-5 averages (3.55 / 14.77) steps/tool calls and GPT-5 Nano (5.13 / 17.97), while Gemini 2.5 Pro averages (3.04 / 6.95).Gemini 2.5 Flash averages (4.04 / 7.12) and Grok-4 Fast (2.91 / 5.93); very short Claude 4.5 Haiku and GLM 4.5v dialogues reflect under-coverage rather than efficiency.Source: §5.3 · p. 8
5.4. Analysis of Error Distributions
- Higher-performing systems have more Successful calls, while Qwen2.5-VL-72B and GLM-4.5V show tool-use hygiene shortfalls rather than primarily multi-hop reasoning limits.Interpreter replay classifies calls as illegal format, unknown tool, invalid arguments, successful with resource not found, or successful.Source: §5.4 · p. 8
6. Conclusion
The conclusion presents M3-Bench as a multimodal MCP benchmark for multi-hop and multi-threaded workflows. Its auditable call alignment and structure-aware metrics reveal persistent gaps in argument fidelity and structural consistency.
6. Conclusion
- M3-Bench targets multimodal tool use under MCP with trajectories supporting multi-hop and multi-threaded execution.The benchmark is tailored to these workflow characteristics.Source: §6 · p. 9
- Predicted and reference calls are serialized, text-encoded, and matched one-to-one using tool-name–bucketed Hungarian alignment.The resulting correspondences are auditable and underpin a structure-aware suite of interpretable metrics.Source: §6 · p. 9
- Across representative MLLMs, results reveal persistent gaps in argument fidelity and structural consistency.The conclusion highlights the need for agents that jointly reason over images, text, and tool graphs.Source: §6 · p. 9
Benchmark
This section presents the benchmark’s image-grounded MCP tasks, workflow prompts, and evaluation methods for task completion, step coverage, and tool-call alignment. It also reports human judgments of trajectories and similarity thresholds.
6.1. Task Descriptions
- Tasks combine image understanding with multi-step MCP workflows for needs such as camping, safety documentation, medical guidance, travel, and NixOS diagnostics.Workflows can require verified campground logistics, annotated hazard slides, medicine checks, air-quality comparisons, or staged package diagnostics.Source: §6.1 · pp. 12–14
6.2. LLM Prompts Used in Executor and Judge
- The Executor first plans tool selection, then executes selected calls as bounded-parallel JSON, checks completion with a yes/no stop decision, and produces a natural-language reply.The prompts direct it to consider conversation history and skip unnecessary calls.Source: §6.2 · pp. 14–15
- The Judge flags incorrect causal order or clearly wrong tool information, then reconstructs an optimal trajectory by removing unnecessary calls and grouping parallel operations.The summarization prompt groups calls by step and removes failed or unnecessary calls.Source: §6.2 · pp. 15–16
- End-task completion is scored by a four-judge trimmed mean on a 0–10 scale.The prompt requests a single scalar based solely on visible prediction content.Source: §6.2 · p. 16
- The judge rubric scores useful tool feedback and task completion, while grounding evaluation rewards reference-step coverage and penalizes missing or deviating steps proportionally.Equivalent or paraphrased steps and minor order changes count as covered; extra steps are not penalized.Source: §6.2 · p. 16
6.3. Bucketed Hungarian sample pairs based on similarity
- Within each tool bucket, weak-threshold-eligible call pairs enter Hungarian assignment, while pairs above the strong threshold count as high-confidence matches.Illustrative high-, medium-, and low-similarity pairs show that semantically close arguments can match despite surface variation, while dissimilar pairs are withheld.Source: §6.3 · p. 16, Figure 8 · p. 17
6.4. Human Study
- The human-optimized Best Trajectory was preferred in 98.16% of cases (589/600 selections, 98.17%).Participants compared randomized, blinded trajectory pairs based on faithfulness and procedural soundness.Source: §6.4 · pp. 16–17
- Thresholds (τstrong, τweak) = (0.8, 0.6) produced the closest agreement with majority human labels on 40 call pairs.Participants categorized pairs as strong, medium, or weak correlation, and the thresholds were selected by sweeping similarity cutoffs.Source: §6.4 · p. 17
7. Trajectory Case Study
The trajectory case studies illustrate multi-step workflows that combine image interpretation with tool-based lookup, planning, or artifact creation. They show examples ranging from fish-trip planning to document reconstruction and ecological reasoning.
Fish
- The fish workflow identifies an American eel and assembles a Sandy Hook fishing plan with weather, route, nearby amenities, and practical guidance.The example gives a 44-mile, one-hour route from Newark to Sandy Hook and lists restaurants and bait shops within 5 km.Source: §7 · p. 18
Generate PowerPoint
- The slide-reconstruction workflow extracts slide text, adds answers to a detected Reddit-related question, and saves a rebuilt presentation.The final slide uses a blue title-and-question theme and includes a Reddit answer list.Source: §7 · pp. 18–19
Barcode Lookup
- The barcode workflow scans an ISBN and retrieves bibliographic metadata identifying The Discovery of Heaven by Harry Mulisch.OpenLibrary metadata gives Viking as publisher, 1996 as the year, and 730 pages.Source: §7 · p. 19
QR Attraction
- The QR workflow retrieves a linked image, locates the Natural History Museum in London, and reports air quality and health guidance.The reported UAQI is 82, with NO2 as the dominant pollutant.Source: §7 · p. 20
Biology Chain
- The Biology Chain example combines organism detection and privacy blurring with ecological summaries to validate a food-chain sequence.It gives Grass → Rabbit → Wolf as one valid chain and excludes the person from the natural chain.Source: §7 · pp. 20–21
8. Error Trajectory Distribution Analysis
The case studies illustrate failure trajectories involving malformed calls, unavailable tools, and mistaken visual grounding. Several examples continue to unsupported or superficially coherent final answers despite execution problems.
Illegal calling format
- GLM-4.5v’s non-JSON tool-call format violates the MCP schema and halts progress in the Fruit Cal task.In a separate movie-task example, repeated unavailable OCR and TMDB tools culminate in an unsupported showtime claim.Source: §8 · pp. 21–22
Unknown Tool Invocation
- Qwen2.5-VL-72B-Instruct repeatedly calls unavailable OCR and TMDB endpoints, misuses a server/tool wrapper, and ends with an unsupported claim.The example’s calls include unsupported air-quality tools, while the final summary remains coherent and informative.Source: §8 · pp. 21, 23, 24
- InternVL-3.5 repeatedly invokes unavailable tools and fails to recover, then presents a conclusion unsupported by executed tools.The example also shows workbook calls failing on missing or unusable file paths.Source: §8 · pp. 22–23
Incorrect image understanding
- Claude-Sonnet-4-5 misidentifies the right-hand landmark, triggering downstream geolocation errors and an unsupported final summary.The landmark was in Tokyo, not Xining, and the air-quality retrieval steps failed after the misidentification.Source: §8 · p. 23
9. Per-tasks M3-Bench Evaluation
This section presents per-task M3-Bench evaluations across models. The supplied text identifies the ordering and task coverage of the tables but does not provide their score values.
Per-task evaluation
- Table 2 reports results for each task across models, ordered by average score from lowest to highest.The passage identifies the table’s scope and ordering, not individual scores.Source: §9 · p. 24
Task coverage
- The task-specific cross-model evaluation captions identify Items Price as Table 5, Fish as Table 10, and NixOS Diagnostic as Table 32, all on M3-Bench.Source: Table 5 · p. 25, Table 10 · p. 26, Table 32 · p. 34
- Separate cross-model evaluations of multimodal MCP tool-calling trajectories on M3-Bench are listed for Attraction (Table 6), Movies (Table 7), and QR Attraction (Table 18).Source: Table 6 · p. 25, Table 7 · p. 25, Table 18 · p. 29