Source-linked AI summary
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
Jie Zhu, Yimin Tian, Boyang Li, Kehao Wu, Zhongzhi Liang, Junhui Li, Xianyin Zhang, Lifan Guo, Feng Chen, Yong Liu, Chi Zhang
TL;DR
Financial LLM agents need evaluation on realistic tasks involving external tools, dependent calls, and multi-step reasoning, but existing financial evaluations typically omit tool use. FinMCP-Bench addresses this gap with a 613-sample benchmark spanning real and synthetic MCP tasks, then evaluates mainstream LLMs using explicit tool-invocation metrics. Results show stronger performance from Qwen3 models overall, while multi-tool dependencies and multi-turn conversations remain challenging.
Problem
Existing financial evaluations are limited to specific tasks and typically do not involve tool use, despite financial agents needing dependent tool calls and multi-step reasoning.
Method
FinMCP-Bench combines real and synthetic financial queries across single-tool, multi-tool, and multi-turn samples, and evaluates LLMs with explicit tool-invocation metrics including EMR.
Results
Qwen3 models generally outperform other evaluated models on TF1 and EMR, while multi-turn samples tend to receive the lowest scores overall, especially in EMR.
Takeaways & Limitations
The benchmark provides a standardized testbed that exposes challenges in complex multi-tool dependencies and multi-turn conversations for financial LLM agents.
Abstract
from arXiv · showhide
This paper introduces \textbf{FinMCP-Bench}, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 real financial MCPs and three types of samples, single tool, multi-tool, and multi-turn, allowing evaluation of models across different levels of task complexity. Using this benchmark, we systematically assess a range of mainstream LLMs and propose metrics that explicitly measure tool invocation accuracy and reasoning capabilities. FinMCP-Bench provides a standardized, practical, and challenging testbed for advancing research on financial LLM agents.
1 Introduction
FinMCP-Bench addresses limited financial tool-use evaluation by benchmarking LLM agents on realistic, dependency-rich MCP tasks. It combines diverse scenarios and sample types with systematic model evaluation and explicit tool-invocation metrics.
- Motivation: Financial LLM agents must interpret requests, invoke external tools, and perform multi-step reasoning with dependencies between tool calls.These dependencies make realistic financial-agent evaluation difficult.
- Motivation: Existing financial evaluations largely focus on specific tasks and typically omit tool use.FinMCP-Bench is introduced to address this evaluation gap through real-world MCP interactions.
- Benchmark: The dataset construction starts from 10K production interaction records spanning 33 real-world scenarios and 65 MCP-integrated financial tools.Records average more than two dependent tool calls, and augmentation creates chains exceeding five steps.
- Benchmark: FinMCP-Bench contains 613 samples across 10 main scenarios and 33 sub-scenarios, combining real and synthetic queries with single-tool, multi-tool, and multi-turn formats.The formats support evaluation across different levels of task complexity.
2 FinMCP-Bench: A Financial MCP Benchmark
FinMCP-Bench combines real financial-agent traces with chain-based multi-tool and role-playing-based multi-turn synthesis. Its 613 samples span multiple scenarios and tool-use complexities, with expert-reviewed construction and dataset statistics that quantify task difficulty.
- Dataset Scope: FinMCP-Bench covers 613 samples across 10 main scenarios and 33 sub-scenarios, including Market Analysis and Research and Investment Planning and Allocation.MAR contains 141 samples and IPA contains 101 samples.
- Dataset Scope: Samples are categorized as single-tool, multi-tool, or multi-turn, containing 145, 249, and 219 samples respectively.Multi-tool calls may be sequential or parallel, while multi-turn samples span multiple conversational turns.
- Data Construction: Real data comes from 10,000 anonymized XiaoGu AI assistant logs filtered for genuine financial needs, successful tool-call resolution, and satisfactory responses.The process initially yields 1,484 single-tool and 183 multi-tool samples.
- Data Construction: The chain-based multi-tool method builds a dependency graph, generates aligned user queries, and expands sampled chains into full trajectories.Candidate dependencies are validated by an LLM, and the final graph contains 65 nodes with 288 edges.
- Data Construction: Experts retain 249 high-quality multi-tool samples after reviewing 1K generated query-trajectory pairs alongside real multi-tool samples.Retained trajectories must preserve the specified dependency relations, while additional tools may also appear.
- Data Construction: The role-playing method assigns a planner to define user personas and goals, then simulates both user and assistant roles to create multi-turn dialogues.From 500 candidates, automated validation and review by six financial experts retain 219 samples.
- Dataset Analysis: The benchmark classifies samples with up to 5 tool calls as easy, up to 10 as medium, and the remainder as hard.Single-tool samples have exactly one call; multi-tool samples average 7.32 calls across 5.72 steps, while multi-turn samples average 5.95 turns and 5.00 tools.
3 Experimentation
The experiments evaluate six LLMs by extracting invoked tools and measuring tool-level accuracy with recall, precision, F1, and exact organization matching. Results show strong overall performance from Qwen3 models, while single-tool and especially multi-turn settings expose distinct tool-selection and conversational challenges.
- Experimental Settings: Six LLMs are evaluated by generating replies from user queries and gold conversation histories, then extracting the tools invoked in those replies.The evaluated models include three Qwen3 variants, DeepSeek-R1, GPT-OSS-20B, and Seed-OSS-36B.
- Evaluation Metrics: Tool Recall measures correctly predicted reference tools, while Tool Precision measures correctly predicted tools among all predicted tools.Both metrics ignore dependency relations between tools.
- Evaluation Metrics: Tool F1 balances Tool Recall and Tool Precision, while Exact Match Rate requires the predicted tool organization to match the reference, ignoring order within parallel groups.Exact Match Rate is stricter because it evaluates the organization of tool calls rather than only tool membership.
- Experimental Results: Overall, the three Qwen3 models generally outperform the other models on both Tool F1 and Exact Match Rate.Model size is not consistently associated with performance: Qwen3-4B-Thinking exceeds Qwen3-30B-A3B-Thinking in EMR, while the larger model has higher TF1.
- Experimental Results: Single-tool samples have higher Tool Recall but lower Tool Precision because models often over-predict multiple tools when only one is needed.The single-tool setting contains only one required tool, making recall easier while exposing unnecessary predictions through precision.
- Experimental Results: Multi-turn samples tend to receive the lowest overall scores, especially in Exact Match Rate, showing that longer conversations with multiple tool calls remain challenging.Scenario-wise gaps also widen for multi-tool planning and cross-source synthesis, whereas simpler single-operation queries narrow them.
4 Conclusion
FinMCP-Bench evaluates LLMs in real-world financial scenarios requiring MCP tool invocation across varying levels of tool-use and dialogue complexity. Its results expose both current model strengths and persistent challenges in complex tool dependencies and multi-turn interactions, motivating further work on reasoning, orchestration, and dialogue capabilities.
- FinMCP-Bench evaluates LLMs in real-world financial scenarios that require invoking MCP tools.
- The benchmark covers single-tool, multi-tool, and multi-turn tasks to capture different levels of tool-use and dialogue complexity.
- Evaluations of several popular LLMs analyze performance across multiple dimensions.
- Results highlight model strengths and challenges, particularly with complex multi-tool dependencies and multi-turn conversations.
- FinMCP-Bench is intended as a standardized, challenging testbed for advancing financial tool-augmented LLM research.
- The benchmark is intended to inspire improvements in reasoning, tool orchestration, and dialogue capabilities in finance.