PaperlayerView exact revision on arXiv

Source-linked AI summary

FinToolBench: Evaluating LLM Agents for Real-World Financial Tool Use

Jiaxuan Lu, Kong Wang, Yemin Wang, Qingmei Tang, Hongwei Zeng, Xiang Chen, Jiahao Pi, Shujian Deng, Lingzhi Chen, Yi Fu, Kehua Yang, Xiao Sun

arXiv:2603.08262v2cs.AI

TL;DR

Financial tool-use evaluation lacks a realistic, finance-specific benchmark that measures more than executable API calls. FinToolBench builds such a benchmark and pairs it with finance-aware routing and compliance metrics; across seven backends, no model dominates capability and compliance simultaneously. Its scope remains a dated, read-only benchmark snapshot rather than deployable financial automation.

  • Problem

    Existing finance evaluations emphasize static or document-based QA, while general tool benchmarks lack finance-specific checks for timeliness, intent, and regulatory domain alignment.

  • Method

    FinToolBench combines 760 executable tools and 295 tool-required questions with auditable traces, finance-aware compliance metrics, and the FATR routing baseline.

  • Results

    Across seven LLM backends, no model dominates capability and compliance: aggressive callers improve coverage but produce noisier traces, whereas conservative models produce cleaner, sparser tool use.

  • Takeaways & Limitations

    FinToolBench provides a reproducible testbed for comparing financial tool-use agents through trace-level capability and compliance evaluation.

  • Takeaways & Limitations

    The benchmark excludes paid terminals, brokerage and order-routing systems, many markets and jurisdictions, and production deployment conditions; FATR is diagnostic rather than deployable.

Abstract

from arXiv · show

The integration of Large Language Models (LLMs) into the financial domain is driving a paradigm shift from passive information retrieval to dynamic, agentic interaction. While general-purpose tool learning has witnessed a surge in benchmarks, the financial sector, characterized by high stakes, strict compliance, and rapid data volatility, remains critically underserved. Existing financial evaluations predominantly focus on static textual analysis or document-based QA, ignoring the complex reality of tool execution. Conversely, general tool benchmarks lack the domain-specific rigor required for finance, often relying on toy environments or a negligible number of financial APIs. To bridge this gap, we introduce FinToolBench, the first real-world, runnable benchmark dedicated to evaluating financial tool learning agents. Unlike prior works limited to a handful of mock tools, FinToolBench establishes a realistic ecosystem coupling 760 executable financial tools with 295 rigorous, tool-required queries. We propose a novel evaluation framework that goes beyond binary execution success, assessing agents on finance-critical dimensions: timeliness, intent type, and regulatory domain alignment. Furthermore, we present FATR, a finance-aware tool retrieval and reasoning baseline that enhances stability and compliance. By providing the first testbed for auditable, agentic financial execution, FinToolBench sets a new standard for trustworthy AI in finance. The tool manifest, execution environment, and evaluation code will be open-sourced to facilitate future research.

1 Introduction

FinToolBench addresses the mismatch between executable tool-use evaluation and finance-specific reliability requirements. It introduces a runnable benchmark and finance-aware metrics and baseline for auditable financial agent execution.

  • Financial agents require evaluation of tool traces beyond invocation success because stale data, drifting endpoints, and wrong-market tools can undermine reliability.
  • Existing general tool benchmarks emphasize API correctness, while finance benchmarks largely rely on static QA or negligible mock interfaces.
  • FinToolBench contains 760 executable financial tools and 295 tool-required questions, including single-tool and multi-tool items.
  • Each tool is annotated for timeliness, intent type, and regulatory domain, enabling call-level compliance mismatch metrics alongside execution measures.
  • FATR provides a lightweight finance-aware routing baseline with candidate retrieval, attribute-enriched tool cards, and execution-stability utilities.

2 Related Work

Related work separately advances general tool-use evaluation and finance knowledge benchmarks, but neither establishes a large standardized financial tool library with call-level compliance metrics. LLM judging offers scalable correctness assessment, though its sensitivity motivates repeated evaluation and explicit separation of execution from correctness.

  • General benchmarks study API calling, tool-interface competence, and long-horizon interaction, whereas finance benchmarks mostly assess knowledge and document-centric QA.
  • Existing finance benchmarks do not release a standardized large tool library or define call-level compliance metrics.
  • LLM judges enable scalable open-ended answer scoring but can vary across runs and prompts.
  • FinToolBench reduces judging variance through repeated scoring and separates tool execution from answer correctness.

3 FinToolBench

FinToolBench is an execution-grounded benchmark that combines a validated, normalized financial tool library with tool-required questions, auditable traces, and separate capability and compliance evaluation.

  • FinToolBench evaluates financial tool use under real execution, with every run producing an auditable tool trace.
  • The construction pipeline filters raw sources, normalizes interfaces, annotates finance attributes, aligns questions with tools, and applies human quality assurance.
  • 760 tools remain after filtering 5,470 candidate interfaces from RapidAPI and AkShare.
  • The benchmark contains 295 questions, including 166 single-tool and 129 multi-tool items, and records complete traces for joint capability and compliance assessment.
  • Evaluation Metrics: Capability metrics measure invocation, execution, conditional execution success, and answer correctness from structured traces.
  • Evaluation Metrics: Compliance metrics use tool finance tags and judged call alignment to compute timeliness, intent, and domain mismatch rates.

4 Finance-Aware Tool Routing (FATR)

FATR makes finance constraints explicit during tool retrieval and ReAct planning while keeping execution infrastructure fixed and model-agnostic. It routes calls using timeliness, intent, and domain attributes and can optionally apply conservative filters.

  • FATR retrieves a small candidate tool set, formats tool cards, and exposes finance attributes to a generic LLM planner.
  • FATR wraps execution with unified validation, dispatch, output normalization, caching, and trace recording while keeping the benchmark independent of the baseline.
  • The ReAct interaction horizon is capped at max_steps=5 tool-augmented steps to limit latency and exposure to tool drift.
  • The planner infers timeliness, intent, and domain requirements before selecting compatible tools and maintaining constraints through the ReAct loop.
  • Optional hard filters can exclude transactional or domain-incompatible tools to reduce unforced compliance errors.

5 Experiments

The experiments evaluate FinToolBench under a reproducible protocol with fixed execution infrastructure and varying LLM planners. They combine capability, answer-quality, and finance-compliance measurements while validating the LLM-based judging pipeline with expert annotations.

  • Evaluation Protocol: FinToolBench evaluates 295 tool-required questions using 760 runnable financial tools under a fixed limit of five tool-use rounds per question.Each tool call has a 60-second timeout and up to two retries, with deterministic caching and full trace logging.
  • Evaluation Protocol: Evaluation stratifies questions by single- versus multi-tool usage and inferred category, reporting metrics defined for capability and compliance analysis.The protocol analyzes tool traces under standardized conditions rather than treating every question as an identical execution case.
  • Evaluation Protocol: The unified FATR framework fixes retrieval and execution infrastructure while varying the LLM planner across seven backends.All models use the same structured tool-call interface, isolating planner behavior under a common stack.
  • Evaluation Protocol: Compliance mismatch rates are conditioned on traces with at least one tool call and should be interpreted alongside invocation and execution-success rates.Conservative models may obtain low mismatch rates by attempting fewer tool-required questions.
  • Validation: Three finance experts annotated 60 questions and 50 traces, yielding 85.3% inter-expert agreement and 83.8% expert-versus-pipeline agreement with F1 0.81.The pipeline used repeated judging to reduce variance and reviewed disagreements in a second round.

6 Results

Results show that tool invocation, execution reliability, answer quality, and finance-specific compliance diverge across models. Attribute injection changes tool selection and improves conditional reliability while reducing mismatch, and usage patterns reveal a broad but selective routing space.

  • 6.1 Main Results: Doubao-Seed-1.6 achieves the best end-to-end execution success, with TESR =0.3254.Qwen3-8B invokes tools most often, but its lower CER shows that frequent invocation does not ensure reliable execution.
  • 6.1 Main Results: Claude-Sonnet-4.5 and Gemini-3.1-Pro obtain stronger semantic scores, while Grok-3-beta has the highest CER and low mismatch rates among attempted tool traces.These patterns separate answer quality from trace precision and compliance.
  • 6.1 Main Results: FinToolBench separates coverage, execution reliability, answer quality, and finance-specific compliance instead of collapsing them into one success rate.Model rankings depend on the operational objective, such as semantic quality for research assistance or low mismatch for monitoring workflows.
  • 6.2 Finance Attribute Injection: Attribute injection reduces marginal calls, improves conditional execution reliability, and lowers mismatch rates across timeliness, intent, and domain dimensions.The ablation keeps the same retriever and executor while removing finance tags from tool cards in the baseline.
  • 6.2 Finance Attribute Injection: 103/114/78 of 295 evaluation instances end with no/single/multiple final-round tool calls.Doubao-Seed-1.6 makes 923 calls across 236 distinct tools, showing that the 760-tool inventory is a routing space rather than a per-tool unit-test suite.
  • 6.2 Finance Attribute Injection: Category-level analysis shows that aggregate scores can hide failures specific to value extraction, macro interpretation, and other tool scopes.The breakdown helps distinguish tool-selection failures from rigid output-format or answer-scoring mismatches.

7 Conclusion

The paper concludes that FinToolBench provides a runnable, auditable benchmark for evaluating financial tool-use agents across capability and compliance. Across seven backends, no model dominates both dimensions, motivating reproducible trace-level comparison.

  • 7 Conclusion: FinToolBench provides 760 free-tier tools and 295 tool-required questions for runnable evaluation of financial tool-use agents.The benchmark separates capability metrics from compliance metrics over timeliness, intent, and domain constraints.
  • 7 Conclusion: Across seven LLM backends, no model dominates both capability and compliance.Aggressive callers improve coverage but produce noisier traces, whereas conservative models produce cleaner but sparser tool use.
  • 7 Conclusion: The released tool manifest, question set, and evaluation scripts support reproducible comparison while preserving trace-level auditability.FATR is provided as a reference baseline rather than the benchmark itself.

Limitations

The benchmark covers free-tier financial tools and a bounded question set rather than the full range of financial systems, markets, and jurisdictions. Its metrics are benchmark checks, and results are a dated snapshot affected by drift and judge sensitivity.

  • Limitations: FinToolBench covers 760 free-tier tools and 295 tool-required questions, excluding paid terminals, brokerage or order-routing systems, and all markets and jurisdictions.The scope boundary limits direct generalization to financial systems outside this inventory and question set.
  • Limitations: Its metrics are benchmark checks rather than professional compliance review, and API, data, and model drift make results a dated protocol snapshot.Residual LLM-judge sensitivity also affects interpretation, while FATR is diagnostic rather than deployable.

Ethical Considerations

FinToolBench constrains its research setting to executable, non-transactional tools and excludes sensitive or non-public financial information. Its benchmark design makes finance-specific compliance auditable through normalized tool interfaces, structured traces, and explicit timeliness, intent, and regulatory-domain annotations.

  • FinToolBench is a research benchmark rather than financial advice or authorization for regulated automation.
  • The benchmark uses read-only or sandboxed tools, disallows transactions, and excludes personal data, credentials, paid-feed content, and non-public financial information.
  • The inventory retains executable endpoints through authentication, billing, rate-limit, schema-mapping, and successful-invocation checks.RapidAPI filtering yields 261 executable endpoints, while finance-domain and executability validation retains 499 AkShare interfaces.
  • All 760 tools are normalized into a common manifest and execution traces record tool names, JSON parameters, outputs, errors, and chronological steps.This structure distinguishes model reasoning errors from system-level failures such as API rejections.
  • Each tool receives timeliness, intent-type, and regulatory-domain annotations that support planning and call-level compliance evaluation.Timeliness spans realtime, daily, as_filed, periodic, and static; intent spans informational, advisory, and transactional.
  • The question set excludes static-knowledge items and contains 295 tool-required questions, including 166 single-tool and 129 multi-tool questions.Candidate retrieval and LLM verification align questions with tools before deduplication and release.

D.4 Human Verification

Human verification checks whether benchmark questions genuinely require tools, whether aligned tools are necessary, and whether compliance annotations and answer formats are plausible. Expert review achieves substantial agreement with both the benchmark artifacts and the automatic pipeline.

  • Three financial-data experts independently review sampled questions and execution traces using a fixed rubric.The sample covers single-tool and multi-tool items as well as all three compliance dimensions.
  • The rubric checks tool necessity, logical necessity of aligned tool sequences, plausibility of timeliness, intent, and regulatory-domain labels, and answer-format compatibility.
  • 85.3% inter-expert agreement and 83.8% agreement with the automatic pipeline indicate substantial validation consistency, with F1 0.81 against the pipeline.Disagreements receive a second review, and unresolved items are removed from the release.

E Metric Definitions

The evaluation separates whether agents invoke and successfully execute tools from whether their tool traces satisfy finance-specific constraints. It also documents scoring, retrieval-depth analysis, judge stability, and artifact-use boundaries.

  • Capability Metrics: Capability metrics measure tool invocation, execution success, conditional execution, answer quality, and answer quality conditional on successful execution.TIR tracks questions with at least one tool call; TESR requires a valid final tool output; CER conditions success on invocation; SOFT SCORE uses repeated LLM judgments; CSS conditions answer quality on successful execution.
  • Compliance Mismatch Rates: Compliance metrics measure whether attempted tool traces align with required timeliness, intent type, and regulatory domain.A trace contains selected tools, arguments, and outputs, while finance attributes are looked up from tool metadata and evaluated with judge-level alignment indicators.
  • Compliance Mismatch Rates: TMR, IMR, and DMR report the fractions of invoked-question traces containing at least one mismatch, with lower values indicating better compliance.Because these rates are conditional on invocation, they are reported jointly with TIR and TESR to distinguish attempted-trace quality from coverage and execution capability.
  • Finance-Aware Evaluation: The planner is instructed to infer required finance tags, require domain overlap, prefer matching timeliness, and avoid transactional tools unless explicitly requested.Tool descriptions expose financial tags during selection, while normalized wrappers align generated function parameters with request-schema keys to reduce argument errors.
  • Retrieval Depth: Retrieval coverage improves sharply through K=20 and then saturates, motivating K=20 as the default candidate depth.Tool Hit@K measures whether the gold required tool set is covered by the retrieved candidate pool.
  • Judge Stability and Prompt Order: Three-repeat GPT-5.1 scoring has a representative SOFT SCORE standard deviation of 0.02 and an approximately ±0.03 95% confidence interval.Reversed prompt order yields 94.0% binary-label agreement and changes overall IMR by 1.0 percentage point on 50 sampled cases.

K.3 Case Study 3: Finance Attributes Do Not Guarantee Numerical Fidelity

The case studies show that injecting finance attributes can alter tool strategy and recover execution, but it does not guarantee correct task framing or numerical answers. Benefits include pruning redundant or incompatible calls while preserving correctness in some cases, whereas proxy-based estimation can remain wrong when the target is unavailable.

  • Failure mode: A required numeric target may not be directly exposed by available tools, forcing a choice between abstaining and using a proxy.This limitation makes strategy improvement insufficient for numerical fidelity.
  • Case Study 1: For AMEX operating-margin evaluation, the baseline produces no final answer after an interface error, while FATR recovers a tool trace but remains incorrect on task framing.The ground truth states that operating margin is not the relevant performance measure for this company.
  • Case Study 2: For Tianhong Yu’e Bao, injected finance attributes prune redundant tools while preserving the correct low-downside-risk assessment.The case concerns evaluating the fund’s downside risk relative to its category.
  • Case Study 3: For Nike’s FY2016–FY2018 cost-of-goods-sold question, injection changes the strategy but the resulting proxy-based estimate remains incorrect against the 55.1% ground truth.The figure describes this as a single-box comparison in which injected finance attributes change the tool strategy without producing a correct estimate.
Loading 2603.08262v2…