FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning

Task design

• Its workflows cover Time-Sensitive Data Fetching, Simple Historical Lookup, and Complex Historical Investigation, from rapid verification to multi-source, multi-period analysis.Historical tasks require retrieving the correct disclosure and aligning reporting calendars, while investigations require consistent, comparable values across time.Source: §1 · p. 2

Quality assurance

• Construction involved 70 financial experts, multi-stage verification, rubric-based scoring, and LLM judging with tolerance bands and expert spot checks.The evaluation accounts for legitimate answer variation, including revisions and rounding across sources.Source: §1 · p. 2

Results

• Global: Grok 4 (web) scores 68.9%, 5.0 pp above GPT-5-Thinking (web) and 6.1 pp below experts; Greater China: DouBao leads, but all models trail humans by more than 34 pp.The study evaluates 21 models (products).Source: §1 · pp. 2–3

Benchmark

• FinSearchComp is a fully open-source, end-to-end benchmark with 635 expert-curated questions across Global and Greater China markets and three analyst-style task families.It also releases deterministic gold answers and an open-source evaluation harness.Source: §1 · p. 3

Analysis

• Web search and financial plugins improve performance; recurring failures include shallow searches, stale or mis-timestamped evidence, unit or currency aggregation, and reporting-calendar misalignment.Case analysis also identifies incorrect extraction from valid sources and unnecessary complexity on simple queries; current systems remain fragile in freshness, reconciliation, and temporal reasoning.Source: §1 · p. 3

Related Works

• General browsing benchmarks [23] [27] [7] favor short lookups, while financial QA [5] [15] preloads context; Finance Agent Benchmark [3] uses a self-built retrieval system and historical data.These limitations leave open-domain search, real-time retrieval, and finance-grade evidence integration under-assessed.Source: §1 · pp. 3–4

2.1 Design Principles

• Design principles prioritize professional task diversity, unambiguous answer targets, and cross-market validity; the benchmark uses mirrored Global and Greater China templates with English and Chinese questions.The design balances entity coverage by sector and size across subsets.Source: §2.1 · p. 4

2.2 Task Design of FinSearchComp

• The benchmark defines three analyst-workflow tasks, each requiring an external tool call and a single objective answer, spanning freshness management, point-in-time fidelity, and multi-period synthesis.The tasks progress in difficulty from time-sensitive fetching to complex historical investigation.Source: §2.2 · pp. 4–5
• T1 targets changing daily or intraday values, T2 fixed historical facts, and T3 multi-period derivations; difficulty rises from freshness handling to long-horizon synthesis.The task types test freshness management, reporting-period fidelity, and multi-step retrieval and reasoning.Source: §2.2 · pp. 4–5

2.3 Construction of FinSearchComp

• Task-specific construction uses API-verified live questions, official-source or database historical facts, and expert-generated or database-table-derived multi-step investigations.Experts finalize 2 to 5 high-quality questions per table; historical questions specify reference times or answer ranges when data revisions could create ambiguity.Source: §2.3 · pp. 7–8

2.4 Quality Control

• Quality controls combine 70 financial experts, cross-validation of authoritative sources, ambiguity screening, and independent blind reviews with senior arbitration for discrepancies.The cohort includes 50 annotators and 20 senior reviewers; construction involved approximately 180 expert hours and 60 senior-expert hours.Source: §2.4 · p. 8

2.5 Data of FinSearchComp

• FinSearchComp covers 10 topics, and most evaluation metrics use expert-annotated acceptable-value ranges rather than a single ground-truth value.Source: §2.5 · pp. 8–9

2.6 Evaluation of FinSearchComp

• The evaluation uses LLM-as-a-Judge and a 0-1 error metric, with static-answer tasks assessed against annotated rubrics such as predefined error margins.For time-sensitive questions, evaluators first obtain the real-time answer by executing API commands; the judge maps candidate answers and rubrics to a binary outcome.Source: §2.6 · p. 9
• Time-sensitive evaluation begins after market close and applies asset-specific rubrics: rounding tolerance for mainstream stocks and indices, daily high-low bounds for regional indices, and buffered high-low ranges for FX.This protocol addresses response-time lags, API latency, and APIs’ limits on querying prices at a specific second.Source: §2.6 · p. 9
• 95% agreement with human-verified labels was achieved by LLM-as-a-Judge on roughly 400 instances per dataset.The human evaluation selected 4–5 models for each benchmark dataset.Source: §2.6 · pp. 9–10, Table 3 · p. 10

3 Experiments

• The human baseline is the average score of 50 financial experts who were not involved in data construction and completed benchmark tasks using search tools.Source: §3 · p. 11

3.1 Main Results

• Grok-4 (web) leads the global subset and approaches expert-level accuracy, while DouBao (web) and YuanBao-HunYuan-T1-Thinking (web) are strong on Greater China but trail experts substantially.The overall model hierarchy retains a sizable gap to human experts.Source: §3.1 · p. 11

3.2 Results Across Different Tasks

• Performance declines monotonically from T1 time-sensitive fetching through T2 historical lookup to T3 complex investigation, which requires multi-hop retrieval, temporal reasoning, entity resolution, and evidence reconciliation.T2–T3 also depend on finance expertise, including interpreting filings and disclosures, distinguishing GAAP from non-GAAP metrics, and understanding corporate actions.Source: §3.2 · p. 11
• US models lead on the global set, while Chinese models lead on Greater China, a pattern attributed to geographic corpus coverage, linguistic and market conventions, and alignment or recency effects.The passage states that these home-field effects do not imply leakage.Source: §3.2 · p. 11
• Although Grok-4 and GPT-5-Thinking broadly underperform human experts, they approach expert-level results on the global subset and widen their advantage over other systems on harder tasks, especially T3.On Greater China, Grok 4 (web) attains the top score on T3 despite aggregate accuracy trailing experts; the passage associates these gains with multi-step reasoning, timeline alignment, and entity disambiguation.Source: §3.2 · p. 11

4.1 How much do search capabilities impact performance on FinSearchComp?

• Search augmentation raises average scores by 40.8 points on T1, 29.0 on T2, and 8.1 on T3, with the largest gain on time-sensitive tasks.Models without search score zero on T1, while their nonzero T2 and T3 scores are attributed to approximate, sometimes outdated or misaligned parametric-memory facts. The tasks also require multi-step query planning, timeline and identifier alignment, and evidence verification and synthesis.Source: §4.1 · p. 12, Figure 6 · p. 13

4.2 How do financial plugins improve the performance on top of general search tools?

• On T1, YuanBao’s financial plugin improves DeepSeek R1 performance by 31.9 pp over DeepSeek web search.The plugin provides real-time data and historical datasets, supporting T1 and T2, but YuanBao-R1 remains suboptimal; outcomes also reflect platform search tools and model capability.Source: §4.2 · pp. 12–13, Figure 7 · p. 13

4.3 How does model origin impact the performance?

• US models favor global assets while Chinese models favor Greater China; US asset-origin ratios exceed 100% on T1 and T2, while most models exceed 100% on T3.Among Chinese models, Doubao and Kimi k2 have the highest asset-origin ratios, indicating more balanced performance across the two subsets than other Chinese models.Source: Figure 8 · p. 13, §4.3 · p. 13

4.5 Does reasoning ability enhance performance on FinSearchComp?

• Adding reasoning capability corresponds to an average 7.0-point decline on T1, while changes on T2 and T3 are negligible.The authors suggest that T1’s low complexity and possible overthinking by reasoning models may explain the decline [1].Source: Figure 9 · p. 14, §4.5 · p. 14

4.4 Where do models excel and falter on FinSearchComp?

• Plugin-equipped GPT-5-Thinking, HunYuan-T1, and DouBao exceed 70% average accuracy on T1; Grok 4 leads T2, and only Grok 4 and GPT-5-Thinking score above 30 on T3.Deepseek-R1 web scores 28.8, 12.4% lower than its YuanBao-integrated performance; T1 failures include inactive plugins, outdated content, and conflicting sources. T2 often suffers from parametric-memory answers or reliance on news reports instead of detailed official filings, while T3 commonly requires API or SQL retrieval beyond web-only products.Source: §4.4 · p. 14

Financial Benchmarks

• Financial benchmarks span report-based program reasoning (FinQA [5], ConvFinQA [6]), broad finance and multilingual suites (FLUE, FinEval, MultiFinBen), and professional or noisy queries (FinanceQA [15], BizFinBench [14]).BizFinBench distilled 6.7K genuine queries from an investment-search app.Source: §5 · p. 14
• Existing datasets often provide financial data by default, while Finance Agent Benchmark [3] searches only static historical data, allowing memorization and limiting assessment of search capability.The default data provision substantially mitigates the challenge of searching open-domain sources [23].Source: §5 · p. 14

Agentic Benchmarks

• Agentic benchmarks evaluate tool-mediated tasks: FinEval scores planning, API use, and long-horizon reasoning, while BrowseComp [23], BrowseComp-ZH [27], and BrowseComp-Plus [7] test persistent web navigation.These benchmarks motivate evaluation that combines financial expertise with realistic tool use.Source: §5 · p. 15

• Even state-of-the-art agents significantly underperform humans, often because they search insufficiently deeply or use outdated information.Source: §6 · p. 15
• FinSearchComp is the first fully publicly available end-to-end benchmark for LLM-based agents searching for financial data in realistic, context-free scenarios.Source: §6 · p. 15
• The benchmark contains 635 expert-curated questions across three demanding tasks requiring agents to use tools such as SQL, APIs, and web search to obtain verifiable answers.Source: §6 · p. 15
• The authors release FinSearchComp as a resource to drive the development of more robust and reliable financial agents.Source: §6 · p. 15

8.1 What is Xpert Platform

• Xpert brings together approximately 3,000 selected academic and industry experts across fields including finance, law, healthcare, and education.Its experts include scholars from China’s 985/211 universities and industry professionals with 2–10 years of practical experience.Source: §8.1 · p. 16

8.2 Xpert Leaderboard Intro

• The Xpert Leaderboard assesses AI’s ability to solve complex expert-level tasks in the real world, unlike mainstream exam-oriented evaluations.It is dedicated to driving AI to create greater economic value.Source: §8.2 · p. 16

A.1 Release Format

• The release contains JSONL files with six fields—id, tier, question, tool_template, answer, and trace—plus a sandboxed harness that replays traces and accompanying documentation and citation files.Source: §A.1 · p. 20

• Forward- and backward-adjusted stock prices differ across databases, so the benchmark uniformly queries non-adjusted prices.Source: §A.2 · p. 20
• PE (TTM) can vary because institutions define “Earnings” differently.Source: §A.2 · p. 20
• For dual-listed companies, market capitalization questions specify the calculation method or are avoided.Source: §A.2 · p. 20
• Futures data can differ because institutions switch main contracts at different times and use different continuous-contract construction algorithms.Source: §A.2 · p. 20
• Cryptocurrency prices vary across exchanges.Source: §A.2 · p. 20

A.3 Guide for Mitigating Ambiguity

• Questions must specify GAAP or Non-GAAP accounting so the answer and ground truth use the same accounting basis.For example, ask for net income under U.S. GAAP rather than asking for income without specifying a standard.Source: §A.3 · p. 21
• Specify the currency and answer units, and identify whether an RMB exchange rate is onshore (CNY), offshore (CNH), or interbank.These specifications make values directly comparable and distinguish rates that can differ.Source: §A.3 · pp. 21–22
• When asking for a company’s industry, name the classification standard and, where relevant, its level or code system.Examples include a specific Shenwan level or CSRC industry codes.Source: §A.3 · p. 21
• For multi-listed companies, specify the market-cap calculation method rather than leaving “total market cap” ambiguous.The guide describes summing market values from each listing as a precise method.Source: §A.3 · p. 21
• Questions must state numerical precision, such as decimal places or rounding, and reference answers should accept equivalent futures quote notation.The guide says both “113.265625” and “113’08’5” are acceptable for the futures quote example.Source: §A.3 · p. 21

B Detailed Scores on FinSearchComp

• Table 5 reports detailed scores for various models on FinSearchComp.Source: B Detailed Scores on FinSearchComp · p. 22

Judge for Time-Sensitive Data Retrieval

• Time-sensitive answers must include all required content; extra information is allowed, and trusted real-time information supplies the ground truth for accuracy checks.An empty student answer scores 0; when the student responds but the real-time data contains no numbers, the prompt specifies a null score.Source: C Prompt · p. 23
• Accuracy rules allow specified rounding, high-low ranges, or absolute tolerances, while judging numerical values rather than date alignment.For high-low criteria, the prompt also permits expanding the range by the stated absolute amount.Source: C Prompt · p. 23

Judge for Single Historical Data Point Retrieval and Complex Data Retrieval

• Historical judges identify the student’s final answer and compare it with the trusted reference, assigning only 1 for correct or 0 for incorrect answers.When a reference contains multiple key points, all must be present and correct; equivalent numerical formats and omitted units can still match.Source: C Prompt · p. 25
• Historical evaluations require the result first, followed by a brief scoring basis and JSON containing the answer_score.The scoring basis is limited to 150 words or tokens.Source: C Prompt · p. 25

Time-Sensitive Data Fetching

• Using financial plugins, a web-based product reported Walmart’s August 27, 2025 close as $96.08, up 0.03% for the day.It also reported a five-trading-day decline of 6.33% and year-to-date gain of 7.13%.Source: D Case · p. 27

Single Historical Data Retrieval

• Without using search sources, a web-based product answered Apple’s 2021 fiscal-year investing cash question with $144,266 million, and the case is identified as a failure.Its reasoning relied on recalled financial-report figures rather than retrieved sources.Source: D Case · p. 28

Complex Historical Investigation

• For Apple’s split investigation, the product searched Nasdaq and reverse-searched to verify a 4-for-1 split and a -$371.65 opening-price change.The response compared the August 31, 2020 opening price of $127.58 with the August 28 closing price of $499.23.Source: D Case · p. 29