PaperlayerView exact revision on arXiv

Source-linked AI summary

FinExam-10K: When Retrieval Helps Financial Reasoning?

Yan Lin, Jingyu Sun, Zhongliang Guo, Qing Li, Zhuohan Xie, Yuxia Wang

arXiv:2608.28155v1cs.CL

TL;DR

Professional financial reasoning lacks a unified benchmark spanning the full CFA and FRM structure while distinguishing coverage from locally grounded answerability. FinExam-10K supplies that benchmark and evaluates whether retrieval helps or harms, finding that selective gated invocation yields only a modest held-out gain. The paper also identifies context completeness as a necessary boundary for interpreting reasoning results.

  • Problem

    Existing benchmarks do not provide a unified, level-resolved CFA–FRM benchmark with item-aligned analysis of failures, repairs, and intervention-induced harms.

  • Method

    The paper introduces 10,198 expert-reannotated questions, complementary Full-Coverage and Context-Complete tracks, matched retrieval interventions, and a public-trained gate for FunctionGraph-RAG.

  • Results

    Across 17 models, performance is strong overall but collapses on Hard items; retrieval rescues errors while also causing harms, and gated FunctionGraph-RAG improves held-out accuracy from 70.83% to 71.23%.

  • Takeaways & Limitations

    The benchmark supports evaluating financial reasoning with both curriculum coverage and local evidence, while the intervention results favor selective rather than uniform retrieval.

  • Takeaways & Limitations

    Full-Coverage results may mix reasoning with reconstruction from detached parent evidence, and gate efficiency is an implied branch-call count rather than measured resource cost.

Abstract

from arXiv · show

Professional financial examinations require models to combine domain knowledge, calculation, and judgment, yet no benchmark covers the full CFA and FRM structure under one protocol. We introduce FinExam-10K, to our knowledge the largest reported English benchmark for this setting, with 10,198 expert-reannotated questions spanning CFA Levels I-III and FRM Parts I-II. We release 5,110 questions and sequester 5,088 for a quarterly maintained leaderboard. To separate coverage from local answerability, we report a 10,198-item Full-Coverage Track and a 7,625-item Context-Complete Reasoning Track, which is the primary basis for claims about reasoning from the supplied record. Across 17 models, the best accuracy is 85.29% overall. On the frozen Hard band, the best score is 34.68% on the Full-Coverage Track and 54.57% on the 372 context-complete items. All 17 models share 47 context-complete failures. Function-RAG and FunctionGraph-RAG rescue hundreds of errors but also overturn many correct answers, producing little or negative net gain. A gate trained only on public data decides from the question and initial response when FunctionGraph-RAG should run. On the 5,088 held-out items, the gate invokes FunctionGraph-RAG for 7.9% of questions and improves accuracy from 70.83% to 71.23% (p = .0446).

1 Introduction

FinExam-10K addresses the lack of a unified CFA- and FRM-aligned benchmark for financial reasoning by combining expert-reviewed questions, complementary coverage tracks, difficulty analysis, and matched retrieval interventions.

  • Benchmark motivation and design: Existing benchmarks do not unify CFA and FRM levels with item-aligned analysis of failures, repairs, and intervention-induced harms.This gap limits controlled comparisons of whether external knowledge repairs an error or disrupts a correct answer.
  • Benchmark motivation and design: 10,198 expert-reannotated questions span all five CFA and FRM stages under one benchmark protocol.The benchmark releases 5,110 questions, holds out 5,088, and maintains a quarterly leaderboard.
  • Benchmark motivation and design: The 10,198-item Full-Coverage Track is complemented by a 7,625-item Context-Complete Reasoning Track for reasoning from locally supplied evidence.The second track excludes questions whose answer-necessary context is not locally available.
  • Empirical difficulty: 85.29% is the best overall accuracy, while the highest Hard-band accuracy is only 34.68% across 17 models.The difficulty bands are consensus-based and separate empirical difficulty from curriculum stage.
  • Retrieval interventions: Function-RAG and FunctionGraph-RAG rescue hundreds of Direct errors, but intervention-induced harms offset their gains.A public-trained gate invokes FunctionGraph-RAG selectively and raises held-out accuracy from 70.83% to 71.23%.

2 Related Work

Prior financial resources cover specialized models, numerical QA, executable reasoning, and professional examinations, but lack one complete multi-stage CFA–FRM curriculum with controlled intervention analysis.

  • Financial reasoning benchmarks: Existing financial benchmarks span report-grounded QA, financial mathematics, executable reasoning, and selected professional assessments.These resources include FinQA, TAT-QA, FinanceMATH, FinanceReasoning, and CFA-focused evaluations.
  • Benchmark positioning: FINEXAM-10K differs by preserving CFA Levels I–III and FRM Parts I–II under one protocol with expert reannotation, rationales, difficulty bands, public and sequestered splits, and intervention diagnostics.Table 1 identifies these dimensions as benchmark comparison criteria.
  • Professional examinations: Professional examination studies have concentrated mainly on CFA, with varied coverage from all levels to Level III and smaller question sets.The cited studies include curriculum-grounded retrieval and recent reasoning-model evaluations.
  • Structured augmentation and verification: FinanceReasoning is the closest methodological predecessor, showing that passage retrieval can reduce accuracy while filtered function retrieval can perform better in specific reasoning settings.The present work extends this line through controlled matched interventions and selective invocation.
  • Financial agents: Adaptive financial-agent research studies retrieval, revision, and stopping, whereas this work uses fixed-budget closed-world interventions and leaves adaptive agentic control for future work.The stated future direction is motivated by remaining retrieval and verification failures.

3 Dataset

The dataset is a 10,198-item, expert-reannotated CFA–FRM benchmark with public and held-out partitions, empirically derived difficulty bands, and separate coverage and context-complete tracks.

  • Dataset composition: FINEXAM-10K contains 10,198 English multiple-choice questions aligned with CFA Levels I–III and FRM Parts I–II.Each item includes a stem, options, gold answer, rationale, program, and stage; CFA items have three options and FRM items four.
  • Context completeness: Context incompleteness affects 2,573 records because flattened item sets may detach shared vignettes, tables, images, or exhibits from subquestions.The Full-Coverage Track preserves coverage, while supplied-record reasoning claims rely primarily on the context-complete track.
  • Construction and reannotation: A four-member finance-qualified team performed filtering, normalization, deduplication, and two-stage expert reannotation across all retained records.Review covered stems, options, answers, rationales, stages, subjects, and metadata.
  • Partitions: The public partition contains 5,110 Mock or Practice Exam records, while 5,088 records are held out for leaderboard evaluation.Access status is based on source metadata rather than inferred provenance.
  • Empirical difficulty: The fixed panel yields 6,578 Easy, 2,183 Medium, and 1,437 Hard items using leave-one-model-out difficulty bands.The bands are based on frozen model predictions and are stable across alternative weighting and split-half checks.
  • Complementary evaluation tracks: The Full-Coverage Track has 10,198 items, while the Context-Complete Reasoning Track has 7,625 items with answer-necessary local evidence attached.The reasoning track preserves partition membership and inherited difficulty labels rather than recomputing them.

4 Evaluation Framework

The evaluation framework compares frozen models on complementary tracks and tests matched retrieval, graph retrieval, verification, and gated invocation while holding core evaluation conditions fixed.

  • Evaluation tracks: The Full-Coverage Track compares 17 frozen models, while the Context-Complete Reasoning Track is the primary basis for supplied-record reasoning claims.Across interventions, question, options, backbone, decoding, answer extraction, and scoring remain fixed.
  • Retrieval interventions: Direct provides no external function knowledge, whereas Function-RAG retrieves and judges chain-specific financial functions.GPT-4o PoT uses Contriever retrieval; DeepSeek-R1 CoT uses BM25 followed by transferred retrieval and judging.
  • Graph retrieval: FunctionGraph-RAG expands retrieval candidates through chain-specific graph construction and frozen selectors that return at most three functions.The GPT-4o graph uses shared article titles and Contriever nearest-neighbor links with k = 4.
  • Verification: GPT-4o verification runs only when Function-RAG and FunctionGraph-RAG disagree, selecting between frozen branches rather than iteratively repairing outputs.The verifier receives the item, options, generated programs or traces, and execution results.
  • Selective invocation: A one-step gate uses a 27-dimensional feature vector from the item and completed Direct call to decide whether FunctionGraph-RAG should run.The gate excludes gold labels, reference rationales, and retrieval state.
  • Gating and measurement: The gate is trained on 5,110 public items and evaluated on held-out items using a frozen policy applied to precomputed predictions.Its trigger rate is an implied branch-call count, not measured latency, token usage, energy, or monetary cost.
  • Outcome analysis: Rescues convert incorrect Direct answers to correct, while harms convert correct Direct answers to incorrect; matched transitions are tested with exact McNemar tests.Static interventions are reported on both tracks, with context-complete results used for supplied-record conclusions.

5 Results and Analysis

Results show substantial variation in financial reasoning difficulty: leading models perform strongly overall but collapse on Hard items, with context loss contributing without fully explaining failures. Retrieval interventions are useful only selectively, while a low-cost gate yields a small held-out improvement by invoking FunctionGraph-RAG sparingly.

  • Difficulty and subject heterogeneity: 54.57% is the best score on the 372-item context-complete Hard subset, indicating that context loss amplifies but does not explain the Hard-band collapse.Only three models exceed the 30.40% item-weighted chance baseline.
  • Shared failures: 47 context-complete items are missed by all 17 models, while erroneous votes concentrate on one distractor far above the item-specific null.On 369 context-complete Hard items, the median concentration is 0.923 versus 0.605 under the exact null; all erroneous votes select one distractor on 148 items.
  • Retrieval interventions: Static Function-RAG and FunctionGraph-RAG produce little or negative net gain because rescues are offset by intervention-induced harms.Under DeepSeek-R1 CoT, Function-RAG yields 505 rescues and 538 harms, while FunctionGraph-RAG yields 509 rescues and 500 harms.
  • Retrieval interventions: 2.92 points is the gain from graph expansion when the relevance judge retains exactly one function, whereas forcing three functions when none are relevant reduces accuracy by 1.10 points.The effects remain significant after Benjamini–Hochberg correction, but larger retained-function strata show no reliable difference.
  • Selective invocation: 71.23% is the gated held-out accuracy, up from 70.83% when FunctionGraph-RAG is invoked for 7.9% of 5,088 questions.The gate produces 55 rescues and 35 harms, for a +0.39-point improvement with p = .0446; on the context-complete subset, it gains 0.55 points.

6 Conclusions

FINEXAM-10K combines an expert-reviewed, sequestered benchmark with matched retrieval diagnostics. The results support selective rather than uniform use of retrieval and graph tools.

  • 7,625-item Context-Complete Reasoning Track confirms that the principal rankings and intervention conclusions persist when answer-necessary evidence is locally attached.
  • Static Function-RAG and chain-specific FunctionGraph-RAG repair many errors but introduce comparable harm.
  • 7.9% of items receive the graph branch under the public-trained Direct-conditioned gate, which recovers a modest held-out gain.
  • Selective access to independent evidence and tools is favored over uniformly adding retrieval or another pass over the same reasoning.

Limitations and Future Work

The benchmark distinguishes curriculum coverage from reasoning with locally supplied evidence, but detached parent context limits Full-Coverage interpretation. Future work targets context preservation, answer-position effects, and broader intervention evaluation.

  • 2,573 standalone records lack parent-level evidence such as shared vignettes, tables, images, or exhibits.Full-Coverage results may therefore combine reasoning with prior exposure, reconstruction from partial cues, or chance.
  • Claims about reasoning from supplied evidence rely primarily on the Context-Complete Reasoning Track.
  • Difficulty bands come from a frozen model panel rather than human psychometric calibration, and slice-specific answer-position effects are observed.Future versions will add option-permutation tests and, where available, human response evidence.
  • FunctionGraph-RAG comparisons are restricted to within-chain matched settings, while gate efficiency is measured by branch-call count rather than latency, tokens, energy, or monetary cost.Verification, abstention, and planning probes are single-backbone, fixed-budget diagnostics rather than a learned multi-step agent.

Ethical Statement

FINEXAM-10K uses reviewed, rationale-bearing preparatory materials with public and sequestered partitions. The paper limits its intended use to research and cautions against interpreting benchmark performance as evidence of autonomous financial decision safety.

  • The benchmark uses CFA-aligned and FRM-aligned preparatory materials, excluding official CFA Institute and GARP examination content.
  • 5,110 mock and practice items are publicly released, while 5,088 items remain sequestered for evaluation.
  • CFA and FRM identify curriculum alignment descriptively, and the work is not affiliated with or endorsed by CFA Institute or GARP.
  • All retained items underwent two-stage review by a four-member finance-qualified team, with external reviewers compensated at or above applicable statutory minimum wage.
  • Benchmark performance should not be interpreted as evidence that a model is safe for autonomous investment, risk-management, compliance, or advisory decisions.
  • 10,198 final questions resulted after rationale filtering, normalization, deduplication, and exclusion of 147 CFA Level III constructed-response items.
  • The final collection retains 7,631 CFA and 2,567 FRM questions, partly reflecting differential rationale availability and screening retention.

B.4.1 Full Hard Context Recheck for Sensitivity Analysis

The full Hard-band recheck establishes context-completeness sensitivity analyses without changing frozen difficulty labels. The resulting Context-Complete Hard subset is distinct from the universal-failure diagnostic subset.

  • Recheck protocol: The lead curator manually rechecked every frozen Hard item after labeler calibration without redefining or overwriting the empirical difficulty bands.
  • Track construction: 7,625 of 10,198 items form the Context-Complete Reasoning Track, with context-complete rates of 66.7% for Public and 82.9% for Held-out.
  • Track construction: 25.9% of Hard items are context-complete, yielding 5,929 Easy, 1,324 Medium, and 372 Hard questions in the reasoning track.
  • Diagnostic subsets: The 372 Context-Complete Hard items and 188 universal-failure items are distinct subsets serving different analytical purposes.
  • Sensitivity results: All models score higher after the context filter, while Full-Coverage and Context-Complete rankings remain nearly identical at Spearman’s ρ = 0.988.
  • Diagnostic subsets: The universal-failure core contains parseable but incorrect answers from every model, rather than parsing or truncation failures.
  • Difficulty definition: Difficulty bands are defined for the frozen model panel, not as expert-assigned curriculum levels.

C.3 Robustness of the Difficulty Partition

The empirical difficulty partition is stable across panel-weighting and model-removal checks, while remaining distinct from examination stage. Diagnostic analyses also separate context coverage, item format, and shared-error concentration as different sources of difficulty.

  • Partition robustness: 95.8% of items receive the same band under Rule C and Rule A, with continuous-score Spearman correlation ρ = 0.998.Rule C and Rule B agree on 90.8% of band assignments, with ρ = 0.991.
  • Partition robustness: 0.902 is the Spearman–Brown corrected split-half reliability, while the worst leave-one-model-out correlation is 0.983.The mean split-half Spearman correlation is 0.821, with range 0.737 to 0.849.
  • Stage distinction: Cramér’s V = 0.174 between empirical bands and five examination stages, indicating that difficulty does not reproduce curriculum progression.The association with examination family alone is smaller, with V = 0.056.
  • Shared-error diagnostics: 188 items form the universal-failure core, and 147 are CFA questions, motivating item-specific rather than single-binomial chance calculations.The paired distractor-concentration analysis compares observed modal shares with null expectations of 0.5982 for CFA and 0.4539 for FRM items.
  • Diagnostic slices: 17–22% is the panel-averaged accuracy range across audited item-format groups in the Context-Complete Hard subset.The remaining difficulty is not explained by one dominant format alone.

D.5 Universal-Failure Core

The universal-failure core isolates items missed by all 17 models and shows concentrated, professionally plausible distractor choices. Its answer-position pattern differs from the Context-Complete Hard subset, so positional effects are slice-specific.

  • Distractor concentration: 0.806 is the median modal distractor share across the universal-failure core, versus a median item-specific uniform null of 0.5982.Errors therefore frequently concentrate on a small number of professionally plausible distractors.
  • Answer-position diagnostics: 44.7% of universal-failure items have gold label A, while models select B most often at 39.0% of predictions.The passage associates this mismatch with possible answer-position preference rather than financial content alone.
  • Answer-position diagnostics: Context-Complete Hard items under-select A and over-select C, unlike the universal-failure core’s A-heavy gold labels and B-heavy predictions.Answer-position effects are therefore treated as slice-specific confounds rather than one fixed benchmark-wide bias.
  • Retrieval diagnostics: FunctionGraph-RAG’s exploratory top-10 no-judge configuration reaches 79.22% on DeepSeek-R1 CoT, 0.53 points below Direct.This configuration is distinct from the reported chain-specific FunctionGraph-RAG results.
  • Retrieval diagnostics: The GPT-4o PoT informed verifier reaches 68.92% on Full Coverage, 0.45 points below Direct, and remains a bounded post-generative verifier.It does not initiate a new retrieval trajectory or unrestricted external search.

F.2 Relevance-Judge and Selector Diagnostics

FunctionGraph-RAG outcomes depend on how many functions the relevance judge retains, with one function the clearest favorable regime and zero functions harmful. Selective routing reduces unnecessary intervention, but exploratory subgroup effects and oracle gains do not establish broad deployable improvements.

  • Relevance-judge strata: Retaining zero functions is harmful, while retaining one function is the clearest favorable FunctionGraph-RAG regime.Larger retained-function strata are smaller and reverse direction on the Context-Complete Reasoning Track, so gains are not monotonic.
  • Subgroup diagnostics: Credit Risk shows +7.73 percentage points over Function-RAG, but BH q = .142; exploratory subject effects are therefore hypothesis generating rather than confirmatory.Economics and Quantitative Methods effects also fail multiplicity correction.
  • Verification: +0.86 points are recovered on Full Coverage when the verifier removes 130 harms at the cost of 43 rescues.On the Context-Complete Reasoning Track, it removes 117 harms at the cost of 19 rescues and recovers 1.29 points relative to FunctionGraph-RAG.
  • Oracle comparison: +7.25 points and +7.46 points are the post hoc oracle ceilings over Direct for DeepSeek-R1 CoT and GPT-4o PoT on Full Coverage.The oracle uses gold correctness and is separate from the deployable gate.
  • Selective routing: The adopted gate runs Direct first and uses 27 inference-visible item and Direct-stage features to decide whether to invoke FunctionGraph-RAG lazily.It excludes gold answers, rationales, retrieval outputs, and other branch information.
  • Selective routing: The gate’s branch-preference target is positive only when FunctionGraph-RAG uniquely repairs a Direct error; all other outcomes favor Direct.Correctness ties are downweighted, and a zero target does not imply that Direct is correct.

G.3 Strict Held-Out Gate Results

The held-out analysis evaluates a frozen gate and several bounded interventions, while distinguishing deployable selection from post hoc oracle ceilings. Retrieval and deliberation offer limited or unstable gains, and same-backbone methods remain constrained by shared misconceptions.

  • Strict Held-Out Gate Results: The evaluation applies a frozen gate to precomputed Direct and FunctionGraph-RAG predictions.The gate uses item and Direct features, while held-out decisions depend on frozen estimator, threshold, and inference-visible features.
  • Strict Held-Out Gate Results: The gate would add one FunctionGraph-RAG invocation for each triggered item under lazy deployment, but the released evaluation did not execute lazily.The implied invocation count is not a measurement of tokens, latency, money, or energy.
  • Oracle Selection Ceiling: The oracle ceiling uses gold correctness after all branch outputs are frozen and therefore cannot serve as a deployable router.It is a post hoc union upper bound and does not affect router fitting, threshold selection, or held-out prediction.
  • Post-Execution Verification: The verifier flags 50 of 369 Context-Complete Hard items and changes 47 answers, yielding 11 rescues and 10 harms for a +0.27-point net change.The exact McNemar test gives p = 1.000, with a bootstrap 95% confidence interval of [−2.17, +2.71] points.
  • Answerability Detection and Selective Abstention: The answerability detector raises selective accuracy from 57.55% at full coverage to 85.92% at 23.7% coverage, but its 53.3% false-positive rate makes it a risk signal rather than a calibrated deployment policy.The audit sample is approximately balanced between locally answerable and context-dependent records and is not representative of the benchmark’s natural 74.8/25.2 prevalence.
  • Joint Interpretation: The two deliberation probes do not improve paired accuracy, and reusing the same backbone over the same evidence is insufficient when errors reflect stable misconceptions.The plan-then-solve sample reaches 46.67% versus Direct’s constructed 50.00%, a −3.33-point change with p = .3533.
Loading 2608.28155v1…