FinGREAT: Evaluating LLMs on Financial Graph-guided Retrieval And Typing

1 Introduction

• Financial analysis requires more than plausible summaries: analysts must locate relevant evidence and interpret the accounting relationships it describes.High-volume disclosure review demands both scale and precision.Source: §1 · p. 1
• Local open-weight and hosted proprietary models represent different deployment settings, and the study asks whether locally deployable models preserve analyst-needed relation-level accuracy.The passage identifies control over confidential inputs, model adaptation, and inference resources as features of local deployment.Source: §1 · p. 1
• The study separates schema-guided discovery and assignment of accounting evidence from classification of supplied passages, addressing three questions about retrieval, relation mapping, and model accuracy.The fixed graph contains 11 accounting relationships grouped into Cash Flow, Net Income, and Equity edges.Source: §1 · p. 2
• Schema-guided retrieval targets accounting relationships across documents, unlike open information extraction without predefined relations or retrieval driven by an individual financial question.Item 7 MD&A provides narrative explanations of financial results, developments, trends, uncertainties, and outlook.Source: §1 · pp. 1–2
• The retrieval study covers five open-weight extractors across 1,300 SEC Item 7 filings from 2000–2025, with three proprietary references on a shared 300-filing subset.A separate benchmark evaluates 34 open-weight classifiers below 36B parameters and ten proprietary models on 483 passages reviewed by three accounting experts.Source: §1 · p. 2
• The paper examines annual evidence allocation, cross-model retrieval and classification agreement, tag–edge organization, vocabulary overlap, and category-specific accuracy.It also compares externally reported API pricing and first-token latency.Source: §1 · p. 2

2 Related Works

• Existing finance work includes numerical reasoning benchmarks, broad task evaluations, and models advancing accessible financial language-model development.FinQA and TAT-QA address numerical reasoning over financial text and tables; FinBen spans extraction, analysis, reasoning, and decision-making.Source: §2 · p. 2
• Schema guidance is established prior art across extraction and accounting tasks; FinGREAT instead studies passage retrieval and classification under a compact graph of accounting flows.The cited examples include entity-pair relations, event-enhanced knowledge graphs, table extraction, canonical mapping, numerical concept linking, and taxonomy-aligned consistency.Source: §2 · p. 2
• Other financial retrieval and extraction assessments cover answer retrieval, embeddings, retrieval-generation separation, multimodal evidence, long reports, and knowledge-graph extraction.FinGREAT’s distinction is the paired examination of evidence discovery and interpretation quality.Source: §2 · pp. 2–3
• FinGREAT complements prior work by comparing graph-conditioned retrieval behavior with expert-reviewed passage classification under the same accounting schema.Cross-model evidence agreement characterizes retrieval behavior, while human labels assess relation correctness.Source: §2 · p. 3

3.1 Underlying Accounting Graph

• The fixed 11-node, 11-edge accounting graph constrains allowable relationships and represents transactions affecting the accounting identity at the level of aggregated concepts.Its directed edges represent bookkeeping flows; the authors state that discussions of transactions affecting the identity must relate to one or more schema edges.Source: §3.1 · p. 3, Figure 2 · p. 3

3.2 Task I: Graph-Conditioned Evidence Extraction

• Graph-conditioned extraction identifies passages throughout an Item 7 filing that correspond to predefined accounting relationships, rather than retrieving evidence for a specific query.Each extraction includes a passage, an assigned graph edge, and model-generated descriptive tags; multiple passages may map to one edge, while some edges may have no evidence.Source: §3.2 · pp. 3–4

3.3 Task II: Financial Relation Classification

• Task II isolates relation understanding by asking models to classify a supplied financial passage using the same graph schema, independently of document-level evidence discovery.Human labels may accept multiple edges, but a prediction is correct if it selects any accepted edge.Source: §3.3 · p. 4

3.4 Evaluation Objectives and Metrics

• Evaluation spans extraction behavior, cross-model agreement, semantic consistency, and human-grounded classification, with the first three assessed on Task I and the last on Task II.The evaluation also reports extraction volume and edge proportions, and examines annual extraction trends.Source: §3.4 · p. 4
• Cross-model agreement measures whether open-weight models retrieve the same passages as proprietary references and whether they assign matching relationships to recalled passages.Retrieval is assessed with recall and Jaccard similarity; conditional classification precision checks edge agreement on recalled passages.Source: §3.4 · p. 4
• Tag–edge purity, entropy, and homogeneity assess consistency between model-generated vocabulary and graph relationships, not factual correctness.These metrics examine how consistently models translate the supplied schema into their internally learned vocabulary.Source: §3.4 · p. 5
• Human-grounded classification compares overall and per-edge model accuracy with expert finance professionals, with category-level performance reported to expose differences in relation understanding.Per-edge accuracy is measured on retrieved passages whose human-labeled edge is the edge being evaluated.Source: §3.4 · p. 5

4 Experimental Methodology

• The methodology evaluates two tasks: retrieving passages and assigning accounting edges, then classifying supplied passages against human-accepted labels.Source: §4 · p. 5

4.1 Financial Corpus and Dataset

• The corpus contains 1,300 Item 7 filings from 2000–2025, with proprietary-model comparisons restricted to the same 300 filings across six selected years.Source: §4.1 · p. 5

4.2 Graph-Conditioned Extraction Study

• Extractors receive Item 7 text and graph definitions, return quoted supporting passages with assigned edges, and process text in batches of about 4,096 tokens.Source: §4.2 · p. 5

4.3 Human-Annotated Classification Benchmark

• Three accounting reviewers labeled passages retrieved by all models; after exclusions, 483 passages from 259 firm-year filings formed the classification benchmark.The candidate pool began with 500 passages from 300 common filings.Source: §4.3 · pp. 5–6

4.4 Model Coverage and Evaluation Setup

• Task II compares 34 open-weight classifiers under 36B parameters with ten proprietary models, using common task formulations and constrained reasoning settings.Open-weight models were evaluated in BF16 on a single 80 GB NVIDIA H100 GPU.Source: §4.4 · pp. 6–7

5.1 Evidence Allocation and Cross-Model Agreement

• Ministral produces the most extractions and highest reference-relative recall, but trails Gemma and Qwen on Jaccard against Gemini and GPT; Gemma leads conditional classification precision consistently.Extraction profiles also differ: Phi and Nemotron emphasize Equity, Gemma and Qwen Net Income, and Ministral Cash Flow.Source: §5.1 · p. 7

5.2 Tag–Edge Consistency

• Qwen leads open-weight models in tag–edge purity and has the lowest conditional entropy, while Ministral and Phi produce more one-off tags.Gemma is close to Qwen in homogeneity; lexical variety does not itself ensure consistent alignment with the graph schema.Source: §5.2 · pp. 7–8

5.3 Human-Grounded Classification

• Figure 4 relates overall accuracy to first-token latency and input price, providing deployment context from externally reported API data.Horizontal positions use log10 latency and log10(1 + cost/0.01).Source: Figure 4 · p. 8, §5.3 · p. 8
• 84.1% accuracy for the strongest open-weight models is 8.7 points below proprietary leaders, with the largest category gap on Cash Flow edges at 14.3 points.The best-model gap is 2.8 points for Net Income and 6.4 points for Equity; Qwen and Gemma lead the open-weight group overall and across categories under the evaluated settings.Source: §5.3 · p. 8, Figure 5 · p. 8
• Equal accuracy can conceal different errors: two Qwen models each score 84.1% but share only 60 of 77 errors, leaving 17 unique errors apiece.Qwen3.6-27B and Gemma4-12B share 58 errors, with a failure-set Jaccard of 58.0%.Source: §5.3 · p. 8

• Open-weight models show competitive reference-relative retrieval, while both groups retain room for semantic consistency and schema alignment.Source: §6 · p. 8

• The evaluation covers narrative evidence in annual US SEC Form 10-K filings under a fixed accounting graph, not numerical reasoning, table interpretation, or general financial decision-making.Task I uses proprietary reference-model extractions rather than exhaustive human-labeled retrieval coverage; Task II annotates passages selected through shared retrieval.Source: §7 · p. 9

8 Ethical Considerations

• Automated extraction and classification may misidentify financial relationships, so the outputs are intended for research and evaluation rather than automated financial decisions.The study uses public SEC Item 7 disclosures and expert-annotated data, without external crowdsourcing.Source: §8 · p. 9

A Accounting Schema and Filing Examples

• The fixed accounting graph has 11 edges, and the supplied Apple filing illustrates that Item 7 contains MD&A while Item 8 contains statements and notes.The Apple screenshots are illustrative and do not establish that the filing entered the benchmark.Source: A Accounting Schema and Filing Examples · p. 11, §A.1 · p. 11

B.1 Coverage and Industry Composition

• The corpus contains 1,300 filings from 2000–2025, but four industry divisions account for 76.0% of filings, so it is not industry-balanced.Current SEC industry classifications were used; 15 registrants remain unknown or unclassified.Source: §B.1 · p. 11

B.2 Reported Total Assets, 2012–2025

• Asset-size analysis covers 700 selected filings from 2012–2025; values are available for 647 (92.4%), with 53 unknown.Assets are nominal USD and measure balance-sheet scale, not market capitalization; the window restriction applies only to size characterization, not the extraction corpus.Source: §B.2 · pp. 11–12, Table 6 · p. 11, Table 8 · p. 12

B.3 Item 7 Document Length

• Across all 1,300 documents, recorded processed-input lengths had a median of 57,395.5 characters (interquartile range 36,616.5–78,265) and ranged from 2,543 to 501,163 characters.These are text-input character counts, not HTML byte sizes, page counts, or tokenizer-dependent token counts.Source: §B.3 · p. 12

C Task I: Retrieval Protocol

• Task I divides documents into approximately 4,096-token chunks at natural text boundaries and requests quoted evidence, graph endpoints, descriptive tags, and supported attributes.Outputs are validated for structured extractions with admissible graph endpoints.Source: C Task I: Retrieval Protocol · p. 12

C.1 Saved Extraction Example

• The saved Qwen extraction quotes a financial-period relationship and supplies graph endpoints, an extraction class, and supporting attributes.Its financial period is reported as generated rather than inferred from the filing year.Source: §C.1 · p. 12

D.1 Annotation Interface

• Reviewers select all supported relationships or None while inspecting highlighted evidence in filing context; screenshots show the interface, not adjudicated answers.Source: §D.1 · p. 12

D.2 Inter-Reviewer Agreement

• Krippendorff’s α is 0.764 across 500 candidate passages, using MASI distance to account for overlap among reviewers’ edge sets.The agreement calculation uses the reviewers’ latest non-test submissions before answer-set construction and exclusions.Source: §D.2 · pp. 12–13, Table 9 · p. 13
• Pairwise Jaccard similarity is 80.6%, and all three reviewers select identical edge sets on 331/500 passages (66.2%).These statistics describe agreement, not annotation correctness.Source: §D.2 · p. 13

D.3 Accepted Answer Sets and Passage Context

• An edge enters the primary answer set when at least two reviewers select it; if that set is empty, the reviewers’ union is used, while majority-None passages are excluded.Source: §D.3 · p. 13
• The final benchmark contains 483 passages from 259 firm-year filings, including 85 passages with multiple accepted edges.Any accepted edge earns classification credit; models receive the passage and source context within an 8,192-token budget.Source: §D.3 · p. 13, Table 10 · p. 13

E Task II: Classification Protocol

• Models receive an evidence passage and source context under an 8,192-token budget, with context drawn from the enclosing section, paragraph, or a bounded local window as it fits.The budget does not imply a fixed input length.Source: E Task II: Classification Protocol · p. 13
• The classifier must return exactly one of the 11 listed undirected financial relationships.The prompt lists the allowed account-to-account relationships and requires the complete relationship name.Source: E Task II: Classification Protocol · p. 13
• All 44 classifiers predict on the same 483 eligible passages, with open-weight and proprietary models using different specified inference settings.Open-weight classifiers used temperature 0, top-p 0.95, a 128-token cap, and reasoning off; proprietary API controls varied by provider and model.Source: E Task II: Classification Protocol · p. 14, Table 11 · p. 14

F Metric Definitions and Statistical Procedures

• Retrieval recall, evidence Jaccard, and conditional classification precision compare model outputs with proprietary references, while human classification scores predictions against accepted answer sets.Family winners and medal highlights summarize observed scores, not statistical significance.Source: F Metric Definitions and Statistical… · p. 14, Table 12 · p. 14
• The Artificial Analysis snapshot reports external hosted-service measurements, not controlled Task II runtimes or local GPU inference costs.Latency uses 1K-token prompts, and provider reasoning configurations may differ from the study’s.Source: F Metric Definitions and Statistical… · p. 14

F.1 Evidence Matching and Vocabulary Normalization

• Evidence matching uses normalized quote overlap and one-to-one assignment at a score threshold of 0.80; vocabulary Jaccard compares mechanically normalized distinct tag sets.The normalization does not use embedding similarity, general stemming, or semantic-bucket merging; tag–edge measures retain supplied tag strings and weight each extraction’s tags to total one.Source: §F.1 · p. 14

F.2 Baselines and Confidence Intervals

• Accuracy is compared with uniform-random and majority-selector baselines, and 95% intervals use a source-filing cluster bootstrap over 2,000 replicates.Paired comparisons resample the same filings; intervals describe sampling variability conditional on benchmark labels.Source: §F.2 · p. 15

G.1 Edge Supports and Baselines

• Edge cohorts may overlap, and edge-level scores test predictions against the complete accepted set rather than only the edge defining a cohort.The accompanying table gives exact baseline accuracies.Source: §G.1 · p. 15, Table 14 · p. 16

G.2 Category-Level Retrieval Agreement

• Category-level retrieval recall and conditional classification precision are reported relative to reference models, with matches partitioned by reference-edge category.The filing-level matches are computed once before category partitioning.Source: §G.2 · p. 16, Table 15 · p. 16

G.3 Majority Voting on Shared Evidence

• Majority voting requires at least three of five open-weight extractors to agree on shared evidence; split votes abstain, and edge agreement is compared on majority-decidable passages.The comparison uses 364, 543, and 448 passages for the three proprietary references.Source: §G.3 · p. 16, Table 16 · p. 16

G.4 Vocabulary Reuse and Cross-Task Stability

• The analysis compares tag recurrence and cross-task edge-label changes, with accuracy measured on the same 483 passages.Tag counts distinguish reusable vocabulary from one-off labels; the cross-task comparison uses corresponding extractors and classifiers.Source: Table 18 · p. 16, Table 17 · p. 16, §G.4 · p. 16

G.5 Annual Edge-Level Allocation

• Annual edge shares describe retrieval emphasis, not classification accuracy.Each panel reports the share of a model’s annual extractions assigned to one edge.Source: §G.5 · pp. 16–17, Figure 9 · p. 22

G.6 Cross-Model Vocabulary Overlap

• Vocabulary Jaccard shows Qwen–Gemini and Gemma–Gemini as the strongest open/proprietary pairs, but lexical overlap alone cannot establish shared training data.The Gemma–Gemini pair also shares Google’s model lineage.Source: Figure 10 · p. 17

G.7 Overlap of Classification Failures

• 63.8% of errors overlap between the tied Qwen leaders, 58.0% between Qwen3.6-27B and Gemma4-12B, and 62.8% between the equally accurate proprietary leaders; only two passages stump all 44 models.Failure overlap is measured by Jaccard similarity on the 483-example cohort.Source: Figure 11 · p. 17, §G.7 · p. 17

H Worked Annotation and Model-Response Examples

• Worked cases contrast overlapping reviewer selections and divergent model predictions with unanimous reviewer agreement, but illustrate patterns rather than their prevalence.The accepted answer sets are E1 and E11 for Converted Organics, E5 and E10 for Micromuse, and E5 for Riviera Holdings.Source: H Worked Annotation and Model-Response… · pp. 26–27