PaperlayerView exact revision on arXiv

Source-linked AI summary

Recognition Is Not Grounding: Benchmarking Multilingual Financial Evidence Extraction

Lingfei Qian, Yan Wang, Xueqing Peng, Polydoros Giannouris, Nadhem Zmandar, Yi Han, Yanguang Liu, Víctor Gutiérrez-Basulto

arXiv:2610.19432v1cs.AIcs.CE

TL;DR

Existing benchmarks provide limited evidence on whether models can both recognize shared accounting concepts across languages and recover all corresponding numerical facts from the same contexts. MULTIFINGROUND pairs these tasks on FY2025 reports in English, French, and Greek; eight reference models show high recognition accuracy alongside frequent omissions in correctly recognized groups.

  • Problem

    Existing evaluations provide limited evidence on whether models recognize shared accounting concepts across languages and recover all corresponding numerical facts from the same evidence.

    Source: §1 · p. 1

  • Method

    MULTIFINGROUND pairs concept recognition with exhaustive fact selection on matched contexts, alongside financial-fact identification and concept discrimination without candidate labels.

    Source: §1 · pp. 2–3

  • Results

    97.3–99.4% T2a accuracy coexists with target omissions in 26.6–41.9% of correctly recognized groups across eight reference models.

    Source: §1 · p. 2, §5 · p. 6

  • Takeaways & Limitations

    Multilingual financial extraction should consider both whether models recognize concepts and whether they find all corresponding evidence.

    Source: §6 · p. 8

    The benchmark covers three languages within one reporting regime, and language differences cannot be isolated from issuer, sector, and document-style differences.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Comparing financial disclosures across languages requires identifying shared accounting concepts and recovering all numerical facts that express them. Existing benchmarks evaluate aspects of these capabilities, but provide limited insight into whether both succeed on the same multilingual evidence. In particular, it remains unclear whether language models can recognize these concepts across languages and exhaustively recover their corresponding numerical evidence from the same contexts. We introduce MULTIFINGROUND, a benchmark designed to investigate this recognition-grounding gap in multilingual financial reports. Constructed from FY2025 disclosures of 90 issuers in English, French, and Greek, MULTIFINGROUND aligns independently authored reports through a shared accounting taxonomy and provides traceable fact-level annotations. The benchmark evaluates financial fact identification, concept recognition, exhaustive fact selection, and concept discrimination without candidate labels. Across 15 evaluated models, eight API and open-weight reference models achieve 97.3-99.4\% concept recognition accuracy, yet miss at least one target fact in 26.6-41.9\% of correctly recognized groups. Further analyses reveal substantial variation in concept discrimination and limited gains from the evaluated finance-specialized models. Our findings demonstrate that strong concept recognition can coexist with incomplete evidence recovery, highlighting the importance of evaluating semantic recognition and evidence completeness together.

1 Introduction

Comparing financial reports across languages requires recognizing shared accounting concepts and recovering all numerical evidence that expresses them. MULTIFINGROUND evaluates these capabilities together and finds that high concept-recognition accuracy can coexist with incomplete fact recovery.

Motivation and gap

  • Investors and analysts comparing companies across countries need comparable financial information from reports published in different languages.The paper illustrates the recognition-grounding gap with a French revenue example in which omitting the prior-year value prevents calculation of annual growth.Source: §1 · p. 1
  • Existing benchmarks cover financial concept linking, extraction, reasoning, and multilingual tasks, but provide limited insight into recognition and complete fact recovery on the same evidence.Different terms, table structures, and reporting periods can express the same concept, so retrieving one correct value may still omit a needed comparison.Source: §1 · p. 1

Benchmark contribution

  • MULTIFINGROUND links FY2025 reports from 90 issuers in English, French, and Greek through shared IFRS concepts and traceable fact annotations.The excerpts come from actual reports by different issuers, rather than translations of the same passages.Source: §1 · pp. 1–2
  • The benchmark pairs concept recognition with exhaustive fact selection on matched evidence, alongside financial-fact identification and concept discrimination without candidate labels.This design tests whether a model recognizes a concept yet misses corresponding evidence in the same contexts.Source: §1 · pp. 2–3

Key findings

  • 97.3–99.4% T2a accuracy coexists with target omissions in 26.6–41.9% of correctly recognized groups across eight reference models.Greek recall is lowest for seven of eight reference models, and evaluated finance-specialized models show limited gains under the study’s protocol.Source: §1 · p. 2

2 Related Work

Prior financial benchmarks cover concept linking, extraction, reasoning, and multilingual tasks, but do not directly test concept recognition and complete numerical evidence recovery together on matched multilingual evidence.

Existing benchmarks

  • Financial benchmarks evaluate number-to-concept linking and extraction, while FinQA and TAT-QA test reasoning over financial evidence.FinTagging examines errors across extraction stages, and FinQA also reports retrieval recall.Source: §2 · p. 3

Remaining gap

  • MultiFin and MultiFinBen evaluate financial tasks across languages, but these evaluations do not directly show whether recognized concepts lead to recovery of all corresponding numerical facts.The gap concerns recovery from the same multilingual evidence, not merely the presence of both capabilities in a benchmark suite.Source: §2 · p. 3
  • The missing comparison is whether models that recognize a shared accounting concept can also recover every corresponding numerical fact from matched evidence.The paper distinguishes paired evaluation from simply including recognition and coverage tasks in one suite.Source: §2 · p. 3

3 The MULTIFINGROUND Benchmark

MULTIFINGROUND builds linked financial-evidence tasks from FY2025 reports by 90 issuers across English, French, and Greek. Its eight benchmark subsections describe collection, evidence processing, four tasks, diagnostic conditions, and expert validation.

3.1 Data Collection

  • 30 issuers per language provided full-year reports ending 31 December 2025, with initial fact-to-concept mappings from issuer iXBRL tags.Each issuer appears in only one language.Source: §3.1 · p. 3

3.2 From iXBRL Reports to Evidence

  • Reports are rendered as text and Markdown tables retaining headers, fact offsets, concepts, and accounting attributes; standard ifrs-full concepts serve T2 tasks, while issuer extensions appear only in T1.The initial annotations come from issuer tags and the taxonomy, with expert corrections or exclusions applied to a sampled subset.Source: §3.2 · p. 3

3.3 T1: Fact Identification with Accounting Attributes

  • T1 identifies financial values and predicts their type, period type, and monetary balance; untagged amount-like numbers are unscored.The output distinguishes monetary amounts, per-share amounts, and shares, as well as instant versus duration periods.Source: §3.3 · p. 3

3.4 T2a: Concept Recognition

  • T2a combines English, French, and Greek contexts for shared concepts, marks target facts, and asks models to choose among four concept options.Of 190 concepts tagged by at least two issuers per language, 187 had three qualifying distractors.Source: §3.4 · p. 4

3.5 T2b: Exhaustive Fact Selection

  • T2b gives the target concept and asks models to select all its numbered facts, including comparative-period values, while rejecting other facts.Groups average 40.6 numbered facts and 7.8 targets; T2b uses separate calls from T2a.Source: §3.5 · p. 4

3.6 T2c: Concept Discrimination

  • Across 534 items over 267 concepts, T2c asks models to select the odd excerpt without concept labels; language rotation includes 275 items with another excerpt in the odd excerpt’s language.The odd concept shares the target’s datatype and period type, making it confusable.Source: §3.6 · pp. 4–5

3.7 Taxonomy Conditions and Diagnostic Variants

  • T2a label variants change which language labels appear, while additional diagnostics remove English evidence or use harder option sets.The label variants retain the same items, options, and option order; harder sets include calculation siblings and distractors ranked by word overlap.Source: §3.7 · p. 5

3.8 Expert Validation

  • κ = 0.87 (T1), 0.74 (T2a), 0.84 (T2b), and 0.88 (T2c); corrected annotations change reference-model scores by at most 1.4 percentage points on audited subsets.Two experts independently reviewed a stratified 10% sample per task, and identified ambiguities among T2a distractors.Source: §3.8 · p. 5

4 Experimental Setup

The experiments evaluate 15 models under shared task and scoring protocols, while leaving model reasoning and generation settings unequalized. The section defines task-specific scores and strict handling of invalid outputs.

4 Experimental Setup

  • The study evaluates 15 API, open-weight, and finance-specialized models, with base models included for the specialists.Calls use English instructions and original-language evidence, default decoding settings, and no system prompt, tools, retrieval, or history.Source: §4 · p. 5

4.1 Evaluation Metrics

  • T1 uses seven-class macro-F1 and joint fact F1, T2a and T2c use accuracy, and T2b reports concept-macro F1, complete target coverage, and exact-set accuracy.T2b recall measures the share of target facts recovered; exact-set accuracy requires neither omissions nor extra selections.Source: §4.1 · pp. 5–6
  • Invalid outputs are scored as wrong and reported alongside scores; T1 intervals resample issuers, while T2 comparisons use exploratory item-level McNemar tests.T2 intervals are not reported because the graph linking groups to issuers forms a single connected component; item-level tests do not account for shared-issuer dependence.Source: §4.1 · p. 6

5 Results

Across 15 models, high concept-recognition scores did not ensure exhaustive fact recovery. Results also varied across evidence and label conditions, and between recognition and discrimination tasks.

5 Results

  • 97.3–99.4% T2a accuracy coexists with best T2b scores of 90.4 concept-macro F1 and 71.8% exact-set accuracy, while T2c accuracy spans 35.6–95.1%.Source: §5 · p. 6

5.1 Does Recognition Imply Coverage?

  • 26.6–41.9% of correctly recognized groups still miss a target in at least one language; Greek has the lowest recall for seven of eight reference models.Figure 2 distinguishes target coverage from exact-set correctness: selecting all targets permits extra selections.Source: §5.1 · p. 6, Figure 2 · p. 6

5.2 How Does Evidence Language Affect Grounding?

  • −0.8 to +0.4 points separates joint from no-English T2a inputs for reference models, while seven of eight score 1.9–3.1 points lower on English-only inputs.The comparisons do not isolate language competence because issuers and contexts differ.Source: §5.2 · p. 7
  • T2c accuracy varies beyond recognition scores: DeepSeek reaches 98.3% on T2a but 35.6% on T2c, and every model clearly above chance does worse when another excerpt shares the odd excerpt’s language.The tasks also differ in evidence and structure, so the score gap does not isolate candidate-label effects.Source: §5.2 · p. 7, Figure 4 · p. 8

5.3 How Does Taxonomy Label Presentation Affect Recognition?

  • On Greek-only items, C2 significantly lowers accuracy for eight models by 1.7–5.0 points; C3 changes reference-model scores by at most 1.0 point, with no significant changes.C2 compares localized labels without identifiers with C1’s English labels and identifiers; it does not isolate label language.Source: §5.3 · p. 7, Figure 3 · p. 8

5.5 Does Financial Specialization Help?

  • All four finance-specialized models underperform their bases on T1, T2a, and T2c, despite Fin-R1’s 9.9-point T2b macro-F1 gain.Source: §5.5 · p. 8

5.6 Robustness to Harder Options and Output Failures

  • 70.2% lexical-overlap baseline T2a accuracy falls to 49.1% with four distractors and 36.3% with twenty, while reference models retain 94.4–98.7% and 89.4–97.5%.The passage cautions that other wording cues may remain.Source: §5.6 · p. 8
  • 23–34% of Qwen3.8’s T1/T2b/T2c items exhaust the budget, ODA-Fin-RL-8B does so on 43% of T2c, and Plutus frequently returns empty answers.The strict scoring protocol retains these generation failures.Source: §5.6 · p. 8

6 Conclusion

MULTIFINGROUND examines recognition and recovery of shared accounting concepts across English, French, and Greek reports. Its findings show that recognition can coexist with missed facts, supporting evaluation of both capabilities.

6 Conclusion

  • MULTIFINGROUND examines whether models recognize shared accounting concepts and recover all corresponding numerical facts across English, French, and Greek reports.Source: §6 · p. 8
  • Greek recall is lowest for seven of eight reference models, while the evaluated finance-specialized models offer limited improvements.Source: §6 · p. 8
  • Models still struggle with accounting attributes and distinctions between related concepts, supporting evaluation of recognition alongside evidence completeness.Source: §6 · p. 8

Limitations

The benchmark’s annotations, coverage, and evaluation setting constrain how its results should be interpreted.

Limitations

  • Issuer-provided iXBRL tags may contain errors or inconsistent concept choices, and ambiguities may remain in unreviewed data.Experts reviewed stratified samples covering 10% of each task; reported results incorporate their corrections and exclusions.Source: Limitations · p. 9
  • T1 leaves untagged amount-like numbers unscored, while most T2 contexts are table windows and shared concepts tend to be common primary-statement concepts.Source: Limitations · p. 9
  • The benchmark covers three languages in one reporting regime, and single-language differences cannot be attributed to language alone because issuer, sector, and document style vary too.Source: Limitations · p. 9

Ethical Considerations

The benchmark uses public filings and the IFRS Accounting Taxonomy, with stated access conditions and no personal data in its inputs.

Ethical Considerations

  • Benchmark inputs come from public regulatory filings and the IFRS Accounting Taxonomy, with access conditions documented for each source.Source: Ethical Considerations · p. 9
  • When full documents cannot be redistributed, the release provides annotations, source information, and reconstruction instructions.Source: Ethical Considerations · p. 9
  • Inputs contain company information but no personal data; the annotation reviewers are co-authors, and benchmark scores do not establish compliance suitability or replace professional review.Source: Ethical Considerations · p. 9

Use of AI Assistants

The appendix formalizes four tasks, their prompts, and model-running conditions, then reports additional score comparisons. It also records AI-assistant use for language polishing, brainstorming, and LaTeX editing, with authors reviewing technical claims and results.

Use of AI Assistants

  • AI assistants supported language polishing, brainstorming, and LaTeX editing; authors reviewed and verified technical claims, results, analysis, and final writing decisions.Source: Use of AI Assistants · p. 9

A Formal Task Definitions

  • The formal tasks distinguish financial-instance and attribute scoring from shared-concept selection, exhaustive target coverage, and label-free identification of the odd excerpt.Source: A Formal Task Definitions · p. 12

B Prompts

  • Prompts are in English but present evidence in its original language and request task-specific structured or single-label outputs.Source: B Prompts · pp. 12–13

C Models and Runs

  • API runs disabled reasoning, and model runs recorded provider-returned names, request identifiers, configurations, data hashes, and code version.Source: C Models and Runs · p. 13

D.1 Additional Score Comparisons

  • Eight reference models achieved 97.3–99.4% T2a recognition accuracy yet missed at least one target in 26.6–41.9% of correctly recognized groups.Source: §D.1 · p. 13
  • On single-language T2a items, Qwen2.5-7B scored 69.5% on Greek versus 86.0% on English, while Greek-oriented Llama-Krikri-8B reached 74.3% on Greek.Source: §D.1 · p. 13
  • Finance specialists lost 3.6–6.9 points on T2a versus their bases; Fin-o1-8B and Plutus also lost 19.7 and 24.7 points on T1.Source: §D.1 · p. 13
  • Qwen2.5-7B T2a accuracy fell from 88.3% with four original options to 83.9% with harder options and 65.8% with twenty options.Source: §D.1 · p. 13

D.2 Taxonomy-Label Comparisons

The appendix compares taxonomy-label conditions and illustrates the tasks with a toy multilingual item. It reports mixed effects from label presentation and clarifies what the comparison conditions change.

D.2 Taxonomy-Label Comparisons

  • C2 supplies localized official labels without identifiers, whereas C1 includes English labels and identifiers; their effects cannot isolate label language from identifier availability.Source: §D.2 · pp. 13, 15
  • Table 4 inventories API, open-weight, and finance-specialist models with adapted bases; Figures 5–6 show attribute-class scores and specialist/base differences.Source: Table 4 · p. 14, Figure 5 · p. 14, Figure 6 · p. 14
  • On Greek-only items, Greek-label C2 significantly lowered accuracy for eight models, while Greek-adapted Plutus alone gained significantly (+5.8 points).Source: §D.2 · pp. 13, 15
  • Adding French and Greek labels to English labels changed reference-model scores by at most 1.0 point, never significantly, but lowered Fin-R1 by 10.2 and Qwen2.5-7B by 5.0.Source: §D.2 · p. 15

E An Illustrative Grounding Item

F Expert Review

Expert reviewers independently audited stratified samples using IFRS-based rules for judging marked facts, concepts, and excerpt context. The section also describes sampling and score-comparison procedures for the audit and diagnostic analyses.

F Expert Review

  • Two qualified financial experts independently reviewed sampled items, then resolved disagreements jointly; agreement is measured before discussion.Reviewers had at least five years of IFRS reporting or audit experience and professional working proficiency in French and Greek.Source: F Expert Review · p. 15
  • Diagnostic tables report task-specific scores by language, label condition, class, and finance-model/base-model comparison, but these captions do not provide outcome values.The captions define T1 macro-F1 and attribute metrics, T2a/T2c accuracy, and T2b macro-F1 or per-language F1.Source: Table 6 · p. 16, Table 7 · p. 16, Table 8 · p. 16
  • A fixed-seed stratified 10% sample was audited, with item-level resampling used for intervals on fact-, option-, and excerpt-level proportions.The audit covered about 4,200 T1 numbers, 2,100 T2b facts, 156 T2a distractors, and 212 T2c excerpts.Source: F Expert Review · pp. 16–17
  • Reviewers judge each marked number against IFRS definitions, with task-specific rules for financial attributes, concept matching, distractor plausibility, and excerpt validity.For T2b, a number matching the target value can still be rejected when it appears under another row label; ambiguous facts are excluded from scoring.Source: F Expert Review · pp. 17–18
  • The audit table compares original and corrected annotations on reviewed items, while the harder-option table contrasts original, harder, and twenty-option T2a settings.The audit score change is the largest absolute difference across eight reference models; the harder-option caption also describes a lexical-overlap baseline.Source: Table 11 · p. 18

G Collection and Preprocessing Details

The benchmark draws one FY2025 report per issuer across English, French, and Greek, then extracts and normalizes tagged financial evidence. It relates this matched-concept evaluation to prior financial extraction and multilingual benchmarks while noting issuer-related variation in coverage.

G Collection and Preprocessing Details

  • Reports are drawn from 90 issuers, with 30 issuers per language and each issuer appearing in only one language.English reports span ten sectors and six jurisdictions; the French collection draws on CAC 40 constituents, and Greek reports are consolidated Athens Exchange filings.Source: G Collection and Preprocessing Details · pp. 18–20
  • Table 12 describes a stratified 10% expert audit and compares scores under original versus corrected annotations; Table 13 lists the issuer reports and sectors.The audit score change is the largest absolute difference across eight reference models on reviewed items.Source: Table 12 · p. 19, Table 13 · p. 19
  • The pipeline renders iXBRL reports, reconstructs tables and paragraphs, normalizes text, and retains each tagged fact’s visible offsets and taxonomy attributes.It excludes empty nil facts and drops numeric columns without tagged facts to avoid mixing in untagged parent-company figures alongside consolidated data.Source: G Collection and Preprocessing Details · p. 20

H Extended Related Work

  • Unlike prior resources focused on single-language extraction, multilingual representations, or parallel tasks, this evaluation matches independently authored excerpts by accounting concept and assesses target coverage conditional on correct recognition.Because excerpts come from different issuers, coverage differences may also reflect sector and document difficulty rather than language alone.Source: H Extended Related Work · p. 20
Loading 2610.19432v1…