PaperlayerView exact revision on arXiv

Source-linked AI summary

IGT @ FinMMEval 2026 Task 2: Question-Type Prompting with Targeted Extraction for Multilingual Financial QA

Yuwen Chiu

arXiv:2609.08139v1cs.CLcs.AI

TL;DR

PolyFiQA Task 2 asks multilingual financial QA systems to answer heterogeneous questions over SEC filings and news, creating a need for approaches matched to question type. IGT uses specialized extraction and passage-selection handlers with format-aware prompting, achieving strong development performance and third-place official-test ranking, while remaining calibrated to task-specific conventions.

  • Problem

    PolyFiQA Task 2 combines English SEC filings with multilingual news for four companies, but its questions require different evidence and answer structures.

  • Method

    IGT routes questions to specialized handlers, applying filing-text extraction to structured numeric types and rule-based multilingual news selection to synthesis types.

  • Results

    ROUGE-1 ≈0.395 on development data represented a 60% relative improvement over a generic RAG baseline, while the official test result was ROUGE-1 = 0.3071 with 3rd of 12 ranked teams.

  • Takeaways & Limitations

    Format precision and company-specific extraction audits are practically important because surface-form matching and filing heterogeneity materially affect performance.

  • Takeaways & Limitations

    Development tuning on a fixed 152-sample set may not generalize to the 256-sample test set, and company-specific normalizations may fail on unseen filing variants.

Abstract

from arXiv · show

We present the IGT system for PolyFiQA Task 2 of the FinMMEval Lab at CLEF 2026, a multilingual financial question answering task over English SEC filings and multilingual news articles (English, Chinese, Japanese, Spanish, Greek) for four companies. Our central observation is that the 344 development questions divide into two families requiring fundamentally different approaches: structured numeric types (R&D ratio, cash flow, capital expenditure) are best answered by direct keyword extraction on filing text, while synthesis types (investment strategy, capital allocation, top-three revenue focuses) require rule-based multilingual news passage selection. A dataset analysis reveals that 17-18 of 19 ground-truth reference answers per synthesis type share an exact evidence label prefix, whose unigram tokens contribute directly to ROUGE-1 overlap. The final system achieves development ROUGE-1 approximately 0.395, a 60% relative improvement over a generic RAG baseline (approximately 0.247), and ranks 3rd of 12 teams on the official test set with ROUGE-1 = 0.3071, Precision = 0.2821, and Recall = 0.4044.

1. Introduction

PolyFiQA exposes financial QA as a heterogeneous problem, motivating a diagnostic system that routes questions to specialized handlers. IGT combines direct extraction for structured financial values with multilingual news selection for synthesis answers.

  • Problem framing: Uniform treatment is inadequate because cash-flow questions permit pattern matching, whereas investment-strategy questions require multilingual evidence synthesis.The paper contrasts deterministic table-derived answers with answers lacking a single correct phrasing.
  • Task motivation: PolyFiQA combines English SEC filings with multilingual news articles across five languages and four companies, using ROUGE-1 for concise evidence-grounded answers.The task includes English, Chinese, Japanese, Spanish, and Greek news sources.
  • Design rationale: A development-set diagnostic found that retrieval sophistication mattered less than generation quality and output-format control.This finding motivated routing, direct ratio computation, and preservation of non-English source text.
  • System contributions: IGT routes questions to eight tailored handlers, using raw-filing keyword extraction for structured numeric types and rule-based multilingual passage selection for synthesis types.The routing strategy assigns different computational approaches to the two question families.
  • System contributions: Company-specific label normalization and original-language revenue-quote preservation target extraction reliability and multilingual evidence matching.The system also reports systematic ablations, per-type and per-company analysis, extraction pseudocode, and prompt templates.

2. Related Work

Prior financial QA work addresses document structure, multilingual modeling, prompt sensitivity, and multi-hop retrieval, but PolyFiQA combines these challenges in a distinct multilingual setting. The paper uses this literature to motivate format-aware, task-specific methods.

  • Financial QA: English financial QA research identifies table-destroying chunking and limited context windows as recurring RAG challenges.Element-based chunking, cross-encoder reranking, and HyDE query expansion are reported as useful approaches.
  • Multilingual QA: Prior multilingual work includes Spanish–English transfer and bilingual Chinese–English financial RAG, but these settings differ from PolyFiQA’s source-language preservation.The cited Spanish–English setting translates content to a common language, while PolyFiQA includes five news languages.
  • Evaluation and prompting: Financial retrieval models are sensitive to prompt placement and formatting, while performance can collapse beyond 32k tokens for multiconcept questions.The related evaluation work also reports that standard Recall can overstate performance relative to F1.
  • Multi-hop retrieval: A two-hop retrieval approach inspired by graph-based multi-hop QA did not transfer to PolyFiQA Expert questions; format-controlled direct prompting outperformed it by 0.087 ROUGE-1.The comparison concerns the questions where multi-hop retrieval was intended to help.

3. Task and Data Analysis

PolyFiQA Task 2 pairs SEC filing excerpts with multilingual news and divides questions into structured numeric and synthesis families. Reference-answer evidence labels also create a direct ROUGE-1 formatting signal.

  • Task and dataset: 344 instances cover MSFT, HON, JNJ, and UVV, with 172 Easy factual questions and 172 Expert analytical questions.Each answer is limited to 100 words, and ROUGE-1 F-measure is the primary metric.
  • Question-type analysis: The analysis identifies eight question types in two information regimes: filing-derived structured numeric answers and multilingual-news synthesis answers.Numeric questions require retrieval precision, while synthesis performance depends on evidence integration and formatting.
  • Evidence-label analysis: Reference answers generally begin with either News Evidence: or Financial Statement Evidence:, whose tokens count toward ROUGE-1 overlap.Using the wrong label can remove up to three unigram matches before answer content is compared.
  • Evidence-label analysis: Table 1 reports NE and FSE label distributions across question types, with 19 questions per type and one plural FSE variant.The table distinguishes News Evidence from Financial Statement Evidence.
  • Evidence-label analysis: Capital-expenditure references are predominantly FSE-labeled, with 17+1 of 19 using the financial-statement label family.The capex prompt therefore begins with Financial Statement Evidence:, while one NE instance creates a label mismatch.

4. System Description

The system routes each question to a specialized handler: direct extraction for structured financial quantities and multilingual news selection for synthesis questions. It combines targeted extraction, format-aware prompting, and type-specific context budgets instead of uniform vector retrieval.

  • Question-type routing: Eight question types are routed to tailored handlers, separating structured extraction from synthesis-oriented news selection.Structured numeric questions use filing-text extraction, while synthesis questions use selected multilingual news passages.
  • Question-type routing: The final architecture bypasses retrieval for structured numeric questions and assembles multilingual news context for synthesis questions before calling Claude Sonnet 4.Both paths use type-specific handling rather than a single retrieval pipeline.
  • Targeted financial extraction: Company-specific cash-flow patterns corrected approximately 30% JNJ misclassification by avoiding generic labels that retrieved the wrong cash-flow category.The added JNJ variants recovered approximately 0.038 ROUGE-1 on JNJ cash-flow questions.
  • Targeted financial extraction: R&D ratio ROUGE-1 increased from ≈0.28 to ≈0.582 when expenditure and revenue were extracted directly and passed to the LLM as a scalar.The LLM formatted the computed result rather than performing the calculation from retrieved text.
  • Token budget management: News budgets were tuned by question type: revenue peaked at 1,500 characters, while top-three questions peaked at 2,000.Larger budgets could produce verbose outputs that were truncated before key points were completed.
  • Token budget management: An explicit word-count instruction reduced over-length outputs from ≈18% to 6%, at a cost of −0.004 ROUGE-1.The system paired a 300-token maximum with sentence-boundary trimming to control output length.

5. Results

The final system shows strong gains from targeted architectural changes, with structured extraction outperforming synthesis routes and performance varying by question type, company, and routing accuracy. Development and official-test results also expose persistent limits from label noise, verbose outputs, and routing mismatches.

  • Development trajectory: 65% of the total baseline-to-final gain came from two interventions: switching the LLM backend and replacing retrieval with direct R&D extraction.The backend switch contributed +0.049 ROUGE-1, while direct R&D extraction contributed +0.054.
  • Per-question-type performance: A 0.281-point ROUGE-1 spread separates R&D (0.582) from top-three questions (0.301), reflecting distinct structured and synthesis regimes.Structured types score higher because extraction returns complete table rows, whereas synthesis types converge near 0.30–0.33.
  • Routing-level breakdown: The +0.195 Easy-question gap between extraction and synthesis routes shows that routing materially affects performance, while Expert extraction falls to 0.1942 under keyword mis-routing.Some Expert analytical questions contain terms such as “cash flow” and are incorrectly sent to the extraction path.
  • Per-company performance: R&D performance is highly company-dependent, reaching 0.841 for HON and 0.869 for UVV but only 0.310 for JNJ and 0.367 for MSFT.Filing-format consistency explains the contrast; JNJ cash-flow normalization recovered approximately 0.038 ROUGE-1, while JNJ R&D remains weak.
  • Precision tightening: Reducing the maximum output to 200 tokens increased precision from ∼0.308 to ∼0.331 but reduced recall from ∼0.489 to ∼0.431, producing net ROUGE-1 −0.008.The two-sentence constraint showed the same trade-off, with precision +0.018, recall −0.041, and net ROUGE-1 −0.006.
  • Official test results: The official system ranked 3rd of 12 with ROUGE-1 = 0.3071, while its 0.4044 recall was highest among the top-five teams.Precision was 0.2821, and the top-three systems were separated by less than 0.005 ROUGE-1.

6. Discussion

The discussion argues that performance depends on delivering the right information in a parseable form and matching reference answer formats, while several gains remain specific to PolyFiQA’s conventions. It also identifies company-specific filing variation and multilingual surface-form evaluation as important constraints.

  • ROUGE-1 = 0.3071 placed the system 3rd of 12 teams on the official test set, consistent with the development trajectory.The largest gains addressed information delivery, while label and format changes addressed output structure.
  • The development-to-test gap reflects iterative tuning on a fixed 152-sample set, with company-specific routing and normalization potentially misfiring on unseen filing variants.The test set contains 256 samples, including 128 Easy and 128 Expert instances.
  • +0.038 ROUGE-1 was recovered on JNJ cash flow questions after company-specific normalization addressed non-standard section headers.The overall gain was +0.009, but the targeted JNJ cash-flow effect was larger.
  • Deterministic regex-based extraction remained competitive with, and often superior to, embedding-based retrieval for structured financial documents.Capex and cash-flow gains relied on pattern matching over raw filing text.
  • +0.017 from preserving source-language revenue quotes reflects ROUGE-1’s penalty for factually correct translations with no lexical overlap.The references quote non-English news verbatim, making surface-form preservation important under this evaluation.
  • Several design choices are calibrated to PolyFiQA and may not transfer to benchmarks without its fixed evidence-label prefixes and top-three format convention.The label-prefix strategy depends on reference answers beginning with one of two fixed strings.

7. Conclusions

The paper treats question-type heterogeneity as the central design problem and builds specialized handlers that align extraction and generation with each type. It reports strong development performance and a third-place official test ranking, while identifying broader lessons about filing variation, multilingual evaluation, and format precision.

  • Eight specialized handlers replace retrieval with direct extraction for structured numeric questions and align synthesis outputs with reference answer structure.The system was built from diagnostic analysis of the development data.
  • ROUGE-1 ≈0.395 on development data was a 60% relative improvement over a generic RAG baseline, while the system ranked 3rd of 12 on the official test.The official test result was ROUGE-1 = 0.3071, Precision = 0.2821, and Recall = 0.4044.
  • Company-specific filing audits are necessary because a single extraction pattern is insufficient across heterogeneous company formats.The conclusion identifies these audits as high-yield beyond this task.
  • Under multilingual ROUGE-1 evaluation, preserving source-language text matters more than factual accuracy of a paraphrase.The paper presents this as a finding that extends beyond the specific task.
  • Format precision outperformed multi-hop retrieval by 0.087 ROUGE-1 on questions where retrieval complexity was expected to help.The comparison is reported as one of three findings intended to transfer beyond PolyFiQA.
  • Future work includes a lightweight trained router, dynamic company-label discovery, improved evaluation metrics, and tighter retrieval–format integration.The proposed router targets Expert mis-routing, identified as the primary remaining failure mode.

Declaration on Generative AI

The author reports using Claude during preparation for code debugging, language review, and initial section outlines. The author states that all content was reviewed and edited personally.

  • Claude assisted with code debugging, grammar and spelling review, and drafting initial section outlines.
  • The author reviewed and edited all content and takes full responsibility for the publication.

A. Extraction Pseudocode

The pseudocode extracts R&D and revenue values from filing lines using keyword checks and numeric parsing, then returns their ratio for LLM formatting. The LLM performs no arithmetic.

  • The extractor scans lowercased filing lines for R&D and revenue keywords and records the first parsed number for each.R&D keywords include “research and development” and “r&d expense”; revenue keywords include “total revenue,” “net revenue,” and “total net revenue.”
  • The returned scalar is rd_value / rev_value, while the LLM formats the result without performing arithmetic.
  • parse_first_number uses a regex to extract the first numeric token, including dollar amounts and millions-or-billions notation.

B. Prompt Templates

The system uses a shared financial-assistant prompt across question types, while type-specific user prompts define the required evidence format and response constraints.

  • B. Prompt Templates: All prompts are sent to claude-sonnet-4-20250514 through the AWS Bedrock Messages API with temperature=0.0 and max_tokens=300.The run uses the us-east-1 region, and one official run was submitted.
  • Shared System Prompt: The shared system prompt requires answers to use only the provided evidence.It also requires a prescribed evidence-label prefix, a maximum length, preservation of non-English figures, and exact source numbers.

R&D Ratio Handler (extraction route)

The question-type templates route numeric questions through extracted filing data and route synthesis questions through multilingual news passages, with strict output formats for each task.

  • R&D Ratio: R&D-ratio prompts provide expenditure, revenue, and a computed percentage for a concise evidence-based answer.The requested response begins with “News Evidence:” and cites the figures used in the computation.
  • Revenue Trends: Revenue-trend prompts combine a filing revenue excerpt with English, Chinese, Japanese, Spanish, and Greek news passages.They require revenue trends and preserve original currency notation and numeric phrasing when quoting non-English sources.
  • Cash Flow: Cash-flow prompts expose cleaned extracted data and prescribe reporting for operating, investing, and financing changes.The format includes amounts, comparisons with prior values, and a percentage change for operating cash flow.
  • Balance Sheet: Balance-sheet prompts pair a filing excerpt with supplementary multilingual news and request relevant figures plus notable trends.The answer is limited to 100 words and begins with “News Evidence:”.
  • Capital Expenditure: Capital-expenditure prompts use extracted capex rows and filtered multilingual news passages to report filing figures with optional context.The template clarifies that values such as 3,767 or 15,441 represent millions.
  • Investment Strategy: Investment-strategy prompts synthesize multilingual news while treating the financial-statement excerpt as supplementary evidence.The answer must begin with “Financial Statement Evidence:” and preserve non-English figures without translation.
  • Top Three Revenue Focuses: Revenue-focus prompts require exactly three numbered items introduced by a fixed sentence.They draw on multilingual news and supplementary filing evidence within a 100-word limit.
  • Capital Allocation: Capital-allocation prompts require a company-led opening or “Answer: None.” when the news is silent.They use multilingual news with supplementary filing evidence and preserve non-English figures.
Loading 2609.08139v1…