PaperlayerView exact revision on arXiv

Source-linked AI summary

CLAIR-Fin: An Adversarial Multi-Agent Framework for Claim-Level Verification and Adaptive Debate in Cross-Modal Financial QA

Fatema Tuj Johora Faria, Mukaffi Bin Moin, Jubayer Al Mahmud, M. F. Mridha, Md. Alam Hossain

arXiv:2608.13706v1cs.CLcs.AI

TL;DR

Existing financial QA systems struggle to reconcile conflicting evidence across long, multimodal reports and verify individual claims before errors reach the final answer. CLAIR-Fin addresses this with a typed claim ledger, claim-aware evidence authority, hand-off verification, and adaptive adversarial debate, evaluated across query types and formats.

  • Problem

    Existing multimodal financial QA systems struggle with realistic questions when evidence is distributed across long reports and presented in conflicting text, table, and chart forms.

  • Method

    CLAIR-Fin decomposes questions into typed atomic claims and combines claim-aware evidence weighting, hand-off grounding checks, adaptive debate, and terminal entailment auditing.

  • Results

    CLAIR-Fin is evaluated across query types, presentation formats, framework-specific metrics, human judgments, and ablations against retrieval-augmented baselines.

  • Takeaways & Limitations

    The framework provides a claim-level approach for reconciling cross-modal financial evidence while supporting explicit citations, selective abstention, and auditable risk tracking.

  • Takeaways & Limitations

    The framework was evaluated only in English and with one model family, leaving multilingual and cross-model validity open.

Abstract

from arXiv · show

Existing defenses against hallucination in retrieval-augmented and multi-agent pipelines remain partial: evidence is trusted despite modality disagreement, debate verifies an aggregate report rather than individual claims, and such verification occurs only after drafting, leaving inter-agent errors undetected until the final text. To close this gap, we present CLAIR-Fin, a nine-agent framework that decomposes each question into atomic claims maintained in a typed Financial Claim Ledger. Each claim is resolved through Asymmetric Evidence Authority, which conditions evidence trust on claim type rather than treating all modalities as equally reliable; Chain-of-Custody Verification, which checks grounding at the hand-off between drafting and adversarial review rather than only at the pipeline's exit; an Adaptive Rebuttal Cycle, which routes contested claims through adversarial debate whose depth scales with what that debate finds; and a terminal entailment audit paired with a continuous Hallucination Risk Index that distinguishes claims that passed scrutiny from claims never contested. We evaluate CLAIR-Fin on BB-FinQA-X, a 500-question cross-modal financial evaluation set built from Bangladesh Bank Annual Report material, stratified by query type, format, and difficulty. Relative to a single-pass retrieval-augmented generation baseline, it raises faithfulness ($0.780 \rightarrow 0.889$) while abstaining on 5.4% of questions when evidence is insufficient rather than forcing an unsupported response, and it exceeds stronger retrieval-strategy baselines such as HyDE and Graph-RAG on faithfulness ($\leq 0.874$).

1 Introduction

Long multimodal financial reports make faithful question answering difficult because models may misread fiscal labels or chart values, fail as context grows, and face conflicting evidence. CLAIR-Fin addresses these gaps with a nine-agent, claim-level framework that weights evidence by claim type, verifies grounding throughout reasoning, debates disputes adaptively, audits entailment, and abstains when evidence is insufficient.

  • Motivation: Financial QA over long multimodal reports is vulnerable to materially consequential misreadings, context-dependent failures, and unsupported answers when evidence is insufficient.The same fact may appear in prose, tables, and charts, while current models struggle when relevant evidence is distant from the document’s start.
  • Research gap: Prior multiagent, multimodal retrieval, and financial debate systems address parts of the problem but do not condition evidence trust on claim type.The cited frameworks coordinate agents, jointly index modalities, or structure debate, yet leave claim-type-aware evidence weighting unresolved.
  • Framework: CLAIR-Fin introduces a nine-agent framework centered on a Financial Claim Ledger of atomic, typed assertions reconciled through Asymmetric Evidence Authority.Three evidence agents populate the ledger, while the Ledger Guardian applies claim-type-aware weighting.
  • Verification pipeline: The framework verifies evidence hand-offs, escalates disputed claims through adaptive adversarial debate, applies terminal entailment auditing, and tracks continuous hallucination risk.Its mechanisms include CoCV, ARC, a Judge-Auditor, equal-weight counterfactuals, HRI, and abstention when evidence cannot support the answer.
  • Evaluation questions: The evaluation examines modality-aware prioritization, hand-off verification, contested-claim debate, continuous risk estimation, and performance across presentation formats.The research questions compare these mechanisms with alternatives and identify challenging single-modal and multimodal configurations.

2 Related Work

Prior work extends retrieval to multimodal financial documents, uses multi-agent debate, or verifies individual claims, but these directions address evidence arbitration, adaptive debate, and hand-off verification separately. CLAIR-Fin combines claim-type-conditioned evidence trust, adaptive adversarial debate, and hand-off grounding verification.

  • Multimodal Financial Retrieval: Multimodal financial RAG coordinates text, tables, and images, escalating to richer context only when needed.General-purpose agents coordinate text and image agents, while financially specialized retrieval batches table and figure images before escalation.
  • Multimodal Financial Retrieval: Retrieval evaluation measures faithfulness, relevancy, and context precision/recall but cannot adjudicate conflicting evidence modalities.A table cell and approximately read chart value may be cited interchangeably, although faithfulness measures entailment by some evidence rather than the correct evidence.
  • Multi-Agent Debate: Multi-agent debate improves factuality and reasoning through critiques, revisions, and specialized financial personas or dialectical roles.Financial frameworks include earnings, market, sentiment, valuation, and risk personas, as well as bull, bear, and devil’s-advocate roles.
  • Claim-Level Verification: Claim-level verification decomposes statements, retrieves evidence, and checks entailment, but typically verifies claims only after they are formed.Graph-structured methods check entity-relationship triplets before a verdict, yet verification remains a one-time check and reproducibility does not guarantee correctness.
  • CLAIR-Fin’s Position: CLAIR-Fin jointly conditions evidence trust on claim type, adaptively routes insufficiently covered claims to adversarial debate, and verifies grounding at hand-off.Its evidence arbitration uses a stated, auditable prior, while debate depth scales with scrutiny rather than a fixed budget.

3 CLAIR-Fin Framework

CLAIR-Fin decomposes multimodal financial questions into independently verifiable atomic claims tracked in a persistent Financial Claim Ledger. Claim-type-specific evidence authority, hand-off grounding checks, and adaptive adversarial verification govern whether claims are audited, escalated, or abstained.

  • Claim decomposition: Questions become 1–8 ordered atomic claims with types, allowing each claim’s evidence, debate, and audit to remain independent and auditable.Whole-answer verification can hide unsupported figures inside supported narratives; claim decomposition enables exact grounding checks at every stage.
  • Financial Claim Ledger: The Financial Claim Ledger is a persistent typed directed multigraph linking claims to text spans, table cells, derived metrics, chart regions, and constraint checks.It serves as the sole verification, audit, and citation artifact throughout a question’s lifetime.
  • Evidence fusion and authority: Asymmetric Evidence Authority assigns fixed modality weights by claim type, so evidence sufficiency depends on the claim rather than a global trust rule.Numeric claims require table cells, attribution claims require prose, and claims escalate when authority coverage A(ci) falls below 0.75.
  • Adversarial verification: Strong agreeing evidence skips debate, while weak or conflicting evidence escalates claims into adversarial verification and adaptive rebuttal before final audit.The framework’s phases fuse modality-specific retrieval, decide fast-path or escalation, and apply authority-weighted terminal auditing.
  • Grounding and risk auditing: Chain-of-Custody Verification checks draft grounding at hand-off before adversarial review, catching drift before counsel reasons over ungrounded content.The Hallucination Risk Index distinguishes claims that survived genuine scrutiny from claims fast-pathed without contest.

4 Dataset Construction

This section introduces BB-FinQA-X, a 500-question multimodal financial question-answering dataset grounded in the Bangladesh Bank Annual Report. Detailed construction, annotation, validation, and distribution statistics are provided in Appendix A.

  • BB-FinQA-X is a 500-question multimodal financial QA dataset grounded in the Bangladesh Bank Annual Report.
  • Appendix A provides detailed dataset construction, annotation, validation, and distribution statistics.

5 Experimental Configuration

The paper places detailed implementation settings, experimental configurations, and evaluation protocols in Appendix B.

  • Detailed implementation settings, experimental configurations, and evaluation protocols are provided in Appendix B.

6 Results and Discussion

CLAIR-Fin’s ablations show that adaptive debate and terminal entailment auditing contribute most to faithfulness and correctness, while performance varies substantially by evidence format and query type. Human evaluation and risk analysis further support risk-sensitive abstention and the framework’s cross-modal evaluation design.

  • Evaluation Design: Evaluation spans automatic retrieval and generation metrics, framework-specific metrics, human evaluation, and ablations on BB-FinQA-X.The study compares the full system with single-pass RAG and reports results across query type and presentation format.
  • Ablation Results: AEA removal produces smaller but consistent declines: faithfulness 0.889 →0.883, context recall 0.897 →0.893, and exact correctness 0.592 →0.585.AEA impact rate is 0.515, indicating asymmetric evidence authority changes the winning modality in roughly half of contested decisions.
  • Ablation Results: Removing the terminal entailment audit reduces faithfulness 0.889 →0.845, exceeding CoCV’s decline to 0.857.Faithfulness rate likewise falls 0.783 →0.741 without the audit versus 0.783 →0.753 without CoCV.
  • Ablation Results: Removing ARC causes the largest degradation: faithfulness falls 0.889 →0.770, exact correctness 0.592 →0.524, and answer coverage 0.946 →0.896.Debate utilization is 0.646, routing nearly two-thirds of claims through adaptive adversarial debate.
  • Risk and Human Evaluation: HRI correlates negatively with correctness (r = −0.072), while abstention appropriateness scores 4.06/3.95 with κ = 0.84.These results support reporting continuous risk alongside binary verification outcomes and align risk-sensitive abstention with human judgment.
  • Format and Query-Type Analysis: Text + Table has the highest faithfulness at 0.915, whereas Chart Only has the lowest at 0.850; Evidence Retrieval and Multi-hop Reasoning score 0.839 and 0.840.Difficulty concentrates in chart-dependent evidence and evidence-synthesis or grounding queries, while combined formats outperform their weakest constituent modality.

7 Conclusion

CLAIR-Fin addresses unresolved challenges in faithful financial question answering across disagreeing text, tables, and charts. Its nine-agent design uses a typed Financial Claim Ledger and conditions evidence trust on claim type through Asymmetric Evidence Authority.

  • 7 Conclusion: CLAIR-Fin targets evidence reconciliation across text, tables, and charts that may disagree in long financial documents.The passage identifies this multimodal reconciliation requirement as a gap left unresolved by prior retrieval, debate, and claim-level verification methods.
  • 7 Conclusion: The framework comprises nine agents organized around a typed Financial Claim Ledger.The ledger is the central structure described for CLAIR-Fin’s framework.
  • 7 Conclusion: Asymmetric Evidence Authority conditions evidence trust on claim type instead of treating all evidence uniformly.This mechanism is presented as one of CLAIR-Fin’s core design principles.

Limitations

CLAIR-Fin’s claims are scoped to faithful, citation-grounded QA over a specific English central-bank document family, while several design and evaluation choices limit broader conclusions. Transfer to other institutions, languages, domains, and adversarial or corrupted inputs remains unestablished, despite architectural components that may generalize.

  • Scope: CLAIR-Fin targets long, multimodal financial documents requiring narrative, table, and chart reconciliation under strict correctness constraints.The authors frame this setting as the scope of their claims rather than a qualification of the contributions.
  • Dataset Scope: BB-FinQA-X covers one Bangladesh Bank report, one institution, one language, and one reporting convention, limiting transferability across document families.Its 500 questions are stratified by query type, format, and difficulty, but not document diversity.
  • Methodological Constraints: Fixed AEA weights, HRI component weights, and escalation and entailment thresholds represent a designed operating point rather than an optimized solution.Cross-modal linking also relies on lexical substring overlap, a coarse heuristic that can miss semantic relationships.
  • Evaluation Scope: LLM-judged automatic metrics share methodology with the framework’s entailment checks, while human evaluation covers two annotators over the full set.The evaluation also uses a fixed nonzero budget of ρmax = 2 rounds for every escalated claim, comparing debate presence against absence rather than adaptive versus fixed allocation.
  • Evaluation Scope: The study does not evaluate robustness to adversarial or out-of-distribution questions, or noisy and corrupted source PDFs beyond the evaluation corpus.These omissions constrain conclusions about performance outside the tested source conditions.
  • Generalizability and Future Extensions: The taxonomy, modalities, and authority weights are specific to financial statistical reporting, although claim decomposition, modality-conditioned authority, hand-off verification, and continuous risk scoring may generalize.The architecture was not designed with transfer to legal contracts or clinical reports in mind.

Ethics Statement · Appendix · A BB-FinQA-X: Dataset Construction and Validation

CLAIR-Fin is intended as a decision-support aid for verifying facts in long, multimodal financial reports, using a public Bangladesh Bank source without personal data or human subjects. The statement emphasizes limited generalizability, risks of overreliance, embedded safeguards, and the need for broader empirical validation.

  • Ethics Statement: CLAIR-Fin supports analysts in locating and verifying facts across narrative text, tables, and charts, but does not replace expert financial or regulatory judgment.Outputs should be reviewed by domain experts.
  • Ethics Statement: BB-FinQA-X uses a publicly available Bangladesh Bank Annual Report and contains aggregate indicators rather than personally identifiable information.The dataset was used for fair, non-commercial academic research, with no preprocessing beyond the extraction pipeline before annotation.
  • Ethics Statement: The work did not require formal institutional ethics approval because it involved no human subjects, personal data, or data collection beyond manual annotation of a public government document.Dataset construction and independent review were performed by the authors; two external banking-sector experts provided voluntary, uncompensated validation and blind evaluation.
  • Ethics Statement: BB-FinQA-X is limited to one institution, language, and reporting convention, so its findings should not be assumed to generalize without further evaluation.The fixed, hand-specified modality-trust prior in Asymmetric Evidence Authority is a design choice not learned or validated against human judgment.
  • Ethics Statement: CLAIR-Fin can remain inaccurate or hallucinated, while its Hallucination Risk Index and audit verdicts are calibration signals rather than correctness guarantees.Use outside the evaluated domain carries unquantified risks, and passing verdicts or low HRI should not replace independent evidence checking.
  • Ethics Statement: Claim-level grounding, auditable evidence arbitration, and selective abstention could make automated financial QA more transparent and support more reliable access to dense public documents.These potential benefits depend on indicating confidence rather than always answering without qualification.
  • Ethics Statement: Automation bias remains a risk, particularly for users without domain literacy, and performance and abstention behavior are unequal across settings beyond the evaluated domain.Confident-sounding outputs may still be trusted without independent verification despite the safeguards.
  • Ethics Statement: CLAIR-Fin mitigates misuse by requiring citations, checking grounding at drafting hand-off, gating unentailed claims through terminal audit, and treating selective abstention as a first-class outcome.These safeguards are incorporated directly into the framework rather than added externally.

A.1 Evaluation Protocol … B Experimental Setup

CLAIR-Fin is evaluated on a manually constructed, cross-modal 500-question benchmark from the Bangladesh Bank Annual Report using automated, human, and ablation protocols. The benchmark spans controlled query, difficulty, and presentation dimensions, with staged expert validation, quality control, and a specified implementation stack.

  • A.1 Evaluation Protocol: The evaluation uses 500 BB-FinQA-X questions stratified by query type, format, and difficulty, with answers mapped to Correct, Partial, Incorrect, or Abstained outcomes.RAGAS metrics are judged by GPT-4.1 mini, while GPT-4o assigns four-way labels; exact correctness, coverage, debate utilization, changed-outcome rate, and HRI calibration are also reported.
  • A.2 Data Sources and Selection Criteria: Questions come from the Bangladesh Bank Annual Report and target material with dense narrative, table, and chart evidence around shared indicators while remaining self-contained.The source is a public statistical and policy publication containing no proprietary or personally identifiable data.
  • A.3 Dataset Construction Pipeline: Dataset construction followed source selection, direct drafting with recorded evidence and labels, three-stage validation, and final quality-control release.Items were revised or discarded during validation, and the final 500-item set was frozen only after balance checks across annotation dimensions.
  • A.4 Manual Annotation Protocol: Four annotators built and validated BB-FinQA-X: two authors handled drafting and independent review, while two external banking experts performed domain validation and joint consensus resolution.Flagged items were resolved against a shared guideline rather than by majority vote or a single adjudicator.
  • A.5 Annotation Schema and Guidelines: Each item is labeled by query type, difficulty, and presentation format, including six query types, three difficulty levels, and six evidence-format combinations.Evidence Retrieval requires locating and grounding evidence, whereas Multi-hop Reasoning combines evidence from multiple locations or modalities.
  • A.6 Quality Control and Validation: Quality control checked correctness, grounding, ambiguity, duplication, numerical accuracy, and all three label dimensions, revising or discarding failures before release.A final pass confirmed target distributions and annotation consistency across all 500 items.
  • A.7 Dataset Statistics: BB-FinQA-X contains 500 English question–answer pairs covering six query types, three difficulty levels, and six presentation formats, with Easy and Medium items predominating.Hard items number 75 and are reserved for multi-hop, cross-modal, or computation-heavy cases; every source-occurring combination is represented.
  • B Experimental Setup: The experimental setup specifies the implementation stack, ingestion pipeline, and inference configuration used to produce the reported results, with code, prompts, and configurations slated for release.This separates each mechanism’s computation from the library or model instantiating it.

B.1 Implementation Framework and Multi-Agent Orchestration … B.7 Configuration Management

CLAIR-Fin is implemented as a configurable nine-agent LangGraph pipeline that processes financial documents across text, tables, and charts, then evaluates retrieval, answer quality, and claim-level outcomes. Its shared typed state, multimodal retrieval, fixed inference settings, runtime tooling, and externalized configuration support mechanism-focused execution and ablation.

  • B.1 Implementation Framework and Multi-Agent Orchestration: Nine agents operate as nodes over one shared typed LangGraph StateGraph, with conditional coverage escalation and severity-gated rebuttal edges.Structured-output mode uses Pydantic schemas with a safe-default fallback on parse failure.
  • B.2 Document Ingestion and Chunking: Source PDFs are parsed page-by-page, rendered to images for vision-based table and chart extraction, and sentence-aware narrative chunks target 1,000 characters.The splitter protects common abbreviations such as “Dec.”, “approx.”, and “e.g.”.
  • B.3 Embedding Model and Vector Database: OpenAI text-embedding-3-large produces 3,072-dimensional vectors for all modalities, which Milvus Lite stores with modality, document, page, and modality-specific metadata.Queries use the same embedding model and shared vector space.
  • B.4 Retrieval Configuration: Retrieval reranks a 4k candidate pool using normalized vector similarity weighted 0.65 and lexical overlap weighted 0.35, with agent widths of Narrative k=8, Tabular k=6, and Visual k=5.Lexical overlap helps distinguish semantically close aggregates without sacrificing dense retrieval recall.
  • B.5 Language Models and Inference Settings: GPT-4o is shared across all nine agents and vision extraction, while temperatures are fixed by call site, including 0 for claim decomposition and terminal-audit drafting and 0.2 for drafting and rebuttal.The Chain-of-Custody checkpoint reuses GPT-4o through the Judge-Auditor’s entailment call.
  • B.6 Evaluation Tooling and Runtime Environment: Evaluation combines automatic retrieval and generation metrics with ledger-derived exact correctness, answer coverage, AEA impact rate, debate utilization, and HRI calibration.Computed metrics include context precision, context recall, faithfulness, and answer relevancy.
  • B.7 Configuration Management: All thresholds and weights are externally configurable through shared settings and YAML files, making mechanism changes configuration edits rather than code changes.Configured values include the 0.75 coverage cutoff, ρmax=2 rebuttal-round cap, 0.5 entailment pass bar, AEA weights, and HRI term weights.

C Supplementary Results

The supplementary results report framework-specific, retrieval and generation quality, human evaluation, and breakdown analyses for CLAIR-Fin on BB-FinQA-X. CLAIR-Fin performs strongest overall across retrieval-strategy baselines, with faithfulness varying by evidence format and query type.

  • Overall comparison: 0.889 faithfulness is achieved by CLAIR-Fin, alongside the highest context precision and context recall among four retrieval-strategy baselines.Answer relevancy remains broadly comparable across the evaluated systems.
  • Presentation-format breakdown: Faithfulness ranges from 0.850 for Chart Only to 0.915 for Text + Table, indicating higher scores for combined evidence formats.The evaluation covers Text, Table, and Chart evidence formats on BB-FinQA-X.
  • Query-type breakdown: 0.920 faithfulness is attained by Fact Extraction, followed by Multi-hop Reasoning at 0.840 and Evidence Retrieval at 0.839.These results come from query-type analysis across retrieval and generation metrics.
  • Dataset composition: The 500 BB-FinQA-X items are distributed across six query categories, with overall scores computed through sample-weighted aggregation.The dataset is also distributed by presentation format, including matched Text Only/Table Only and Chart Only/Text + Chart pairs.

D Detailed Analysis of Research Questions · E Prompts

The detailed analysis finds that each CLAIR-Fin safeguard materially contributes to faithfulness, while evidence format strongly affects performance. The prompts operationalize claim decomposition, modality-aware authority, adversarial scrutiny, entailment gating, and evidence-only drafting.

  • D Detailed Analysis of Research Questions: AEA ablation lowers faithfulness from 0.889 to 0.883, context recall from 0.897 to 0.893, and exact correct answer rate from 0.592 to 0.585.AEA changes the followed modality in roughly half of scored decisions, with an impact rate of 0.515.
  • D Detailed Analysis of Research Questions: Removing the terminal audit reduces faithfulness from 0.889 to 0.845, compared with 0.889 to 0.857 without CoCV.Faithfulness rate likewise falls from 0.783 to 0.741 without the terminal audit and from 0.783 to 0.753 without CoCV.
  • D Detailed Analysis of Research Questions: Without debate, faithfulness falls from 0.889 to 0.770, exact correct answer rate from 0.592 to 0.524, and answer coverage from 0.946 to 0.896.Debate utilization is 0.646, routing nearly two-thirds of claims through debate rather than fast-pathing.
  • D Detailed Analysis of Research Questions: HRI calibration is −0.072 alongside a faithfulness rate of 0.783, while abstention appropriateness receives ratings of 4.06 and 3.95 with quadratic weighted Cohen’s κ of 0.84.The negative HRI-correctness correlation is the theoretically expected direction.
  • D Detailed Analysis of Research Questions: Text + Table achieves the highest faithfulness at 0.915, while Chart Only is lowest at 0.850; Table Only reaches 0.900 versus 0.870 for Text Only.Adding Text or Table to Chart raises faithfulness to 0.875 or 0.880, respectively.
  • E Prompts: The Planner–Orchestrator decomposes each question into 1–8 atomic, independently checkable claims, assigns one of four claim types, and uses previews only for grounded wording.Claim typing determines modality authority and escalation to debate versus fast-path judgment; unsupported numbers must not be invented.
  • E Prompts: The Adversarial Counsel cross-examines briefs for numeric, scope, fiscal-period, causal, citation, and visual-precision weaknesses, assigns severity, and recommends abstention only when revision cannot make the brief sound.Attacks must be evidence-based, and high severity is reserved for errors that make the answer wrong or unfaithful.
  • E Prompts: The shared Entailment Judge gates both CoCV and final publication by classifying evidence–hypothesis pairs as entails, neutral, or contradicts, while the Judge-Auditor drafts concise evidence-only answers using authority-ordered sources.It fails closed on truncation, requires exact support, and reserves INSUFFICIENT EVIDENCE for genuinely unanswered claims or irreconcilable same-authority disagreement.
Loading 2608.13706v1…