CLAIR-Fin: An Adversarial Multi-Agent Framework for Claim-Level Verification and Adaptive Debate in Cross-Modal Financial QA

1 Introduction

• Prior approaches leave gaps in claim-type-sensitive evidence trust, timely verification, measurable abstention, and coverage of non-calendar fiscal years and dense South Asian central-bank tables.Long multimodal reports also require systems to retrieve relevant content, reconcile cross-modal conflicts, and answer faithfully or decline when evidence is insufficient.Source: §1 · pp. 1–2
• CLAIR-Fin uses a nine-agent pipeline to decompose financial questions into typed atomic claims and resolve them with claim-type-aware evidence weighting.The weighting prioritizes tables for precise numerical values and narrative text for causal attribution.Source: §1 · p. 2
• Chain-of-Custody Verification restores evidence grounding before review, while the Adaptive Rebuttal Cycle escalates disputed claims with debate depth tied to unresolved findings.This is intended to prevent attribution drift from propagating and avoids a predetermined debate budget.Source: §1 · p. 2
• A Judge-Auditor applies a terminal entailment gate, records evidence-authority determinations against an equal-weight counterfactual, and uses a continuous risk index to distinguish scrutinized from unchallenged claims.The Brief Synthesizer abstains when available evidence cannot adequately support an answer.Source: §1 · p. 2
• The evaluation agenda tests modality-aware prioritization, hand-off verification, contested-claim debate, continuous risk estimation, and performance across single-modal and multimodal formats.It also asks which configurations remain most challenging.Source: §1 · p. 2

2.1 Multimodal Financial Document Understanding and Retrieval-Augmented Generation

• Multimodal financial RAG coordinates text and image agents or escalates table-and-figure retrieval, but does not adjudicate conflicting cited evidence.A reference-free protocol scores faithfulness, relevancy, and context precision/recall, yet faithfulness can reward entailment by the wrong evidence when modalities disagree.Source: §2.1 · pp. 2–3

2.2 Multi-Agent Debate and Financial Multi-Agent Systems

• Financial debate frameworks use specialist or dialectical roles, but treat debate as report-level and keep its depth fixed despite contested claims.Prior work also asks whether coordination improves outcomes relative to cost (Nguyen and Pham, 2026).Source: §2.2 · p. 3

2.3 Claim-Level Verification, Extraction Reliability, and Faithfulness Evaluation

• Claim-level methods check entailment once claims are formed, while graph verification targets open-domain claims and self-consistency assumes reproducibility signals reliability.Open-book QA depends on trustworthy extraction, and generated summaries can omit source figures despite high surface scores; reproducibility alone does not ensure correctness.Source: §2.3 · p. 3

2.4 Research Gap and Positioning of CLAIR-Fin

• CLAIR-Fin jointly conditions evidence trust on claim type, routes insufficiently covered claims to depth-scaled adversarial debate, and checks grounding at drafting-to-review hand-off.Its stated prior makes cross-modal arbitration auditable, and even self-consistent evidence remains suspect.Source: §2.4 · p. 3

3.2 Phase I: Document Ingestion; 3.4 Phase III: Multimodal Evidence Retrieval

• Offline page processing uses native extraction only for clean text without chart signals; otherwise vision extracts modalities, which dedicated agents retrieve as grounded evidence.The retrieved modality lists pass to the Ledger Guardian for fusion into the claim ledger.Source: §3.2 · p. 4, Figure 2 · p. 4, §3.4 · p. 5

3.3 Phase II: Claim Decomposition

• Decomposing answers into 1–8 atomic claims prevents unsupported figures from hiding inside a supported narrative and enables independent claim-level checking.Claims are typed and have no dependency structure; a decomposition failure falls back to one CAUSE_ATTRIBUTION claim, which may fast-path past debate.Source: §3.1 · p. 3, §3.3 · pp. 4–5

3.5 Phase IV: Evidence Fusion and Authority-Weighted Escalation

• Claim-type authority weights govern modality coverage and escalation below 0.75, but one grounded prose passage lets a causal claim fast-path without debate.The weights are fixed design priors; prose is treated as the sole authoritative modality for causal attribution, leaving causal overclaim least likely to reach adversarial review.Source: §3.5 · pp. 5–6, Table 1 · p. 5

3.6 Phase V: Adversarial Verification and Adaptive Rebuttal

• For escalated claims, adaptive rebuttal routes high-severity attacks through up to two debate rounds, so review depth and cost track claim difficulty.The adversarial scorecard covers six attack types and includes an abstain flag; even an abstain recommendation proceeds to audit.Source: §3.6 · p. 6

3.7 Phase VI: Chain-of-Custody Verification

• Chain-of-Custody Verification checks draft grounding at the hand-off to adversarial review, catching drift before the reviewer reasons over ungrounded content.It allows one bounded repair; if grounding still fails, custody is marked broken and the claim routes to audit as InsufficientEvidence.Source: §3.7 · p. 6

3.8 Phase VII: Authority-Weighted Terminal Audit

• The terminal Judge-Auditor resolves claims using claim-type authority, applies an entailment gate, and assigns passing claims StronglySupported or Supported.The Hallucination Risk Index combines entailment confidence, custody repairs, and high-severity findings, distinguishing claims that survived scrutiny from claims fast-pathed without contest.Source: §3.8 · p. 6

3.9 Phase VIII: Answer Synthesis and Selective Abstention

• The synthesizer rephrases and reorders only resolved claim lines, retaining citations and abstaining only when no claim passed audit.A supported claim can still yield a partial substantive answer when other claims fail; if rephrasing drops a citation label, synthesis falls back to raw concatenation.Source: §3.9 · pp. 6–7

• BB-FinQA-X contains 500 multimodal financial QA questions grounded in the Bangladesh Bank Annual Report.Source: §4 · p. 7
• Appendix A provides detailed dataset construction, annotation, validation, and distribution statistics.Source: §4 · p. 7

5 Experimental Configuration

• Appendix B provides the detailed implementation settings, experimental configurations, and evaluation protocols.Source: §5 · p. 7

6 Results and Discussion

• The evaluation spans automatic retrieval and generation metrics by query type and format, framework-specific metrics, two-annotator human evaluation, and ablations against the full system and single-pass RAG.The reported axes use Tables 8, 10, and 11; Table 7; Table 9; and Tables 2 and 3, respectively.Source: §6 · p. 7

6.1 RQ1: Modality-Aware Evidence Prioritization

• AEA removal causes the smallest but consistent ablation degradation: faithfulness falls 0.889 →0.883, context recall 0.897 →0.893, and exact correctness 0.592 →0.585.Its impact rate is 0.515, indicating the asymmetric prior changes the winning modality in roughly half of contested decisions.Source: §6.1 · p. 7

6.2 RQ2: Verification at the Drafting-to-Review Hand-off

• Removing the terminal entailment audit lowers faithfulness more than removing CoCV (0.889 →0.845 vs. 0.889 →0.857), with the same ordering for faithfulness rate (0.783 →0.741 vs. 0.783 →0.753).The terminal gate carries more of the faithfulness guarantee than any single upstream check, while CoCV contributes independently at the drafting-to-review hand-off.Source: §6.2 · p. 7

6.3 RQ3: Adaptive Allocation of Adversarial Debate

• Removing adaptive debate causes the largest ablation degradation: faithfulness falls 0.889 →0.770, exact correctness 0.592 →0.524, and answer coverage 0.946 →0.896.Debate utilization is 0.646; debate is reserved for claims below the fast-path coverage threshold, so removing it eliminates their only verification opportunity.Source: §6.3 · p. 8

6.4 RQ4: Continuous Risk Estimation versus Binary Gating

• HRI correlates negatively with correctness (r = −0.072), while human-rated abstention appropriateness is 4.06/3.95 (κ = 0.84).The modest correlation suggests HRI adds information beyond the binary gate; it is directionally consistent and human-corroborated, but not formally calibrated as a probability.Source: §6.4 · p. 8

6.5 RQ5: Sensitivity to Evidence Presentation Format

• Text + Table has the highest faithfulness (0.915), while Chart Only has the lowest (0.850); Evidence Retrieval (0.839) and Multi-hop Reasoning (0.840) are the lowest-scoring query types.Matched-content comparisons favor Table Only over Text Only by +0.030 and Text+Chart over Chart Only by +0.025; combined formats outperform their weakest constituent modality.Source: §6.5 · p. 8

7 Conclusion

• On BB-FinQA-X, CLAIR-Fin raises faithfulness over Vanilla RAG from 0.780 to 0.889; removing ARC produces the largest ablation drop, from 0.889 to 0.770.The paper reports that ablation confirms all four mechanisms are non-redundant.Source: §7 · p. 8
• CLAIR-Fin’s nine-agent typed Financial Claim Ledger conditions evidence trust on claim type, checks grounding at the drafting-to-review hand-off, adaptively debates contested claims, and pairs terminal audit with continuous Hallucination Risk Index.These mechanisms are Asymmetric Evidence Authority (AEA), Chain-of-Custody Verification (CoCV), and the Adaptive Rebuttal Cycle (ARC).Source: §7 · p. 8
• Future work will replace substring matching with a semantic cross-modal evidence matcher, test generalization across institutions, languages, and models, and evaluate multi-turn financial analyst settings.Source: §7 · p. 8

Dataset Scope

• BB-FinQA-X covers one institution, one English-language report, and one central-bank reporting convention, limiting transfer claims across document families.Its 500 questions are stratified by query type, format, and difficulty, but all come from the same source corpus; cross-modal conflict, extraction reliability, and abstention findings need validation on other sources, conventions, and languages.Source: Limitations · pp. 8–9

Methodological Constraints

• Fixed design choices, lexical cross-modal linking, and LLM-as-judge checks constrain methodological reliability and the degree of claim resolution.AEA and HRI weights and escalation and entailment thresholds are hand-specified; lexical overlap can miss equivalent references or over-link coincidental matches, and the same judge bias may affect both checkpoints. Debate allows at most two rebuttal rounds and grounding repair one attempt.Source: Limitations · p. 9

Evaluation Scope

• Evaluation does not quantify accuracy-cost trade-offs or establish that adaptive debate outperforms a fixed debate budget.Automatic metrics use an LLM judge, while human evaluation uses two annotators without formal inter-rater sampling; ablations isolate mechanisms individually, omit some combinations, and do not measure latency or added LLM calls. The w/o ARC comparison tests debate versus no debate, not adaptive allocation versus uniformly applying ρmax = 2 rounds; robustness to adversarial or out-of-distribution questions and corrupted PDFs is also untested.Source: Limitations · p. 9

Generalizability and Future Extensions

• The financial-reporting taxonomy and authority weights were not designed for other high-stakes domains, and evaluation covers only English, one model family, and offline per-question use.Future work includes adapting the architecture to other domains, testing multilingual use and whether weights and thresholds transfer across models, and studying calibration drift, user trust, and analyst-workflow integration in deployed multi-turn settings.Source: Limitations · p. 9

Ethics Statement

• The authors caution that CLAIR-Fin is limited to the evaluated single-institution setting and should support, not replace, domain-literate financial or regulatory judgment.Low-confidence and abstained claims warrant review by a domain-literate user.Source: Ethics Statement · p. 9
• BB-FinQA-X uses aggregate statistics from a public Bangladesh Bank report, contains no personal information, and involved no human subjects or formal institutional ethics approval.The authors report no preprocessing beyond the extraction pipeline before annotation.Source: Ethics Statement · pp. 9–10
• The authors constructed and independently reviewed the dataset, while two unpaid external banking-sector experts performed domain validation and blind evaluation of system outputs.The experts were not employed by Bangladesh Bank.Source: Ethics Statement · p. 10
• The fixed, hand-specified evidence-authority prior may be wrong, while inherited language-model bias and LLM-mediated retrieval and entailment judgments can also affect support decisions.The Authority Docket logs decisions against an equal-weight counterfactual, making the prior’s influence auditable.Source: Ethics Statement · p. 10
• Passing audit verdicts and low Hallucination Risk Index scores are not correctness guarantees, and overreliance and automation bias remain risks, especially beyond the evaluated setting.The audit indicates grounding, not external factual correctness; performance and abstention may differ across settings.Source: Ethics Statement · p. 10
• Safeguards include citations for published claims, hand-off grounding checks, a terminal entailment gate, selective abstention, and auditable evidence arbitration.The paper also describes blind evaluation by two external domain experts as an independent check on outputs.Source: Ethics Statement · p. 10
• Future work should test evidence-authority weights and thresholds against human-labeled ground truth, evaluate other institutions, languages, and reporting conventions, and study analyst workflows and automation bias.The paper calls for broader human-centered evaluation with domain experts and end users.Source: Ethics Statement · pp. 10–11

A.1 Evaluation Protocol

• A four-way outcome harness evaluates answers and abstentions against gold references, alongside automatic, correctness, coverage, debate, and risk-calibration measures.RAGAS metrics are judged by GPT-4.1 mini, while GPT-4o assigns the outcome label; two external banking experts also rate all 500 answers across six dimensions, blind to confidence/HRI. Ablations disable AEA, CoCV, ARC, or the terminal entailment audit, with single-pass RAG as a baseline.Source: §A.1 · pp. 12–13

A.2 Data Sources and Selection Criteria

• Questions were sourced from Bangladesh Bank’s Annual Report, selecting self-contained material where narrative, tables, and charts describe shared indicators.The source is a publicly available central-bank publication used for non-commercial academic research; annotators worked from its original text, tables, and charts without automated preprocessing.Source: §A.2 · p. 13

A.3 Dataset Construction Pipeline

• Dataset construction selected cross-modal sources, drafted evidence-linked questions and gold answers, validated items, and quality-checked the final 500-item release.Each draft recorded its supporting text passage, table cell, or chart element and preliminary query-type, difficulty, and format labels; the final pass checked target balance across all three dimensions.Source: §A.3 · p. 13

A.4 Manual Annotation Protocol

• Four annotators developed and validated the dataset through author drafting and review, external banking-expert validation, and joint consensus resolution of flagged items.The domain experts reviewed financial correctness, terminology, numerical accuracy, source faithfulness, practical relevance, and difficulty consistency; flagged items were jointly resolved rather than decided by majority vote or one adjudicator.Source: §A.4 · pp. 13–14

A.5 Annotation Schema and Guidelines

• Items are labeled by six query types, three difficulty levels, and six evidence presentation formats, including cross-location or cross-modal multi-hop reasoning.Evidence Retrieval asks respondents to identify and ground the relevant evidence itself, rather than relying on a pre-identified passage or cell.Source: §A.5 · p. 14

A.6 Quality Control and Validation

• Quality control checks correctness, grounding, ambiguity, duplication, numerical accuracy, and all three annotation labels; failing items are revised or discarded before release.A final pass confirmed consistency across all 500 items and the target distribution across query type, difficulty, and format.Source: §A.6 · pp. 14–15

A.7 Dataset Statistics

• BB-FinQA-X contains 500 questions spanning six query types, three difficulties, and six formats, with every source-occurring combination represented.The set includes 175 Easy, 250 Medium, and 75 Hard items; counts range from 150 Fact Extraction questions to 40 Evidence Retrieval questions, and from 150 Text + Table items to 50 each for Chart Only, Text + Chart, and Table + Chart.Source: §A.7 · p. 15

B Experimental Setup

• The experimental setup specifies the implementation stack, ingestion pipeline, and inference configuration used for the reported experiments, with code, prompts, and configurations slated for release.The setup distinguishes each mechanism’s computation from the library or model used to instantiate it.Source: B Experimental Setup · p. 15

B.1 Implementation Framework and Multi-Agent Orchestration

• CLAIR-Fin is a Python LangGraph StateGraph with nine agents sharing typed state; conditional routing implements escalation and rebuttal, while custody verification reuses the Judge-Auditor rather than adding a tenth agent.Typed-output calls use Pydantic schemas with a safe-default fallback on parse failure.Source: §B.1 · pp. 15–16

B.2 Document Ingestion and Chunking

• PDFs are parsed page by page and also rendered for vision-LLM table and chart extraction; narrative chunks target 1,000 characters and overlap by the previous two sentences.Sentence-aware splitting protects common abbreviations and keeps chunk boundaries at sentence ends; tables and chart descriptions are embedded separately from narrative chunks.Source: §B.2 · p. 16

B.3 Embedding Model and Vector Database

• All modalities use 3,072-dimensional text-embedding-3-large vectors in Milvus Lite, with metadata enabling modality-filtered retrieval from one collection.Queries share the same vector space; similarity search uses Milvus’s default L2 distance index.Source: §B.3 · p. 16

B.4 Retrieval Configuration

• Retrieval reranks a 4k vector-search pool using normalized similarity at weight 0.65 and lexical overlap at 0.35, returning top-k passages with agent-specific widths.The lexical component addresses similarly embedded but lexically distinct aggregates; Narrative, Tabular, and Visual agents use k=8, k=6, and k=5, respectively.Source: §B.4 · pp. 16–17

B.5 Language Models and Inference Settings

• All nine agents and ingestion-time vision extraction use GPT-4o, with call-site-specific temperatures and per-agent output-token caps.Temperature is 0 for claim decomposition and the terminal audit draft, and 0.2 for drafting, rebuttal, and answer composition; example caps are 700 tokens for counsel calls, 400 for audit, and 300 for entailment checks.Source: §B.5 · p. 17

B.6 Evaluation Tooling and Runtime Environment

• Evaluation combines retrieval and generation metrics with ledger-derived correctness, coverage, authority impact, debate utilization, and HRI calibration; callbacks track API costs.The pipeline is API-based, with Milvus Lite as its only local service.Source: §B.6 · p. 17

B.7 Configuration Management

• Mechanism thresholds and weights are externally configurable through shared settings and YAML files, so ablations change a setting or routing edge rather than the code path.Configured values include the 0.75 coverage-escalation cutoff, ARC round cap ρmax=2, and terminal entailment pass bar of 0.5.Source: §B.7 · pp. 17–18

Supplementary Results

• 0.889 faithfulness gives CLAIR-Fin the strongest overall performance across four retrieval-strategy baselines, with highest context precision and recall and broadly comparable answer relevancy.The comparison is on BB-FinQA-X across the evaluated RAGAS metrics.Source: Figure 5 · p. 19
• 0.915 faithfulness for Text + Table is the highest reported format result, compared with 0.850 for Chart Only.Combined formats improve faithfulness across Text, Table, and Chart evidence formats.Source: Figure 6 · p. 19
• 0.920 faithfulness for Fact Extraction exceeds Multi-hop Reasoning at 0.840 and Evidence Retrieval at 0.839.Table 11 reports metrics for six query categories and calculates the overall score using sample-weighted aggregation.Source: Figure 7 · p. 20, Table 11 · p. 20
• BB-FinQA-X’s 500 items are distributed by query type, presentation format, and difficulty, with Text Only/Table Only and Chart Only/Text + Chart matched by design.Text + Table and Table + Chart were constructed independently.Source: Table 12 · p. 20

RQ1: Modality-Aware Evidence Prioritization

• 0.515 AEA impact rate indicates that asymmetric evidence weighting changes the selected modality in about half of scored decisions, alongside improved faithfulness and context recall.Disabling AEA lowers faithfulness (0.889 →0.883) and context recall (0.897 → 0.893); exact correct answer rate falls (0.592 →0.585).Source: RQ1: Modality-Aware Evidence… · p. 21

RQ2: Verification at the Drafting-to-Review Hand-off

• 0.889 →0.845 faithfulness without the terminal audit is a larger decline than 0.889 →0.857 without CoCV, while neither ablation matches full-system performance.The terminal audit is a mandatory final check, whereas CoCV checks grounding at the drafting-to-review hand-off; the remaining CoCV cost indicates the checks are complementary.Source: RQ2: Verification at the… · p. 21

RQ3: Adaptive Allocation of Adversarial Debate

• 0.889 →0.770 faithfulness without debate is the largest single ablation drop; debate is routed to claims with coverage below the fast-path threshold.Debate utilization is 0.646, and removing debate also lowers exact correct answer rate (0.592 →0.524) and answer coverage (0.946 →0.896). The result supports targeted debate on contested claims, not the untested claim that debating every claim would improve performance.Source: RQ3: Adaptive Allocation of… · p. 22

RQ4: Continuous Risk Estimation versus Binary Gating

• −0.072 HRI calibration accompanies a faithfulness rate of 0.783; human-rated abstention appropriateness is 4.06 and 3.95, with quadratic weighted Cohen’s κ of 0.84.The negative correlation is directionally consistent with higher predicted risk accompanying lower correctness, but it is not a formal probability-calibration validation.Source: RQ4: Continuous Risk Estimation versus… · p. 22

RQ5: Sensitivity to Evidence Presentation Format

• 0.915 faithfulness for Text + Table exceeds 0.850 for Chart Only; Table Only scores 0.900, Text Only 0.870, Text + Chart 0.875, and Table + Chart 0.880.Evidence Retrieval (0.839) and Multi-hop Reasoning (0.840) are the lowest-scoring query types, while Fact Extraction is highest (0.920); Numerical Calculation scores 0.865. Combined formats outperform single modalities, but chart-dependent and evidence-synthesis or grounding tasks remain difficult.Source: RQ5: Sensitivity to Evidence… · p. 23

E Prompts

• The Planner–Orchestrator decomposes each question into 1–8 atomic, typed claims and uses source previews to word them accurately without inventing unsupported figures.Claim type determines evidence authority and escalation; the preview is for wording only, and its absence of information is not proof that the source lacks it.Source: E Prompts · p. 24
• The Adversarial Counsel cross-examines briefs against evidence, assigns severity to real weaknesses, and recommends abstention only when no reasonable revision could fix the grounding problem.The prompt directs attacks at issues such as numeric mismatch, scope, fiscal-year confusion, causal overclaim, and citation gaps.Source: E Prompts · p. 25
• The Judge-Auditor drafts independently from raw evidence, then gates publication on entailment and orders evidence by claim-specific authority.When sources differ in precision, it prefers the higher-authority evidence and distinguishes percentage-point change from relative growth; failed entailment leads to abstention rather than publication.Source: E Prompts · pp. 26–27