PaperlayerView exact revision on arXiv

Source-linked AI summary

FinCARDS: Card-Based Analyst Reranking for Financial Document Question Answering

Yixi Zhou, Fan Zhang, Yu Chen, Haipeng Zhang, Preslav Nakov, Zhuohan Xie

arXiv:2601.06992v2cs.IRcs.AIcs.CL

TL;DR

Financial QA over long corporate filings requires evidence that satisfies entity, metric, period, and numeric constraints, while conventional LLM rerankers can be unstable and opaque. FINCARDS represents questions and filing chunks with aligned finance-aware fields and applies staged tournament reranking with stability-aware procedures. Across its evaluation, it improves retrieval over lexical and zero-shot LLM baselines without task-specific fine-tuning, while its multi-document scope remains unevaluated.

  • Problem

    Financial QA over long corporate filings requires evidence matching entities, metrics, fiscal periods, and numeric values, but generic LLM rerankers emphasize semantic relevance and can be opaque or unstable.

  • Method

    FINCARDS maps filing chunks and questions into aligned structured cards and intent specifications, then uses staged reranking under explicit constraints with auditable traces.

  • Results

    FINCARDS consistently outperforms lexical baselines and strong zero-shot LLM rerankers while reducing candidate sets, systematic errors, and ranking variance.

  • Takeaways & Limitations

    Explicit intermediate representations and stability-aware ranking improve reliability for intra-document financial evidence selection without task-specific fine-tuning.

  • Takeaways & Limitations

    The study evaluates only intra-document retrieval within a single SEC filing; its tournament design has not been evaluated for multi-document or heterogeneous-source analysis.

Abstract

from arXiv · show

Financial question answering (QA) over long corporate filings requires evidence to satisfy strict constraints on entities, financial metrics, fiscal periods, and numeric values. However, existing LLM-based rerankers primarily optimize semantic relevance, leading to unstable rankings and opaque decisions on long documents. We propose FinCards, a structured reranking framework that reframes financial evidence selection as constraint satisfaction under a finance-aware schema. FinCards represents filing chunks and questions using aligned schema fields (entities, metrics, periods, and numeric spans), enabling deterministic field-level matching. Evidence is selected via a multi-stage tournament reranking with stability-aware aggregation, producing auditable decision traces. Across two corporate filing QA benchmarks, FinCards substantially improves early-rank retrieval over both lexical and LLM-based reranking baselines, while reducing ranking variance, without requiring model fine-tuning or unpredictable inference budgets. Our code is available at https://github.com/XanderZhou2022/FINCARDS.

1 Introduction

Financial QA over long corporate filings is best treated as constrained evidence reranking rather than generic retrieval. FINCARDS addresses this by aligning structured financial fields and progressively reranking candidates to produce stable, auditable evidence selections.

  • Motivation: Financial QA evidence must jointly match the queried metric, fiscal period, entity, and often an explicit numerical value.These signals are sparsely distributed across long filings containing boilerplate and recurring disclosures.
  • Motivation: Large-batch LLM reranking faces context-scale, latency, and opacity problems in multi-hundred-page corporate filings.These limitations can yield wrong-period, metric-misaligned, or temporally ambiguous evidence.
  • Approach: The framework models analyst reasoning as progressive screening, global ordering, and adjudication under a shared alignment contract.Document chunks become structured cards, while questions become intent specifications over the same fields.
  • Scope: FINCARDS reformulates long-filing financial QA as intra-document evidence reranking under numeric, temporal, and entity constraints.The setting assumes a single pre-selected filing and focuses on locating grounded evidence before generation.
  • Approach: FINCARDS maps document chunks and questions to explicit financial fields, enabling deterministic, interpretable alignment and auditable decision traces.The representation shifts ranking away from monolithic semantic relevance toward explicit constraint satisfaction.
  • Outcome: The zero-shot tournament-style pipeline produces stable ranked evidence sets and improves early precision over strong baselines.The introduction presents this as a central contribution without requiring task-specific fine-tuning.

2 Related Work

Related work spans structured financial QA benchmarks, lexical and LLM-based reranking, agentic reasoning, model-based evaluation, and retrieval-augmented systems. These strands motivate evidence-sensitive retrieval but leave domain-specific constraint satisfaction as a distinct focus.

  • Financial QA benchmarks: Financial QA benchmarks progressed from locally contained classification and extraction toward structured numerical, hybrid text-table, conversational, and verification tasks.FinQA and TAT-QA formalized numerical and hybrid reasoning, while ConvFinQA and FinChain extended conversational and intermediate-step verification settings.
  • Retrieval and LLM reranking: Retrieval systems range from BM25 to dense retrievers, crossencoders, and zero-shot LLM rerankers such as RankGPT and LRL.Listwise prompting can be input-order sensitive and context-length constrained, whereas pairwise prompting improves calibration and stability.
  • Agentic reasoning: Agentic reasoning methods use external actions, structured search, self-critique, or aggregated reasoning paths to improve decision processes.The cited approaches primarily target generation and problem solving rather than evidence selection under domain-specific constraints.
  • Evaluation and document understanding: Model-based evaluation research highlights variance and calibration challenges, while layout-aware and hybrid-reasoning benchmarks address visually structured financial documents.LayoutLMv3, ChartQA, and TAT-QA represent complementary efforts around document understanding and textual-tabular reasoning.
  • RAG-based systems: RAG surveys identify retrieval quality as a frequent performance bottleneck, motivating retrieval-aware reasoning, adaptive retrieval, and multi-agent information planning.These approaches dynamically decide what to retrieve or how to use retrieved evidence, but the passage does not specialize them to FINCARDS-style financial constraints.

3 FINCARDS

FINCARDS reranks evidence within a single SEC filing by converting chunks and questions into aligned structured representations, then progressively filtering and ordering candidates under financial constraints. The pipeline records field matches and ranking decisions for auditability.

  • Problem: FINCARDS targets intra-document reranking so top-k chunks satisfy required metrics, fiscal periods, entities, and often explicit numbers.The setting assumes a pre-selected filing and focuses on locating grounded evidence before generation.
  • Card Abstraction: Card abstraction converts filing chunks into compact records containing finance-relevant fields for auditable matching.Cards encode entities, metrics, periods, numeric spans, section cues, and auxiliary screening signals.
  • Auditability: The pipeline produces an audit trace for each selected chunk, recording matched Card fields, stagewise retention or filtering, and final-rank determination.This supports inspection of how evidence was selected rather than providing only a final ordering.
  • Query Intent Mapping: Query intent mapping represents entities, metrics, relations, temporal constraints, numeric requirements, and keywords for direct alignment with Cards.This decomposition addresses underspecified financial questions by exposing the dimensions needed for matching.
  • Tournament Reranking: The tournament reranker uses three stages: recall-oriented candidate generation, Card-based semantic filtering, and stability-aware listwise aggregation.Its staged procedure mirrors progressive analyst-style narrowing, ordering, and adjudication under numeric and temporal constraints.
  • Tournament Reranking: Stage 1 applies BM25 within the filing and retains a length-adaptive top-N candidate set for downstream reranking.The cutoff uses r = 0.5, Nmin = 60, and Nmax = 150, with all chunks retained when the filing is shorter than Nmin.

4 Experiments and Evaluation

The evaluation tests FINCARDS for intra-document evidence ranking on SEC-filing QA, using controlled comparisons, early-rank retrieval metrics, and component ablations. The full pipeline improves accuracy over lexical and LLM-based baselines while reducing candidate sets and ranking instability.

  • Experimental setup: The evaluation uses FinAgentBench SEC-filing questions, with each system ranking evidence chunks from one document under a controlled inference protocol.Systems share document chunking and evaluation conditions, with deterministic decoding and structured outputs.
  • Systems: The comparison includes BM25 retrieval, zero-shot LLM reranking, and staged FINCARDS variants using common candidate pools.The full pipeline combines lexical retrieval, Card-based filtering, and bootstrap listwise stabilization.
  • Main results: The full three-stage pipeline substantially improves early-rank retrieval over traditional and LLM-based baselines across nDCG@10, MAP@10, and MRR@10.Compared with Stage 1, it improves nDCG@10 by over 27 points and MRR@10 by nearly 20 points; versus zero-shot LLM reranking, nDCG@10 rises by +15.8.
  • Main results: Accuracy gains accompany candidate reduction from roughly 100 chunks to fewer than 25, indicating improved ranking quality and retrieval efficiency.The reported metrics are evaluated at rank 10 because relevant evidence sets are typically small.
  • Component analysis: Card-based filtering addresses temporal mismatch, scope ambiguity, and boilerplate interference more effectively than unstructured LLM judgments.Adding Card-based filtering yields +7.9 nDCG@10 over zero-shot reranking, while removing temporal information causes the largest Stage 2 degradation.
  • Component analysis: Bootstrap aggregation further improves early-rank metrics and reduces ranking variance, while random regrouping outperforms fixed grouping.The analysis attributes these gains to stabilization across comparison contexts; rankings typically converge within 3–4 rounds.
  • Conclusion: The results support structured reasoning, progressive filtering, and stability-aware aggregation without task-specific fine-tuning or additional supervision.These design principles remain robust across model backbones and evaluation conditions.
  • Robustness: Across diverse LLM backbones, Stage 2 and Stage 3 provide consistent improvements, indicating that the framework is model-agnostic.The reported robustness is attributed to the structured intermediate representation and bootstrap aggregation rather than backbone-specific capability.

5 Conclusions and Future Work

FINCARDS replaces monolithic relevance ranking with structured, tournament-style evidence selection that enforces financial constraints. Experiments report consistent gains over lexical and zero-shot LLM rerankers, while performance can degrade on complex reasoning and the framework remains limited to single-document retrieval.

  • FINCARDS is a zero-shot intra-document reranking framework that combines Card abstractions with staged enforcement of metric, temporal, and scope constraints.
  • Across extensive FinAgentBench experiments, FINCARDS consistently outperforms lexical baselines and strong zero-shot LLM rerankers while progressively reducing candidate-set size.The reported gains do not rely on task-specific fine-tuning.
  • By making constraints explicit and decomposing ranking into recall, alignment, and stabilization, FINCARDS reduces systematic errors and ranking variance in long, noisy documents.
  • Performance degrades on queries requiring implicit reasoning, cross-chunk aggregation, or causal interpretation, where structured attributes do not fully capture the reasoning required.
  • Future work targets adaptive computation budgets, cross-document evidence selection, and richer Card schemas for complex reasoning and downstream answer generation.

Limitations

The framework has practical limitations involving computational cost, evaluation scope, and sensitivity to prompt or schema choices. These constraints bound scalability, generalization to multi-document settings, and robustness under changing configurations.

  • The multi-stage pipeline incurs non-trivial computational cost because listwise and bootstrap-based ranking require multiple LLM calls per query.Candidate compression and early stopping mitigate the cost, but efficiency remains important.
  • The study evaluates only intra-document retrieval within a single SEC filing, leaving effectiveness in multi-document settings unevaluated.Real-world analysis may require combining filings with press releases or earnings calls.
  • The approach may remain sensitive to prompt design and schema choices despite deterministic decoding and structured outputs.The authors identify robustness under prompt variation and evolving model behavior as requiring further study.

Ethical Considerations

The work uses publicly available textual disclosures to improve evidence selection and interpretability in high-stakes financial analysis. It is intended to assist analysts, with human oversight needed to reduce risks from automation and evidence misinterpretation.

  • The framework operates on publicly available regulatory filings and does not involve personal data, user profiling, or sensitive individual information.
  • Structured intermediate representations and transparent ranking procedures are intended to support more responsible use of LLMs in financial decision support.
  • The system is intended as an assistive tool for analysts rather than a replacement for professional judgment because over-reliance and evidence misinterpretation are potential risks.
  • Experiments use publicly available datasets derived from U.S. SEC filings, whose underlying 10-K and 10-Q documents are freely accessible for research and academic purposes.

A Candidate Set and Evaluation Fairness

The evaluation controls candidate-pool comparability by giving reranking methods the same BM25-retrieved inputs, while the pipeline uses structured Cards, intent specifications, staged filtering, and stability-oriented ranking. Stage-wise traces and error labels support analysis of both retrieval quality and failure modes.

  • Candidate Set and Evaluation Fairness: All reranking systems operate on identical Stage 1 BM25 candidate pools, so reported differences reflect reranking quality rather than candidate-recall advantages.The pool contains the top-N chunks, with N varying from 60 to 150 according to document length.
  • Stage 1: Structured Representations: Chunk Cards encode evidence role, temporal anchoring, scope, and verifiability signals for downstream alignment.
  • Stage 1: Structured Representations: Query Intents encode topical focus, requested metrics, temporal constraints, and relational form as requirements for alignment with Chunk Cards.
  • Stage 2: Batch Selection: Stage 2 performs group-wise evidence selection using only Card-level information, filtering the Stage 1 candidate pool without external retrieval scores.
  • Stage 3: Listwise Ranking: Stage 3 produces a complete ordinal ordering from filtered candidates, avoiding absolute scores and aggregating stable judgments across multiple rounds.
  • Error Analysis: The error study examines six representative success and failure cases across stages, labeling lexical mismatch, temporal misalignment, schema gaps, and implicit reasoning.The taxonomy distinguishes quantitative lookup, trend/comparison, and qualitative-impact query intents.

C.3 Findings

The error study shows that FinCARDS recovers evidence when financial intent is expressible through explicit schema constraints, while failures concentrate in lexical retrieval and implicit reasoning gaps. Bootstrap aggregation stabilizes rankings but cannot repair structurally misaligned candidates.

  • Lexical retrieval failures dominate early-stage errors, with BM25 sometimes retrieving no gold evidence and yielding zero nDCG@10.Abbreviations, paraphrases, and implicit temporal constraints create mismatches that term-based retrieval cannot resolve.
  • Card-based alignment substantially improves quantitative lookup retrieval by matching financial metrics, temporal annotations, and structured tables.Gold chunks can be promoted even when Stage 1 recall is zero.
  • Card-based methods degrade predictably on implicit or explanatory queries because the schema lacks fields for implicit reasoning, causal effects, and cross-section synthesis.These gaps can cause over-selection of superficially related but nonanswering chunks.
  • Bootstrap reranking reduces rank variance and stabilizes Top-K sets when Stage 2 candidates are well aligned, but reinforces errors when candidates are structurally mismatched.Stage 3 therefore acts primarily as a stabilizer rather than a corrective mechanism.
  • Overall, the pipeline is most effective when query intent decomposes into explicit constraints such as metric, time, scope, and evidence type.Failures mainly reflect schema expressiveness limits rather than ranking noise or model stochasticity.

D Rank Variance Definition

The paper defines rank variance as the average across queries of per-query retrieval-score variance over five independently randomized runs. Although named rank variance, the metric captures score-based ranking stability, and the Single Round setting still uses five runs.

  • Each configuration uses five independent runs with different random seeds, shuffling group assignments and stochastic components such as bootstrap grouping.This setup supplies the repeated-run measurements used for the variance calculation.
  • For each query, the paper computes variance over retrieval scores such as nDCG@10 across the five runs.The reported quantity is based on score variability rather than directly on rank positions.
  • The reported rank variance averages the per-query variance across all evaluation queries.Q denotes the set of evaluation queries.
  • Despite its name, rank variance measures retrieval-score variability across runs, reflecting stability and consistency under different random seeds and evaluation conditions.This interpretation follows the paper’s clarification of the metric.
  • The Single Round (R=1) setting still estimates variance across five independent runs rather than executing the pipeline only once.The paper gives 0.0856 as an example of the resulting cross-run estimate.

E Token Cost

FinCARDS replaces one very long-context reranking call with multiple bounded calls over compact Card representations. This yields predictable per-query resource use while improving early-rank quality.

  • A typical filing has approximately 300 chunks, making top-100 listwise reranking require approximately 75k input tokens in one call.Prompt overhead raises the estimate to approximately 78k tokens.
  • FinCARDS uses approximately 150–200-token Cards, with Stage 2 processing groups of 25 and Stage 3 operating on a reduced pool.The stages require approximately 22k and 30k total tokens, respectively, under the described configuration.
  • Approximately 50k tokens and 15–20 LLM calls per query replace one extremely long-context call, using bounded 4k–6k-token calls.The design supports predictable memory usage and parallel execution while improving early-rank quality.

F Stronger Retrieval and Reranking Baselines

Against dense, hybrid, and cross-encoder baselines evaluated on the same Stage 1 candidate pool, FinCARDS consistently improves retrieval and reranking performance. Its gains include both ranking quality and evidence coverage.

  • The evaluation compares E5-base-v2 dense retrieval, BM25–dense hybrid retrieval, and a cross-encoder reranker.The baselines represent semantic, combined lexical-semantic, and fine-grained neural relevance modeling.
  • All rerankers use the identical Stage 1 candidate pool, isolating differences in reranking quality from candidate-recall differences.This provides a controlled comparison of ranking effects.
  • FinCARDS consistently outperforms dense, hybrid, and cross-encoder baselines across metrics at both @5 and @10.The comparison is conducted under a unified evaluation protocol.
  • nDCG@10 increases from 0.5548 with the cross-encoder to 0.7652 with FinCARDS, a gain of +0.2104.This reports the stated cross-encoder comparison for ranking quality.
  • Recall@10 increases from 0.4923 with the cross-encoder to 0.7242 with FinCARDS, a gain of +0.2319.Similar improvements are also reported at @5, indicating gains beyond deeper ranks.
Loading 2601.06992v2…