PaperlayerView exact revision on arXiv

Source-linked AI summary

Sustainable Hybrid Document-Routed Retrieval for Financial RAG: Resolving the Robustness-Precision Trade-off

Zhiyuan Cheng, Longying Lai, Yue Liu

arXiv:2603.26815v3cs.CLcs.AIcs.IR

TL;DR

Chunk-based retrieval suffers cross-document confusion in structurally homogeneous financial filings, while whole-document routing sacrifices targeted precision. HDRR combines document routing with scoped chunk retrieval and achieves the best performance across all four metrics, including a 7.54 average score.

  • Problem

    Structurally homogeneous financial filings make globally retrieved chunks from different companies nearly indistinguishable, while existing remedies trade accuracy against efficiency.

  • Method

    HDRR uses SFR document routing as a first-stage filter, then performs chunk retrieval scoped to the identified document or documents.

  • Results

    HDRR achieves the best performance on all four metrics simultaneously, including a highest average score of 7.54 and lowest failure rate of 6.4%.

  • Takeaways & Limitations

    Across experimental groups, document routing eliminates cross-document confusion and HDRR produces robust, consistent improvements.

  • Takeaways & Limitations

    Ambiguous or non-standard company identifiers can prevent routing, forcing full-corpus fallback and reducing performance.

Abstract

from arXiv · show

Retrieval-Augmented Generation (RAG) systems for financial document QA typically follow a chunk-based paradigm: documents are split into fragments, embedded, and retrieved by similarity. In structurally homogeneous corpora such as regulatory filings, this suffers from cross-document chunk confusion. Semantic File Routing (SFR), which uses LLM structured output to route queries to whole documents, reduces catastrophic failures but sacrifices targeted-chunk precision. We identify this robustness-precision trade-off on the FinDER benchmark (1,500 queries across five groups): SFR achieves higher average scores (6.45 vs. 6.02) and fewer failures (10.3% vs. 22.5%), while chunk-based retrieval (CBR) yields more perfect answers (13.8% vs. 8.5%). To resolve it, we propose Hybrid Document-Routed Retrieval (HDRR), a two-stage architecture that uses SFR as a document filter followed by chunk retrieval scoped to the identified document(s), eliminating cross-document confusion while preserving chunk precision. HDRR achieves the best performance on every metric: an average score of 7.54 (25.2% above CBR, 16.9% above SFR), a 6.4% failure rate, 67.7% correctness (+18.7 pp over CBR), and a 20.1% perfect-answer rate (+6.3 pp over CBR, +11.6 pp over SFR), simultaneously attaining the lowest failure rate and highest precision across all five groups. Beyond accuracy, HDRR is also the most efficient of the high-quality systems: it preserves CBR's compact per-query token budget (~5K-15K, an order of magnitude below SFR's ~50K-200K), incurs no indexing-time LLM spend (versus the one-time ~$100 cost of contextual indexing), and uses fewer per-query LLM calls than self-correcting agentic baselines, translating directly to lower API spend and inference-time energy at deployment scale.

1. Introduction

Financial document QA applies RAG to large 10-K collections, but structurally similar filings make chunk retrieval vulnerable to cross-document confusion. The paper proposes HDRR, combining document routing with scoped chunk retrieval to improve robustness, precision, and efficiency.

  • Background and Motivation: 10-K collections contain dense, standardized filings whose similar sections can make semantically relevant chunks difficult to distinguish across companies.Large filings include financial statements, MD&A, risk factors, and supplementary schedules.
  • Background and Motivation: Cross-document chunk confusion is a fundamental limitation of chunk-level retrieval in homogeneous corpora.A query about one company can retrieve analogous disclosures from another company.
  • Background and Motivation: Existing remedies trade accuracy against efficiency: whole-document routing increases token budgets, contextual indexing requires corpus-wide LLM processing, and agentic designs add verification calls.These costs make computational efficiency relevant alongside answer quality.
  • Proposed Approach: SFR resolves document identity from query metadata, while HDRR routes first and then retrieves chunks only from the identified document or documents.HDRR is designed to retain targeted-chunk precision while avoiding wrong-document candidates.
  • Proposed Approach: HDRR achieves the best reported accuracy and efficiency profile among the compared high-quality systems.It preserves CBR-level per-query context of about 5K–15K tokens and requires no indexing-time LLM spend.

2. Related Work

Related work improves RAG through retrieval, indexing, graph, and agentic strategies, but this paper focuses on document identity and scoped retrieval for local financial queries. It positions HDRR as a systematic hybrid architecture for structurally homogeneous document collections.

  • RAG and Retrieval Strategies: RAG combines retrieval from an external knowledge base with language-model generation conditioned on the retrieved context.The survey framing distinguishes indexing, retrieval, and generation strategies.
  • RAG and Retrieval Strategies: Hybrid search, rank fusion, and neural reranking refine candidate retrieval by combining sparse, dense, and cross-encoder signals.These methods improve retrieval quality but do not by themselves resolve document identity.
  • Financial Document QA: FinDER targets realistic financial inquiries over 10-K filings, while prior results show weak closed-book performance for advanced models.The benchmark contains query-evidence-answer triplets derived from real financial inquiries.
  • Contextual and Graph Retrieval: Contextual indexing enriches chunks with document-level information, but the chunk retriever still searches across the corpus.Static metadata prefixes and LLM-generated context form complementary contextual baselines.
  • Contextual and Graph Retrieval: GraphRAG is designed around corpus graphs and community traversal, whereas FinDER questions are local and require within-document retrieval.HDRR searches all chunks of the routed document, including non-adjacent sections when needed.
  • Agentic Retrieval: Agentic systems use reflection, verification, and retries to correct retrieval after the fact.The paper contrasts this post-hoc strategy with HDRR’s upfront document routing and evaluates a minimal agentic baseline.
  • Research Gap: Structured output has been used broadly, but document identity resolution as a retrieval-routing mechanism has not been systematically studied.The paper exploits few-shot classification capabilities without task-specific retraining.
  • Research Gap: Systematic evaluation of LLM-routed hybrid retrieval for RAG remains limited despite prior enterprise search work combining document- and chunk-level retrieval.The paper compares HDRR with contextual-indexing and agentic-retrieval families on the same benchmark.

3. Problem Formulation

The paper formulates financial QA over named 10-K documents and compares chunk retrieval, whole-document routing, and hybrid retrieval. Its design relies on naming regularity, structural homogeneity, and complementary evaluation metrics.

  • 3.1. Task Definition: The task is to answer a natural-language question about a specific company’s financial performance, operations, or disclosures from a financial-document corpus.The answer should accurately and completely address the query using corpus information.
  • 3.1. Task Definition: The corpus contains S&P 500 10-K reports, with each document identified by a ticker and fiscal year.Each ticker-year pair uniquely identifies one document.
  • 3.2.1. Chunk-Based Retrieval (CBR): CBR partitions each document into overlapping chunks, embeds them, retrieves top-k candidates by similarity, and generates an answer from the retrieved context.The implementation also uses full-text search, rank fusion, and neural reranking.
  • 3.2.2. Semantic File Routing (SFR): SFR extracts ticker-year metadata with structured output, resolves matching file paths, and supplies the full document or documents as generation context.SFR bypasses offline indexing and operates on whole documents.
  • 3.2.3. Hybrid Document-Routed Retrieval (HDRR): HDRR uses SFR to identify target documents, restricts chunk retrieval to those documents, and falls back to unrestricted CBR when routing fails.Answer generation uses the resulting scoped chunk context.
  • 3.2.3. Hybrid Document-Routed Retrieval (HDRR): HDRR combines immunity to cross-document confusion with the ability to retrieve precisely relevant fragments within the routed document.This is the intended robustness-precision combination of the architecture.
  • 3.3. Corpus Structure Assumption: SFR assumes naming regularity: metadata can be reliably extracted from queries and deterministically mapped to file paths.The mapping must be injective over document metadata.
  • 3.3. Corpus Structure Assumption: Structural homogeneity means documents share organizational templates while corresponding sections contain semantically similar but entity-distinct content.Financial filings exhibit both structural homogeneity and naming regularity.

4. System Architectures

The study compares three retrieval architectures under a shared GPT-4.1 generation and evaluation pipeline. CBR retrieves chunks globally, SFR routes to whole documents, and HDRR combines routing with scoped chunk retrieval.

  • Architecture Comparison: All three systems share GPT-4.1 and the same evaluation pipeline, differing in how they retrieve context.This isolates retrieval architecture as the principal system difference.
  • Architecture Comparison: CBR splits documents into overlapping fragments, embeds them in a shared space, and retrieves candidates through hybrid search and neural reranking.It follows the canonical chunk-based RAG design.
  • Architecture Comparison: SFR bypasses chunking by extracting company and fiscal-year metadata with structured output and providing the resolved full document as context.The metadata is mapped to a file path in a pre-organized repository.
  • Architecture Comparison: HDRR combines document routing with chunk retrieval so the second stage operates only over chunks from routed documents.This architecture targets both document-level robustness and chunk-level precision.

4.1. Chunk-Based RAG (CBR)

CBR indexes a corpus into complementary keyword and semantic stores, then retrieves, fuses, reranks, and generates from selected chunks. Its global embedding-space assumption can fail when structurally similar filings cause cross-document chunk confusion.

  • Pipeline: CBR uses offline corpus indexing and online retrieval-generation to answer questions from selected document chunks.The pipeline preprocesses the corpus once, then retrieves and generates at query time.
  • Retrieval: The implementation combines SQLite FTS5 keyword search with FAISS semantic search for complementary retrieval coverage.FTS5 supports keyword and BM25-based relevance ranking, while FAISS enables nearest-neighbor semantic retrieval.
  • Retrieval: Queries are rewritten into a clarified semantic query and extracted keywords before parallel full-text and semantic retrieval.The two result lists are fused with RRF, using k=60.
  • Retrieval: Neural reranking selects the final context from the fused candidate list using cumulative-probability and score-cliff cutoffs.The cumulative threshold retains chunks until normalized scores reach 55%, while score-cliff detection excludes chunks after a drop greater than 0.15.
  • Design rationale and limitations: CBR’s global embedding-space assumption breaks down in homogeneous filing collections, where similar sections from different companies become difficult to distinguish.This produces cross-document chunk confusion, such as retrieving another company’s disclosure for a company-specific query.

4.2. Semantic File Routing (SFR)

SFR routes queries directly to whole documents using structured metadata rather than offline chunk indexing. It resolves extracted ticker-year pairs to files and supplies the located documents to the generation model.

  • Pipeline: SFR eliminates offline indexing and instead relies on documents organized under a predictable directory naming convention.The corpus uses a data/{year}/{ticker}.{ext} hierarchy with approximately 500 files per year.
  • Document routing: GPT-4.1 extracts company tickers and fiscal years from each query as structured metadata.Years default to the configured year when the query does not specify one.
  • Document routing: Structured output guarantees the expected JSON schema, eliminating parsing failures during metadata extraction.The extraction result is used by the routing procedure shown in Algorithm 1.
  • Document routing: The extracted ticker-year pairs are resolved to PDF or TXT paths, with a default-year retry when no files are found.PDF is tried first, followed by TXT, and the fallback handles queries without explicit fiscal years.
  • Answer generation: Located files are encoded and sent with the original query to GPT-4.1 for answer generation over the complete document.Full-document input preserves formatted text, tables, and layout information available through file reading.

4.3. Hybrid Document-Routed Retrieval (HDRR)

HDRR combines SFR document routing with CBR chunk retrieval: it first identifies the relevant document, then performs hybrid search only within that document’s chunks. This preserves targeted context while preventing cross-document confusion.

  • Architecture: HDRR reuses CBR’s offline index and adds a document-routing stage before retrieval.The architecture requires no additional indexing.
  • Stage 1: Document routing: Stage 1 parses the query into ticker-year metadata and resolves those values to document IDs, falling back to full-corpus search if routing finds none.The fallback makes routing failure degrade to standard CBR retrieval.
  • Stage 2: Scoped chunk retrieval: Stage 2 applies query rewriting, FTS, semantic search, RRF fusion, and reranking within the routed document scope.The underlying retrieval algorithms remain unchanged; the chunk ID space is filtered before search.
  • Design rationale: Restricting retrieval to a single document, typically 50–200 chunks, eliminates cross-document chunk confusion by construction.The reranker can focus on query relevance because candidates originate from the correct document.
  • Answer generation: HDRR supplies GPT-4.1 with compact, reranked chunks from the routed document for targeted answer generation.This combines document-level routing with chunk-level precision.

4.4. Architectural Comparison

HDRR is presented as combining the strengths of SFR and CBR while avoiding their principal weaknesses. Its trade-offs are an additional routing call and a requirement for regular corpus naming.

  • Architectural comparison: HDRR eliminates cross-document confusion like SFR while avoiding CBR’s context dilution.The design aims to retain document-level accuracy and chunk-level precision in one architecture.
  • Architectural comparison: HDRR adds an LLM API call for document routing and requires regular naming conventions in the corpus.These are the stated costs of combining routing with scoped chunk retrieval.

5. Experimental Setup

The experiments compare retrieval paradigms and contextual or agentic variants on FinDER using paired, stratified query groups, deterministic GPT-4.1 generation and evaluation, and four complementary metrics. The design also tests cross-evaluator robustness while documenting a potential same-model evaluation risk.

  • Dataset and sampling: The benchmark contains mostly qualitative questions, ambiguous analyst-style queries, and evidence scattered across non-adjacent filing sections.These characteristics require narrative synthesis, domain-specific interpretation, and cross-referencing.
  • Dataset and sampling: 1,500 queries are sampled into five independent 300-query groups from FinDER, a financial QA benchmark based on S&P 500 10-K filings.The groups are sampled uniformly at random and reused across all system evaluations.
  • Experimental design: The same five query groups are used for every system, enabling paired comparisons, inter-group variance estimation, and reproducibility checks.Per-system aggregates are computed over the full 1,500-query set unless otherwise noted.
  • Systems: The compared systems include CBR, SFR, HDRR, metadata-prefix contextual indexing, LLM-generated contextual indexing, and Agentic RAG.HDRR combines SFR document routing with CBR retrieval and falls back to full-corpus chunk retrieval when routing fails.
  • Evaluation: All systems use deterministic GPT-4.1 generation, while GPT-4.1 also performs routing where applicable and evaluates answers on a 1–10 rubric.The evaluator receives the query, ground-truth answer, and generated answer, producing structured scores and justifications.
  • Evaluation: The study reports Average Score, Failure Rate, Correctness Rate, and Perfect-Answer Rate to capture quality, worst-case reliability, practical correctness, and completeness.Failure Rate counts score-1 answers, while Correctness Rate counts scores of 8 or above.
  • Evaluation caveat: Using GPT-4.1 for both generation and scoring creates a potential circular-evaluation risk, although a stratified subset is rescored with Claude Sonnet 4.6.The authors describe symmetry across systems as a mitigation and use the alternate-model check as direct evidence.

6. Results

Across five experimental groups, HDRR resolves the CBR–SFR robustness–precision trade-off, outperforming both baselines on every metric while maintaining consistent gains and efficient retrieval. Its distributional profile indicates that document-scoped chunk retrieval shifts answers toward the high-score band without SFR’s context dilution.

  • Key Findings: SFR scores higher than CBR on average, but CBR yields more high-quality and perfect answers.SFR records 6.45 versus 6.02 average score and 10.3% versus 22.5% failures; CBR records 49.0% versus 44.0% correctness and 13.8% versus 8.5% perfect answers.
  • Key Findings: HDRR dominates CBR and SFR on every metric across the five experimental groups.It achieves the best average score, failure rate, correctness rate, and perfect rate, with the result holding in every group.
  • Key Findings: HDRR reaches a 7.54 average score, 6.4% failure rate, 67.7% correctness rate, and 20.1% perfect rate.These are the best values across all four reported metrics.
  • Key Findings: Compared with CBR, HDRR raises average score by 1.52 points and correctness by 18.7 percentage points while reducing failures by 16.1 points.Perfect-answer rate also increases by 6.3 percentage points.
  • Key Findings: Compared with SFR, HDRR improves average score by 1.09 points, reduces failures by 3.9 percentage points, and increases correctness by 23.7 points.Its perfect-answer rate is also 11.6 percentage points higher than SFR’s.
  • Paired Group Analysis: HDRR’s gains are stable across groups, with lower average-score variance than CBR and SFR and preserved system rankings.Average-score standard deviations are 0.18 for HDRR, 0.24 for CBR, and 0.22 for SFR.
  • Score Distribution Analysis: HDRR shifts probability mass from both low and medium scores into the high-score band, unlike SFR’s concentration in the medium band.Over CBR, HDRR reduces the low band by 18.2 percentage points and channels 18.7 points into the high band, avoiding context dilution and residual wrong-document chunks.
  • Comparison with Baselines: Static document-identity tagging improves CBR at zero LLM cost, while contextual indexing narrows but does not close HDRR’s lead.V-A raises average score from 6.02 to 6.92, and V-B reaches 7.26 while remaining behind HDRR on average score, failures, and correctness.

7. Discussion

HDRR resolves the robustness–precision trade-off by combining document routing with scoped chunk retrieval, while remaining efficient. Its residual failures mainly arise from routing and corpus-coverage limitations, whereas some queries remain intrinsically difficult for every system.

  • Resolving the trade-off: HDRR removes cross-document noise before reranking, allowing finer-grained relevance distinctions among chunks from the correct document.Unlike reranking over the full corpus, scoped candidates can be compared without enforcing document identity after retrieval.
  • Residual failures: 6.4% failure rate leaves HDRR below SFR’s 10.3% and CBR’s 22.5%, while successful routing accounts for 93.07% of queries.Successful-routing queries average 7.79, compared with 4.26 for fallback queries.
  • Residual failures: Routing failures can degrade HDRR to CBR-equivalent full-corpus retrieval rather than complete failure, helping explain its lower aggregate failure rate than SFR.This fallback preserves a retrieval path when identifier resolution fails, but does not eliminate cross-document confusion.
  • Routing limitations: Identifier-based routing fails on ticker history, descriptive company references, ambiguous abbreviations, and tickers absent from the corpus.Rebranding requires an external ticker-history table; descriptive references could use a company-name-to-ticker lookup.
  • Residual failures: Some failures are inherent to queries requiring absent information or multi-hop reasoning across sections, regardless of retrieval strategy.Targeted retrieval cannot fully capture every such query.
  • Efficiency: HDRR’s efficiency comes from CBR-scale generation context, zero indexing-time LLM spend, and deterministic two-call queries.It uses approximately 5K–15K generation tokens, compared with SFR’s approximately 50K–200K, while contextual indexing costs approximately $100 once.
  • Efficiency: HDRR offers the strongest quality–cost profile in the deterministic-cost regime, while V-B may be preferable when per-query calls are constrained and indexing spend is acceptable.The quality comparison favors HDRR regardless of query volume, but amortized LLM cost depends on deployment scale.

8. Conclusion and Future Work

The paper concludes that HDRR resolves the robustness–precision trade-off identified between CBR and SFR, while outlining routing, multi-document, domain, confidence, and cost directions for future work.

  • Summary of Findings: HDRR achieves the best performance on all four metrics simultaneously, with a 7.54 average score, 6.4% failure rate, 67.7% correctness rate, and 20.1% perfect rate.It also outperforms CBR by 25.2% in average score, reduces failure rate by 71.6% relative to CBR, and improves perfect rate by 6.3 percentage points over CBR and 11.6 percentage points over SFR.
  • Summary of Findings: HDRR dominates both baselines in every experimental group, with low inter-group variance of 0.18 for average score.The paper reports that this demonstrates robust and consistent improvements.
  • Summary of Findings: Re-scoring a stratified 99-query subset with Claude Sonnet 4.6 preserves the top-three and bottom system rankings, with system-level Spearman ρ = 0.886.On a separate 25-query cross-company subset, HDRR attains 1.000 routing coverage and the highest retrieval coverage.
  • Future Directions for the Routing Layer: HDRR’s routing stage remains sensitive to ticker mismatches, including rebrandings, descriptive references, ambiguous abbreviations, and out-of-corpus tickers.Proposed remedies include ticker-history aliases and company-name-to-ticker dictionaries as secondary routing signals.
  • Comparative and Multi-document Queries: The main benchmark consists almost entirely of single-company queries, while a handcrafted 25-query subset evaluates comparative cross-company retrieval.The paper proposes per-routed-document reranking as a small extension for multi-document queries.
  • Future Work: Future work includes testing HDRR on legal filings, medical records, and technical documentation, whose routing challenges may require domain-specific metadata schemas.Adaptive routing confidence and combining routing with query rewriting are also proposed for reducing residual failures and API overhead.

A. System Hyperparameters

The supplied passages identify the complete configuration tables for the CBR, SFR, and HDRR systems.

  • A. System Hyperparameters: Table A1 provides the complete configuration for the Chunk-Based RAG system.
  • A. System Hyperparameters: Table A2 provides the complete configuration for the Semantic File Routing system.
  • A. System Hyperparameters: Table A3 provides the complete configuration for the Hybrid Document-Routed Retrieval system.

CRediT authorship contribution statement

The authors’ contributions span conceptualization, methodology, software, data curation, validation, visualization, and manuscript writing.

  • CRediT authorship contribution statement: Zhiyuan Cheng contributed conceptualization, methodology, software, the original draft, and visualization.
  • CRediT authorship contribution statement: Longying Lai contributed data curation, validation, and review and editing.
  • CRediT authorship contribution statement: Yue Liu contributed validation and review and editing.
Loading 2603.26815v3…