Sustainable Hybrid Document-Routed Retrieval for Financial RAG: Resolving the Robustness-Precision Trade-off

1.1. Background and Motivation

• In homogeneous 10-K corpora, similar regulatory sections can make chunk retrieval return another company’s filing, a failure the paper terms cross-document chunk confusion.Routing whole documents avoids that confusion but increases per-query token budgets; contextual indexing adds one-time LLM processing, while self-correcting agents make multiple LLM calls per query. These costs affect deployment-time spend and inference energy.Source: §1.1 · pp. 1–2

1.2. Proposed Approach

• HDRR first routes a query to document identity, then retrieves chunks only from the identified document(s), combining document-level robustness with targeted-chunk precision.If routing fails, HDRR gracefully falls back to full-corpus search. Its per-query generation context stays at CBR’s level (∼5K–15K tokens), and it requires no indexing-time LLM spend.Source: §1.2 · p. 2

1.3. Contributions

• On 1,500 FinDER queries across five groups, the paper reports that HDRR achieves the best performance on all four evaluation metrics, with the lowest failure rate and highest answer precision.The paper also compares CBR, SFR, and HDRR on identical query sets, evaluates costs against five comparison systems, and formalizes naming regularity and structural homogeneity as conditions for routing’s benefit.Source: §1.3 · p. 2

2.1–2.2. Retrieval-Augmented Generation; Hybrid Retrieval and Reranking

• RAG retrieves external passages to condition generation, and this work extends its indexing, retrieval, and generation taxonomy with retrieval granularity: chunk-level retrieval versus document-level routing.Hybrid search, rank fusion, and neural reranking refine candidate retrieval; prior work reports a 15.5 percentage point correctness gain from neural reranking in financial RAG.Source: §2.1 · p. 3, §2.2 · p. 3

2.3. Financial Document Analysis

• FinDER contains 5,703 query-evidence-answer triplets from 10-K filings, with concise, jargon-rich queries and often dispersed evidence; GPT-4-Turbo achieves 9% accuracy on its closed-book questions.The related work also situates financial RAG among benchmarks and studies of quantitative reasoning, sentiment, multi-hop retrieval, context pruning, and financial filing analysis.Source: §2.3 · p. 3

2.4. Contextual Retrieval; 2.6. Self-Correcting and Agentic Retrieval

• Contextual Retrieval prepends LLM-generated document context to chunks, while Agentic RAG verifies and retries retrieval; both retain full-corpus chunk search, whereas HDRR routes before retrieval.The contextual baselines include static company-and-year tags and LLM-generated context; the agentic baseline uses one verification call and up to two entity-anchored retries.Source: §2.4 · p. 4, §2.6 · p. 4

2.5. Knowledge-Graph and Global-Synthesis RAG

• GraphRAG builds a knowledge graph for global cross-document queries, while FinDER targets local company-specific facts; HDRR instead searches all chunks of the routed filing for within-document evidence.The passage describes graph construction for a 500-document corpus as much more expensive than chunking and embedding, so GraphRAG is treated as complementary rather than a comparison baseline.Source: §2.5 · p. 4

2.7. LLM Structured Output; 2.9. Document Routing and Classification

• LLM structured output enables SFR to extract ticker and fiscal-year metadata from queries, treating document identification as classification; this application of structured output to retrieval is described as understudied.SFR treats each filing as a shard and uses predictable company-to-filing mappings; traditional document routing instead uses statistical models of shard contents.Source: §2.7 · pp. 4–5, §2.9 · p. 5

2.8. Long-Context LLMs and Document-Level Reasoning

• LongRAG retrieves longer segments, while SFR takes document-level retrieval further by supplying entire financial filings—typically 50K–200K tokens—to long-context models.The related work notes the “lost in the middle” challenge for long-context models and prior work on positional interpolation.Source: §2.8 · p. 5

2.10. Multi-Stage and Cascaded Retrieval

• HDRR follows multi-stage retrieval by using LLM structured output for zero-shot document selection before scoped chunk search, adapting retrieval scope to whether document identity can be resolved.This relates to distributed shard selection and retrieve-then-rerank pipelines, while systematic evaluation of LLM-routed hybrid RAG for structured-document corpora remains limited.Source: §2.10 · p. 5

2.11. Gap Analysis

• HDRR addresses a gap in RAG integration by combining document routing with scoped chunk retrieval, unlike contextual indexing and agentic retry, which leave chunk search open to full-corpus confusion.The paper also compares the hybrid against contextual-indexing and agentic-retrieval systems on the same benchmark.Source: §2.11 · p. 6

3.1. Task Definition

• The task is to answer natural-language questions about a company’s financial performance, operations, or disclosures using information in a financial-document corpus.The study corpus comprises S&P 500 companies’ 10-K annual reports, with each ticker-and-year pair uniquely identifying a document.Source: §3.1 · p. 6

3.2. Retrieval Paradigms

• The paper compares three retrieval paradigms that differ in retrieval unit, mechanism, and scope.Source: §3.2 · p. 6

3.2.1. Chunk-Based Retrieval (CBR)

• CBR partitions documents into overlapping chunks, embeds them, and retrieves the top-k chunks by similarity before generating an answer.The implementation also uses full-text search, rank fusion, and neural reranking.Source: §3.2.1 · p. 6

3.2.2. Semantic File Routing (SFR)

• SFR uses an LLM to extract ticker-and-year metadata from a query, resolves it to document paths, and generates an answer using the full document context.It bypasses offline indexing.Source: §3.2.2 · p. 6

3.2.3. Hybrid Document-Routed Retrieval (HDRR)

• HDRR routes to target documents first, then retrieves chunks only within them; if routing resolves no documents, it falls back to unrestricted CBR.This combines SFR’s immunity to cross-document chunk confusion with CBR’s ability to surface precisely relevant text fragments.Source: §3.2.3 · p. 6

3.3. Corpus Structure Assumption

• The corpus assumption is that financial filings have naming regularity and structural homogeneity: ticker and year identify reports, while standardized sections contain semantically similar but entity-distinct content.The paper hypothesizes that homogeneity causes chunk-retrieval confusion, while naming regularity enables SFR to bypass it.Source: §3.3 · p. 7

3.4. Evaluation Framework

• Evaluation uses a separate LLM judge to score each answer against a reference on a 1-to-10 scale.The reported metrics are average score, failure rate (score 1), correctness rate (score at least 8), and perfect rate (score 10).Source: §3.4 · p. 7

• All three systems use GPT-4.1 and the same evaluation pipeline; they differ only in how they retrieve relevant context for a query.Source: §4 · p. 7

HDRR

• HDRR uses SFR to identify a target document, then applies CBR’s chunk-level retrieval within that document to address cross-document confusion.This combines document routing with targeted chunk retrieval.Source: §4 · p. 7

SFR

• SFR uses LLM structured output to identify a document from the query and provides the whole document as context, without chunking.Source: §4 · p. 7

CBR

• CBR splits documents into overlapping fragments, embeds them in a shared vector space, and retrieves them using hybrid search and neural reranking.Source: §4 · p. 7

4.1. Chunk-Based RAG (CBR)

• CBR uses an offline indexing phase followed by online retrieval and generation, as illustrated in Figure 1.The dual-store pipeline indexes documents in FTS5 for keyword search and FAISS for semantic search before retrieving, fusing, reranking, and generating.Source: §4.1 · p. 7, Figure 1 · p. 9

4.1.1. Offline Indexing Pipeline

• The offline pipeline creates 2,500-character chunks with 50% overlap and stores them in complementary FTS5 and FAISS indices.FTS5 supports keyword and BM25 search; FAISS uses 1,024-dimensional embeddings and exact inner-product search over normalized vectors, avoiding approximate-search quantization errors at the corpus’s roughly 500K-chunk scale.Source: §4.1.1 · pp. 7–8

4.1.2. Online Query Pipeline

• GPT-4.1 rewrites each query into a clarified semantic-search query and extracted keywords, then FTS5 and FAISS retrieve candidates for rank-fusion.FTS returns the top 20 chunks, semantic search returns the top 30 with a distance threshold of 2.0, and Reciprocal Rank Fusion combines their ranks with k = 60.Source: §4.1.2 · p. 8
• Jina Reranker v2 rescoring and adaptive cutoffs select a compact context, which GPT-4.1 uses to generate answers solely from the retrieved chunks.Chunks are retained to a 55% cumulative normalized score threshold and excluded below a score drop of more than 0.15 from the top; this typically yields 5–10 chunks. RRF is rank-based and unsupervised, while adaptive cutoff reranking avoids a fixed-size context window.Source: §4.1.2 · p. 8, §4.1.3 · p. 8

4.1.3. Design Rationale and Limitations

• CBR’s global embedding-space assumption fails when similar sections from different 10-K filings cluster together, causing cross-document chunk confusion.The paper identifies this as CBR’s primary failure mode and the motivation for routing-based architectures.Source: §4.1.3 · p. 8

4.2. Semantic File Routing (SFR)

• SFR eliminates the offline indexing phase entirely and requires documents to be organized in a directory structure that follows a predictable naming convention.Source: §4.2 · p. 10

4.2.1. Corpus Organization

• The corpus stores files as data/{year}/{ticker}.{ext}, yielding approximately 500 files per year for the S&P 500 corpus.Here, year is the fiscal year, ticker is the company’s stock ticker symbol, and ext is pdf or txt.Source: §4.2.1 · p. 10

4.2.2. Online Query Pipeline

• GPT-4.1 extracts ticker and fiscal-year arrays in JSON structured-output mode, using a default year when none is specified.The structured-output guarantee ensures responses conform to the expected schema and eliminates parsing failures.Source: §4.2.2 · p. 10
• For each ticker-year pair, SFR checks PDF then TXT paths, retries with the default year if no files are found, and sends located full documents for answer generation.Located files are base64-encoded as API file inputs, allowing the model to process PDF text, tables, and layout information; the experiments use 2023 as the default year.Source: §4.2.2 · p. 10

4.3. Hybrid Document-Routed Retrieval (HDRR)

• HDRR adds document routing before CBR’s chunk retrieval while reusing CBR’s offline index, with no additional indexing required.Source: §4.3 · pp. 10, 12

4.3.1. Stage 1: Document Routing

• GPT-4.1 extracts ticker symbols and fiscal years for document lookup; if no document is identified, HDRR falls back to full-corpus retrieval.If exact-path lookup fails, the system tries filename-based matching; full-corpus fallback is equivalent to standard CBR.Source: §4.3.1 · p. 12

4.3.2. Stage 2: Scoped Chunk Retrieval

• By limiting retrieval to chunks from routed documents, HDRR eliminates cross-document chunk confusion and lets reranking focus on query-relevant passages.A typical routed document contains 50–200 chunks.Source: §4.3.2 · p. 12

4.3.3. Stage 3: Answer Generation

• GPT-4.1 receives reranked chunks from the routed document with CBR’s prompt template, yielding compact, targeted context for answer generation.Source: §4.3.3 · p. 12

4.4. Architectural Comparison

• HDRR eliminates cross-document confusion like SFR and avoids context dilution like CBR, but requires an extra routing LLM call and naming regularity in the corpus.Source: §4.4 · p. 12

5.1. Dataset

• FinDER contains 5,703 financial QA query-evidence-answer triplets from S&P 500 10-K filings, with ambiguous analyst queries and evidence often scattered across sections.The benchmark includes qualitative and quantitative questions, with half of the quantitative questions requiring multi-step reasoning.Source: §5.1 · p. 12

5.2. Sampling Strategy

• Five stratified random groups of 300 queries each yield 1,500 evaluations, with the same groups used across all six systems for paired comparisons.The multi-group design supports inter-group variance estimation; aggregates use the full 1,500-query set unless otherwise noted.Source: §5.2 · pp. 12–13

5.3. System Configurations

• CBR uses its strongest reranked configuration, SFR routes with GPT-4.1, and HDRR scopes CBR retrieval to routed documents, falling back to full-corpus retrieval if routing fails.HDRR reuses CBR’s chunk index and SFR’s structured-output parsing; all systems generate with GPT-4.1 at temperature 0.0.Source: §5.3 · p. 13
• CBR+Meta adds static company-year tags, CBR+LLM-Ctx prepends GPT-4.1-mini-generated chunk context, and Agentic RAG verifies retrieved chunks before limited retries.CBR+Meta has zero indexing-time LLM cost; CBR+LLM-Ctx’s 497-document, 169,640-chunk indexing cost stayed under its $100 budget cap, with per-query cost matching CBR.Source: §5.3 · p. 13

5.4.1. LLM-as-Judge Scoring

• GPT-4.1 evaluates each answer against the original query and FinDER ground truth on a deterministic integer scale from 1 to 10.It returns structured JSON with a score and brief justification at temperature 0.0.Source: §5.4.1 · p. 14

5.4.2. Evaluation Metrics

• Four metrics capture average answer quality, completely incorrect answers, substantively correct answers, and fully correct answers.They are Average Score (mean on the 1–10 scale), Failure Rate (score 1), Correctness Rate (score ≥8), and Perfect Rate (score 10).Source: §5.4.2 · p. 14

5.4.3. Statistical Controls

• The five-group design estimates inter-group variance, enables paired comparisons on identical samples, and tests reproducibility across query selections.These controls reduce the influence of query difficulty and help establish that findings are not artifacts of a particular sample.Source: §5.4.3 · p. 14

5.4.4. LLM-as-Judge Symmetry and Cross-Evaluator Robustness

• GPT-4.1 scores every system, so its potential stylistic bias applies uniformly; a cross-evaluator check re-scores a stratified subset with Claude.The subset contains 99 unique queries and yields 594 scores across six systems; the same rubric is used, and system rankings are checked across evaluators.Source: §5.4.4 · p. 14

6.2. Key Findings

• SFR scores higher and fails less often than CBR, but CBR produces more high-quality and perfect answers.SFR averages 6.45 vs. CBR’s 6.02 and has 10.3% vs. 22.5% failures; CBR has 49.0% vs. 44.0% high-quality answers and 13.8% vs. 8.5% perfect answers.Source: §6.2 · p. 15
• HDRR leads all systems on all four metrics, scoring 7.54 average, with 6.4% failures, 67.7% correctness, and 20.1% perfect answers across every group.Against CBR, HDRR improves average score by 1.52 points (+25.2%), reduces failures by 16.1 pp, increases correctness by 18.7 pp, and increases perfect answers by 6.3 pp. Against SFR, the respective changes are +1.09 points (+16.9%), −3.9 pp, +23.7 pp, and +11.6 pp.Source: §6.2 · p. 15, Table 2 · p. 16

6.3. Score Distribution Analysis

• HDRR shifts score mass from both low and medium bands into the high band, reducing low-band scores by 18.2 pp and increasing high-band scores by 18.7 pp over CBR.SFR mainly shifts mass from low to medium scores, while contextual-indexing variants retain a low band of at least 15.9%; Agentic’s distribution resembles CBR’s.Source: §6.3 · p. 15, Table 3 · p. 16

6.4. Paired Group Analysis

• HDRR improves on CBR in every metric and group; average-score gains range from +1.18 to +1.75, with a standard deviation of 0.21.Improvements over SFR are also consistent: average-score standard deviation is 0.14, with mean gains of +23.7 pp in correctness and +11.6 pp in perfect answers.Source: §6.4 · pp. 15–16, Table 4 · p. 16, Table 5 · p. 16

6.5. Comparison with Contextual-Indexing and Agentic Baselines

• V-A raises average score from 6.02 to 6.92 and cuts failures from 22.5% to 11.7% at zero LLM cost, while V-B narrows but does not close HDRR’s lead.V-B averages 7.26 and has 9.3% failures, versus HDRR’s 0.28-point average-score lead and 2.9 pp failure-rate lead; V-B’s perfect rate is 20.3% versus HDRR’s 20.1%.Source: §6.5 · p. 17, Table 6 · p. 17
• Agentic improves over CBR but trails upfront routing, averaging 6.36 with 17.3% failures and 3.36 LLM calls per query, versus HDRR’s 2 calls.Post-hoc verification cannot prevent wrong-company chunks from entering the reranker on the first pass.Source: §6.5 · p. 17, Table 6 · p. 17

6.6. Multi-Document Query Handling

• HDRR routed every named ticker in the 25-query cross-company subset, achieving 1.000 routing coverage, although reranking sometimes dropped lower-relevance company chunks.Agentic’s verifier failed on 4 of 25 queries and triggered retries on at least 5; its answer coverage was 0.920 versus CBR’s 0.960.Source: §6.6 · pp. 17–18, Table 7 · p. 17

6.7. Cross-Evaluator Robustness

• Cross-evaluator rankings were strongly aligned: per-system Spearman correlations ranged from 0.85 to 0.94, and system-level correlation was 0.886.Claude scored more strictly, but failure rates differed by at most 3 pp; absolute correctness rates fell 12–34 pp under Claude, so correctness comparisons should use a fixed evaluator.Source: §6.7 · p. 18

7.1. The CBR–SFR Trade-off: Root Causes

• The trade-off arises because cross-document chunk confusion drives CBR failures, while context dilution reduces SFR precision.Source: §7.1 · p. 18

7.1.1. Why CBR Suffers Catastrophic Failures

• CBR’s failure rate is 22.5% because standardized 10-K sections make chunks from different companies semantically similar, and reranking cannot recover correct chunks missing from its candidate pool.Chunk-level reranking also cannot enforce that candidates come from the same company.Source: §7.1.1 · pp. 18–19

7.1.2. Why SFR Sacrifices Precision

• SFR avoids cross-document confusion but sends entire 50,000–200,000-token filings, diluting relevant evidence and shifting scores from both tails toward the middle.The passage attributes this precision loss to long-context attention difficulties, including the “lost in the middle” phenomenon (Liu et al., 2024).Source: §7.1.2 · p. 19

7.2. How HDRR Resolves the Trade-off

• 20.1% of HDRR answers are perfect, versus 13.8% for CBR and 8.5% for SFR, while HDRR’s failure rate is 6.4%, versus 10.3% and 22.5%.Routing first confines candidates to the target company’s filing; reranking then selects relevant chunks, avoiding both cross-document noise and full-document context dilution.Source: §7.2 · p. 19

7.3. Error Analysis of Remaining HDRR Failures

• 93.07% of queries route successfully, scoring 7.79 on average with 4.0% catastrophic failures and 70.8% correctness; the 6.93% fallback bucket averages 4.26 with 38.5% failures.The successful filename_fallback branch is the intended route for FinDER’s path layout; fallback_* rows instead indicate failed document binding and corpus-wide retrieval.Source: Table 9 · p. 19, §7.3 · p. 19

7.3.1. Ticker Symbol Mismatch with Successful Fallback

• Residual routing failures include legacy ticker rebrandings, descriptive company references without tickers, ambiguous abbreviations, and tickers for filings absent from the corpus.Fallback converts these into full-corpus retrieval rather than complete routing failure; rebrandings require ticker-history knowledge, while ambiguity can make declining to guess appropriate.Source: §7.3.1 · p. 20

7.3.2. Inherent Query Difficulty

• Some failures affect all three systems because answers require information absent from the corpus or multi-hop reasoning across sections.Source: §7.3.2 · p. 20

7.3.3. Routing to Correct Document but Insufficient Chunk Relevance

• Even after correct document routing, HDRR can miss relevant information when it is dispersed across small fragments that score poorly in both FTS and semantic search.Source: §7.3.3 · p. 20

7.4. Cost and Efficiency Analysis

• HDRR’s deterministic 2-call cost is below Agentic’s variable ∼3.4-call cost, while HDRR preserves CBR’s 5K–15K generation-token budget.Agentic averages ∼3.36 calls per query across 1,500 queries; its additional retrieval and verification pass is triggered by the 27% of queries with verification NO.Source: §7.4 · p. 21
• V-B incurs a one-time indexing cost for context generation across ∼170,000 chunks, bounded by a $100 budget cap, while V-A pays no indexing-time LLM cost.The V-B spend is amortized across subsequent queries; V-A’s indexing cost is zero.Source: §7.4 · p. 21
• A 0.28-point average-score lead over V-B (7.54 vs. 7.26) and 2.9 pp failure-rate advantage (6.4% vs. 9.3%) cost HDRR one extra LLM call per query.V-B reaches HDRR’s perfect rate (20.3% vs. 20.1%) without that call; the break-even favors V-B at high query volume and HDRR at low volume, while HDRR wins on answer quality at either scale.Source: §7.4 · p. 21
• HDRR uses ∼5K–15K generation tokens, zero indexing-time LLM spend, and 2 LLM calls per query—1.7× fewer than Agentic’s ∼3.4 calls.The paper states that lower per-query token and call counts translate to lower API spend and inference energy at deployment scale.Source: §7.4 · p. 21
• For production deployments, HDRR offers the highest answer quality at a per-query cost one routing call above CBR; V-B avoids that overhead, while V-A gains 0.90 average-score points over CBR at zero LLM cost.V-B suits settings where per-query calls are budget-constrained and one-time indexing spend is acceptable; V-A is the cheapest non-trivial improvement over CBR.Source: §7.4 · p. 21

7.5. Scalability and Maintenance

• HDRR’s quality is invariant to corpus expansion when routing succeeds and degrades gracefully to CBR quality when it fails; new indexed documents need no routing-logic changes.For multi-year corpora, (ticker, year) metadata avoids compounded confusion across companies and years.Source: §7.5 · p. 22

7.6. Generalizability to Other Domains

• HDRR applies to domains with naming regularity and structural homogeneity, including regulatory filings, medical records, and technical documentation.These examples use identifiers such as case numbers and filing dates, patient identifiers and encounter dates, or product names and versions for routing before targeted retrieval.Source: §7.6 · p. 22
• On heterogeneous corpora without naming regularity, routing frequently fails and HDRR falls back to full-corpus retrieval, reducing to standard CBR with neither advantage nor disadvantage.Source: §7.6 · p. 22

8.2. Contributions

• The study evaluates six systems on 1,500 FinDER queries across five groups and formalizes corpus regularity and homogeneity as conditions for document routing.Its contributions include a controlled paired evaluation, a second-family evaluator check on a 99-query subset, and a framework for assessing when routing may generalize beyond finance.Source: §8.1 · p. 22, §8.2 · p. 23

8.1. Summary of Findings

• 7.54 average score, 6.4% failure rate, 67.7% correctness, and 20.1% perfect rate make HDRR best across all four metrics.These correspond to +25.2% average score and −71.6% failure rate relative to CBR, +18.7 pp correctness relative to CBR, and +6.3 pp perfect rate relative to CBR (+11.6 pp relative to SFR).Source: §8.1 · p. 22
• SFR improves average score and reduces failures versus CBR, while CBR yields more perfect answers; HDRR scopes chunk retrieval to routed documents.The study attributes CBR failures to cross-document chunk confusion and SFR’s lower precision to context dilution; HDRR combines document routing with scoped chunk retrieval to address both limitations.Source: §8.1 · p. 22, §8.2 · p. 23
• HDRR dominates both baselines in every experimental group, with an average-score standard deviation of 0.18.A 99-query re-scoring found system-level Spearman ρ = 0.886 between GPT-4.1 and Claude rankings, preserving the top three and bottom-ranked systems; on a 25-query cross-company subset, HDRR achieved 1.000 routing coverage and the highest retrieval coverage.Source: §8.1 · pp. 22–23
• ∼5K–15K tokens per query, zero indexing-time LLM cost, and two deterministic LLM calls make HDRR the most efficient high-quality system.The conclusion contrasts this with SFR’s ∼50K–200K tokens per query, V-B’s one-time ∼$100 contextualization pass, and Agentic’s ∼3.4 calls per query; it connects lower tokens and call counts to reduced API spend and inference energy.Source: §8.1 · p. 22

8.3. Future Directions

• Ticker mismatches remain a routing weakness; future work proposes ticker-history aliases, company-name dictionaries, entity resolution, or external knowledge bases.These signals could be used before routing fallback to address rebrandings and descriptive references without changing HDRR’s architecture.Source: §8.3 · p. 23
• Larger comparative-query benchmarks and per-document reranking for multiple routed targets are proposed to improve multi-document retrieval.The existing handcrafted 25-query subset showed HDRR’s multi-pair routing extracted every named ticker and surfaced every routed document, while rerankers sometimes dropped lower-relevance company chunks.Source: §8.3 · p. 23
• Future work proposes testing HDRR on legal, medical, and technical documents, adapting retrieval scope to routing confidence, and reducing routing-call cost.Confidence-based expansion or full-corpus fallback could reduce residual failures; combining routing with query rewriting or using a smaller routing model could lower API overhead.Source: §8.3 · p. 23

• Table A1 provides the complete configuration for the Chunk-Based RAG system.Source: Table A1 · p. 24
• Table A2 provides the complete configuration for the Semantic File Routing system.Source: Table A2 · p. 24
• Table A3 provides the complete configuration for the Hybrid Document-Routed Retrieval system.Source: Table A3 · p. 24