PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Expert Financial QA via Self-Improving RAG

Junjie Xiong, Shawheen Ghezavat, Aum Hirpara

arXiv:2608.26706v1cs.CL

TL;DR

Financial QA needs verification against evidence and auditability because single-pass RAG can return incorrect, unverifiable answers. Self-Improving RAG addresses this with specialized agents and feedback-driven retries, achieving 86% LLM Judge accuracy and a 36.4% Lazarus Rate on FinanceBench.

  • Problem

    Financial QA lacks reliable grounded verification and audit trails in standard single-pass RAG, despite risks from numeric reasoning, temporal filtering, and entity disambiguation.

  • Method

    Self-Improving RAG coordinates Retrieval, Reasoning, and Judge agents through an orchestrator that escalates retrieval and prompting when quality falls below a dynamic threshold.

  • Results

    86% LLM Judge accuracy and a 36.4% Lazarus Rate are reported on FinanceBench, recovering nearly 4 in 10 initially incorrect answers.

  • Takeaways & Limitations

    The fixed retrieval pipeline with judge-driven retry achieves strong results without dynamic routing and provides full interpretability.

  • Takeaways & Limitations

    The reported numeric-matching difference of +2.1% falls within confidence intervals, so its interpretation requires caution.

Abstract

from arXiv · show

Expert-level financial question answering requires both grounded verification to catch numeric hallucinations and audit trails for regulatory compliance, attributes that standard single-pass RAG systems lack. We take a step toward this goal with Self-Improving RAG, a framework that decomposes document QA into three specialized agents (Retrieval, Reasoning, and Judge) coordinated by an orchestrator with feedback-driven self-correction. When the Judge Agent scores an answer below a dynamic threshold, the system triggers retry with escalated strategies: broader retrieval, more careful prompting, and relaxed acceptance criteria. We evaluate on FinanceBench (SEC filing QA), where Self-Improving RAG achieves 86% oracle-guided accuracy (measuring agreement with gold answers) with a 36.4% Lazarus Rate, recovering nearly 4 in 10 initially incorrect answers through targeted retry. A key finding is that a fixed retrieval pipeline with judge-driven retry achieves strong results without dynamic routing, providing full interpretability. Every decision is logged with confidence scores, enabling the audit trails required for regulated financial applications.

1 INTRODUCTION

Self-Improving RAG addresses failures in single-pass financial QA by combining specialized agents, feedback-driven escalation, grounded verification, and audit logging. On FinanceBench, it reports 86% LLM Judge accuracy and a 36.4% Lazarus Rate.

  • Motivation: Single-pass financial RAG lacks mechanisms to recognize and correct failures, while incorrect answers and absent audit trails create risks for financial professionals.Financial QA also requires numeric reasoning, temporal filtering, and entity disambiguation.
  • Motivation: Financial QA retrieval must remain within authorized document corpora, eliminating web-search fallback as a recovery mechanism.This closed-domain constraint demands alternative self-correction approaches.
  • Approach: Self-Improving RAG decomposes QA into Retrieval, Reasoning, and Judge agents coordinated by an orchestrator with feedback-driven self-correction.The Retrieval Agent escalates pipelines, the Reasoning Agent adapts prompting, and the Judge accepts or requests retry using dynamic thresholds.
  • Approach: Targeted escalation retrieves more documents, prompts more carefully, and applies grounded verification with programmatic numeric checking.The design logs provenance, confidence scores, and reasoning traces for compliance review.
  • Results: 86% LLM Judge accuracy and a 36.4% Lazarus Rate on FinanceBench show recovery of nearly 4 in 10 initially incorrect answers.A fixed retrieval pipeline with judge-driven retry achieves strong results without dynamic routing, providing full interpretability.

2 RELATED WORK

Related work spans self-correction, adaptive retrieval, and multi-agent coordination, while Self-Improving RAG targets financial QA’s closed-domain and audit requirements. Its positioning combines within-session retry, specialized verification, authorized-corpus retrieval, and audit trails.

  • Self-Correction in Language Models: Agentic RAG and self-correction methods extend beyond single-pass systems through reinforcement, critique-refine loops, reflection tokens, or corrective retrieval.The cited methods differ in whether correction occurs within a session and whether fine-tuning is required.
  • Adaptive Retrieval and Document QA: Adaptive-RAG routes queries by complexity, whereas Self-Improving RAG adds judge-driven retry when an initial attempt fails.The related-work discussion presents fixed-pipeline retry as a simpler alternative relevant to explainability.
  • Multi-Agent Coordination: General-purpose multi-agent frameworks coordinate agents, but this work tailors roles to document QA with finance-domain constraints and audit requirements.The distinction is specialization for regulated financial applications.
  • Positioning: The Walled Garden Constraint: Self-Improving RAG operates in a closed authorized-document corpus, where web-search fallbacks are precluded by financial data governance policies.The walled-garden constraint is a central positioning dimension against prior approaches.
  • Positioning: The Walled Garden Constraint: The comparison identifies Self-Improving RAG as combining within-session self-correction, no task-specific fine-tuning, walled-garden retrieval, and full regulatory audit trails.These dimensions distinguish the proposed system in the related-method comparison.

3 METHOD: SELF-IMPROVING RAG

Self-Improving RAG orchestrates repeated retrieval, reasoning, and judging attempts within an authorized corpus, escalating effort when quality is insufficient. Its design combines numeric-focused evaluation, logged decisions, and retry mechanisms that preserve the best answer.

  • Problem Setting: The problem setting produces an evidence-supported answer from a natural-language query and authorized financial-document corpus under a bounded retry budget.The system accepts when utility exceeds threshold or returns the best answer after budget exhaustion.
  • Preliminaries and Notation: Each attempt contains retrieved evidence, a candidate answer, a quality vector, and an acceptance threshold, coordinated by Retrieval, Reasoning, and Judge agents.The state representation supports orchestration across attempts.
  • Retrieval Agent: Retrieval uses a hybrid dense, BM25, and reranking pipeline, adding 10 documents per retry up to k=30 and enabling RSE on the final attempt.The escalation configuration progressively expands recall while preserving a staged retrieval design.
  • Routing Heuristics: Rule-based routing selects retrieval settings using ticker, company, numerical-comparison, and exploratory-query characteristics.Entity recognition uses an S&P 500 ticker gazetteer and fiscal-year regex patterns with reported latency below 10ms.
  • Judge Agent: The Judge separates grounding, completeness, and numeric faithfulness, weighting numeric faithfulness at 0.5 to penalize unsupported financial numbers.Strict normalized-number coverage requires every answer number to be explicitly supported by retrieved evidence.
  • Orchestrator: The orchestrator retries below-threshold answers with broader retrieval, more careful prompting, and relaxed acceptance criteria while logging decisions for audit compliance.The algorithm maintains the best answer seen so retries do not degrade output quality; the paper states failure probability decays multiplicatively across attempts.

4 EXPERIMENTS AND RESULTS

Experiments on FinanceBench evaluate the orchestrated retry loop under oracle-guided and deployment settings, measuring accuracy, correction recovery, component effects, and retrieval design. Results show strong oracle-guided performance and targeted recovery, while blind judging and numeric verification remain important constraints.

  • Experimental setup: FinanceBench contains 150 SEC filing questions, 66% requiring numerical calculations; experiments use retry budget B=2 and initial threshold τ0=0.5.The implementation uses GPT-4o-mini, BGE-large embeddings with ChromaDB, cross-encoder reranking, and a Judge combining LLM evaluation with programmatic numeric verification.
  • Main results: 86% LLM Judge accuracy is achieved in oracle-guided evaluation, while deployment-mode blind judging reaches only 31% acceptance.The oracle-guided Judge has gold-answer access; the production Judge performs blind verification.
  • Question-type analysis: +81.1% is the largest self-correction gain, occurring on domain-relevant questions that often require synthesizing information across multiple document sections.Table 4 reports correctness by question type and identifies domain-relevant questions as benefiting most from self-correction.
  • Correction recovery: 36.4% Lazarus Rate recovers 12 of 33 initially incorrect answers among questions triggering retry.The 33 retry-triggered questions represented 22% of the total FinanceBench set.
  • Component ablation: −10.6% results from removing prompt escalation, while B=1 versus B=2 produces a −25.5% drop, highlighting the value of careful prompting and multiple attempts.The ablation is conducted in deployment mode with a blind Judge and no gold answers.
  • Design implications and limitations: A fixed hybrid retrieval pipeline with judge-driven retry provides a simpler interpretable alternative to dynamic routing, but numeric verification can trigger unnecessary retries.Over-sensitive matching treats equivalent formats such as “$394.3 billion” and “$394,300 million” as incorrect, and the second attempt may introduce errors.

5 CONCLUSION

Self-Improving RAG uses specialized agents and feedback-driven escalation to recover from failures in financial document QA while supporting auditability. On FinanceBench, it achieves strong semantic performance, but deployment remains constrained by latency, judge reliability, and limited evaluation power.

  • Self-Improving RAG decomposes retrieval-augmented generation into specialized agents with a self-correction feedback loop.
  • 86% LLM Judge accuracy and a 36.4% Lazarus Rate show strong FinanceBench performance and recovery of nearly 4 in 10 initially incorrect answers.
  • 15-25 seconds of retry latency versus 5-8 seconds for single-pass processing limits the system primarily to analyst support rather than real-time applications.
  • Future work includes learned escalation, unit-aware normalization, expansion to other high-stakes domains, and human-in-the-loop oversight.
  • Every decision is logged with provenance and confidence scores, enabling post-hoc analysis and regulatory compliance review in regulated industries.

A EXTENDED METHODOLOGY

The extended methodology specifies the system’s retrieval, reasoning, judging, and governance components, including escalation policies, confidence signals, and formal agent functions. It emphasizes auditable operation within an authorized financial document corpus.

  • Retrieval: Retry retrieval uses a conservative-to-aggressive configuration, beginning with k = 10 and progressively expanding recall.
  • Retrieval: A rule-based router maps ticker or company queries to metadata filtering, numerical comparisons to precision-focused reranking, and exploratory questions to broad-recall hybrid retrieval.
  • Reasoning: The Reasoning Agent escalates from standard to conservative to detailed prompts, progressively increasing instructions for uncertainty, reasoning, and source attribution.
  • Judging: The Judge evaluates grounding, completeness, numeric verification, and confidence signals, with numeric verification receiving the highest weight for financial queries.
  • Trustworthiness: Audit trails record timestamps, confidence scores, reasoning traces, and intervention points for post-hoc analysis, compliance, and human review.
  • Formalization: The framework defines retrieval as returning top-k relevant documents, reasoning as generating an answer from the query, documents, and prompt strategy, and judging as computing a quality score.

B.2 CONVERGENCE ANALYSIS

The convergence analysis formalizes bounded self-correction, best-answer retention, and multiplicative failure reduction under repeated attempts. Retry cannot worsen the selected output, while the theoretical reduction depends on non-zero success probability.

  • The loop terminates after at most Tmax = B + 1 iterations, where B is the retry budget.
  • The observed first-attempt retry probability is p1 ≈ 0.22 on FinanceBench, informing the expected cost analysis.
  • The Orchestrator maintains the best answer across attempts, so later retries cannot degrade the final output quality.
  • The stated proposition is that self-correction reduces failure probability multiplicatively with the number of attempts T.
  • Under the Markovian state assumption, total failure requires failure at every attempt, yielding multiplicative factorization through conditional failure probabilities.
  • If each stage has a non-zero success probability, Pfail approaches 0 exponentially as T approaches infinity.

C.2 CORRECTION FLOW ANALYSIS

The correction-flow analysis examines FinanceBench component contributions and shows how a fixed retrieval pipeline uses judge-triggered retries as a safety net. Lazarus Rate measures semantic recovery rather than numeric exact-match improvement.

  • Table 9 analyzes the complete self-correction flow on FinanceBench, including the Lazarus Rate for successful retry corrections.
  • The Lazarus Rate measures semantic correctness recovery, while self-correction does not improve numeric exact-match accuracy.
  • A fixed hybrid filter-rerank pipeline prioritizes simplicity and reproducibility over dynamic routing.
  • When the Judge identifies a low-quality answer, retrieval escalates through higher k and RSE segment merging as a recovery safety net.
  • Figure 2 follows 150 questions through the pipeline and reports a 36.4% Lazarus Rate for successful corrections.

D.1 UNIT/SCALE CONFUSION ERROR

Unit/scale confusion occurs when a model extracts the correct number but mishandles its stated units, producing materially incorrect financial answers. The Judge Agent can detect this inconsistency and trigger a retry that restores the unit interpretation.

  • D.1 UNIT/SCALE CONFUSION ERROR: A unit/scale confusion error occurs when a correctly extracted number is reported without applying its source unit qualifier.In the example, omitting “in millions” makes the answer wrong by a factor of 1,000.
  • D.1 UNIT/SCALE CONFUSION ERROR: The Judge Agent detected the unit inconsistency and triggered a retry that correctly interpreted the unit context.The second attempt corrected the initial answer after the qualifier was applied.
  • D.1 UNIT/SCALE CONFUSION ERROR: Unit/scale confusion is identified as one failure mode in cases where Self-Improving RAG does not improve over single-pass baselines.The broader failure analysis includes misinterpreting thousands versus millions and mixing absolute values with percentages.

E.1 DATASETS

The evaluation uses FinanceBench, a benchmark of SEC filing questions requiring financial extraction and reasoning, alongside comparisons with several single-pass retrieval baselines. The described system combines specialized agents, escalating retrieval, judge-based retry, and analyst-oriented latency trade-offs.

  • E.1 DATASETS: FinanceBench contains questions about publicly traded companies that require extraction and reasoning over SEC 10-K and 10-Q filings.The benchmark includes questions with numerical calculations, temporal context, and multi-document reasoning challenges.
  • E.1 DATASETS: The comparison includes semantic, hybrid, hybrid-plus-filter, and hybrid-plus-filter-plus-rerank single-pass retrieval baselines.These baselines progress from dense BGE-large retrieval to hybrid retrieval with metadata filtering and cross-encoder reranking.
  • E.1 DATASETS: The evaluation measures semantic similarity, numeric verification, LLM Judge score, and Lazarus Rate.Lazarus Rate is the percentage of initially incorrect answers successfully corrected through retry.
  • E.1 DATASETS: The system uses GPT-4o-mini, 512-token document chunks with 50-token overlap, BGE-large embeddings, reranking, and default retrieval of k = 10 documents.The Retrieval Agent can escalate to k = 20 and k = 30 with RSE, while Judge thresholds decrease from τ = 0.5 to τ = 0.4 across attempts.
  • E.1 DATASETS: Retry is capped at two retries, and latency rises from approximately 5–8 seconds for single-pass to 15–25 seconds when retry is triggered.The system is positioned for analyst support rather than real-time chatbot use.
  • E.1 DATASETS: In the Apple example, the Judge assigns 0.4 because the first answer omits the FY2022 comparison, causing escalation below τ1 = 0.5.The example illustrates judge-driven retry for semantic incompleteness and provenance logging for each decision.

F.1 ERROR ANALYSIS

The error analysis identifies recurring situations where Self-Improving RAG cannot improve over single-pass baselines, with missing corpus evidence as the largest reported failure category. It also points to corpus expansion, chunking improvements, and judge calibration as potential areas for further gains.

  • F.1 ERROR ANALYSIS: Retrieval ceiling accounts for 28% of failures because escalation cannot recover information absent from the document corpus.This is the largest reported failure category.
  • F.1 ERROR ANALYSIS: Arithmetic errors account for 22% of failures when multi-step calculations accumulate rounding errors or use incorrect formulas.These errors remain unresolved when the calculation itself is wrong.
  • F.1 ERROR ANALYSIS: Unit/scale confusion accounts for 18% of failures, including misinterpreting thousands versus millions or mixing absolute values with percentages.The category concerns interpretation of financial quantities rather than document absence.
  • F.1 ERROR ANALYSIS: Temporal misalignment accounts for 15% of failures when figures are extracted from incorrect fiscal periods.The failure concerns selecting the wrong reporting period.
  • F.1 ERROR ANALYSIS: Judge miscalibration accounts for 12% of failures when the Judge scores an incorrect answer highly and prevents beneficial retry.The analysis identifies calibration tuning as an opportunity.
Loading 2608.26706v1…