PaperlayerView exact revision on arXiv

Source-linked AI summary

CIFQA: A Deterministic Tool-Grounded Multi-Agent LLM Framework for Financial Query Answering

Kunjesh Parekh, Anil Kumar Tiwari, Divya Saxena

arXiv:2608.26114v1cs.AIcs.CLq-fin.CP

TL;DR

Financial QA often requires exact arithmetic, temporal reasoning, and rule application, yet LLMs can produce plausible numerical errors. CIFQA addresses this with specialized language agents and deterministic computation engines, achieving high fixed-deposit accuracy and outperforming larger frontier models. The evaluation remains focused on fixed-deposit scenarios, while policy-heavy queries perform comparatively worse.

  • Problem

    Financial question answering requires exact reasoning over structured rates, temporal conditions, formulas, and rules, but LLMs can fail on multi-step financial calculations despite strong language capabilities.

  • Method

    CIFQA assigns LLM agents to interpretation, routing, extraction, planning, and response generation while deterministic tools execute financial calculations and rule application.

  • Results

    95.54% accuracy on calculation-intensive tasks and 90.87% overall accuracy were achieved on fixed-deposit queries, while a 17B backbone outperformed substantially larger frontier models.

  • Takeaways & Limitations

    The results support strict separation of language understanding from deterministic computation as a framework for reliable calculation-intensive financial reasoning.

  • Takeaways & Limitations

    The evaluation focuses on fixed-deposit scenarios, and policy-heavy RAG-based queries perform comparatively worse because they require complex rule interpretation.

Abstract

from arXiv · show

Calculation-intensive financial question answering requires exact reasoning over structured rates, temporal conditions, numerical formulas, and rule-based constraints. Although Large Language Models (LLMs) perform strongly on natural language tasks, they often produce numerically incorrect yet plausible answers when solving multi-step financial calculations. To address this limitation, we introduce CIFQA (Calculation-Intensive Financial Query Answering), a deterministic tool-grounded multi-agent LLM framework for financial question answering. CIFQA separates language understanding from numerical execution by assigning specialized agents to query interpretation, routing, parameter extraction, computation planning, and response generation, while deterministic Python-based tools perform financial calculations and rule application. We instantiate CIFQA for fixed deposit query answering and evaluate it on a curated benchmark of fixed deposit queries. CIFQA achieves 95.54% accuracy on calculation-intensive queries and 90.87% overall accuracy, substantially outperforming direct LLM baselines even when provided with complete formulas, rate cards, and benchmark instructions. Ablation studies show that deterministic components such as exact rate lookup, tenure computation, rolling-year adjustment, and premature-withdrawal logic are critical contributors to performance. Notably, a 17B open-source backbone operating within CIFQA outperforms substantially larger frontier models evaluated with the same financial information, demonstrating that architectural design is a more important determinant of numerical reliability than model scale. While evaluated on fixed deposit queries, CIFQA provides a generalizable framework for calculation-intensive financial reasoning tasks.

1 Introduction

CIFQA targets financial queries requiring exact computation over structured data, temporal conditions, formulas, and domain rules, where frontier LLMs show systematic execution failures. The framework separates language processing from deterministic numerical execution and achieves high accuracy on fixed-deposit queries.

  • Motivation: Frontier LLMs systematically fail on fixed-deposit tasks involving rate selection, rolling-year denominators, payout reconstruction, and premature-withdrawal settlement.These failures persist despite complete formulas, rate cards, and benchmark instructions.
  • Evaluation: 95.54% accuracy on calculation-intensive queries and 90.87% overall accuracy were achieved on a curated fixed-deposit benchmark.The benchmark contains 126 fixed-deposit queries spanning calculation-intensive reasoning, rate lookup, premature-withdrawal handling, policy interpretation, and edge cases.
  • Motivation: Calculation-intensive financial QA requires exact executable reasoning over structured financial data, temporal conditions, numerical formulas, and domain-specific rules.Unlike conventional retrieval or text-generation tasks, these queries involve rate selection, calendar-aware computation, compounding, payout handling, and rule application.
  • Framework: CIFQA separates language-centric tasks from numerical execution through specialized LLM agents and deterministic tools for financial computations and rule operations.Agents handle interpretation, routing, parameter extraction, planning, and response formulation, while deterministic tools execute computations verifiably.
  • Evaluation: Deterministic modules including rolling-year adjustment, exact rate lookup, tenure computation, and premature-withdrawal logic were important for calculation correctness.The framework was evaluated against direct LLM baselines supplied with relevant formulas and rate information.

2 Related Work

Prior financial QA and tool-augmented systems improve language understanding or provide external computation, but often leave execution planning or arithmetic to LLMs. CIFQA instead removes arithmetic execution from the LLM loop and delegates financial calculations to deterministic engines.

  • Financial QA: Financial NLP has emphasized text understanding, information extraction, sentiment analysis, retrieval, and domain adaptation of language models.Examples include FinBERT, BloombergGPT, and FinGPT.
  • Retrieval-Augmented Systems: RAG systems retrieve external knowledge but do not guarantee correct execution of multi-step arithmetic computations.This limitation is especially relevant to calculation-intensive financial settings.
  • Tool-Augmented Reasoning: Tool-augmented and program-aided approaches improve numerical reasoning but still rely on LLMs to determine computation processes, generate programs, select tools, or orchestrate steps.The cited approaches augment LLMs with external computation capabilities while retaining substantial LLM responsibility for execution planning.
  • CIFQA's Distinction: CIFQA removes arithmetic computation from the LLM reasoning loop and delegates all financial calculations to deterministic engines.This stricter separation is evaluated for tasks involving calendar-aware computation, structured rate lookup, payout scheduling, and financial rules.

3 Problem Definition: CIFQA

CIFQA defines a class of financial QA tasks requiring exact numerical computation over structured data together with rule-based reasoning under domain constraints. The problem formulation maps natural-language queries to structured parameters and computes exact outputs using financial data and computational rules.

  • Definition: Calculation-Intensive Financial Query Answering requires exact numerical computation over structured financial data and rule-based reasoning under domain-specific constraints.Representative tasks include fixed-deposit interest, loan amortization, bond yields, investment returns, and tax-related settlements.
  • Formalization: A CIFQA query maps natural language into structured parameters such as principal, rate, tenure, compounding frequency, and applicable rules.The goal is to compute an exact numerical output from this representation.
  • Formalization: The formal problem uses structured financial data R and domain-specific computational rules C, including calendar-aware calculations, compounding formulas, and regulatory constraints.These components determine the computation performed on the query representation.
  • Defining Characteristics: CIFQA is characterized by exactness, chained multi-step computation, structured-data dependence, and conditional rule application.Examples include compounding, time-based calculations, rate-table retrieval, tax deductions, penalties, and payout frequencies.
  • Evaluation Setting: The evaluation uses 126 expert-designed fixed-deposit queries spanning calculation, rate lookup, and policy-related categories with strict numerical and formatting criteria.Calculation ground truth was manually computed and validated using spreadsheet-based implementations.
  • Problem Distinction: CIFQA targets exact executable financial reasoning rather than primarily retrieval, summarization, or textual reasoning.Numerical correctness is the primary evaluation criterion.

4 CIFQA Framework

CIFQA uses a modular multi-agent pipeline for interpreting and planning financial queries, while deterministic computation engines perform all numerical operations. This separation is intended to make calculation execution transparent, reproducible, and verifiable.

  • 4.1 Pipeline: The pipeline comprises routing, parameter extraction, planning, deterministic execution, and response generation.Each stage has a distinct role in transforming a natural-language financial query into a final answer.
  • 4.2 Agents: The router classifies queries and selects an execution path, while the extractor structures parameters such as principal, tenure, rate, payout frequency, and conditions.Accurate extracted inputs are passed to downstream computation.
  • 4.2 Agents: The planner produces a computation plan covering formula selection, compounding intervals, partial periods, and calendar-aware adjustments.The plan is based on extracted parameters and domain rules.
  • 4.4 Execution Flow: Deterministic Python-based engines perform rate lookup, interest calculation, calendar-aware day counting, and rule-based operations such as tax deduction or payout handling.The response generator then formats computed outputs into a coherent natural-language response.
  • 4.3 Separation of Language Understanding and Computation: CIFQA strictly separates language understanding from numerical computation, limiting LLMs to interpretation, planning, and response synthesis.Arithmetic execution is delegated entirely to deterministic engines outside the LLM reasoning loop.
  • 4.3 Separation of Language Understanding and Computation: The architecture's complete removal of arithmetic from the LLM loop is designed to eliminate numerical drift and arithmetic hallucinations while ensuring reproducible computation.The stated pipeline is transparent, modular, and verifiable for high-stakes financial applications.

5 Implementation

CIFQA combines LLM-based language processing with deterministic Python computation engines to support reproducible, modular, and exact financial calculations. Its sequential pipeline separates interpretation and planning from numerical execution and can be extended across financial domains.

  • 5 Implementation: CIFQA combines LLM inference with deterministic Python-based computation engines for reproducible and exact numerical processing.The implementation is designed around modularity and clear separation between language-driven and computational components.
  • 5 Implementation: The LLM handles query routing, parameter extraction, computation planning, and response generation, but performs no arithmetic computation.This separation keeps numerical operations deterministic.
  • 5 Implementation: The computation layer includes rate lookup, interest computation, calendar handling, and rule application engines.These engines address rate tables, compounding and payout schedules, leap years and rolling periods, and conditional financial rules.
  • 5 Implementation: The end-to-end flow covers query sanitization, validation, routing, parameter extraction, planning, deterministic computation, and response generation.Arithmetic is executed by deterministic engines, whereas LLM agents are restricted to language understanding, planning, and response synthesis.
  • 5 Implementation: All computation engines use deterministic logic, while structured rate tables and policy rules are incorporated as lookup modules.This design reduces ambiguity in parameter selection and aligns calculations with financial specifications.
  • 5 Implementation: A sequential Python pipeline passes structured outputs between agents and computation modules, allowing components to be independently updated or extended.The modular design supports adaptation to other financial domains without changing the overall framework.

6 Experimental Setup

The evaluation uses a domain-expert-curated fixed-deposit benchmark spanning computation, rate lookup, policy interpretation, and edge cases. Strict correctness criteria require numerical agreement, valid intermediate logic, and appropriate financial formatting.

  • 6 Experimental Setup: 126 domain-expert-designed fixed-deposit queries cover calculation-intensive, interest-rate lookup, policy-related, and edge-case scenarios.The benchmark includes irregular tenures, large principals, ambiguous phrasing, and calendar-sensitive calculations.
  • 6 Experimental Setup: 101 calculation-intensive queries have manually computed ground-truth answers verified through spreadsheet-based implementations.This subset targets exact numerical correctness.
  • 6 Experimental Setup: A response is correct only when its final numerical value is within ±1 INR of ground truth, its intermediate logic is correct, and its formatting is appropriate.The protocol treats numerical deviations beyond tolerance as errors.
  • 6 Experimental Setup: Arithmetic hallucinations include incorrect arithmetic, day-count handling, rate application, compounding, or domain-specific rule execution.The criteria distinguish numerically meaningful errors from approximate semantic correctness accepted by conventional QA benchmarks.
  • 6 Experimental Setup: The comparison includes Claude Sonnet 4.6 among the evaluated frontier language models.The setup evaluates CIFQA against state-of-the-art frontier LLMs.
  • 6 Experimental Setup: Frontier LLM baselines receive complete financial knowledge and computational instructions to isolate numerical-execution failures from missing information.Models are evaluated through native conversational interfaces, and outputs are manually compared with ground truth.

7 Results and Analysis

CIFQA improves numerical reliability through structured multi-agent execution and deterministic computation, outperforming direct LLM baselines on calculation-intensive financial queries. Results and ablations identify calendar-aware handling and parameter grounding as especially important, while policy and analytical retrieval tasks remain comparatively weaker.

  • 7 Results and Analysis: 95.54% accuracy on calculation-intensive queries significantly outperforms proprietary frontier models and larger open-source baselines.CIFQA uses structured multi-agent execution, whereas the baselines use direct single-prompt generation.
  • 7 Results and Analysis: A 17B CIFQA backbone remains superior to larger models, showing that model scale alone does not resolve arithmetic unreliability.The results attribute the primary limitation to unreliable arithmetic execution rather than insufficient model capacity.
  • 7 Results and Analysis: Deterministic computation substantially reduces arithmetic hallucinations compared with proprietary and open-source LLM baselines.The reported low hallucination rate supports deterministic execution for calculation-intensive financial reasoning.
  • 7 Results and Analysis: CIFQA achieves near-perfect performance across most calculation-intensive categories, with the largest gains in premature-withdrawal and payout-based calculations.These categories require calendar-aware execution, exact rate grounding, and rule-based computation.
  • 7 Results and Analysis: CIFQA is comparatively weaker on Rate Analytics and RAG Rules because these tasks require higher-level table analysis, policy interpretation, and retrieval.The weakness does not reflect rate-retrieval failure, since rate lookup is deterministic.
  • 7 Results and Analysis: 90.87% overall accuracy is lower than calculation-intensive accuracy, primarily because of policy-heavy queries.The overall result includes the broader benchmark beyond the calculation-intensive subset.
  • 7 Results and Analysis: Baseline errors include structured parameter misselection, calendar-aware execution errors, and numerical precision drift in chained calculations.CIFQA avoids these failure modes by delegating numerical operations to deterministic engines.
  • 7 Results and Analysis: 48.00% accuracy after removing rolling-year adjustment makes it the most damaging ablation.This component is especially important for multiperiod calculations across year boundaries, leap years, and partial-year segments.

8 Discussion

CIFQA’s separation of language understanding from deterministic computation improves reliability beyond fixed-deposit answering. Its modular design supports domain extension while revealing remaining limitations in policy interpretation and broader validation.

  • Architectural implications: CIFQA separates probabilistic language understanding from deterministic computation, reducing arithmetic hallucinations and improving reliability and reproducibility.LLM agents handle interpretation and planning, while external deterministic engines perform numerical execution.
  • Domain generalization: CIFQA’s modular architecture extends across financial domains by replacing only domain-specific computation engines while retaining the router, extractor, planner, and response generator.The demonstrated loan EMI extension preserves the multi-agent components and swaps deterministic modules.
  • Architecture vs. scale: CIFQA enables smaller models to outperform larger ones by removing arithmetic reasoning from the LLM, making architectural design more important than model scale for exact computation.The 17B instantiation outperforms substantially larger frontier models under the evaluated conditions.
  • Limitations: Performance is comparatively lower on policy-heavy queries because complex rule interpretation extends beyond deterministic computation.This limitation reflects CIFQA’s emphasis on reliable financial computation rather than broad policy interpretation.
  • Limitations: Errors in parameter extraction and routing can propagate downstream, while the current evaluation remains focused on fixed-deposit scenarios.Broader validation across multiple financial domains remains future work.
  • Practical significance: Hybrid architectures combining LLM reasoning with deterministic computation offer a more robust approach for financial systems where numerical precision is critical.The supported scope is reliable financial computation through separation of concerns.

9 Conclusion

The conclusion presents CIFQA as a deterministic tool-grounded framework that converts financial question answering into executable reasoning. On fixed-deposit queries, it achieves high accuracy and shows that modular deterministic execution can outperform substantially larger models while supporting extension to other financial domains.

  • Conclusion: CIFQA addresses unreliable exact financial computation by separating language understanding from numerical execution in a tool-grounded multi-agent framework.The framework treats financial question answering as executable reasoning rather than only text generation.
  • Conclusion: 95.54% accuracy on calculation-intensive tasks and 90.87% overall accuracy were achieved on a curated fixed-deposit benchmark.CIFQA significantly outperformed GPT-5.3, Gemini 3, Claude Sonnet 4.6, and other evaluated models.
  • Conclusion: A 17B open-source backbone within CIFQA outperformed substantially larger frontier models, linking numerical reliability to architectural separation rather than model scale.The comparison used models evaluated with complete formulas and rate information.
  • Conclusion: Ablation results identify calendar-aware rolling-year computation and exact rate lookup as primary drivers of CIFQA’s performance advantage.The finding supports deterministic execution as the key source of reliability in calculation-intensive queries.
  • Conclusion: CIFQA generalizes beyond fixed deposits because extending it to loan amortization, bond yields, or tax settlement requires replacing domain-specific computation modules.Language-understanding components remain unchanged across these extensions.
  • Future work: Future work should extend CIFQA toward proactive financial decision support involving cross-instrument, time-horizon, and penalty comparisons.Such extensions require multi-objective planning while retaining deterministic execution.
Loading 2608.26114v1…