PaperlayerView exact revision on arXiv

Source-linked AI summary

FiMI Banking: A Sovereign Model for Indian Retail Banking

NPCI AI Research Team, Aman Kumar, Asit Desai, Chandra Bhushan, Harsh Sharma, Harshit Bhushan, Hrithik Kadam, Keyur Doshi, Kolisetty Sai Kapardheeswar, Krishanu Adhikary, Nadeem Shaik, Navya Prakash, Nitin Kukreja, Prashant Devadiga, Shamanth MH, Shantanu Pandey, Suvradip Paul, Yatharth Dedhia

arXiv:2609.03960v1cs.AIcs.CL

TL;DR

Banking agents need grounded, policy-compliant answers and reliable multi-turn account actions, but general-purpose models do not consistently provide both. FiMI Banking builds a controlled Indian retail-banking environment and evaluates preference optimization alongside reinforcement learning with verifiable rewards. Preference optimization improves behavioral alignment, while reinforcement learning improves edge-case and order-sensitive task execution with lower generation cost.

  • Problem

    General-purpose language models do not reliably combine grounded banking answers, correct tool use, and cautious handling of sensitive bank-specific situations.

  • Method

    FiMI Banking uses vetted banking documents, structured scenarios, synthetic customer data, banking tools, preference optimization, and reinforcement learning with verifiable rewards.

  • Results

    Preference optimization improves behavioral alignment, while reinforcement learning improves edge-case and order-sensitive task execution and reduces generated-token cost.

  • Takeaways & Limitations

    Preference optimization and verifiable-reward reinforcement learning address complementary requirements for reliable banking agents.

  • Takeaways & Limitations

    The reported evaluation does not isolate whether teacher-sourced or self-rephrased preference pairs are better.

Abstract

from arXiv · show

Banks need conversational systems that can answer product questions, assist customers with account-related requests, and operate safely within strict operational and regulatory constraints. General-purpose language models do not reliably meet these requirements. They fall short when a task requires grounded information, correct tool use, or cautious handling of bank-specific sensitive situations. We introduce FiMI Banking, a controlled Indian retail-banking setting. We build it from vetted banking documents, structured ground truth, synthetic customer backgrounds, and banking tools. We evaluate two post-training approaches: preference optimization for response-level behavior, and reinforcement learning with verifiable rewards for multi-turn tool-use tasks. Preference optimization improves safe behavior substantially: out-of-scope refusal rises from 52% to 80%. Reinforcement learning improves edge-case performance from 0.509 to 0.718 and order-sensitive task performance from 0.590 to 0.679, while using 29% fewer generated tokens. These results show that preference optimization and verifiable-reward reinforcement learning address complementary requirements for reliable banking agents.

1 Introduction

FiMI Banking targets Indian retail-banking conversations where safe, grounded answers must be combined with correct multi-turn account actions. It evaluates preference optimization for response behavior and reinforcement learning for tool-use trajectories.

  • Motivation: Banking assistants must follow bank-specific policies, protect sensitive information, use operational tools correctly, and ground answers in customer and product context.Supported requests include KYC, EMIs, government schemes, insurance claims, and tax deducted at source.
  • Contribution: FiMI Banking introduces a small Indian retail-banking model intended for bank-controlled hardware, including fully air-gapped settings.The model uses an open family released under Apache 2.0 so banks can control weights and specialize products and tool contracts.
  • Motivation: Correct account actions require valid tool arguments, ordered workflows, authoritative grounding, missing-detail requests, confirmation before state changes, and out-of-scope refusal.Single-response metrics cannot capture these interaction-level requirements.
  • Environment: The environment contains five retail use cases, executable scenarios, banking tools, and a correct gold chain of tool calls for every task.Training and evaluation follow the τ-bench family, with simulated customers acting on shared account state.
  • Post-training: Preference optimization improves response-level behavior, whereas reinforcement learning improves complete multi-turn tool-use trajectories against verifiable rewards.The studies use different corpora, evaluation sets, judges, and metrics, so they are reported separately.

2 Background and Related Work

Prior finance-domain work establishes domain specialization but largely targets question answering rather than account actions. FiMI Banking addresses the gap with a customizable environment whose tools, scenarios, and verifiable reward jointly support training and evaluation.

  • Finance-domain models: Finance-domain modeling has used frontier-scale financial pre-training and open-model adaptation with financial instruction data.These routes establish domain specialization and increasingly include small open-weight families suitable for bank specialization.
  • Synthetic tool-use data: Synthetic generation with verification is the standard response when tool-use data must connect requests to structured actions over changing system state.Related approaches generate verified task blueprints and simulate persona-conditioned interactions.
  • Preference learning: Preference learning depends strongly on how response pairs are constructed, while response differences can also reflect unrelated factors such as length.Direct preference optimization uses a log-ratio margin against a frozen reference policy.
  • Reinforcement learning over dialogs: Multi-turn reinforcement learning scores complete trajectories, where out-of-order calls can surface later and turns share credit for a delayed score.GRPO samples groups of complete dialogs and is usable when rewards are program-checkable.
  • The gap: Existing strands do not provide a bank-customizable environment that can remain aligned with deployment-specific use cases and tool contracts.Public benchmarks use fixed domains and rules, limiting modification for a particular bank.
  • The gap: FiMI Banking builds the needed instrument from use cases, executable tools and scenarios, and a verifiable reward shared by training and evaluation.The reward is designed so score improvement corresponds directly to solving the specified use cases.

3 The Banking Setting

The banking setting combines five retail use cases with vetted knowledge sources, synthetic replayable data, and executable tools in a simulated shared-state environment. The same implementation supports isolated training and bank-controlled deployment.

  • Use cases: FiMI Banking centers on five retail-banking use cases and the core behaviors each requires.The scope covers retail tasks involving multi-turn reasoning and tool-mediated action.
  • Use cases: The assistant must order tools correctly, check eligibility before irreversible actions, obtain confirmation after disclosing charges, ground answers in documents, request missing information, and refuse out-of-scope requests.These requirements are shared across all five use cases and both post-training studies.
  • Corpora: The system creates separate conversation and task corpora: persona-conditioned dialogs support preference pairs, while task families with gold chains support reinforcement-learning rollouts.The datasets use separate evaluations and are not combined.
  • Environment: Each episode is a complete agent-customer conversation over shared account state, with tool calls recorded as a trajectory and customer-held details revealed only when requested.Passing requires both appropriate customer interaction and correct calls against the account database.
  • Serving, isolation, and determinism: The same tool code runs in isolated per-rollout databases during training and through deployment infrastructure, so parallel rollouts cannot affect one another.This makes training rewards match the implementation served in deployment.
  • Serving, isolation, and determinism: Seeded databases make outcomes deterministic: replaying a task reproduces its result and score, allowing the environment to serve as the evaluation harness.The setup also constrains simulator drift and catches agents that mirror customers without making correct tool calls.
  • Grounding: The knowledge tool retrieves only from vetted, licensed banking documents filtered for relevance, currency, and duplication, with source and date retained for traceability.Sources include RBI material, official scheme documents, product terms, and bank operational material.
  • Corpora: The corpora are fully synthetic, including personas, identities, account data, transactions, and database states, and are filtered before use.Neither corpus contains a real customer conversation.

4 Preference Optimization

FiMI Banking constructs preference data from validated synthetic banking scenarios and evaluates preference optimization for response-level behavior. The approach improves conduct and safety substantially, while gains do not extend to multi-tool ordering or complete-trajectory reliability.

  • Preference data construction: Preference pairs come from candidate-model failures replayed against validated conversations, with each divergent action rejected and a constructed preferred response.The pairs drive DPO updates against a frozen reference policy.
  • Preference data construction: Teacher-sourced preferred responses use a larger model, but the reported checkpoint is not attributed to either preferred-response construction.The evaluation therefore measures preference optimization on candidate failures rather than comparing the two constructions.
  • Evaluation setup: The benchmark uses synthetic personas, account states, expected tool calls, ordering constraints, and reproducible simulated-backend trajectories.Each case includes a target behavior axis and disclosures required before acting.
  • Results: Preference optimization moves all four aggregate metrics and reaches the range of substantially larger models, while DeepSeek V4 Pro remains highest on this set.The trained checkpoint also passes MiniMax M3 on achievability.
  • Results: Capability cases rise 22 points, whereas banking-task categories move only a few points at most because the training data target conduct failures.The capability tier covers safety and adversarial framing.
  • Results: Conduct and safety axes gain by as much as 42 points on social engineering, but multi-tool chains do not improve.A turn-level preference signal does not reward completing all steps of a multi-tool trajectory.

5 Reinforcement Learning in the Verifiable Environment

The reinforcement-learning setting evaluates agents on structured Indian retail-banking tasks requiring ordered tool use, customer information gathering, and safe state changes. Its corpus contains 48,245 generated tasks, with 10,000 used for training and a disjoint, family-stratified 1,000-task held-out set.

  • Environment and task design: The task environment requires valid tool arguments, ordered workflows, grounded policy claims, missing-detail requests, confirmation before state changes, and out-of-scope refusal.These requirements make action order part of correctness rather than merely a response-quality attribute.
  • Environment and task design: Each family template instantiates a scenario with a customer, product, amounts, and database state, and each task belongs to one of four task kinds.The four kinds are happy path, sequence, edge, and tools.
  • Environment and task design: Tasks specify gold tool calls and arguments in order, while the simulated customer reveals key fields incrementally during the interaction.The example gold chain is balance, rates, book, then balance again.
  • Corpus construction: 48,245 tasks span 100 families, expanding to 408 named variants across product and channel prefixes.The corpus includes shallow families with chains up to 6 actions and deeper families with chains up to 8 actions.
  • Corpus construction: 10,000 training tasks pass replay, argument, corpus-gate, and family-stratification requirements, while the held-out set contains 1,000 fixed, disjoint, family-stratified tasks.The held-out set is drawn from shallow-slice families; graded and communication families are excluded from training.

5.2 The reward and its audit

The environment scores complete trajectories using task-specific weighted checks for tool order, database state, customer communication, and judged assertions. An audit found exact agreement for deterministic checks on real trajectories, while only judged assertions require a live judge.

  • Trajectory scoring: A rollout alternates policy and simulated-customer turns against a private seeded bank copy, then converts the completed trajectory into a dense reward.The same loop produces a GRPO advantage during training.
  • Reward checks: The four checks assess ordered tool calls, final database state, required customer-facing values, and correctness of unverifiable assertions.The judged-assertion check is the only non-deterministic component.
  • Reward computation: For tasks declaring all checks, the reward is 0.40 r_seq(τ) + 0.25 r_db(τ) + 0.15 r_comm(τ) + 0.20 r_judge(τ).Tasks declaring fewer checks average only over their declared checks and weights.
  • Reward computation: The sequence metric uses longest-common-subsequence order matching, but one out-of-order step makes the sequence component zero through an all-or-nothing gate.A swapped step in an eight-step chain yields seq_frac = 7/8 before the gate is applied.
  • Audit: On 600 real trajectories, the two deterministic checks matched the independent reference scorer’s pass/fail decisions in every case.Only the judged-assertion check remains unauditable offline; 91.3% of the corpus uses only deterministic checks.

5.3 Evaluation protocol

TauIndianBankBench is a fixed held-out evaluation for Indian retail-banking tool-use tasks. Models use identical task lists and seeded databases, with average dense reward reported alongside category breakdowns.

  • Benchmark: TauIndianBankBench uses the tasks, scorer, and simulated customer defined for this held-out evaluation route.The split, subsets, and metrics were fixed before training began.
  • Evaluation protocol: Each model attempts every capability-ladder and before-and-after task once using identical task lists and seeded databases.Learnable-band measurements use two attempts per task, with no resampling between models or model versions.
  • Evaluation protocol: Average dense reward from Eq. 3 is the primary metric, reported with per-category breakdowns to expose differences hidden by aggregate scores.The protocol uses one trial per task for the main score.
  • Evaluation protocol: The held-out set ranks base models, defines the training target, and evaluates before-and-after performance, while the deep slice supports later analysis.The held-out set provides the main results.

5.4 The ladder and the learnable band

The capability ladder shows how banking performance changes with model size, while the learnable band estimates which tasks remain inconsistent across repeated trials. Small models offer the largest adjacent reward gains, and the middle band remains near one-fifth of tasks for most models.

  • Capability ladder: E2B, E4B, 12B, and 31B form an increasing-size ladder, supplemented by 26B-A4B and MiniMax-M2.7 mixture-of-experts references.All models are scored on the same held-out tasks.
  • Capability ladder: The adjacent reward gains are +0.127 from E2B to E4B, +0.080 from E4B to 12B, and +0.070 from 12B to 31B.MiniMax-M2.7 scores above the 31B rung.
  • Capability ladder: The happy-path column stays flat across the ladder’s 13× parameter range, providing a stable control while other task axes increase with size.It is the only column where the size ordering breaks.
  • Capability ladder: The 31B reference nearly matches MiniMax-M2.7 on edge cases but trails it on tool coverage, while the 12B reference leads the control.These comparisons show that model-size ordering differs by task category.
  • Capability ladder: E4B is selected for training because it is positioned to gain the most, with the 12B rung serving as the target near three times its effective size.The 31B reference is about seven times E4B’s effective size.
  • Learnable band: The learnable band consists of tasks that pass once and fail once across two trials, rather than tasks that always fail or always pass.Band width reflects trial-to-trial consistency as well as capability.
  • Learnable band: The learnable band stays near a fifth of tasks for five of six models, while always-fail shrinks and always-pass grows across the ladder.Only the 31B reference has a clearly narrower band.

5.5 The training run

The run trains E4B with GRPO using grouped rollout rewards and selects step 180 because held-out reward peaks there despite continued training-set improvement.

  • Setup: E4B is trained with GRPO using groups of four rollouts and 64 trajectories per step on the policy node.The setup uses five nodes: one for policy training, two for the simulated customer, and two for the judge.
  • GRPO update: Each rollout’s advantage compares its reward with the group average, scaled by the group’s standard deviation.Positive advantages reinforce actions, while negative advantages discourage them.
  • GRPO update: Tasks produce no gradient when all four rollouts receive the same reward, making the learnable band the effective training subset.The update assigns the same episode-level advantage to every generated token in a rollout.
  • Training progress: Held-out reward peaks at step 180 and then falls back, although training reward continues rising after roughly the first 150 steps.Step 180 is therefore selected as the final trained checkpoint for subsequent evaluations.

5.6 Results and serving cost

GRPO improves held-out banking performance, especially on edge cases, while preserving general capability and reducing generated-token and inference-compute costs.

  • Results: Average held-out reward rises from 0.610 to 0.697, exceeding the 12B reference at close to a third of its effective parameters.The trained model reaches the ladder’s 12B target and covers 45% of the distance from base E4B to MiniMax-M2.7.
  • General capability: The trained model preserves general capability because benchmark shifts across broad knowledge, science, instruction following, coding, and reasoning are small and bidirectional.These benchmarks are not included in training.
  • Results by axis: Edge reward rises from 0.509 to 0.718, while seq rises from 0.655 to 0.713 and tools from 0.487 to 0.526.The edge gain is +0.209, more than three times any other axis, and ends within a hundredth of the 31B reference.
  • Dialog behavior: Generated tokens per dialog fall by 29%, mainly through shorter turns rather than fewer turns, while tool calls increase because missing steps are supplied.The trained model asks for the needed detail, calls the tool, and reports the result instead of restating requests and listing unnecessary options.
  • Serving cost: The trained model generates fewer tokens than every ladder model except the 31B reference, and decode cost is 30% below its own base.Total compute also falls because shorter contexts offset the cost of additional tool calls.
  • Serving cost: The trained model exceeds the 12B reference’s reward at less than a third of its compute per dialog while holding 2.7 times fewer weights in memory.The 31B reference gains 0.063 more reward for seven times the compute.

5.7 Additional analyses

Additional analyses show that strict tool ordering matters mainly for deeper chains, training narrows the learnable band, and the observed gains align with the corpus’s available room for improvement.

  • Order-strict scoring: Ordering barely affects shallow chains: the strict-scoring penalty stays under 0.04 despite overall scores spanning 0.48 to 0.80.At this depth, remaining errors involve arguments and missed steps rather than tool order.
  • Order-strict scoring: On deep chains, strict scoring lowers E4B from 0.861 to 0.820 and MiniMax-M2.7 from 0.922 to 0.892.Ordering separates models where set-based scoring does not.
  • Order-strict scoring: Under the strict gate, trained E4B rises from 0.590 to 0.679 and its in-order match fraction from 0.861 to 0.919.Because the gate discounts correctly selected tools used out of order, the gain reflects sequencing skill rather than scoring leniency.
  • Graded families: Adding easier task tiers leaves E2B’s deep-slice learnable share unchanged at 20.5% when graded families comprise just under 10% of the corpus.Band width follows the difficulty mix of the whole corpus.
  • Band migration under training: After training, the held-out score rises from 0.595 to 0.694, always-pass increases fourteen points, and always-fail falls about six points.The learnable band narrows from 26.0% to 17.7%, while 17.7% still splits across trials.
  • Synthesis: The analyses jointly indicate that ordering separates models on deep chains, easier tiers matter in proportion to corpus share, and E4B gains target axes with the most room.The band supplying those gains becomes measurably smaller after training.

6 Discussion and Analysis

The two post-training approaches address different reliability needs: preference optimization mainly improves response-level behavior, while reinforcement learning improves complete tool-use trajectories. The gains include stronger safety behavior, better task execution, preserved general capability, and lower serving cost, but preference-pair construction and single-run judging remain unresolved limitations.

  • Preference optimization: Preference optimization sharply improves behavioral axes, while capability axes change little.Behavioral gains include safer refusal and handling of sensitive requests; multitool reasoning barely improves.
  • Preference optimization: Out-of-scope refusal rises from 52% to 80%, while multitool chains move only from 20% to 21%.Asking rather than guessing rises from 38% to 46%, and credentials and inappropriate content reach 100%.
  • Limitations: The evaluation does not isolate teacher-sourced versus self-rephrased preference pairs, and judged results come from one model and one judge without intervals.These caveats leave the mechanism comparison unsettled and make the reported estimates point estimates.
  • Reinforcement learning: Reinforcement learning improves edge cases from 0.509 to 0.718, sequencing from 0.655 to 0.713, and tool coverage from 0.487 to 0.526.These metrics measure tool selection, arguments, and ordering.
  • Reinforcement learning: 0.697 held-out reward exceeds the 12B reference’s 0.690, while the order-strict gate rises from 0.590 to 0.679.The in-order match fraction also rises from 0.861 to 0.919.
  • Efficiency and validation: 29% fewer generated tokens per dialog accompany lower compute than the 12B rung, while six public benchmarks move only slightly in both directions.The reported reward agrees with an independent reference scorer on 97.2% of reward mass.

7 Conclusion

The paper builds a controlled Indian retail-banking environment and evaluates two post-training routes on a small open-weight model. Preference optimization improves conduct, while verifiable reinforcement learning improves tool-sequence execution above a much larger base model with fewer generated tokens.

  • Contribution: FiMI Banking combines five use cases, executable scenarios and tools, a simulated bank, and independently checked scoring.The environment is designed so claims are measured within a replayable banking setting.
  • Results: Two post-training routes on the same 4.5B-parameter model produce complementary gains: preference optimization improves conduct, while verifiable reward improves task execution.Preference gains are largest on adversarial and out-of-scope requests.
  • Results: The trained model completes tool sequences above a base model nearly three times its size while generating fewer tokens.This result concerns banking-task execution under verifiable evaluation.
  • Model choice: E4B pairs sufficient capability for the full task set with the lowest serving cost among the evaluated models.The model was selected because smaller models failed many tasks and larger models added less capability at greater cost.
  • Indian banking scope: The learned behavior covers Indian banking particulars including government-scheme eligibility, insurance claims, and TDS on deposit interest.The paper frames these behaviors and tool sequences as requirements of the regulatory environment.
  • Reuse: The replayable task corpus, executable environment, and audited reward make the evaluation re-runnable and adaptable to other regulated domains.Changing the tool catalog and use cases is the stated route to reuse.
Loading 2609.03960v1…