PaperlayerView exact revision on arXiv

Source-linked AI summary

IndicBankBench: Evaluating Safety and Reliability of Language Model Assistants in Indian Retail Banking

Suvradip Paul, Chandra Bhushan, Harsh Sharma, Nitin Kukreja, Yatharth Dedhia, Keyur Doshi, Prashant Devadiga

arXiv:2609.29167v1cs.AI

TL;DR

Banking assistants must use customer-specific records and may need to take actions, yet aggregate final-response evaluation can miss interaction failures. IndicBankBench evaluates multi-turn banking requests through staged checks, repeated trials, and case-level diagnostics. No evaluated model exceeded 60% strict reliability, and the benchmark’s scope is limited to its released synthetic environment and English Indian retail-banking interactions.

  • Problem

    Aggregate evaluation can miss where banking assistants fail when using customer-specific information or taking actions through tools.

    Source: §1 · p. 1

  • Method

    IndicBankBench evaluates 799 multi-turn Indian retail-banking cases with four-stage checks, deterministic mock tools, and three trials per case.

    Source: §1 · p. 2, §3 · p. 3, §7 · p. 8

  • Results

    No model in the eleven-model evaluation exceeded 60% strict reliability, and case-level diagnostics distinguish failure patterns.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Repeated trials and case-level diagnostics show differences in consistency and failure patterns within the released benchmark.

    Source: §7 · p. 8

    The benchmark uses synthetic customer state and deterministic mock tools, covers English Indian retail banking, and does not establish production deployment readiness.

    Source: §8 · p. 8

Abstract

from arXiv · show

Banking assistants must use account-specific information to answer requests and, in many cases, take actions through tools. Evaluating only the final response misses important errors. An assistant may ask for information it already has, rely on stale context, select the wrong account, or write an invalid value after stating the correct one. We introduce IndicBankBench, a 799-case benchmark for Indian retail banking spanning five operational domains, a capability/refusal domain, and twenty primary axes. Cases are evaluated at four stages: safety, action and tool use, response adequacy, and advisory quality. Tool use and most safety checks are deterministic. A narrow resolver handles only ambiguous confirmation-before-write cases, while a separate LLM judge evaluates semantic response adequacy. We run every case three times and report strict pass^3, which requires success on all trials. Across the eleven evaluated models, strict reliability ranges from 43.7% to 58.2%, whereas at-least-once success ranges from 60% to 74%. This gap shows that at-least-once success can overstate dependable banking behavior. The case-level diagnostics also distinguish systems that ask unnecessary questions from those that act but fail to reconcile customer context or fully resolve the request. We release the cases, mock environment, and evaluation harness.

1 Introduction

Existing aggregate evaluations can obscure where a banking interaction fails, while dependable assistance requires using customer-specific information and sometimes taking account actions. IndicBankBench evaluates these interactions in stages and pairs repeated trials with case-level diagnostics.

1 Introduction

  • Aggregate tool-calling scores can hide both where a system failed and whether it fabricated an answer or requested information already available.The paper frames this as a practical gap for banking interactions involving customer-specific records and possible financial actions.Source: §1 · p. 1
  • IndicBankBench evaluates banking interactions through ordered safety, action and tool-use, response-adequacy, and advisory-quality stages.Most checks are deterministic; an LLM judge assesses response adequacy, and a narrow resolver is used for ambiguous confirmation-before-write cases.Source: §1 · pp. 1–2
  • The benchmark releases multi-turn, account-grounded interactions with adversarial context and deterministic mock tools.Its evaluation procedure records the stage at which an interaction fails.Source: §1 · p. 2
  • The evaluation runs eleven models three times per case and reports strict repeated-run reliability alongside case-level failure diagnostics.The introduction reports a consistent gap between occasional success and reliable completion; leading systems have similar scores, with no clear winner, while their case-level failures differ.Source: §1 · p. 2

2 Related Work

Prior work evaluates tool calling, interactive agents, and financial knowledge or reasoning, but the paper positions IndicBankBench around account-grounded customer service and diagnosis of banking-interaction failures. It constructs cases for the Indian retail-banking setting rather than translating an existing benchmark.

Tool-calling and agentic benchmarks

  • Tool-calling and agent benchmarks measure tool selection or task completion, while IndicBankBench asks where a banking interaction failed across independently checked stages.The paper notes that AgentBench analyzes failure reasons but does not separate these independently checked stages.Source: §2 · pp. 2–3

Banking and finance NLP

  • Financial benchmarks span intent classification, financial knowledge and document reasoning, instruction data, and financial-agent tasks.Examples include Banking77, FinBen, FinQA (Chen et al., 2021b), PIXIU, and FinGPT (Yang et al., 2023).Source: §2 · p. 3
  • Indian-setting benchmarks cover financial knowledge and broader context grounding, whereas this work focuses on account-grounded customer service that may change account state.The paper contrasts this setting with MILU, BhashaBench-Finance (Devane et al., 2025), and IndicContextEval (Joshi et al., 2026).Source: §2 · p. 3
  • IndicBankBench targets Indian retail banking with setting-specific payment systems, identifiers, and regulatory requirements rather than translating an existing benchmark.It focuses on authenticated conversations using customer records, confirmation before account changes, and repeated cases to measure consistency.Source: §2 · p. 3

3 Benchmark Design

IndicBankBench combines 799 multi-turn cases, Indian banking context, and deterministic mock tools to evaluate grounded decisions. Its cases vary contextual and capability challenges, while ordered grading stages distinguish pass/fail gates from advisory quality.

3 Benchmark Design

  • IndicBankBench contains 799 cases across five operational banking domains, a capability/refusal domain, and 20 primary axes, using Indian payment systems and deterministic mock tools.The mock environment makes each interaction self-contained and reproducible.Source: §3 · p. 3

3.1 Interaction Environment

  • Each case supplies authenticated customer context and scripted turns; deterministic read and write mocks enforce case-specific tool arguments and call order.Cases can test the path from interpreting a request to retrieving state, responding, and taking a safe action when needed.Source: §3.1 · p. 4

3.2 The Twenty Axes

  • The 799 cases assign one primary axis each: 12 task and behavioral axes test grounding challenges, while eight test capability and refusal behavior.The operational coverage figure divides 731 task and behavioral cases across five domains and 68 capability/refusal cases across eight axes; uneven counts reflect intended emphasis, not customer-traffic prevalence.Source: §3.2 · pp. 4–5, Figure 2 · p. 5
  • The axes distinguish error sources, enabling case-level diagnosis of weaknesses such as asking before checking evidence or acting on an incorrect customer value.Similar overall scores can therefore correspond to different weaknesses.Source: §3.2 · pp. 4–5

3.3 The S/A/R/Q Grading Model

  • Safety, action/tool use, and response adequacy gate case success; advisory quality records groundedness and other qualities without changing the verdict.Safety checks are deterministic, with a narrow resolver for inconclusive confirmation; an LLM judge compares the answer, clarification, or decline with the case criterion.Source: §3.3 · p. 5

4 Experimental Setup

The experiments evaluate eleven instruction-tuned models under a shared repeated-trial protocol. Strict success requires passing all three runs, and response-stage judgments use a fixed evaluator with a separate resolver for inconclusive confirmation checks.

4 Experimental Setup

  • Eleven instruction-tuned models from eight families were evaluated with the same system prompt.Three Gemma 4 checkpoints provide within-family coverage; other candidates span seven model families.Source: §4 · p. 5
  • Each case ran three times at temperature 0.7, and pass3 required all three runs to pass.The protocol yields 2,397 trajectories per model and also reports at-least-once success and mean single-run success.Source: §4 · pp. 5–6
  • The evaluated snapshots retain the same 799 case IDs, case design, tool logic, expected behavior, and grading semantics, though some surface text differs.The linked repositories provide cases, evaluation code, and prompts.Source: §4 · p. 6
  • The response-stage evaluator was fixed at temperature 0 with seed 42, while a separate resolver handled only inconclusive confirmation checks.In a blinded audit, the evaluator agreed with adjudicated human response decisions on 31 of 38 decidable cases (81.6%).Source: §4 · p. 6

5 Results

Across eleven models, strict reliability ranges from 43.7% to 58.2%, and at-least-once success exceeds strict success for every model. Results also identify recurring weak axes and distinct safety/action versus response failures.

5.1 Overall Reliability

  • 43.7% (349/799, Gemma 4 E4B IT) to 58.2% (465/799, DeepSeek-V4-Pro-0813) is the strict pass3 range; the top five span only 11 cases.The top-five range is 56.8% to 58.2%.Source: §5.1 · p. 6, Table 5 · p. 7
  • No pair among the top five has a reliably detected difference under the paired analysis.All ten bootstrap intervals include zero, and every exact McNemar test has unadjusted and Holm-adjusted p > .05.Source: §5.1 · p. 6

5.2 Where Reliability Breaks

  • Wrong-information is the lowest-scoring axis for seven of eleven models, and no model passes more than 40.1% of its 142 cases.Three models score lowest on multi-tool tasks, and one scores lowest on bad tool responses.Source: §5.2 · p. 6

5.1 Overall Reliability

  • 10.8–21.4 percentage points separate at-least-once success from strict pass3 across all eleven models.GLM-5.3-Flash has the highest at-least-once rate, 74.3% (594/799), but a 56.8% strict rate.Source: §5.1 · p. 6, Figure 3 · p. 7

5.2 Where Reliability Breaks

  • Equal weighting of the 12 task axes swaps two adjacent model pairs in the rankings.DeepSeek-V4-Pro-0813 and Grok 4.6, and GLM-5.3-Flash and Claude Opus 5, exchange positions.Source: §5.2 · p. 7
  • The most common first blocking gates across all models are A1, a missed required tool call, and R, a response missing case-specific requirements.Among 26,291 trajectories with both decisions available, 3,390 (12.9%) passed safety and action but failed R; 1,497 (5.7%) passed R but failed a safety or action gate.Source: §5.2 · p. 7

6 Analysis & Discussion

The discussion groups recurring failures into a taxonomy and describes how to use the benchmark for fair comparisons and maintenance.

6.1 Behavioral Failure Taxonomy

  • Nine recurring failure patterns form three practical families: trusting the surface, knowing versus doing, and miscalibrated action.The taxonomy was developed from case-level gate records and transcripts in a separate five-model screening run at temperature 0, with one run per case; it describes behaviors, not evaluation-wide frequencies.Source: §6.1 · p. 8
  • Write-after-correction and filter-trap errors show how assistants can state the right fact yet act on the wrong one or accept a misleading empty result.Aggregate accuracy cannot distinguish these failure modes.Source: §6.1 · p. 8

6.2 Using and Maintaining the Benchmark

  • Fair comparisons keep cases and evaluation settings fixed, then use per-axis scores and gate outcomes to locate where a change helps or hurts.Changes to cases or grading rules require a new benchmark version and rerun; prompt comparisons also keep the model endpoint, judge, and decoding settings fixed.Source: §6.2 · p. 8

7 Conclusion

IndicBankBench evaluates whether assistants can safely and reliably complete grounded, multi-turn banking requests. In the eleven-model evaluation, no model exceeds 60% strict reliability, and the benchmark is released as a diagnostic resource.

7 Conclusion

  • IndicBankBench combines account-specific context, deterministic mock tools, adversarial interactions, and staged checks of safety, action, and response.It is designed for grounded, multi-turn customer requests.Source: §7 · p. 8
  • No model in the eleven-model evaluation exceeds 60% strict reliability.Repeated trials and case-level diagnostics distinguish over-clarification, wrong-target selection, omitted context, and acting before confirmation.Source: §7 · p. 8
  • The authors release the cases, mock environment, and evaluation harness as a diagnostic resource.The conclusion also notes release of the code and case data.Source: §7 · p. 8

8 Limitations

The limitations concern judge validity, evaluated configurations, and the benchmark’s coverage of banking interactions.

8 Limitations

  • The blinded judge audit covers only 40 transcripts from the original eight evaluated models, so its agreement may not generalize to the full benchmark.One candidate model shares the evaluator’s broader family, leaving family-level bias unresolved.Source: §8 · p. 8
  • Three trials provide only a limited estimate of run-to-run reliability, and reported rates apply to tested configurations rather than every deployment of the model families.Results may also depend on provider endpoints, serving behavior, and reasoning controls.Source: §8 · p. 8
  • Synthetic cases and deterministic mock tools do not cover every production policy or customer preference; the benchmark covers English Indian retail banking, not all languages, jurisdictions, or deployment systems.Its strict reliability results are measurements within the released environment, not deployment-readiness claims.Source: §8 · p. 8

Ethical considerations

The benchmark uses synthetic data and is intended for defensive evaluation, not as standalone evidence of live-banking safety.

Ethical considerations

  • All cases are synthetic; no real customer data or personally identifying information was used.Mock tools cannot access live financial systems or make transactions.Source: Ethical considerations · p. 9
  • Because the cases are public, future systems could be tuned to the benchmark without becoming safer in live banking; a high score alone should not justify deployment.The stated intended use is defensive evaluation, including tests of social engineering, credential extraction, and impersonation.Source: Ethical considerations · p. 9
  • Generative-AI tools assisted with case drafting, writing support, and boilerplate implementation; the authors designed the benchmark and verified the experiments, analyses, results, figures, and claims.Source: Ethical considerations · p. 9

A.1 Dataset and release information

This appendix documents the benchmark dataset and release, its axis registry and automated case checks, and a small blinded audit of the response-stage evaluator.

A.1 Dataset and release information

  • IndicBankBench contains 799 cases, 32 tools, and 20 primary axes; each JSON case specifies scripted turns, a fixed login context, deterministic mocks, and expected actions and grading rules.The cases can be replayed without a live banking backend; the released tool-definition file gives exact names, input schemas, and read/write designations.Source: §A.1 · p. 11
  • Fictional profile names and selected demographic details derive from a synthetic dataset; code is released under MIT and case-bank data under CC BY 4.0.The source dataset is NVIDIA Nemotron-Personas-India.Source: §A.1 · pp. 11–12

A.2 Axis registry

A.3 Automated case checks

  • Automated checks validate case structure and tool contracts, and ensure required tool-call identifiers are available from authenticated context, prior messages, customer references, or earlier mock results.The checks also ensure expected calls are schema-valid and mock outputs use declared fields.Source: §A.3 · p. 12

A.4 Response-Stage Evaluator Audit

  • On 38 decidable cases, the evaluator agreed with human consensus on 31 (81.6%; 95% bootstrap confidence interval: 68.4–92.1%); counting two unclear cases as disagreements gives 31/40 (77.5%; 95% confidence interval: 65.0–90.0%).The audit covers 40 cases.Source: Table 8 · p. 12, §A.4 · p. 12
  • The blinded audit was small and cannot support model-level claims or measure overall agreement; disagreements are not necessarily evaluator errors.Source: §A.4 · p. 12

B Supplementary Results

The supplementary results document uncertainty, gate-failure, and per-axis lookup analyses alongside the evaluation prompts, grading artifacts, and information supplied to the judges. They also specify the main judge’s output constraint and a separate resolver for ambiguous confirmation-before-write cases.

B.1 Uncertainty analyses

  • All paired bootstrap intervals include zero, and no McNemar comparison among the five leading models is significant before or after Holm adjustment.The analysis uses 799 case-level outcomes; because the cases were authored, its intervals do not estimate performance in live banking traffic.Source: Table 9 · p. 13, §B.1 · p. 13

B.2 Gate failure locations

  • Of 26,367 scored trajectories, 26,291 enter cross-stage comparison; the other 76 lack saved results for ambiguous S2 confirmation checks.Table 10 counts each trajectory at its first failed gate, not distinct cases; one case can contribute up to three counts. Among remaining gates, each model's largest count is A2 or A3, while Figure 4 identifies A1 and R as the two largest first-failure gates for every model.Source: §B.2 · p. 13, Figure 4 · p. 13, Table 10 · p. 14

B.3 Per-axis results and failure taxonomy

  • Table 11 reports strict pass^3 by primary axis, while capability/refusal axes have smaller denominators than behavioral axes and should not be compared directly.Table 12 defines the nine descriptive failure patterns used in §6.1.Source: §B.3 · p. 13, Table 11 · p. 14, Table 12 · p. 14

C Prompt and Grading Artifacts

  • Table 13 summarizes the three prompt artifacts and their responsibilities; the response-stage evaluator and confirmation resolver provide structured signals, not final case verdicts.The section reproduces the instructions used in the reported evaluations.Source: C Prompt and Grading Artifacts · p. 15, Table 13 · p. 15

C.1 Candidate system prompt

  • The candidate system prompt applies general banking-assistant instructions to each case, filling date, time, login context, and tool-schema placeholders at render time.It directs the assistant to rely on authenticated-session context and tool outputs, avoid inventing IDs or account-specific facts, require explicit confirmation before writes, and stay within retail-banking self-service scope.Source: §C.1 · p. 15

C.2 Response-stage evaluator

  • The response-stage evaluator receives a frozen instruction and a per-case message built from expected_resolution, judge_gate, and judge_advisory; its signals are combined with safety and action checks.A narrow confirmation resolver is separately used for inconclusive S2 cases.Source: §C.2 · p. 16

C.2.1 Main judge system instruction

  • The main judge produces structured signals against each case's grading contract, while a separate grader deterministically combines them to decide pass or fail.Its output includes behavior class, axis gate, sub-scores, and a short rationale tied to transcript turns.Source: §C.2.1 · p. 16
  • The main judge system instruction says not to add other keys.Source: §C.2.1 · p. 16

C.2.2 Per-case user message template

  • The per-case user message template supplies the case ID, axis, target behavior, expected resolution, and required behavior class and axis-gate rule.Source: §C.2.2 · p. 16
  • The template also specifies advisory metrics scored 0.0, 0.5, or 1.0, includes the transcript, and requires the judge to echo the case ID.Source: §C.2.2 · p. 16

C.3 Confirmation resolver

  • The confirmation resolver uses a separate, narrow judge to determine whether the customer explicitly confirmed the specific action before a write call.An affirmation about another topic, an ambiguous affirmation, or no clear affirmation before the call counts as unconfirmed; this resolver does not decide the case verdict.Source: §C.3 · p. 16
  • The resolver treats a clear affirmation naming the mandate or cheque, or an unambiguous yes to the confirmation prompt, as confirmation.Source: §C.3 · p. 16

Related papers

Loading 2609.29167v1…