PaperlayerView exact revision on arXiv

Source-linked AI summary

TauIndianBankBench: Procedure-Aware, Action-Oriented Evaluation of Tool-Using Conversational Agents

Krishanu Adhikary, Prashant Devadiga

arXiv:2610.30161v1cs.CLcs.AI

TL;DR

Final-record checks can miss required banking steps that leave records unchanged, limiting evaluation of whether assistants follow bank procedures. TauIndianBankBench tests simulated Indian retail-banking tasks with action and final-state checks. Across the evaluated models, final-state checks alone accept procedural violations, while each check catches failures the other misses.

  • Problem

    Many policy-required banking steps leave records unchanged, so final-state-only benchmarks can miss whether an assistant followed the required procedure.

    Source: §1 · p. 1

  • Method

    TauIndianBankBench evaluates simulated Indian retail-banking tasks with checks for required calls, unrequested changes, and final records.

    Source: §1 · p. 2

  • Results

    The final-state check accepts procedural violations for every evaluated model, and each check fails sessions that the other passes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The benchmark’s authors recommend reporting action checks alongside final-state checks for service assistants in regulated domains.

    Source: §6 · p. 7

    Scores depend on the simulated customer, whose error rate was not measured, and synthetic data make edge-case frequencies unlike established real-bank traffic.

    Source: Limitations · p. 7

Abstract

from arXiv · show

Many Indian banks serve customers through FAQ pages and scripted chatbots. Assistants based on language models could respond faster and carry out requests such as blocking a card, but only if they give correct information and follow the bank's rules. These rules prescribe a procedure, a set of required steps, such as obtaining a closure quote before closing a deposit. Adapting models to this domain requires measuring how well they follow the procedure, before and after training. Benchmarks usually judge only the records left at the end of a conversation. Many required steps, however, leave the records unchanged, so an assistant that skips them can still be judged successful. We present TauIndianBankBench, an action-oriented benchmark of 1,000 Indian retail-banking tasks where an assistant helps a simulated customer. Besides the final records, it checks that the assistant took every required step and made no unrequested change. For every model, 6.5-12.0% of sessions are judged successful when only the final records are checked, although the assistant skipped a required step or made an unrequested change. Scoring both procedure and outcome shows more accurately how well language models perform in Indian retail banking.

1 Introduction

Language-model assistants could serve Indian bank customers, but evaluation must measure both correct outcomes and compliance with required banking procedures. TauIndianBankBench evaluates both through simulated customer-service tasks and shows that final-state checks alone miss procedural violations.

1 Introduction

  • Existing service benchmarks usually check resulting records, but policy-required steps such as obtaining a closure quote may leave records unchanged and therefore go undetected.An assistant can pass a final-state check after skipping a step intended to provide information to the customer.Source: §1 · p. 1
  • TauIndianBankBench provides 1,000 Indian retail-banking tasks, a synthetic bank, servicing tools, written policy, and a knowledge base for evaluating assistants with simulated customers.Tasks include dependent-call sequencing, edge cases, infrequently used tools, and straightforward controls; smaller train and test splits support adaptation.Source: §1 · p. 2
  • The action check requires every gold-chain call, including required reads, and rejects unrequested changes; a session passes only when both action and final-state checks pass.The benchmark evaluates six open-weight models and reports that each check catches sessions the other passes.Source: §1 · p. 2
  • Final-state checks accept sessions with omitted required calls or unrequested changes, while the action check detects these procedural violations for every evaluated model.For the three strongest models, most sessions failing exactly one check are procedural violations; for the weakest, most such sessions include all required calls but leave incorrect records.Source: §1 · p. 2
  • The paper releases code, data, and reference results alongside the benchmark and evaluates models under both final-state and action checks.Its stated contributions include an environment with tasks derived from written policy and an action check that complements final-state scoring.Source: §1 · p. 2

2 Related Work

Prior work covers tool use, interactive task outcomes, process compliance, and financial or Indian-language evaluation, but these strands do not together assess Indian banking service procedures and outcomes. TauIndianBankBench combines a shared database and final-state scoring with policy-mandated action checks.

The 𝜏-bench family

  • TauIndianBankBench requires every read mandated by its written policy and fails sessions where the assistant makes an unrequested change, even if it later reverts that change.Source: §2 · p. 2

Conversational and enterprise tool-use benchmarks

  • Interactive and enterprise benchmarks evaluate tool use and outcomes, but terminal-state checks cannot detect required steps that leave records unchanged.AppWorld checks collateral damage, while Thinkingbox rejects wrong, missing, or extra terminal-state effects. (Trivedi et al., 2024; Li et al., 2026)Source: §2 · pp. 2–3

Process and policy-compliance scoring

  • Process-compliance benchmarks score agent steps, but TauIndianBankBench combines policy-required reads and write purity with shared-database final-state checking on every session.ToolSandbox checks milestones and forbidden minefields; JourneyBench rejects missing, extra, or out-of-order calls but uses pre-generated tool responses. (Lu et al., 2025)Source: §2 · p. 3

Financial and Indian-language benchmarks

  • Most financial LLM benchmarks test knowledge and numerical reasoning, while others enable tool use; Indian-context evaluation has focused on Indian languages and India-specific knowledge, with a bilingual finance subset.Source: §2 · p. 3

Validity of agent evaluation

3 Benchmark Design

TauIndianBankBench pairs a synthetic Indian banking environment and policy-derived servicing tasks with deterministic checks of both required actions and final records. A scorer audit found agreement between the released scorer and an independent implementation on all audited sessions and paired tool calls.

3.1 Environment and tasks

  • The environment provides 944 synthetic customers, 24 policy knowledge-base documents, and 34 tools, including 33 servicing tools for banking products and a human-transfer tool.Thirteen servicing tools are writes and twenty are reads; two writes request services without changing records.Source: §3.1 · p. 3
  • Each task specifies a date, logged-in customer, persona, scenario, and gold chain of tool calls with matching arguments; gold chains average 3.54 calls and range from one to six.Tasks span 50 request families, four categories, and everyday banking, deposits, loans, and cards.Source: §3.1 · p. 4
  • Task construction filters require varied first messages, at least three calls for graded tasks, no train-test overlap, and a gold-chain replay that passes both scoring checks.The release includes base, lite, train, and test splits; scores across different splits are not comparable.Source: §3.1 · p. 4

3.2 Scoring

  • A session passes only if it makes every gold-chain call, makes no unrequested change, ends normally, and leaves records matching the expected state; otherwise it receives zero.There is no partial credit: missing one of four required calls scores the same as doing nothing.Source: §3.2 · p. 4
  • The final-state check compares normalized session records with records from replaying the gold chain, while the action check searches for each required call and its specified matching arguments.Because reads do not alter records, the final-state check cannot detect an omitted read; the action check requires policy-mandated reads to appear.Source: §3.2 · pp. 4–5

Scorer audit.

  • Write purity fails a session for an unrequested change even when the change is later undone, and required steps are ordered by dependencies or numbered customer instructions.The independent scorer reimplementation matched the released scorer on 600 sessions and 705 paired tool calls; a 5,000-session packaged-scorer audit also reproduced every score.Source: §3.2 · p. 5, Scorer audit · p. 5

4 Evaluation Protocol

Evaluation sessions alternate between assistant and simulated-customer turns under fixed stopping rules. Reported comparisons specify the assistant model, customer simulator, and split together because the simulator shapes conversations.

4 Evaluation Protocol

  • Each session alternates assistant and customer turns; the assistant sends a message or makes tool calls on a turn, and the session ends on resolution, assistant termination, or human transfer.Sessions reaching 200 steps, 10 failed tool calls, or an error score zero; the simulated customer reveals information stepwise and raises graded-task steps in order.Source: §4 · p. 5
  • Results are reported with the assistant model, customer-simulator model, and split because the simulator shapes every conversation; the reference results use MiniMax-M2.7 at temperature zero and one attempt per task on the base split.A run may repeat each task k times to estimate pass_k, the probability that all k attempts at a task pass.Source: §4 · p. 5

5 Results

On the 1,000-task base split, six open-weight models were evaluated with the same simulated customer; the results compare their pass rates under procedure-aware scoring. Scoring only final records overstates success, and performance varies across task categories and procedure lengths.

5.1 Reference results

  • Among the four dense Gemma models, each larger model passed more tasks than the next smaller one, with the same ranking in all three graded categories.Each adjacent-model comparison was significant at p < 0.005.Source: §5.1 · p. 6

5.2 Violations missed by the final-state check

  • 6.5–12.0 points of each model’s pass rate came from sessions that passed the final-state check despite procedural violations.Missing required writes were the most frequent action-check failure for four models, and missing required reads for the two smallest models; loan tasks had the highest violation rate.Source: §5.2 · p. 6

5.3 Complementarity of the two checks

  • The checks were complementary: the strongest models more often failed only the action check, while E2B more often passed action checks but ended with incorrect records.For the 12B and E4B, the two disagreement types did not differ significantly.Source: §5.3 · p. 6

5.4 Performance by task category

  • Sequencing separated models most widely; every model scored lower on edge cases, while rare-tool tasks were every model’s lowest-scoring category.Sequencing pass rates ranged from 0.455 for E2B to 0.921 for Qwen3.8-27B; rare-tool scores for Gemma models ranged from 0.388–0.559, and Qwen3.8-27B exceeded the 31B by 0.198.Source: §5.4 · pp. 6–7

5.5 Effect of procedure length

  • More required calls did not uniformly make tasks harder; long sequencing and edge-case tasks affected weaker models most, while Qwen3.8-27B declined most on rare-tool tasks.E4B fell from 0.810 on short sequencing tasks to 0.445 on long ones; the three strongest models changed by no more than 0.086 on sequencing or edge-case tasks.Source: §5.5 · p. 7

5.6 Control category

  • Across the four dense Gemma models, control pass rates stayed between 0.804 and 0.929 without a size trend, unlike sequencing pass rates.The highest control rate belonged to the 12B model.Source: §5.6 · p. 7

5.7 Trivial-agent baselines

  • None of the four scripted agents passed any task: agents that omitted required calls failed the action check, while one that made every available change failed write purity.The first three agents respectively only replied, looked up records without making a required call, or claimed completion persuasively.Source: §5.7 · p. 7

5.8 Further analyses

  • Stopping short of the full procedure was the most frequent failure type for every model; a second Gemma run changed pass rates by at most 2.7 points.Between 14–26% of tasks changed outcome between the two runs.Source: §5.8 · p. 7

6 Conclusion

TauIndianBankBench evaluates whether Indian retail-banking assistants both follow required procedures and reach correct outcomes. Its results support reporting action and final-state checks together.

6 Conclusion

  • Final-state scoring alone is incomplete because many required steps do not change records.The benchmark recommends reporting an action check alongside the final-state check.Source: §6 · p. 7
  • The action check scores required steps, while the final-state check evaluates whether records are correct.The release provides separate train and test splits for measuring adapted models before and after training.Source: §6 · p. 7
  • Each check fails sessions that the other passes, and the final-state check alone accepts procedural violations for every evaluated model.Source: §6 · p. 7

Limitations

The reported evaluation has limits tied to its simulated customer, synthetic records, domain coverage, and computational cost.

Limitations

  • Reported scores depend on the simulated customer, whose own error rate was not measured.Thus, the numbers hold only for the simulator used.Source: Limitations · p. 7
  • The synthetic records determine the frequency of edge cases, which may not reflect actual bank traffic.Version 1.0.0 covers Indian retail banking in English only; other languages and domains are left to future versions.Source: Limitations · p. 7
  • A base-split run requires 1,000 live conversations of up to 200 steps each, and pass_k estimation multiplies that cost by k.The 200-task lite split is for development and ablations; the base split is for final reporting of untuned models.Source: Limitations · p. 8

Ethics Statement

The benchmark uses synthetic customer data and is intended for research and evaluation, not financial advice or deployment certification.

Ethics Statement

  • All customer data are synthetic, with no field for a national identification number.Source: Ethics Statement · p. 8
  • A passing result indicates correct records and required calls with the simulated customer, not that an assistant is suitable for bank deployment.Deployment would require further safety, privacy, and regulatory review.Source: Ethics Statement · p. 8
  • The benchmark is intended for research on and evaluation of tool-using agents and does not constitute financial advice.Source: Ethics Statement · p. 8

A Comparison with Related Benchmarks

Table 3 compares benchmarks across the state and action properties they score, as well as whether call order matters. The tool catalogue groups servicing tools by domain and distinguishes writes from reads.

A Comparison with Related Benchmarks

  • Table 3 compares benchmarks on final-state scoring, required-call presence, read checks, write purity, and call-order dependence.Its legend distinguishes full, partial, and absent coverage.Source: Table 3 · p. 11
  • Call order matters for a pass when the scorer checks it or the task design fixes it.Source: Table 3 · p. 11

B Tool Catalogue

  • The catalogue lists servicing tools by entity domain; 13 tools are treated as writes, while all other tools are reads.A 34th tool transfers the customer to a human staff member.Source: Table 4 · p. 11

C Example Task

The example task requires the assistant to review a projected fixed-deposit maturity value before booking the deposit and then show the post-debit balance. Its five-action gold chain includes several reads, while only the deposit booking changes the records.

C Example Task

  • The customer requests a Rs.2,00,000 fixed deposit for 12 months only after reviewing its projected maturity value, then asks to see the post-debit savings balance.The maturity projection uses the current rate card; the customer’s instructions are for the simulator and are not shown to the assistant.Source: C Example Task · p. 11
  • Three of the four distinct calls are reads; the final balance read repeats the initial one, so a single balance read satisfies both.The gold chain has five calls in total.Source: C Example Task · p. 11, Figure 2 · p. 12
  • Only create_fd changes records, so the final-state check cannot establish whether the assistant consulted the rate card and maturity projection.The action check can determine whether those reads occurred.Source: C Example Task · p. 11, Figure 2 · p. 12

D Dataset Statistics by Category

The dataset groups tasks into categories and families, with task chains recording the calls and conditions needed to resolve each request. The scoring pseudocode describes session scoring, including normalized state comparison for the final-state check.

D Dataset Statistics by Category

  • The released base split is organized into four task categories, with Table 5 reporting the number of tasks in each category.Table 6 describes category composition through family counts, gold-chain length, short and long tasks, read-call share, and tasks containing a write.Source: Table 5 · p. 12, Table 6 · p. 12

E Scoring Pseudocode

  • Algorithm 1 gives the scoring procedure for one session, and its final-state check replays the gold chain to produce the expected state.The replay uses the task seed; normalized hashes are compared in the displayed final-state check.Source: E Scoring Pseudocode · pp. 12–13

D Dataset Statistics by Category

  • The 50 task families cover sequencing, control, edge-case, and rare-tool tasks, with typical call chains or conditions specifying what each family requires.The edge-case families include requests to decline unsafe actions, correct or clarify information, distinguish similar items, or limit commitments to what an account supports.Source: D Dataset Statistics by Category · p. 12, Table 7 · p. 13

E Scoring Pseudocode

  • The scorer’s MATCH operation compares tool names and specified arguments while ignoring fields that vary between runs; NORM removes those fields and ignores collection-entry order.The scorer was compared with an independent re-implementation, as summarized in Table 8.Source: E Scoring Pseudocode · p. 13, Table 8 · p. 13

G Additional Results

Additional analyses compare procedural violations across service areas, pass rates across chain lengths, failure types, and outcomes across repeated attempts. They show how these patterns vary by task area, chain length, and model strength.

G Additional Results

G.1 Violations by service area

  • 20.8% of pooled loan-task sessions are procedural violations, compared with 7.9–10.5% in each of the other three service areas.The pooled counts cover six models and the tasks assigned to each area.Source: §G.1 · p. 14, Table 10 · p. 14

G.2 Pass rates by gold-chain length

  • E2B scores lowest on five-call tasks, while every other model scores higher on five- and six-call tasks than on four-call tasks.On six-call tasks, the gap between Qwen3.8-27B and E2B narrows to 0.423; Qwen3.8-27B also scores higher on long than short sequencing and edge-case tasks, while E2B falls from 0.543 to 0.364 on edge cases.Source: §G.2 · p. 14, Table 11 · p. 14, Table 12 · p. 15

G.3 Failure types

  • Stopping short accounts for 49.6–64.0% of failed sessions, while unrequested writes are more common among failures for the two strongest models than for the three weakest.Unrequested writes account for 34.4% and 29.0% of failures for Qwen3.8-27B and the 31B, versus 9.7–13.5% for the three weakest; finishing with errors accounts for 13.8–15.9% for the three strongest and 28.2–36.9% for the three weakest.Source: §G.3 · p. 15, Table 13 · p. 15, Figure 6 · p. 15

G.4 Outcomes of two attempts

  • The two-attempt analysis groups tasks by their pair of outcomes, and the share passing both attempts rises with model strength.The repeated-run analysis used the base split for the five Gemma models.Source: Table 14 · p. 15, §G.4 · p. 15

Related papers

Loading 2610.30161v1…