J-tau: A Fully Localized Japanese Benchmark for Tool-Agent-User Interaction

1 Introduction

• English evaluations may not reflect agent utility in other languages, while machine translation alone can leave entities such as names unlocalized and tasks unnatural or invalid.Source: §1 · p. 1
• The authors release J-tau publicly.Source: §1 · p. 1
• About 10% of airline and retail tasks were invalid, and flawed aggregation inflated one model’s score by up to 64.6 points; J-tau fixes both.Source: §1 · p. 1
• J-tau is the first publicly available Japanese benchmark for interactive tool-using LLM agents, localizing three domains of tau-bench.Source: §1 · p. 1

2.1 Overview

• tau-bench evaluates agents handling customer-support tasks through conversation and tool use, with domain-specific policies and simulated users.In telecom, agents must also guide users in using their separate tools.Source: §2.1 · p. 2

2.2 Limitations

• tau-bench is English-centric, and its machine-translated multilingual extension can produce degraded data and unnatural conversations.Source: §2.2 · p. 2
• Some tau-bench tasks remain unsolvable or have non-deterministic ground truths.Source: §2.2 · p. 2
• Its score aggregation can overestimate the performance of some models.Source: §2.2 · p. 2

3.1 Construction Pipeline

• J-tau translates benchmark content into Japanese, adapts locale-specific entities, and has experts review the resulting data.Translation uses a domain-specific glossary; entity adaptation replaces names, addresses, and phone numbers and converts amounts to yen and EST to JST.Source: §3.1 · p. 2
• Manual inspection and test runs identify tasks made unsolvable by localization and invalid tasks inherited from tau-bench, which J-tau repairs.Repairs can add missing information to user instructions.Source: §3.1 · pp. 2–3
• J-tau counts agent errors as failures and discards trials with user-simulator errors, rather than excluding persistent errors from aggregation.Source: §3.1 · p. 3

3.2 Benchmark Details

• J-tau covers airline, retail, and telecom customer-support domains.Source: §3.2 · p. 3
• GLM-5.2-FP8 is the default user simulator because it distorted evaluation results the least among the recent LLMs compared.Source: §3.2 · p. 3

Setup

• The evaluation reports mean and standard deviation over three trials per task.For retail conversation-based evaluation, gpt-oss-120b serves as the LLM judge.Source: §4 · p. 3, Table 1 · p. 3

Quantitative Results

• Claude Fable 5 has the highest overall score among proprietary models, while GLM-5.2-FP8 and Qwen3.5-397B-A17B-FP8 perform best among open-weight models.Within the Qwen3.5 series, performance generally improves with model size.Source: §4 · p. 3, Table 1 · p. 3

Case Study

• Qwen3.5-4B sometimes switches from Japanese to another language mid-conversation and fails to recognize Japanese family-name-first order.The example is Kimura Tsubasa (木村翼) in Fig. 1.Source: §4 · p. 3

5.1 Effect of Task and Scoring Fixes

• On repaired tasks, high-performing models score substantially higher on J-tau than tau*, while unmodified-task scores tend to be lower on J-tau.Invalid tasks in tau* cap achievable accuracy regardless of model capability; the repaired and unmodified group comparison is shown in Table 2.Source: §5.1 · pp. 3–4, Table 2 · p. 4
• For small models such as Qwen3.5-4B, scores drop substantially on tau* because empty and malformed outputs count as failures, correcting tau-bench score inflation.The difference is small for high-performing models; Table 3 compares tau and tau* by domain.Source: §5.1 · p. 4, Table 3 · p. 4

5.2 Comparison with an MT-Centric Pipeline

• J-tau has fewer framework-caused failures and more successes than SEATauja, especially in retail, where untranslated database names can conflict with katakana conversation names.Tool descriptions are in Japanese while tool implementations still expect English arguments.Source: Table 4 · p. 4, §5.2 · p. 4

• J-tau localizes tau-bench through entity adaptation and human review, fixes invalid tasks and a scoring flaw, and the analysis finds machine translation alone increases framework-caused failures.The authors hope J-tau advances Japanese LLM agents and that their findings benefit benchmark localization for other languages.Source: §6 · p. 5

Limitations

• J-tau covers airline, retail, and telecom, but excludes banking_knowledge and cannot evaluate agents in languages other than Japanese.Source: Limitations · p. 5
• Airline and retail user information uses Faker-generated names and addresses, including nonexistent places, limiting fidelity to real-world data.Source: Limitations · p. 5
• Solvability verification relies only on GLM-5.2-FP8, so model-specific biases may leave some invalid tasks undetected.Source: Limitations · p. 5
• The original JSON-output instruction can prompt some models, especially in Japanese, to return JSON strings instead of natural-language responses, which are counted as failures and may lower their scores.Source: Limitations · p. 5

Ethical Considerations

• J-tau contains no personally identifiable information because all user names, addresses, and phone numbers are fictitious; in-house annotators reviewed it as part of regular duties for standard salaries.Source: Ethical Considerations · p. 5

A Additional Related Work

• Prior multilingual tau-bench extensions leave locale-specific entities unadapted or unreleased, and the cited concurrent works do not check task validity or revise score aggregation.Source: A Additional Related Work · pp. 7–8

B.1 Translation

• The airline, retail, and telecom domain glossaries support consistent translation, with the airline glossary also accompanied by an airport-mapping table.Source: §B.1 · p. 8, Table 5 · p. 8, Table 6 · p. 8

B.2 Entity Adaptation

• Entity adaptation replaces original user and product entities with Japanese-suited fictitious counterparts, domestic airports, yen, and JST, consistently updating surface text and tool/database values.Source: §B.2 · p. 8

B.3 Human Review

• Three annotators review each domain’s policies; airline and retail review also covers task-specific text, terminology, airport names, and currencies, while telecom reviewers have customer-support experience.Source: §B.3 · pp. 8, 10

B.4 Fix Tasks

• Task fixing has two stages: addressing localization-caused issues and repairing invalid tasks inherited from tau-bench.Source: §B.4 · p. 10
• All task instructions and evaluation criteria are manually inspected to identify tasks made unsolvable or substantially changed in characteristics, such as difficulty, by localization.Source: §B.4 · p. 10
• Each original tau-bench task is run three times with GLM-5.2-FP8, with attention to tasks that fail at least twice, to find inherited invalid tasks.Source: §B.4 · p. 10

C Settings of the Compared Benchmarks

• tau is the original benchmark, tau∗ applies only the Step 5 scoring fix, and J-tau en-tool keeps tool-related components in English.Source: C Settings of the Compared Benchmarks · pp. 10–11

D Statistics of J-tau

• J-tau task counts are divided into Unmod., Rep. only, Loc. only, and Both; the Repaired group in Table 2 combines Rep. only and Both.Source: D Statistics of J-tau · p. 11, Table 12 · p. 11

E Details of the Evaluated Models

• Evaluations use proprietary and open-weight models, with default generation or API settings and high reasoning effort only for the gpt-oss series.Source: E Details of the Evaluated Models · p. 11, Proprietary Models · p. 11, Open-Weight Models · p. 11

Proprietary Models

• The proprietary models evaluated include GPT-5.6 Sol, Gemini 3.1 Pro, Claude Opus 4.8, and Claude Fable 5.Source: Proprietary Models · p. 11

Open-Weight Models

• The open-weight evaluation includes Kimi-K3, GLM-5.2, gpt-oss, Qwen3.5, gemma-4, and llm-jp 4.1 series.Source: Open-Weight Models · p. 11

F Example Simulations on J-tau

• Example simulations include successful and failed GLM-5.2-FP8 conversations across domains, as well as failures from gemma-4-E2B-it and Qwen3.5-4B.Source: F Example Simulations on J-tau · p. 11, Table 13 · p. 12, Table 14 · p. 13

G Details of the Experiments with SEATau

• The SEATauja appendix details its construction, evaluation, failure-cause analysis, and comparisons with J-tau, including telecom scores and user-simulator failures.Source: G Details of the Experiments with SEATau · p. 19

G.1 Construction of SEATauja

• SEATauja uses the SEATau pipeline with Gemini 3.1 Pro and its l2_domain scenario, translating conversations, tool documentation, policies, database, and task definitions.Source: §G.1 · p. 19

G.2 Evaluation on SEATauja

• Most models score lower on SEATauja than J-tau in airline and retail; Kimi-K3 is the only model scoring higher in both, while many models score higher in telecom.Kimi-K3 consistently scores higher on SEATauja than J-tau. Table 25 compares domain results across the benchmarks.Source: §G.2 · p. 19, Table 25 · p. 20

G.4 Cause of Higher Telecom Scores on SEATauja

• Added SEATau instructions tell agents to ask users to perform user-side tool actions and tell the simulator to call its tools, likely explaining higher telecom scores.Telecom is the only domain with user-side tools.Source: §G.4 · p. 19, Figure 5 · p. 21

G.3 Setup of the Failure Cause Analysis

• Figures 3 and 4 show the system and user prompts used to classify failed-trial causes.Source: Figure 3 · p. 20, Figure 4 · p. 21

G.5 Cause of Fewer User Simulator Failures on SEATauja

• Using SEATauja’s English user-simulator prompt cuts retail user-simulator failures by about half or more for all models, likely because the guideline is followed more faithfully in English.Most J-tau user-simulator failures involve ending conversations before task completion; the guideline specifies when to end. The Japanese guideline is retained for full localization, with simulator choice left for future work.Source: §G.5 · p. 22, Table 26 · p. 22

G.6 Example Simulations on SEATauja

• SEATauja simulations illustrate errors when tool-accepted values remain English despite Japanese dialogue, or entities conflict with the assumed cultural context.The examples are shown in Tables 27 and 28.Source: §G.6 · p. 22, Table 27 · p. 23, Table 28 · p. 24

H.1 Effect of Tool Language

• For models with sufficient capability, keeping tool descriptions, values, returns, and error messages in English does not substantially degrade telecom performance.J-tauen-tool retains Japanese conversation and locale-specific entities while keeping tool-related terms in policies and scenarios English.Source: §H.1 · p. 22, Table 29 · p. 25

H.2 Effect of System Prompt

• Removing the Japanese JSON-output instruction raises Kimi-K3’s telecom score from 67.0 to 84.8, near tau∗’s 88.9.The authors suspect the inherited instruction affects Kimi-K3 more strongly in Japanese; they plan to remove it because it conflicts with the evaluation framework.Source: §H.2 · p. 22, Table 30 · p. 25

I Selection of the User Simulator

• GLM-5.2-FP8 has the lowest user-error rates across all three agents, with no lower error rate specifically when paired with itself, so the authors adopt it as J-tau’s simulator.The evaluation measures simulations with any user error and those with critical errors affecting the result.Source: I Selection of the User Simulator · p. 25, Table 31 · p. 25