Source-linked AI summary
$\tau$-Multilingual: Benchmarking Voice Agents Across Languages
Soham Ray, Edgard dos Santos Paiva, Ruben Valenzuela, Karthik Narasimhan, Keshav Dhandhania, Victor Barres
TL;DR
English-only benchmarks cover only a narrow slice of voice-agent behavior. τ-Multilingual extends τ-Voice with native-reviewed multilingual evaluation; task completion remains broadly preserved in some languages but falls significantly in Korean and Mandarin. The paper argues that task completion, interaction, and generation should be reported jointly to expose failures hidden by narrower benchmarks.
Problem
Leading end-to-end voice-agent benchmarks evaluate English alone, while multilingual evaluation needs to capture task completion, conversational behavior, linguistic naturalness, and speech fidelity.
Source: §1 · p. 1Method
τ-Multilingual extends τ-Voice with native-speaker-reviewed language packs and evaluation of task completion, interaction, linguistic naturalness, and speech fidelity across multilingual voice-agent configurations.
Source: §1 · p. 1, §3.4 · p. 2Results
Korean and Mandarin show significant English-relative task-completion losses of 18.8 points (95% CI: 14.8–23.3) and 9.5 points (6.0–13.4), while the other three language differences are not significant.
Source: §4.1 · p. 3Takeaways & Limitations
Jointly reporting task completion, interaction, and generation exposes failures hidden by English-only or outcome-only benchmarks.
Source: §6 · p. 4The study covers six languages, three domains, three hosted model families, and one speech stack; simulated callers may not reproduce human timing or repair.
Source: §5 · p. 4
Abstract
from arXiv · showhide
English-only benchmarks expose only a narrow slice of voice-agent behavior. We introduce $\tau$-Multilingual, extending $\tau$-Voice to Spanish, Brazilian Portuguese, Hindi, Korean, and Mandarin with native-speaker review and evaluation of generated language and spoken output. Across 4,500 full-duplex calls and five voice configurations, Spanish, Portuguese, and Hindi remain within 3.2 task-completion points of English, but Korean and Mandarin fall by 14.7 and 8.4 points. The failure modes also vary: Korean systems miss more responses, Mandarin systems interrupt more often, and both struggle with tools and entities. Grok leads task completion but scores lowest on generation quality, motivating separate task, interaction, and generation reporting. We release language packs, validated judges, and tools for community-built multilingual voice-agent evaluation.
1. INTRODUCTION
English-only benchmarks cover only a narrow slice of voice-agent behavior, despite English being spoken by 19% of the world’s population. τ-Multilingual extends τ-Voice with native-reviewed language packs and evaluation of task completion, interaction, linguistic naturalness, and speech fidelity.
1. INTRODUCTION
- English-only voice-agent benchmarks cover a narrow slice of behavior, while multilingual evaluation must capture task completion, conversational behavior, linguistic naturalness, and speech fidelity.Source: §1 · p. 1
- τ-Multilingual extends τ-Voice with native-speaker-reviewed localization, language-quality evaluation, and testing across five voice configurations and two text systems in six languages and three domains.LanguageFactory localizes caller personas, speech rules, and identities while preserving the underlying tasks.Source: §1 · p. 1
- The released codebase supports community-built callers reflecting local linguistic and cultural norms, alongside language-specific audits of English-centric agent behavior.Source: §1 · p. 1
2. RELATED WORK
Prior benchmarks leave a gap in jointly evaluating multilingual full-duplex speech, executable task success, interaction quality, and language quality. τ-Multilingual combines these capabilities with native-speaker-authored diagnostic factors.
2. RELATED WORK
- τ-Voice evaluates task completion and conversational dynamics in full-duplex speech but not generation quality, while EVA-Bench and FDB-v3 are English-only.M3-DuplexBench covers English and Japanese turn-taking without executed tool tasks.Source: §2 · p. 1
- Table 1 compares related benchmarks across spoken, full-duplex, multilingual, interaction-quality, and language-quality coverage; check marks indicate direct evaluation and tildes partial coverage.Source: Table 1 · p. 1
- No prior benchmark jointly evaluates multilingual full-duplex speech, executable task success, interaction quality, and language-quality auditing.τ-Multilingual combines these capabilities with native-speaker-authored diagnostic factors.Source: §2 · p. 1
3. METHODOLOGY
The methodology localizes τ-Voice task specifications across languages, evaluates matched voice and text cohorts, and combines task, interaction, and generation measures with validation and statistical testing.
3.1. Matched Multilingual Benchmark
- Each language pack localizes caller instructions, persona, voice, identity entities, and evaluation rules while preserving tasks and scoring; LanguageFactory stores the specification as a typed, versioned artifact.Cross-language comparisons concern complete localized systems, not an isolated causal effect of language.Source: §3.1 · p. 2, Table 2 · p. 2
3.2. Evaluation Cohorts
- The study runs 4,500 primary calls across five voice configurations and evaluates the same 900 localized task instances in a 1,800-call half-duplex text control.Because model and interaction regime both change, the text control is end-to-end rather than a pure acoustic intervention.Source: §3.2 · p. 2
3.3. Retail Localization Ablations
- The retail ablations compare source-English identities and native-script database names against localized baselines on 30 Hindi and Mandarin retail tasks, with one trial per task and descriptive reporting.Source: §3.3 · p. 2
3.4. Evaluation Measures
- Task completion is call-level environment-reward pass rate; Interaction summarizes five call-level failure rates, and Generation measures aligned utterances passing naturalness and speech-fidelity checks.These are uncalibrated composites, so language-specific diagnostics remain separate; English is excluded only from Generation.Source: §3.4 · p. 2
3.5. Human Validation
- LLM-based judges are used only after meeting precision, recall, F1, and Cohen’s κ thresholds; native-language human labels assess naturalness and language-specific diagnostics.Reviewers also audit 30 completed calls per language, attributing the first critical error and scoring eight caller-quality dimensions.Source: §3.5 · p. 2
3.6. Statistical Analysis
- Task-completion inference compares each language with English over 150 matched domain–task clusters using paired permutation tests and paired BCa 95% intervals.Interaction and Generation use paired cluster-level label swaps with Holm correction; retail ablations are descriptive. Figures 2 and 3 present headline outcomes and lower-is-better main-metric breakdowns.Source: §3.6 · pp. 2–3, Figure 2 · p. 3, Figure 3 · p. 3
4. RESULTS
Task completion is broadly preserved in Spanish, Portuguese, and Hindi, but Korean and Mandarin show significant English-relative losses alongside interaction declines. The results also reveal language-specific failure patterns, model trade-offs, and validation evidence across task, interaction, and generation measures.
4.1. Task Completion
- Korean and Mandarin task completion fall 18.8 and 9.5 points relative to English, while differences for Spanish, Portuguese, and Hindi are not significant.Telecom has the lowest task completion (44.1%) but the highest Generation (63.2%).Source: §4.1 · p. 3
4.2. Interaction
- Interaction declines significantly versus English in all five languages, with Korean non-response and tool-use failures and Mandarin interruption and tool-use failures standing out.Selectivity failure is high across languages (86.7–93.5%), while Spanish and Portuguese show more monologuing than English.Source: §4.2 · p. 3
4.3. Generation Quality
- Generation leaders vary by language, while native-speaker audits find literal or interface-like phrasing and language-specific naturalness issues.GPT xhigh leads Spanish, Portuguese, and Hindi; Gemini minimal leads Korean; GPT minimal leads Mandarin. Audits also identify gender and modal-particle errors, distinct from Generation components.Source: §4.3 · p. 3
- Grok has the lowest speech-fidelity score: 80.0% of its utterances contain no material audible error, versus 91.3–93.8% for other systems.Across 72,164 utterances, audible punctuation or formatting artifacts are the most common material errors, followed by mispronunciations and number, date, or currency errors.Source: §4.3 · p. 4
4.4. Localization and Entity Handling
- Korean and Mandarin task losses concentrate in authentication, where exact lookup succeeds less often than in English across repeated GPT/Gemini systems.Localized family-first names are often mapped into given-first lookups; exploratory 30-task entity ablations show mixed effects, and exact authentication alone does not ensure task completion.Source: §4.4 · p. 4, Table 3 · p. 4
4.5. Model Choice Across Outcomes
- No configuration leads every headline outcome: Grok leads task completion, while GPT xhigh leads Generation and GPT minimal leads Interaction.Grok's six-language mean task completion is 75.1% but its Generation score is 52.3%; higher-reasoning GPT/Gemini configurations show longer turn-taking latency.Source: §4.5 · p. 4
4.6. Validation and Robustness
- Validation meets pooled thresholds for tool-use, naturalness, and speech-fidelity judges, with naturalness and speech-fidelity F1 varying across languages.Caller audits attribute few first critical errors to the simulator or infrastructure, and system pairs differ significantly on utterance Generation.Source: §4.6 · p. 4
5. LIMITATIONS
The study is bounded by limited language, model, and trial coverage, and by simulation and validation constraints that may affect how results generalize.
5. LIMITATIONS
- The study covers six languages, three domains, three hosted model families, and one speech stack; cost limited additional languages, models, and trials.No documented open-weight model covered all five localized languages, full-duplex speech, and live tool use.Source: §5 · p. 4
- Simulated callers may not reproduce human timing or repair, and transcript-based caller input prevents synthesis errors from affecting task outcomes.Source: §5 · p. 4
- GPT and Gemini have two trials versus Grok’s one, validation sets are finite, and the frozen cohort includes earlier Korean and Mandarin retail runs with underspecified name-role prompts.Source: §5 · p. 4
6. CONCLUSION
The benchmark finds that multilingual voice-agent reliability varies by language and extends beyond task completion. It argues for reporting task, interaction, and generation outcomes jointly, and releases evaluation resources for community use.
6. CONCLUSION
- Spanish, Portuguese, and Hindi largely match English, whereas Korean and Mandarin lose ground through authentication, entity, and interaction failures.Source: §6 · p. 4
- Model rankings depend on the objective, so jointly reporting task completion, interaction, and generation exposes failures hidden by English-only or outcome-only benchmarks.Source: §6 · p. 4
- The release includes language packs, validated judges, and tools for community-built evaluation.Source: §6 · p. 4
7. COMPLIANCE WITH ETHICAL STANDARDS
The benchmark uses synthetic, non-sensitive records and simulated callers, with reviewer-authored materials and annotated outputs; its results are not measures of human ability or cultural quality.
7. COMPLIANCE WITH ETHICAL STANDARDS
- The benchmark uses no real customer records or recordings; records are synthetic and non-sensitive, and calls use simulated callers and provider voices.Source: §7 · p. 5
- Reviewers authored benchmark materials and annotated outputs, and they consented and were compensated.Source: §7 · p. 5
- Before release, artifacts were screened for incidental personal data or secrets, and flagged items were excluded.Source: §7 · p. 5
- Benchmark results should not be interpreted as measures of human ability or cultural quality.Source: §7 · p. 5