Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

Zheng Luo; T Pranav Kutralingam; Ogochukwu N Okoani; Wanpeng Xu; Hua Wei; Xiyang Hu

Source-linked AI summary

Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models

Zheng Luo, T Pranav Kutralingam, Ogochukwu N Okoani, Wanpeng Xu, Hua Wei, Xiyang Hu

arXiv:2601.05366v2Jan 8, 2026cs.CLcs.AIcs.LG

TL;DR

Multilingual tool calling remains underexplored despite English-centric evaluations and shared language-invariant tool interfaces. The paper introduces MLCL to diagnose execution failures across Chinese, Hindi, and Igbo, finding that parameter value language mismatch is dominant even with correct intent and tool selection. Inference-time mitigation reduces some errors but does not recover English-level performance, motivating execution-aware system design.

Problem
Existing tool-calling benchmarks largely assume English queries, leaving multilingual reliability at the language–tool boundary underexamined.
Method
MLCL systematically evaluates multilingual tool calling across Chinese, Hindi, and Igbo using controlled query variation and fine-grained error analysis.
Results
Execution-interface violations, especially parameter value language mismatch, primarily drive multilingual degradation despite correct intent and tool selection; mitigation strategies do not recover English-level performance.
Takeaways & Limitations
Improving multilingual tool calling requires execution-aware system design that aligns natural-language variability with programmatic interfaces.
Takeaways & Limitations
The evaluation uses single-turn scenarios with predefined interfaces and only three languages, so it does not provide exhaustive multilingual coverage.

Abstract

from arXiv · show

Large Language Models (LLMs) are increasingly deployed as agents that invoke external tools through structured function calls. While recent work reports strong tool-calling performance under standard English-centric evaluations, the robustness of tool calling under multilingual user interactions remains underexplored. In this work, we introduce MLCL, a diagnostic benchmark, and conduct a systematic evaluation of multilingual tool calling across Chinese, Hindi, and the low-resource language Igbo. Through fine-grained error analysis, we show that many failures occur despite correct intent understanding and tool selection. We identify parameter value language mismatch as a dominant failure mode, where models generate semantically appropriate parameter values in the user's language, violating language-invariant execution conventions. We further evaluate several inference-time system strategies and find that while these strategies substantially reduce language-induced execution errors, none of them can fully recover English-level performance.

1 Introduction

LLM tool-calling evaluations largely assume English, leaving multilingual reliability at the language–tool boundary underexamined. MLCL diagnoses execution-level failures across Chinese, Hindi, and Igbo, showing that semantically correct calls can become non-executable when parameter values violate English-only conventions.

LLM agents increasingly use structured tool calls to retrieve information, perform computations, and trigger downstream actions.
Existing tool-calling evaluations largely assume English queries, despite agents using language-invariant interfaces with multilingual users.This leaves the effect of linguistic context on tool-calling reliability largely unexamined.
Models can select the correct function and generate semantically appropriate arguments while producing non-executable calls through parameter value language mismatch.The failure occurs when parameter values copied from non-English queries violate English-only execution conventions.
MLCL extends BFCL with controlled query-language composition, semantic perturbations, and an error taxonomy distinguishing execution-level violations from semantic errors.The benchmark covers Chinese, Hindi, and Igbo to compare high-resource and low-resource settings.
Inference-time strategies reduce some language-induced errors but do not consistently recover English-level performance.The evaluated strategies include partial translation, explicit prompting, and pre- or post-translation.

2 Related Work

Prior tool-calling benchmarks show strong performance mainly under standardized English-language settings and typically abstract away linguistic variation. Related multilingual robustness work studies linguistic perturbations and cross-lingual consistency, but rarely examines language-sensitive tool or API interfaces.

Tool Calling and Tool Learning: Tool-calling research has improved structured API interaction through methods such as supervised fine-tuning and reinforcement learning.Early work also identified syntactically valid but operationally incorrect calls as a reliability problem.
Tool Calling and Tool Learning: Benchmarks such as BFCL typically assess tool selection and parameter accuracy under predominantly English, language-consistent execution environments.These settings largely abstract away how linguistic variation interacts with execution conventions.
Tool Calling and Tool Learning: MLCL complements prior work by analyzing execution failures induced by language shifts rather than proposing new training objectives or architectures.Its diagnostic design varies query language composition and semantic perturbations under a fixed, language-invariant interface.
Robustness in Large Language Models: Robustness studies show that paraphrases, synonym substitutions, and distribution shifts can significantly affect model behavior, motivating evaluation beyond aggregate accuracy.
Robustness in Large Language Models: Multilingual evaluations demonstrate cross-lingual reasoning but rarely test outputs that must interface with external language-sensitive tools or APIs.Prior structured-output work nevertheless suggests that cross-lingual consistency remains challenging even with high translation quality.

3 Multilingual Tool-Calling Benchmark

MLCL extends an English tool-calling benchmark into a controlled multilingual diagnostic suite, varying query language and semantic form while keeping execution interfaces fixed. Across Chinese, Hindi, and Igbo, fully translated queries increase execution-level errors, especially parameter value language mismatch.

Benchmark design: MLCL extends BFCL with controlled query-language composition and semantic perturbations across Chinese, Hindi, and Igbo.The benchmark keeps the execution interface fixed to isolate effects of natural-language input.
Benchmark design: The benchmark preserves function names, parameter keys, tool descriptions, and execution conventions while translating only natural-language query content.Fully translated queries and partially translated queries differ in whether ground-truth parameter values remain English.
Evaluation protocol: The evaluation combines strict execution matching with an error taxonomy separating execution-interface violations from semantic misunderstanding.Errors are reported by composition and severity rather than overall accuracy.
Results: Fully translated queries substantially increase execution failures across all three languages, dominated by parameter value language mismatch despite usually correct semantics.Models often copy non-English query tokens into parameter values that must satisfy the English-only interface.
Results: Parameter value mismatch is most prevalent for Chinese, followed by Hindi, and least frequent for Igbo, where semantic misunderstanding forms a larger error share.This distribution indicates different contributions from execution mismatch and language comprehension across languages.
Results: Partial translation reduces execution-level errors and can match or exceed the English reference by preserving English parameter strings while translating context.This setting isolates execution-interface violations from multilingual query understanding.

4 Inference Time Mitigation Strategies

The paper evaluates lightweight inference-time interventions targeting language mismatch without changing model weights. These interventions reduce execution-interface violations, but their effectiveness varies by language and does not consistently restore English-level performance.

Scope and methods: The mitigation study tests lightweight interventions that target execution-level language mismatch without retraining or modifying model parameters.The strategies are evaluated under the same task abstraction and error taxonomy as the benchmark.
Mitigation methods: Prompt-level instruction fully translates queries and asks models to output English parameter values while preserving tool selection and argument structure.This directly targets parameter value language mismatch.
Results: Pre-translation and post-translation reduce language mismatch errors, with pre-translation generally more effective because it accesses the full query context.Prompting reduces mismatch as well, but compliance is inconsistent.
Results: None of the mitigation strategies eliminates execution failures or recovers English-level performance.Translation can introduce semantic drift and surface-form normalization, creating new execution-level errors under strict matching.
Low-resource language results: For Igbo, translation-based mitigations provide limited benefit and can increase errors because parameter value mismatch is already rare.Remaining failures are dominated by query understanding errors rather than execution-level language mismatch.

5 Discussion

Multilingual degradation is driven primarily by execution-interface violations, especially parameter-value language mismatch, rather than failures in intent understanding. Inference-time mitigation reduces some errors but does not restore English-level performance, leaving deployment reliability gaps.

Execution-interface violations, especially parameter value language mismatch, drive multilingual degradation despite correct tool selection and semantically appropriate arguments.This reflects a mismatch between flexible natural-language generation and rigid programmatic constraints.
Prompting and translation-based strategies reduce language mismatch errors but none recover English-level performance.They can introduce semantic drift and normalization effects that create new execution failures.
Non-English queries can create reliability gaps for multilingual users even when intent understanding succeeds.Without explicit handling of execution-interface constraints, these failures can limit global deployment robustness.
Improving multilingual tool calling requires execution-aware system design that aligns natural-language variability with programmatic interfaces.

6 Conclusion

The paper investigates multilingual tool calling under controlled execution settings and finds that degradation primarily reflects execution-interface violations rather than failed intent understanding. It shows that inference-time mitigation reduces specific errors but does not recover English-level performance, motivating closer alignment between language generation and execution interfaces.

Multilingual tool-calling degradation is driven primarily by execution-interface violations rather than failures in intent understanding.
Inference-time mitigation reduces specific error classes but does not recover English-level performance.The strategies can also introduce new execution failures through semantic drift or surface-form variation.
The findings motivate alignment between language generation and execution interfaces for multilingual tool calling.

Limitations

The evaluation isolates multilingual execution effects in controlled single-turn settings, but its scope is limited by the benchmark subset, language coverage, English-value interface convention, and intentionally constrained mitigation strategies.

The evaluation uses the vanilla Berkeley Function Call Leaderboard subset to study single-turn tool calling with predefined interfaces.This isolates execution-level effects but limits interpretation to the single-turn setting.
The study examines Chinese, Hindi, and one low-resource language, leaving linguistic families and writing systems beyond this set uncovered.The authors seek representative rather than exhaustive multilingual coverage.
Execution correctness assumes that tool interfaces expect parameter values in English.This convention reflects common current system and benchmark design rather than a universal interface requirement.
The evaluation's inference-time strategies are intentionally constrained, so the reported mitigation results do not exhaust possible interventions.

Ethical Considerations

The study evaluates multilingual tool calling with translated and perturbed benchmark queries under an English-only execution interface, without collecting user data or conducting human-subject studies. It frames non-English execution failures as a deployment risk and reports limited language coverage and possible translation artifacts as boundaries on interpretation.

The study evaluates Chinese, Hindi, and Igbo queries under a controlled English-only execution interface using translated and perturbed benchmark data.
The study collected no user data and conducted no human-subject studies.
Non-English tool-calling failures can cause silent non-executable calls or incorrect downstream actions that may disproportionately affect non-English users.
Partial language coverage and possible translation artifacts, including semantic drift, limit generalization from the reported language-specific patterns.

A.2 Detailed Error Taxonomy

The paper classifies multilingual tool-calling failures by execution impact and separates semantic correctness from language conformity. It also evaluates translation and prompting strategies designed to enforce English parameter values before execution.

Error severity: The taxonomy orders errors by whether generated calls can be parsed, executed, or reliably recovered.Syntax errors prevent parsing, while function-level errors violate the tool schema despite syntactic validity.
Parameter-value errors: Language-mismatch errors range from unrelated wrong values to relevant but incorrect or same-meaning values expressed in non-English forms.These categories isolate failures at the language–execution interface after intent and tool selection have succeeded.
Parameter-value errors: Non-mismatch parameter errors include wrong, relevant-but-incorrect, and exactly same-meaning values that fail strict surface-form matching.The last category is semantically equivalent but differs from the benchmark’s required surface form.
Inference-time strategies: Prompt-level instruction translates queries while explicitly requiring English parameter values without changing inputs or post-processing outputs.Pre-translation normalizes inputs before tool calling, whereas post-translation converts generated parameter values before execution.
Dataset construction: The dataset construction preserves query meaning while allowing natural wording, and partial translation keeps ground-truth parameter keywords in English.The evaluation still counts semantically appropriate alternatives as failures when they do not match the required ground truth surface form.

B Additional Results

Pre-translation can preserve user intent while changing parameter surface forms through normalization or synonym substitution. Under strict surface-form matching, that semantic preservation does not prevent execution failure.

Pre-translation: Pre-translation alters parameter surface forms even when the translated query preserves the original intent.Normalization or synonym substitution can produce a tool call that fails strict surface-form matching.
Case study: The representative booking query illustrates that semantic equivalence alone cannot guarantee successful tool execution after translation.The example concerns a suite, a queen-size bed, a three-night stay, and Hilton New York.

B.2 Effects of Model Scale and Architecture

Model scale and architecture affect multilingual tool-calling robustness unevenly across model families. Larger Llama 3.1 models improve robustness, whereas Qwen3 and Granite4 show non-monotonic or language-dependent scaling patterns.

GPT-5: GPT-5 performance is consistent across scales, with the standard model slightly outperforming distilled variants.The discussion links accuracy to query comprehension and selecting tool-calling conventions under limited instruction.
Llama 3.1: Larger Llama 3.1 models produce fewer syntax errors and better follow English-parameter instructions, improving overall multilingual robustness.The improvement reflects more stable valid calls, greater awareness of parameter conventions, and stronger instruction following.
Llama 3.1: In partially translated experiments, Llama 3.1 70B performs slightly worse than 8B despite fewer syntax errors because it outputs integers for numeric values whose ground truths are decimals.The authors exclude parser bugs, but the phenomenon remains unexplained.
Qwen3: Qwen3 models do not improve monotonically with scale: Qwen3-30B-A3B underperforms Qwen3-14B, while Qwen3-Next-80B-A3B gains little over Qwen3-32B.The authors associate this pattern with mixture-of-experts activation and uneven expertise across multilingual understanding and tool-call formation.
Granite4: Granite4 shows an inverse language-dependent trend: larger models perform worse on Chinese but better on Igbo.The authors suggest tighter coupling between tool-execution behavior and training-time conventions may limit instruction-based mitigation.

B.3 Additional Results of Semantic Perturbations

Semantic perturbations have limited impact in fully translated settings, where execution failures are dominated by parameter-value language mismatch. Their effects become more relevant when language and execution conventions interact under partial translation or English-parameter requirements.

Cross-setting pattern: In fully translated settings, semantic perturbations have limited impact because execution failures are dominated by parameter value language mismatch.This pattern is reported across the supplementary analysis of paraphrases and synonym substitutions.
GPT-5: GPT-5 variants show low semantic error rates across model sizes, with fully translated errors primarily driven by parameter value language mismatch.The figure compares fully translated, partially translated, and English-parameter-prompted settings.
Llama 3.1: Llama 3.1 evaluations separate syntax, function-level, language-mismatch, and ordinary parameter-value error categories across fully translated and partially translated settings.The supplementary plots compare 8B and 70B models across English, Chinese, Hindi, and Igbo conditions.
Qwen3: Qwen3 evaluations compare multiple model sizes across fully translated, partially translated, and English-parameter-prompted conditions using the same structured error categories.The plotted categories include syntax, function-level, language-mismatch, and other parameter-value errors.
Granite4: Granite4 evaluations compare small and tiny models across the three execution settings and report the same execution-error breakdown.The comparison covers English, Chinese, Hindi, and Igbo conditions in the plotted results.
Chinese and Hindi: Chinese and Hindi semantic-perturbation analyses focus on whether paraphrases and synonym substitutions affect tool calling when English parameter surface forms are required.Both figures compare fully translated, partially translated, and English-parameter-prompted settings.