PaperlayerView exact revision on arXiv

Source-linked AI summary

From Text to Voice: A Reproducible and Verifiable Framework for Evaluating Tool Calling LLM Agents

Md Tahmid Rahman Laskar, Xue-Yong Fu, Seyyed Saeed Sarfjoo, Quinten McNamara, Jonas Robertson, Shashi Bhushan TN

arXiv:2605.15104v2cs.CL

TL;DR

Validated tool-calling benchmarks are largely text-based, while existing audio benchmarks often use newly constructed corpora. This paper converts text benchmarks into paired audio evaluations while preserving their annotations. Results show that performance depends on both model and task, with model rankings changing across benchmarks.

  • Problem

    Prominent validated tool-calling benchmarks remain text-based, while newly constructed audio benchmarks are often limited to selected domains and tool schemas.

    Source: §1 · p. 2

  • Method

    A dataset-agnostic TTS framework creates paired text-audio evaluations while preserving the original tool schemas and gold labels.

    Source: §1 · p. 2

  • Results

    Performance is model- and task-dependent: Gemini-3.1-Flash-Live leads on Confetti, while GPT-Realtime-1.5 leads on When2Call.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    The framework provides a reproducible first-stage comparison for cascade-versus-omni deployment decisions and complements purpose-built audio datasets.

    Source: §5 · p. 13

    The study evaluates two datasets and a finite set of models, and its TTS audio is a controlled first-stage proxy rather than spontaneous calls.

    Source: §3.2 · p. 5, §5 · p. 13

Abstract

from arXiv · show

Voice agents increasingly require reliable tool use from speech, whereas prominent tool-calling benchmarks remain text-based. We study whether verified text benchmarks can be converted into controlled audio-based tool calling evaluations without re-annotating the tool schema and gold labels. Our dataset-agnostic framework uses text-to-speech, speaker variation, and environmental noise to create paired text-audio instances while preserving the original dataset annotations. Based on extensive evaluation of 7 omni-modal models on audio-converted versions of Confetti and When2Call, our framework demonstrates that the performance is strongly model- and task-dependent: Gemini-3.1-Flash-Live obtains the highest Confetti score (70.4), whereas GPT-Realtime-1.5 performs best on When2Call (71.9). On Confetti, the text-to-voice gap ranges from 1.8 points for Qwen3-Omni to 4.8 points for GPT-Realtime-1.5. A targeted analysis of failure cases demonstrates that degradations most often reflect misunderstandings of argument values in the speech. Considering real-world deployment scenarios, we further report text-only results, an ambiguity-based reformulation stress test, and a reference-free LLM-as-judge protocol validated against human preferences. Notably, we find that open-source Qwen3 judges with at least 8B parameters exceed 80% agreement with proprietary judges, supporting privacy-preserving evaluation. Overall, our framework provides a verifiable and reproducible first-stage diagnostic that complements purpose-built audio corpora.

1 Introduction

The introduction presents a reusable way to convert validated text tool-calling benchmarks into paired audio evaluations while retaining their schemas and gold labels. It uses these pairs to compare model behavior across modalities and reports model- and task-dependent performance and audio failure patterns.

1 Introduction

  • The dataset-agnostic framework converts text benchmarks into controlled audio evaluations while preserving original tool schemas, gold labels, and evaluation protocols.The reusable recipe is intended to apply to proprietary text-based tool-calling data.Source: §1 · p. 2
  • The evaluation covers seven omni-modal models on Confetti and When2Call under clean and noisy audio, with diverse voices and additional reformulation and judge tests.The study also includes text-only baselines and Qwen3 text-mode scaling from 0.6B to 32B parameters.Source: §1 · p. 2
  • Model rankings change across benchmarks, and Confetti text-to-voice gaps range from 1.8 to 4.8 points across models.The introduction motivates model- and task-specific cascade-versus-omni diagnostics.Source: §1 · p. 3
  • Paired text and audio instances support counterfactual analysis of cases where models succeed on text but fail on audio, often through incorrect argument values.These failures can retain the broad tool-call structure, especially for Gemini-3.1-Flash-Live and GPT-Realtime-1.5.Source: §1 · pp. 2–3
  • Open-source Qwen3 judges with at least 8B parameters reach over 80% agreement with proprietary judges.A human preference study finds no clear preference between GPT-5 and Gemini-2.5-Pro.Source: §1 · p. 3

2 Related Work

Related work describes costly text tool-calling benchmarks, emerging audio benchmarks, and omni-modal systems whose tool-calling reliability remains unsettled. This paper complements fixed audio corpora by converting existing text benchmarks into paired speech evaluations.

2 Related Work

  • Existing text tool-calling benchmarks have already invested in validated tool specifications, gold parameter values, and scoring protocols, but remain primarily text-only.Examples include ToolBench and API-Bank (Qin et al., 2023), and τ-Bench.Source: §2 · p. 3
  • Most audio benchmarks target perception tasks such as ASR, captioning, and event classification rather than tool-calling evaluation.VoiceBench and WildSpeech-Bench assess spoken-interaction robustness, but do not provide the tool-calling setup targeted here.Source: §2 · p. 3
  • Audio function-calling benchmarks build new corpora in selected domains, whereas this framework converts existing text benchmarks into paired speech evaluations.The paired clean-text examples share gold labels and support text-to-audio gap measurement and instance-level error decomposition.Source: §2 · pp. 3–4

3 Methodology

The methodology converts text tool-calling benchmarks into paired speech evaluations while keeping the task, tool schema, and gold labels fixed. It varies generated speech, speakers, and noise, then compares text and audio inputs across selected datasets and omni-modal models.

3.1 Text-to-Speech Conversion Pipeline

  • The controlled pipeline varies TTS provider, speaker voice, and environmental noise, with noise mixed at 5, 10, 15, and 20 dB SNR.It compares Gemini-2.5-Flash-TTS with higher-capacity Gemini-2.5-Pro-TTS and uses GPT-4o-Mini-TTS as a cross-provider alternative.Source: §3.1 · p. 4

3.2 TTS Audio as a Controlled First-Stage Evaluation

  • Aligned text-only and direct-audio inputs keep examples and labels fixed, enabling estimation of performance loss when input changes from text to speech.The authors treat TTS audio as an optimistic first-stage proxy, not a replacement for spontaneous real calls.Source: §3.2 · p. 5

3.3 Preserving Verifiability through Benchmark Conversion

  • Conversion reuses validated tool schemas, gold labels, and scoring protocols, preserving verifiability while enabling controlled text-to-speech comparison.The converted benchmark is intended to complement speech-native corpora, not replace them.Source: §3.3 · p. 5

3.4 Datasets

  • The study uses 313 Confetti tool-call examples and 300 non-MCQ When2Call examples, and the conversion pipeline can integrate other text benchmarks without architectural modification.Confetti focuses on tool execution correctness; When2Call tests whether a tool call is needed, versus responding or requesting clarification.Source: §3.4 · pp. 5–6

3.5 Omni-Modal Models

  • Seven omni-modal models from four providers span multiple tiers, including GPT-Realtime, Gemini-Flash-Live, Qwen3-Omni, and Phi-4-Multimodal.The evaluated set includes compact and open-source models, including the 5.6B-parameter Phi-4-Multimodal.Source: §3.5 · p. 6

4 Experiments

The experiments evaluate synthetic speech and tool calling on Confetti and When2Call, comparing direct voice with clean-text and ASR-to-text settings. They also analyze modality-related errors, LLM judges, text-only scaling, and ambiguous queries.

4.3.1 TTS Performance

  • GPT-4o-Mini has the lowest WER across clean and noisy conditions, while Gemini-2.5-Flash-TTS narrowly leads clean-condition UTMOS.Human evaluators marked 97.7% of clean clips and 94.3% of noisy clips as content-faithful.Source: §4.3.1 · p. 7

4.3.2 Omni-Modal Model Performance

  • 70.4% AST soft accuracy makes Gemini-3.1-Flash-Live the Confetti leader, while GPT-Realtime-1.5 leads When2Call at 71.9% F1.On When2Call, GPT-Realtime-4o scores 70.1% F1 and GPT-Realtime-Mini scores 68.6%; Gemini-3.1-Flash-Live ranks fourth at 63.4%.Source: Table 2 · p. 8, Table 3 · p. 8, §4.3.2 · p. 8

4.3.3 Cascade vs. Omni Performance Comparison

  • 1.8 points is Qwen3-Omni’s smallest text-to-voice gap; neither the cascade nor direct voice consistently wins across the three models.The cascade beats direct voice by 0.9 points for Gemini-3.1-Flash-Live; direct voice leads by 0.4 points for GPT-Realtime-1.5 and 1.5 points for Qwen3-Omni.Source: Table 4 · p. 9, §4.3.3 · p. 9

4.3.4 Error Analysis

  • Argument-value errors are the largest Confetti audio-induced failure category for Gemini-3.1-Flash-Live (57.2%) and GPT-Realtime-1.5 (54.3%).Decision errors range from 25.8% for Gemini-3.1-Flash-Live to 37.4% for GPT-Realtime-1.5, so failures also involve high-level tool-use behavior.Source: §4.3.4 · pp. 9–10, Table 5 · p. 10
  • On When2Call, Qwen3-Omni correctly identifies tool-call cases in 92.7% of examples but no-tool-call cases in only 43.9%.GPT-Realtime-1.5 is more balanced: 85.2% accuracy on tool-call cases and 73.1% on no-tool-call cases.Source: Figure 3 · p. 10, §4.3.4 · p. 10

4.3.5 Evaluation using LLM Judges

  • Reference-free judge scores are consistently higher than reference-aware scores, while Qwen3 judges with at least 8B parameters achieve 80% agreement with proprietary judges.In 200 cases where GPT-5 and Gemini-2.5-Pro disagreed, human preferences split 48% for GPT-5 and 52% for Gemini-2.5-Pro.Source: §4.3.5 · pp. 10–11, Figure 4 · p. 11

4.3.6 Text-Only Scaling Analysis

  • 75.9% is Gemini-3.1-Flash-Lite’s text-only Confetti score, while ambiguous reformulations reduce Qwen3-Omni from 62.2% to 37.6%.Within the Qwen3 family, text-mode accuracy rises from 26.6% for Qwen3-0.6B to 70.7% for Qwen3-32B.Source: §4.3.6 · pp. 11–12, Figure 5 · p. 12

4.3.7 Practical Implications for Deployment

  • The authors recommend model- and task-specific deployment evaluation rather than assuming cascade or omni-modal systems are uniformly better.Their framework is presented as a first-stage benchmark; stack selection should consider accuracy, latency, and cost in the target environment.Source: §4.3.7 · p. 12

5 Conclusion

The framework converts text-based tool-calling benchmarks into controlled voice evaluations while preserving verifiability. Results vary by model and task, and the study identifies limits to the evaluation’s scope.

5 Conclusion

  • Performance is strongly model- and task-dependent: Gemini-3.1-Flash-Live leads on Confetti, while GPT-Realtime-1.5 leads on When2Call.The text-to-voice gap on Confetti ranges from 1.8 points for Qwen3-Omni to 4.8 points for GPT-Realtime-1.5.Source: §5 · p. 13
  • The framework preserves tool schemas and gold labels to support reproducible first-stage comparisons for cascade-versus-omni deployment decisions.The study evaluates two benchmarks, three TTS systems, multiple voices, and several LLMs.Source: §5 · p. 13
  • The study covers two datasets and a finite set of current models, and its TTS-generated audio is a controlled first-stage proxy rather than spontaneous speech.The authors identify naturally spoken customer-support data as an important extension.Source: §5 · p. 13

A.1 Ethical Considerations

The appendix records licensing requirements and reports that co-authors conducted the human evaluation without additional compensation.

A.1 Ethical Considerations

  • The authors state that they maintained licensing requirements while using tools including OpenAI and Gemini models.Source: §A.1 · p. 17
  • Co-authors conducted the human evaluation, so no additional compensation was required.Source: §A.1 · p. 17

A.2 Sample Prompts for Omni-Modal LLMs

The appendix provides sample prompts for omni-modal models using audio input and available tools, with different instructions for the two datasets.

A.2 Sample Prompts for Omni-Modal LLMs

  • The When2Call prompt supplies conversation context, tools, and audio, and asks the model to return a clear-JSON tool call when appropriate or plain text otherwise.Source: §A.2 · p. 17
  • The Confetti prompt instructs the model to choose among a tool call, follow-up question, cannot-answer response, or direct answer.Source: §A.2 · p. 18
  • For a tool call, the Confetti prompt requires appropriate arguments; for a follow-up, it directs the model to ask for needed information.Source: §A.2 · p. 18

A.3 Sample Prompts for LLM Judge

The sample reference-free judge prompt assesses agent responses against the conversation and available tools, then requires a binary correctness judgment.

A.3 Sample Prompts for LLM Judge

  • The judge evaluates whether the agent selected the right tool or correctly made no tool call when none of the tools were relevant.Source: §A.3 · p. 18
  • The judge must output 1 for correct or 0 for incorrect, without additional information.Source: §A.3 · p. 18
  • The prompt provides the conversation, available tools, and agent response as the judge’s inputs.Source: §A.3 · p. 18

A.4 LLM Judge Evaluation Analysis

The examples show that reference-free judgments can accept useful responses that diverge from the gold tool path, while judges can disagree about whether a clarifying question is acceptable.

Example 1: Comparing Judgments in Reference-Wise vs Reference-Free Settings

  • In the Jamaica example, judges marked a factually correct direct answer incorrect in reference-wise evaluation because the gold label required a RAG_query tool call, but reference-free judges accepted it.The authors cite this as an explanation for higher reference-free accuracy in some cases.Source: Example 1: Comparing Judgments in… · p. 19
  • The two cases illustrate how reference-free judgments may reward a helpful answer or conversational step even when it does not follow the gold tool-use path.The Portland judges did not agree, and human annotators preferred Gemini-2.5-Pro’s judgment.Source: Example 1: Comparing Judgments in… · p. 19
  • In the Portland example, a clarifying question was judged incorrect reference-wise because the gold answer called for an immediate hotel search using the available location.Gemini-2.5-Pro accepted the response in reference-free evaluation, while GPT-5 rejected it; human annotators preferred Gemini-2.5-Pro’s judgment.Source: Example 1: Comparing Judgments in… · p. 19

A.5 Sample Prompts for Ambiguous Query Generation

The sample prompt asks a model to rewrite a user’s final query ambiguously without changing its intent. It permits replacing specific details with generic references when the conversation context keeps the query answerable.

A.5 Sample Prompts for Ambiguous Query Generation

  • The prompt directs the model to rewrite the final user query in an ambiguous form while preserving the user’s intent and grounding it in conversation context.Source: §A.5 · p. 19
  • Specific entities, dates, numbers, and parameters may be replaced by pronouns or generic descriptors, provided the query remains answerable with the conversation context.Source: §A.5 · p. 19

A.6 Additional details in model performance

Table 6 presents LLM-judge results on Confetti across TTS configurations for three models, with two judges and reference-aware and reference-free settings.

A.6 Additional details in model performance

  • Table 6 reports Confetti LLM-judge results across TTS configurations for GPT-Realtime-1.5, Gemini-3.1-Flash-Live, and Qwen3-Omni; higher values indicate better performance.GPT-5 and Gemini-2.5-Pro are the judges, and Avg. is their mean score for each model.Source: §A.6 · p. 19, Table 6 · p. 20
Loading 2605.15104v2…