PaperlayerView exact revision on arXiv

Source-linked AI summary

VoiceAgentBench: Are Voice Assistants ready for agentic tasks?

Dhruv Jain, Harshit Shukla, Gautam Rajeev, Ashish Kulkarni, Chandra Khatri, Shubham Agarwal

arXiv:2510.07978v3cs.AIcs.CLcs.LG

TL;DR

Speech benchmarks have largely emphasized isolated capabilities rather than agentic behavior and adversarial robustness. VoiceAgentBench introduces over 6,000 spoken queries across English and six Indic languages, with varied tool-use tasks and speaker-diverse speech. ASR–LLM pipelines outperform end-to-end SpeechLMs, while models show persistent difficulty with Indic languages, sequential workflows, and safety.

  • Problem

    Existing speech benchmarks largely focus on isolated capabilities and do not systematically evaluate agentic behavior or adversarial robustness.

    Source: §1 · p. 2

  • Method

    VoiceAgentBench evaluates spoken tool use, multi-turn dialogue, and safety across languages, using speaker-embedding-based sampling to diversify TTS audio and metrics for tool selection, structure, argument correctness, and refusal.

    Source: §1 · p. 2, §3.1.2 · p. 6, §3.2 · p. 7

  • Results

    ASR–LLM pipelines consistently outperform end-to-end SpeechLMs; Whisperv3–Llama3 70B reaches 60.6 Avg PF on English, while sequential-dependent tool calling yields only 14.8% PF for the best English pipeline and 4.3% PF for the strongest multilingual models.

    Source: §4.2 · pp. 7–8

  • Takeaways & Limitations

    Results point to persistent challenges in sequential and multi-tool reasoning, multi-turn dialogue, safety robustness, and cross-lingual generalization.

    Source: §5 · p. 10

    The evaluation excludes speech with background noise, so it does not measure noise effects on tool invocation.

    Source: §5 · p. 9

Abstract

from arXiv · show

Large scale Speech Language Models have enabled voice assistants capable of understanding natural spoken queries and performing complex tasks. However, existing speech benchmarks largely focus on isolated capabilities such as transcription or question answering and do not systematically evaluate agentic behavior or adversarial robustness. To address this, we introduce VoiceAgentBench, a comprehensive benchmark for evaluating SpeechLMs in realistic spoken agentic settings, comprising 6,000+ synthetic spoken queries spanning single-tool invocations, multi-tool workflows, multi-turn dialogue, and safety evaluations across English and six Indic languages. To ensure speaker diversity, we further simulate speaker variability using a novel sampling strategy that selects audios for TTS voice conversion based on speaker embeddings to maximize acoustic diversity. Our evaluation measures tool selection accuracy, structural consistency, and the correctness of tool invocations, including adversarial robustness. Across agentic tasks, ASR-LLM pipelines outperform end-to-end SpeechLMs, achieving up to 60.6% average parameter-filling accuracy on English, while SpeechLMs exhibit lower performance and sharper degradation on Indic languages. All models struggle in sequential workflows and safety evaluations, highlighting persistent limitations in tool orchestration, multilingual generalization, and safety robustness. VoiceAgentBench is publicly available on Hugging Face at https://huggingface.co/datasets/krutrim-ai-labs/VoiceAgentBench, and the codebase is released at https://github.com/ola-krutrim/VoiceAgentBench.

1 INTRODUCTION

Agentic research has largely centered on text, while speech evaluations have focused on isolated tasks rather than complex tool use and dialogue. VoiceAgentBench fills this gap with multilingual spoken agentic tasks and speaker-diverse audio; its strongest models still reach only 60.6% average performance on English.

1 INTRODUCTION

  • Existing agent research largely focuses on text, and SpeechLM evaluations emphasize isolated recognition, question-answering, or instruction-following tasks rather than complex tool use and multi-turn interaction.SpeechLMs can avoid the sequential transcription overhead of ASR–LLM pipelines and support natural real-time interactions.Source: §1 · pp. 1–2
  • 60.6% average performance on English is the ceiling reported for even the strongest models, underscoring the difficulty of speech-based agentic reasoning.The benchmark also compares ASR–LLM pipelines with end-to-end SpeechLMs and reports substantial performance gaps, particularly for SpeechLMs.Source: §1 · p. 2
  • VoiceAgentBench contains over 6,000 spoken queries across English and six Indic languages, covering single- and multi-tool use, multi-turn dialogue, and adversarial safety.Its speaker-embedding–based sampling strategy selects audio for TTS voice conversion to simulate diverse accents, speaking styles, and vocal characteristics.Source: §1 · p. 2

2 RELATED WORK

Prior work evaluates text-based tool use and a range of speech capabilities, but the cited benchmarks do not combine speech input with realistic agentic tool use and safety evaluation.

2 RELATED WORK

  • Text-based agent benchmarks cover API invocation, nested or sequential calls, dialogue-driven tool use, and safety or adversarial robustness.Examples include ToolBench, NESTful, API-Bank, Tau-bench, AgentHarmBench, and DoomArena.Source: §2 · p. 3
  • The related-work review identifies no speech benchmark that explicitly evaluates SpeechLMs in realistic agentic and safety-critical settings.This gap remains distinct from both text-based tool-use benchmarks and speech benchmarks focused on simpler tasks or audio reasoning.Source: §2 · p. 3
  • Speech datasets and benchmarks primarily target transcription, translation, recognition, and instruction following, while newer suites also assess audio reasoning and robustness.The cited work includes reasoning benchmarks such as Audio-CoT and MMAU and instruction-following or robustness suites such as AIR-Bench, AudioBench, and VoiceBench.Source: §2 · p. 3

3 VOICEAGENTBENCH

VoiceAgentBench evaluates spoken agentic behavior across multiple tool-use settings, dialogue, safety, and languages. Its construction combines benchmark-derived and custom queries, multilingual translation, diversity-oriented speech generation, and structured evaluation metrics.

3 VOICEAGENTBENCH

  • VoiceAgentBench pairs more than 6,000 TTS-generated spoken queries with expected tool invocations or safety scenarios to evaluate agentic capabilities in realistic spoken interactions.Its categories include single-tool calls, retrieval, parallel and sequential-dependent calls, multi-turn dialogue, and safety evaluations.Source: §3 · p. 3
  • The benchmark combines linguistic coverage, structured evaluation targets, and adversarial robustness testing to assess SpeechLMs’ real-world agentic competence.Its task categories are designed to isolate reasoning, retrieval, long-context, and tool-orchestration capabilities.Source: §3 · pp. 3–4

3.1 DATA CONSTRUCTION

  • The data pipeline sources seed tools and dialogues, creates original and Indian-context English queries, translates them into six Indic languages, then converts speech and pairs it with validated tool calls and instructions.The benchmark’s English subset includes original queries and Indian-context variants; the multilingual subset derives from the latter.Source: Figure 2 · p. 4, §3.1.1 · p. 4

3.1.1 TOOL SOURCING AND QUERY GENERATION

  • Queries draw on BFCL tool-calling tasks, API-Bank dialogues, AgentHarm safety tasks, and custom workflows using 21 tools across cab booking, food ordering, and payment services.Sequential workflows encode ordered dependencies and are checked with Pydantic schemas and manual verification; Indian-context dialogue and safety variants preserve the underlying structure or harmful intent.Source: §3.1.1 · pp. 4–5
  • The benchmark contains 6,134 queries, with English making up 38.75% and the six Indic languages together accounting for 61%.Table 1 reports example counts and average audio duration across tasks, languages, and sources.Source: Table 1 · p. 5, §3.1.1 · p. 6

3.1.2 DIVERSITY BASED TTS GENERATION

  • FPS achieves the highest diversity scores among four tested audio-sampling strategies, selecting samples by embedding-space distance for voice conversion.The method uses ECAPA-TDNN speaker embeddings; selected audios come from IndicSuperb and IndicST, and TTS engines vary by language.Source: §3.1.2 · p. 6

3.1.3 MODEL INSTRUCTIONS

  • Task-wise one-shot instructions require Python-syntax tool calls, tell multilingual models to output calls in English, and add refusal prompts for safety tasks.The prescribed output format is intended to avoid free-form or mixed outputs that cannot be deterministically parsed.Source: §3.1.3 · p. 6

3.2 EVALUATION FRAMEWORK

  • The evaluation framework scores tool selection, call structure, parameter filling, and refusal behavior to expose distinct failure modes.Tool selection uses exact tool-name matching; structure is checked against Pydantic schemas, parameter faithfulness is judged semantically, and refusal behavior is assessed by a semantic judge.Source: §3.2 · p. 7

4 EXPERIMENTS

The experiments compare end-to-end SpeechLMs with ASR–LLM pipelines across English and multilingual agentic tasks. Pipelines generally outperform SpeechLMs, while sequential workflows, Indic-language generalization, and safety remain challenging.

4.1 MODELS

  • The comparison covers three 7B SpeechLMs and three ASR–LLM pipelines, which transcribe speech with Whisper v3 Large before passing text to an LLM.The pipeline backends are Qwen-3 8B, Gemma3 27B, and LLaMA 3.3 70B.Source: §4.1 · p. 7

4.2 RESULTS

  • ASR–LLM pipelines outperform end-to-end SpeechLMs across English and multilingual tool-calling, especially in parameter filling; KimiAudio approaches them only on simpler English single-tool tasks.KimiAudio remains behind similarly sized ASR–LLM systems on retrieval, parallel, and sequential-dependent tasks.Source: Table 2 · p. 7, §4.2 · pp. 7–8, Table 3 · p. 8
  • Indic-language parameter filling trails English across models, with sharper declines in sequential-dependent and multi-turn tasks and larger drops for lighter models.Whisperv3–Llama3 70B averages 60.6 PF on English and 39.2 on Indic languages; Whisperv3–Gemma3 27B averages 59.3 and 35.3, respectively.Source: §4.2 · p. 8
  • Sequential-dependent calling is the weakest setting: Whisperv3–Qwen3 8B reaches 14.8% PF on English, while the strongest multilingual models reach only 4.3%.The authors characterize this persistent degradation as fragility under interdependent tool use.Source: §4.2 · p. 8
  • Whisperv3–Gemma3 27B leads English refusal at 59.56%, while Whisperv3–Llama3 70B leads Indic refusal at 47.08%; SpeechLM refusal performance varies sharply across languages.Whisperv3–Qwen3 8B has a 55.97% English refusal rate and 46.47% Indic rate; Whisperv3–Gemma3 27B has a 38.20% Indic rate.Source: Table 4 · p. 8, §4.2 · pp. 8–9

4.3 ABLATION STUDIES & ANALYSIS

  • Replacing Whisper transcripts with ground-truth text improves average PF by at least +24% across non-Hindi Indic languages; IndicConformer7 recovers about 40–55% of ASR-related PF loss.The analysis attributes a significant fraction of multilingual degradation to weaker Whisper transcription in low-resource Indic settings.Source: §4.3 · p. 9
  • Removing one-shot examples drops PF by 10–17% for Parallel Tool Calling and SinTC with Retrieval, while SinTC changes little: 0% in English and 1.5% in Hindi.The authors suggest the simpler SinTC task may explain its relative stability.Source: §4.3 · p. 9
  • On English and Hindi safety queries, removing refusal prompts sharply lowers refusal rates, and adversarial hints reduce all models to 35–40%.Without prompts, Whisperv3–Gemma3 27B and Whisperv3–Llama3 70B fall fourfold; the hint ablation favors Gemma3, Qwen3, and KimiAudio over Llama3 on English.Source: Figure 3 · p. 9, §4.3 · p. 9

5 LIMITATIONS AND CONCLUSION

VoiceAgentBench reveals persistent challenges in sequential and multi-tool reasoning, multi-turn dialogue, safety robustness, and cross-lingual generalization, with SpeechLMs trailing ASR–LLM pipelines. The evaluation also leaves out several conditions and systems, including background noise and dynamic real-time tool invocation.

5 LIMITATIONS AND CONCLUSION

  • The evaluation excludes speech with background noise and therefore does not measure its effects on tool invocation.It also omits multi-turn dialogue for some Indic languages.Source: §5 · p. 9
  • The evaluation excludes closed-source voice assistants because of cost constraints and does not study dynamic, real-time tool invocation with interactive conversations.Source: §5 · p. 9
  • VoiceAgentBench results reveal persistent challenges in sequential and multi-tool reasoning, multi-turn dialogue, safety robustness, and cross-lingual generalization, with SpeechLMs trailing ASR–LLM pipelines.Source: §5 · p. 10
  • Performance degrades in low-resource and culturally grounded scenarios, and the authors hope the benchmark will support robust, culturally inclusive, reliable speech agents.Source: §5 · p. 10

ETHICS AND REPRODUCIBILITY STATEMENT

The work describes safeguards and documentation for creating and releasing its multilingual spoken-agent benchmark. It reports using strict content filtering and a controlled TTS audio-generation pipeline, while collecting no sensitive or personally identifiable information.

ETHICS AND REPRODUCIBILITY STATEMENT

A VOICEAGENTBENCH COMPARISON WITH OTHER AGENTIC BENCHMARKS

VoiceAgentBench combines speech-based tool-use scenarios, multi-turn dialogue, safety evaluation, multilingual coverage, and cultural diversity in one benchmark. The related-work discussion traces speech models’ growing capabilities while noting that recent spoken tool-use studies lack a unified benchmark across diverse scenarios.

A VOICEAGENTBENCH COMPARISON WITH OTHER AGENTIC BENCHMARKS

  • Existing speech benchmarks leave complementary gaps: VoiceBench evaluates speech-alignment safety without tool use, while AudioBench offers multilingual speech data without agentic tool-calling tasks.Source: A VOICEAGENTBENCH COMPARISON WITH… · p. 16
  • VoiceAgentBench covers speech-based single, parallel, and sequential tool calls, multi-turn dialogues, safety, multilingual coverage, and cultural diversity, unlike the compared text and speech benchmarks.Table 5 contrasts the benchmark with existing datasets across nine evaluation axes.Source: Table 5 · p. 16, A VOICEAGENTBENCH COMPARISON WITH… · p. 16

B ADDITIONAL RELATED WORK: SPEECH MODELS

C DIVERSITY-SAMPLING METHODOLOGIES

The diversity-sampling section compares four strategies for selecting synthetic speech samples using ECAPA-TDNN embedding representations. Its density-based method favors sparsely populated regions probabilistically, while farthest point sampling achieves the highest diversity in the reported comparison.

C DIVERSITY-SAMPLING METHODOLOGIES

  • The comparison evaluates Random Sampling, Density-Based Probabilistic Sampling, DPP, and FPS as strategies for maximizing embedding-space coverage under a fixed sample budget.Source: C DIVERSITY-SAMPLING METHODOLOGIES · p. 17

C.1 DENSITY-BASED PROBABILISTIC METHOD

  • The density-based method assigns higher scores to samples with fewer neighbors and lower scores to dense-cluster samples, then normalizes scores into probabilities for selection.Probabilistic selection preserves randomness to avoid bias toward extreme outliers.Source: §C.1 · pp. 17–18
  • Audio samples are represented by fixed-dimensional ECAPA-TDNN embeddings; pairwise distances measure similarity, and a radius based on the mean pairwise distance defines each sample’s neighbors.Neighbor counts tally samples within that radius.Source: §C.1 · pp. 17–18

C.2 COMPARISION OF METHODOLOGIES FOR SELECTION OF DIVERSE AUDIOS

  • Farthest Point Sampling consistently achieves the highest diversity across runs, outperforming Density-Based Sampling, DPP, and especially Random Sampling.Figure 5 presents mean-distance distributions and t-SNE visualizations for a representative run.Source: Figure 5 · p. 17, §C.2 · p. 18

D ABLATION RESULTS

The ablations examine how ASR transcription errors affect Indic performance and how one-shot prompting changes KimiAudio 7B results. They compare transcript sources across three LLMs and prompting settings across English and Hindi.

D ABLATION RESULTS

  • Replacing ground-truth transcripts with ASR outputs is evaluated for its accuracy impact across three models and three Indic tool-calling tasks.The comparison includes WhisperV3 and IndicConformer transcripts; difference rows measure degradation across tool selection, call structure, and parameter filling.Source: D ABLATION RESULTS · pp. 18–19, Table 6 · p. 19
  • One-shot prompting gives KimiAudio 7B gains in tool-call structure and response following over zero-shot prompting.The comparison covers English and Hindi, and the table caption also reports zero-shot decreases in TCS and PF accuracy.Source: D ABLATION RESULTS · p. 19, Table 7 · p. 19

E.1 INDIC MULTILINGUAL RESULTS

Across Indic languages, ASR–LLM pipelines generally outperform SpeechLMs, while performance worsens as tool-use tasks become more complex. Comparisons also report language-specific gaps, model significance, latency, and annotation and judge reliability.

E.1 INDIC MULTILINGUAL RESULTS

  • Across Indic languages and task categories, Whisper-based ASR–LLM pipelines outperform SpeechLMs, which lag across tool-selection, call-structure, and parameter-filling scores.The comparison covers single-tool, retrieval-augmented, and parallel-tool calling tasks.Source: §E.1 · pp. 20–21, Table 8 · p. 20
  • SpeechLMs deteriorate on retrieval and parallel calls; in sequential-dependent calling, their PF is typically below 2%, while pipelines generally remain below 10%.Sequential-dependent calling produces the lowest PF scores across models, and pipeline results remain modest and inconsistent.Source: §E.1 · p. 21
  • Hindi PF exceeds non-Hindi Indic results across tasks, including 62.7% versus 33–50% for KimiAudio 7B on single-tool calling.The passage also reports Hindi advantages in retrieval and parallel-tool calling.Source: §E.1 · p. 21

E.2.1 SPEECH LMS VS ASR-LLM MODELS

  • ASR–LLM pipelines generally outperform SpeechLMs in single-tool, retrieval, and parallel calling, though significance is not consistent against KimiAudio 7B.The reported exceptions include nonsignificant comparisons and parameter-filling results.Source: §E.2.1 · p. 21, Table 9 · p. 22, Table 10 · p. 22

E.2.2 KIMIAUDIO 7B VS OTHER SPEECH LMS

  • KimiAudio 7B significantly outperforms other SpeechLMs across categories and languages, with fairly narrow reported confidence intervals.The significance tests report p < 0.0001; the authors describe the intervals as supporting reliable accuracy estimates.Source: §E.2.2 · pp. 21, 23, Table 12 · p. 23

F TIME TAKEN FOR FIRST TOKEN (TTFT) GENERATION: SPEECHLM VS ASR-LLM

  • Under the same hardware conditions, p90 TTFT is approximately 40 ms for Qwen2.5-Omni 7B and around 800 ms for Whisper-large-v3 with Qwen3 8B.Measurements used 100 queries averaging 3.5 seconds on a single H100 GPU; the passage associates pipeline latency with transcription overhead and SpeechLMs with interactive use.Source: F TIME TAKEN FOR FIRST TOKEN (TTFT)… · pp. 23–24

G HUMAN VALIDATION OF LLM-GENERATED GROUND TRUTH

  • Human annotators judged 496 of 507 sampled ground-truth entries correct, for 97.83% human-verified accuracy.The remaining cases involved minor partial inconsistencies, often differences in location granularity.Source: G HUMAN VALIDATION OF LLM-GENERATED… · p. 24

H.1 PAIRWISE AGREEMENT

H.2 COHEN’S KAPPA

Cohen’s κ results indicate substantial agreement between GPT-4o-mini and human annotators.

H.2 COHEN’S KAPPA

  • Average Cohen’s κ was 0.79 between GPT-4o-mini and human annotators, indicating substantial agreement.The passage describes κ as discounting agreement expected by chance.Source: §H.2 · p. 24

I TTS QUALITY EVALUATION

This section presents human MOS evaluations supporting the English and Hindi TTS engine choices, defines tool-call evaluation metrics and schemas, and illustrates spoken agentic tasks across cab, food, payment, electricity-bill lookup, and medical dialogue workflows.

I TTS QUALITY EVALUATION

  • ElevenLabs leads the English MOS results, while Krutrim has the highest Hindi prosody, clarity, and pronunciation scores; both support the selected scalable synthesis engines.The pilot rated 50 samples per language from LJSpeech for English and IISc SYSPIN for Hindi across naturalness, prosody, pronunciation, and clarity; ElevenLabs was also strong on Hindi naturalness and prosody, while Krutrim offered a quality-cost balance for Indic synthesis.Source: I TTS QUALITY EVALUATION · p. 25, Table 13 · p. 25, Table 14 · p. 25

J EVALUATION FRAMEWORK IMPLEMENTATION

  • The evaluation framework assesses model performance in progressively layered dimensions: tool selection, tool-call structure, and parameter filling.Representative examples illustrate the metrics.Source: J EVALUATION FRAMEWORK IMPLEMENTATION · p. 25

J.1 TOOL SELECTION

  • Tool selection accuracy requires an exact match between the predicted and gold function names, penalizing incorrect choices and tool hallucinations, including in single-tool calls.Source: §J.1 · p. 25

J.2 TOOL CALL STRUCTURE

  • Schema accuracy checks whether predicted calls parse against Pydantic models generated from JSON schemas, flagging syntax failures and structural mismatches such as missing fields or incorrect types.Pydantic provides strict type and structural enforcement with reproducible automated validation.Source: §J.2 · p. 26

J.3 PARAMETER FILLING

  • Parameter filling uses GPT-4o-mini as a judge to assess semantic fidelity to the query and gold answer, rather than relying on rigid exact-string matching.The judge reasons about alignment before returning a binary correct/incorrect judgment.Source: §J.3 · p. 27

K CUSTOM AGENT TOOLS

  • The custom sequential-calling tools are organized around three representative agents: Cab, Food, and Payment.Source: K CUSTOM AGENT TOOLS · p. 28

K.1 CAB AGENT

  • The Cab Agent schema includes functions for geocoding, trip pricing, vehicle availability, booking confirmation, payment lookup, and cancellation.Source: §K.1 · pp. 28–30

K.2 FOOD AGENT

  • The Food Agent schema supports filtered vendor or product searches, retrieval of a user’s order history, and listing saved addresses.Source: §K.2 · pp. 30–31
  • Food-agent schemas cover provider and address lookup, basket management, customized items, and checkout.The listed operations include adding or removing products, viewing or clearing the basket, retrieving customization options, and starting checkout.Source: §K.2 · pp. 31–32

K.3 PAYMENT AGENT

  • Payment-agent schemas define provider and category discovery and billing retrieval with service, provider, and user-field inputs.The billing schema also includes an authentication-token field and returns bill details such as amount, due date, and status.Source: §K.3 · pp. 32–34

L VOICEAGENTBENCH EXAMPLES

  • The examples appendix covers single-tool calls with and without retrieval, parallel and sequential workflows, multi-turn dialogue, and safety evaluation.It also contrasts source-native English examples with Indian-context examples in its topic overview.Source: L VOICEAGENTBENCH EXAMPLES · p. 34

L.1 EXAMPLES OF SINGLE, SINGLE WITH RETRIEVAL AND PARALLEL TOOL CALLING

  • Examples demonstrate a nearby-restaurant lookup, ticket booking with a camping add-on, and parallel retrieval of historical information.The parallel example asks about the Battle of Plassey and the Treaty of Allahabad.Source: §L.1 · pp. 34–35

L.2 EXAMPLES OF SEQUENTIAL DEPENDENT TOOL CALLING

  • The sequential-calling subsection introduces examples across three custom-agent tools and includes a cab-availability query from Jayanagar to Majestic in Bangalore.The cab example provides user information and an associated audio path.Source: §L.2 · p. 36, Cab Agent · p. 36

Cab Agent.

  • The cab example resolves Jayanagar and Majestic in Bangalore and checks vehicle availability between the locations.Its expected calls include coordinate lookup and vehicle availability checking.Source: Cab Agent · pp. 36–37

Food Agent.

  • The food example searches for pizza from Domino’s in Whitefield, fetches topping options, and adds a customized item to the basket.The expected calls pass product and option references into the customized-item addition.Source: Food Agent · pp. 37–38

Payment Agent.

  • The payment example requests an electricity-bill lookup using the user’s account number and authentication token.Expected calls list providers and fetch billing information for the electricity category.Source: Payment Agent · p. 38

L.3 EXAMPLE OF MULTI TURN DIALOGUES TOOL CALLING

  • The multi-turn dialogue moves from fatigue-related disease questions to an appointment request, with the assistant asking for the patient’s name before registration.The example includes follow-up questions about chronic fatigue syndrome before the appointment request.Source: §L.3 · pp. 39–40

L.4 EXAMPLES OF SAFETY EVALUATION

This section presents safety-evaluation examples involving harmful spoken requests, contrasting safety hints and model instructions with and without a refusal prompt. It also documents GPT-4o-mini’s evaluation role and the limited use of LLMs in the research process.

L.4 EXAMPLES OF SAFETY EVALUATION

  • The safety subset contrasts a malicious software request with an explicit tool hint and a fraudulent purchase request without a hint.The examples cover cybercrime and fraud, respectively.Source: §L.4 · p. 40

M.1 MODEL INSTRUCTIONS

  • Safety instructions are evaluated in two variants: with an explicit refusal prompt and without one.Both variants otherwise direct the agent to select tools and return tool calls without extra text.Source: §M.1 · p. 41
  • The model instructions mark the hotel-booking exchange as an example only, not a template for final tool calls.The example depicts a user booking a Mumbai hotel, supplying the hotel name after a follow-up, and the assistant confirming the stay.Source: §M.1 · p. 46

M.2 LLM AS A JUDGE

  • GPT-4o-mini evaluates parameter filling across tool-calling tasks and refusal behavior on harmful requests in the safety subset.The corresponding evaluation prompts are provided in the appendix.Source: §M.2 · p. 46

N USE OF LARGE LANGUAGE MODELS (LLMS)

  • LLMs were used only for writing assistance, not for research ideation, experimental design, or results analysis.Assistance included polishing grammar, improving clarity, and suggesting alternative phrasings.Source: N USE OF LARGE LANGUAGE MODELS (LLMS) · p. 48
Loading 2510.07978v3…