ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models

Dataset and motivation

• Existing TALM benchmarks largely use short or single-turn interactions, omitting clarification, missing-information, and chained-API scenarios found in real-world tool use.These omissions make it difficult to assess models under complex user-system interactions.Source: §1 · p. 1
• ToolDial contains 11,111 multi-turn dialogues averaging 8.95 turns, with 16 user and system actions that model realistic tool-use interactions.The action set includes clarifying intent and handling failures to provide requested information.Source: §1 · pp. 1–2

Generation framework

• The generation pipeline builds an API compatibility graph, selects API pairs and action skeletons, augments dialogue states, and converts them into natural utterances.API-graph edges connect an API’s output to another API’s compatible input.Source: §1 · p. 2

Evaluation

• Evaluation covers next-action prediction, API and dialogue-state prediction, and faithfulness to API outputs across multiple language models.The tasks test both reasoning traces and responses grounded in tool results.Source: §1 · p. 2

Results

• GPT-based models struggle with dialogue-state prediction and clarification or parameter-request actions, while fine-tuned TD-Llama reaches 77.5% and 91.0% action accuracy with GT and without GT.Performance also declines as dialogue length increases, whereas fine-tuning substantially improves smaller Llama models.Source: §1 · p. 2

Contributions

• The work contributes a large-scale benchmark and framework for studying TALM abilities across multi-turn interactions with external APIs.The authors position ToolDial as both an assessment and improvement resource for complex tool use.Source: §1 · p. 2

Tool-Augmented Language Models

• Most TALM datasets focus on single-turn tool selection and reasoning, whereas ToolDial targets multi-turn interactions with richer dialogue dynamics.ToolNet uses an API graph without input-output compatibility, and TaskBench focuses on single-turn API-chain inference.Source: §2 · pp. 2–3
• ToolDial’s dataset comparison counts action types using its taxonomy as the reference for comparing coverage across TALM datasets.The comparison is presented in Table 1.Source: Table 1 · p. 3
• ToolDial provides longer or larger-scale interactions than prior multi-turn datasets, including 11,111 dialogues compared with ToolTalk’s 78 human-annotated dialogues.API-Bank averages 2.84 turns and ToolTalk averages 6.44 turns, while ToolDial averages 8.95 turns.Source: §2 · pp. 2–3

Task-Oriented Dialogue System

• Task-oriented dialogue benchmarks supply dialogue-state formats and action taxonomies, but typically lack user failures to provide information required for API calls.ToolDial adapts these concepts while adding difficult tool-agent situations and chained API use.Source: §2 · p. 3

3.1 GRAPH CONSTRUCTION

• ToolDial constructs an API graph by matching input and output entities with description embeddings, keyword-augmented embeddings, and longest-common-subsequence similarity.The resulting graph contains 4,857 edges from approximately 500 million candidates; automated edge evaluation achieved a Matthews Correlation Coefficient of 0.868, 70.9% precision, and 95.0% negative predictive value.Source: §3.1 · pp. 4–5

3.2 ACTION SEQUENCES

• The dataset defines 16 user and system actions and combines them into 23 plausible sequences whose structure depends on API inputs, outputs, intent clarity, and user failures to provide information.Actions include Request, Clarify, Suggest, Retriever call, Call, Inform, Fail inform, Affirm, and Negate, enabling chained API interactions and parameter collection.Source: §3.2 · pp. 5–6

3.3 SCENARIO INSTRUCTION GENERATION

• Each dialogue samples one API or a connected API pair and action sequence, then uses GPT-4o to generate a relevant query, concrete dialogue states, scenario instructions, and aligned user and system utterances.Dialogue states specify the target API, required inputs, and user-provided values, serving both utterance generation and ground-truth dialogue-state-tracking evaluation.Source: §3.3 · p. 6, §3.4 · p. 7

3.4 DIALOGUE GENERATION

• 11,111 dialogues average 8.95 turns and cover 23 action sequences across varied real-world scenarios.The dataset uses 473 RapidAPI APIs spanning 23 domains and incorporates predefined speaking styles to reduce repetitive generated interactions.Source: §3.4 · pp. 7–8
• Quality evaluation of 100 sampled dialogues used G-Eval and human annotators across naturalness, coherence, efficiency, and faithfulness, with particularly high G-Eval scores for efficiency.The evaluation emphasizes whether TALM actions efficiently collect information and execute APIs while keeping responses consistent with API outputs.Source: §3.4 · p. 7

4.1 EVALUATION TASKS

• Dialogue-state tracking evaluates selecting the API and extracting user-provided input values, with 6,747 annotated states matched exactly against predictions.Matching compares API names, input parameters, and corresponding values after normalization.Source: §4.1 · p. 8
• Action prediction evaluates the next system action from dialogue history and reasoning steps using 9,200 annotated actions and exact matching.Ground-truth thought is excluded because it directly cues the action.Source: §4.1 · p. 9

4.2 EXPERIMENT SETTINGS

• Without ground truth, performance reflects accumulated prediction errors, while faithfulness is evaluated only without ground truth because final responses lack ground-truth labels.Experiments compare with-GT and without-GT settings across listed baseline and fine-tuned models.Source: §4.2 · p. 9

4.3 RESULTS

• GPT models exceeded 90% on faithfulness, whereas smaller Llama-based models remained around 88.4%.Faithfulness measures whether final responses reflect API call outputs.Source: §4.3 · p. 10
• 77.1% of TD-Llama’s reasoning steps were correct, but only approximately 28.3% of 1,166 test dialogues had correct reasoning traces across every turn.A reasoning step required both the dialogue state and action to be correct.Source: §4.3 · p. 10

• ToolDial models realistic multi-turn TALM interactions using an API graph, 16 user and system actions, dialogue states, and GPT-4o-generated utterances.The generation process samples APIs and action sequences, augments them with API-specific dialogue states, and converts them into utterances.Source: §5 · p. 10
• Modern language models perform poorly when predicting appropriate actions and dialogue states in complex multi-turn interactions.Source: §5 · p. 10

• The authors provide prompts for generating RapidAPI API input and output descriptions.Source: §A.1 · p. 13

• The authors provide the prompt used for keyword extraction from RapidAPI APIs.Source: §A.2 · p. 13

• The interaction framework defines 8 user actions and 8 system actions.Source: §A.3 · p. 13
• Table 6 lists the user actions and their descriptions.Source: Table 6 · p. 13
• Table 7 lists the TALM actions and their descriptions.Source: Table 7 · p. 13

• Retriever scores determine whether the system requests API parameters, suggests an API, or asks for query clarification.Scores of at least 0.6 trigger parameter requests, scores from 0.5 to below 0.6 trigger suggestions, and lower scores trigger clarification.Source: §A.4 · p. 14
• Retriever status records whether retrieval occurred, which APIs or output components were retrieved, and their scores.It distinguishes API discovery from obtaining an input parameter through a prior API’s output.Source: §A.4 · pp. 13–14
• Dialogue state records the currently attempted API and parameter values extracted from the dialogue history.The format represents unconfirmed APIs, confirmed APIs with required and optional slots, and partially populated parameters.Source: §A.4 · p. 14

• Users can either provide the other known parameters while stating they do not know one parameter, or simply report that they do not know the missing parameter.Source: §A.5 · p. 14

A.7 SCENARIO INSTRUCTION

• Detailed scenario instructions constrain generated dialogues so predefined interactions and required entities are represented correctly.The appendix provides an example scenario instruction format.Source: §A.7 · p. 15, Table 8 · p. 15

A.8 DST ERROR ANALYSIS

• DST errors include missing states, incorrect API confirmation, incomplete fields or slots, and incorrectly extracted parameter values.Value errors include selecting incorrectly from multiple API results and failing to add a value already stated in the user query.Source: §A.8 · pp. 15–16
• Multiple-result errors arise when a model arbitrarily selects one returned value instead of asking the user to disambiguate.In the payment example, two execution dates are returned, but the prediction selects one identifier without user confirmation.Source: §A.8 · pp. 16–17

------------------------------------------------------------------ <Example2. Inform Intent Add Error Error>

• An inform-intent-add error occurs when the user provides a required value, but the predicted dialogue state leaves that slot empty.The createPlan example states the name “Basic Plan,” yet the prediction omits it.Source: ---------------------------------------… · p. 17

A.9 COMPREHENSIVE ACTION SEQUENCES

• The generation space contains 23 action sequences: 15 with two APIs, 7 with one API, and 1 with no API call.Two-API sequences distinguish direct parameter requests from requests to choose among multiple returned values.Source: §A.9 · p. 18, Table 11 · p. 18, Table 12 · p. 19

A.10 DST ACCURACY BASED ON TURN LENGTH

• DST accuracy is evaluated as dialogue length increases, with Figure 5 organizing results by model and number of turns.Source: Figure 5 · p. 19

A.11 REMOVING MISMATCH ERRORS

• The appendix identifies domain and entity mismatches as errors that can occur when constructing API-graph edges.Examples pair semantically similar tournament identifiers across different sports domains, illustrating why both domain and entity compatibility matter.Source: §A.11 · p. 19, API 1 · p. 19, API 2 · pp. 19–20

A.12 PROMPT FORMAT FOR THE EXPERIMENT

• The experiment prompt format separates dialogue-state tracking, action prediction, and faithfulness evaluation, including settings with or without gold annotations.The appendix also releases prompts for graph-edge, DST, action-prediction, and faithfulness evaluations.Source: §A.12 · p. 20, §A.13 · p. 20, Table 14 · p. 21