PaperlayerView exact revision on arXiv

Source-linked AI summary

Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuning

Mingyang Chen, Haoze Sun, Tianpeng Li, Fan Yang, Hao Liang, Keer Lu, Bin Cui, Wentao Zhang, Zenan Zhou, Weipeng Chen

arXiv:2410.12952v2cs.CL

TL;DR

Most function-calling research centers on single-turn use, while complex queries can require planning across multiple calls. BUTTON constructs compositional instruction data and multi-turn trajectories, and models tuned with this data perform better across GTA and Tool-Query.

  • Problem

    Prior function-calling research largely evaluates single-turn function selection and argument provision, while complex real-world queries can require task decomposition and sequential call planning.

    Source: §1 · pp. 1–2

  • Method

    BUTTON builds atomic tasks into compositional instructions and functions, then uses simulated user, assistant, and tool agents to generate multi-turn trajectories.

    Source: §1 · p. 2, §2.2 · p. 5

  • Results

    Across GTA and Tool-Query, BUTTON-tuned models performed better; Llama3-8B answer accuracy increased from 1.4% to 30.5%.

    Source: §3.2 · p. 8

  • Takeaways & Limitations

    The findings identify planning with functions, not only correct function use, as a challenge in multi-turn function calling.

    Source: §3.2 · p. 8

    The pipeline does not strictly define atomic or compositional tasks, relying instead on prompts and instruction-following LLMs to align generated data with expectations.

    Source: §2.1 · p. 4

Abstract

from arXiv · show

Large Language Models (LLMs) have exhibited significant potential in performing diverse tasks, including the ability to call functions or use external tools to enhance their performance. While current research on function calling by LLMs primarily focuses on single-turn interactions, this paper addresses the overlooked necessity for LLMs to engage in multi-turn function calling--critical for handling compositional, real-world queries that require planning with functions but not only use functions. To facilitate this, we introduce an approach, BUTTON, which generates synthetic compositional instruction tuning data via bottom-up instruction construction and top-down trajectory generation. In the bottom-up phase, we generate simple atomic tasks based on real-world scenarios and build compositional tasks using heuristic strategies based on atomic tasks. Corresponding function definitions are then synthesized for these compositional tasks. The top-down phase features a multi-agent environment where interactions among simulated humans, assistants, and tools are utilized to gather multi-turn function calling trajectories. This approach ensures task compositionality and allows for effective function and trajectory generation by examining atomic tasks within compositional tasks. We produce a dataset BUTTONInstruct comprising 8k data points and demonstrate its effectiveness through extensive experiments across various LLMs.

1 INTRODUCTION

BUTTON addresses the gap between single-turn function use and real-world queries that require planning across multiple calls by generating compositional instruction-tuning data with multi-turn function-call trajectories. Its pipeline builds tasks and functions bottom-up, then uses simulated human, assistant, and tool interactions to create the trajectories.

1 INTRODUCTION

  • Prior function-calling alignment research predominantly trains and evaluates single-turn function selection and argument generation, often overlooking planning (Patil et al., 2023; Liu et al., 2024b).The introduction contrasts retrieving a flight schedule with booking the first flight, which requires sequential calls.Source: §1 · p. 1
  • BUTTON targets complex compositional queries by training models to decompose them into multi-turn function calls, addressing the need to plan sequences of calls.The paper frames this as a response to real-world tasks that cannot be completed in a single step.Source: §1 · pp. 1–2
  • Synthetic data generation must ensure compositional instructions are reasonable and solvable, functions match those instructions, and multi-turn trajectories are high-quality without human supervision.The paper identifies these requirements as barriers to obtaining suitable data from existing sources without extensive manual annotation.Source: §1 · p. 2
  • BUTTONInstruct contains 8k data points, and the paper reports that fine-tuning LLMs on it improves performance on multi-turn function-calling benchmarks.The introduction does not provide benchmark scores or comparisons.Source: §1 · p. 2
  • BUTTON’s bottom-up phase generates atomic tasks, combines them through sequential or parallel-then-sequential composition, and creates functions informed by the component tasks.Its top-down phase simulates humans, assistants, and tools with generative LLMs to collect multi-turn trajectories, then packages the trajectories and function definitions as instruction data.Source: §1 · p. 2

2 METHOD

BUTTON builds multi-turn function-calling data through bottom-up composition of tasks and functions, followed by top-down simulation of interaction trajectories. Its dataset contains 8,000 examples, with trajectories spanning multiple assistant turns and function calls.

2 METHOD

  • The pipeline collects real-world scenarios, builds simple atomic tasks and compositional tasks from them, then synthesizes corresponding function definitions.BUTTON retains task breakdowns to support trajectory generation, while the functions and trajectories are integrated into instruction-tuning data.Source: §2 · pp. 2–3

2.1 BOTTOM-UP INSTRUCTION CONSTRUCTION

  • Atomic tasks are designed to be realistic, self-contained, and function-agnostic, but prompts—not strict definitions—guide generation to meet these expectations.They should be simple enough to solve without complex planning and serve as units for composing more complex tasks.Source: §2.1 · p. 4
  • Two heuristics compose atomic tasks: sequential composition adds a dependent task, while parallel-then-sequential composition adds an independent task and then a task dependent on both.Compositional tasks are filtered by checking whether their atomic subtasks can complete them; the heuristics still allow parallel calls beyond the first turn.Source: §2.1 · p. 4
  • Function definitions are generated after tasks, using task breakdowns as hints; they prioritize descriptive, general, and consistent inputs and outputs, and a subtask may need zero, one, or multiple functions.Logic, comparison, set operations, and calculations manageable by language models may require no function.Source: §2.1 · p. 5

2.2 TOP-DOWN TRAJECTORY GENERATION

  • A user agent poses the task, an assistant agent decomposes it and calls functions, and function-specific tool agents simulate responses without actual implementations.The assistant has access to the task breakdown and available tools; its responses combine free-text observations and thoughts with function calls before a final answer.Source: §2.2 · pp. 5–6

2.3 DATASET COLLECTION VIA BUTTON

  • Using GPT-4o2 for generation at each pipeline step, the authors collected 8,000 BUTTONInstruct multi-turn function-calling data points with system, user, assistant, and tool content.The available functions for each user question are listed in its system prompt.Source: §2.3 · p. 6
  • Most BUTTONInstruct trajectories contain three or more assistant turns and more than two function calls, averaging over one function call per turn.These distributions indicate diversity across trajectory length and function-call frequency, despite using simple task-composition heuristics.Source: §2.3 · p. 6, Figure 2 · p. 6

3 EXPERIMENTS

Experiments evaluate BUTTONInstruct-tuned Llama3 and Qwen2 models on multi-turn function-calling benchmarks; the reported results show improvements over original instruction-tuned models and a benefit from parallel calling.

3.1 EXPERIMENTS SETUP

  • GTA comprises 229 human-crafted multimodal queries, each accompanied by one or two images; Tool-Query comprises 60 complex multi-round tasks across weather, movies, and academia.The evaluated model families were Llama3-8B, Llama3-70B, Qwen2-7B, and Qwen2-72B.Source: §3 · p. 6, §3.1 · p. 7

3.2 MAIN RESULTS

  • Llama3-8B final answer accuracy on GTA rose from 1.4% to 30.5%, and BUTTON-tuned models outperformed corresponding instruction-tuned versions on GTA and Tool-Query.On Tool-Query, grounding accuracy was already near or above 95% for both model groups, while BUTTON-tuned models improved process and success rates.Source: §3.2 · pp. 7–8, Table 2 · p. 8, Table 3 · p. 8

3.3 FURTHER ANALYSIS

  • Disabling parallel calling reduced Llama3-8B-BUTTON Tool-Query success rate from 35.0% to 28.3% and process rate from 63.2% to 58.7%.The authors interpret these decreases as evidence that parallel calling improves performance, particularly under restricted step conditions.Source: §3.3 · p. 9

4 RELATED WORK

Related work covers synthetic data and function-calling datasets, but prior compositional instruction work does not address multi-turn function calling, and APIGen focuses predominantly on single-turn scenarios (Liu et al., 2024b).

Synthetic Data

  • Synthetic data offers a scalable alternative to resource-intensive collection of large, diverse, high-quality datasets.Prior work has explored improving its quality, diversity, and utility across domains.Source: §4 · p. 10
  • Chain-of-Instruct (Hayati et al., 2024) builds step-by-step compositional instructions, while Sequential Instruction Tuning (Hu et al., 2024) seeds intermediate tasks from single-task instructions.Neither focuses on compositional instructions for multi-turn function calling that requires identifying, invoking, and planning with functions.Source: §4 · p. 10

Function Calling

  • Function calling enables modular, structured, interactive workflows that extend LLMs to practical tasks requiring computational or domain-specific tools.Patil et al. (2023) generated synthetic prompts for repository APIs, and Qin et al. (2023) expanded API data and interaction scenarios using RapidAPI.Source: §4 · p. 10
  • Unclear API descriptions and availability or runtime failures complicate accurate solution-path annotation, while APIGen (Liu et al., 2024b) uses rigorous verification but focuses predominantly on single-turn function calling.Earlier API-dataset approaches drew from TorchHub, TensorHub, and HuggingFace, with Qin et al. (2023) adding RapidAPI data.Source: §4 · p. 10

5 CONCLUSION · A PROMPT DETAILS

This section outlines BUTTON’s bottom-up and top-down data-generation pipeline and documents the prompts and procedures for scenario collection, task construction, function generation, and simulated multi-turn interactions among human, assistant, and tool agents.

5 CONCLUSION

  • BUTTONInstruct contains 8,000 high-quality multi-turn function-call trajectories, and fine-tuned LLMs improved on multi-turn function-calling benchmarks.The pipeline generates atomic tasks, composes them, synthesizes functions, and simulates multi-agent interactions without human supervision; future work proposes data verification and extension to applications such as embodied AI.Source: §5 · p. 10

A.1 SCENARIO COLLECTION

  • Scenario collection seeds from glaive-function-calling-v2 and ToolLLama, extracting concise scenario types and expanding them into substantially different, realistic daily-life scenarios.The extraction prompt limits descriptions to 10 words; the expansion prompt also limits its single-sentence output to 10 words and retains the original format.Source: §A.1 · p. 14

A.2 ATOMIC TASK CONSTRUCTION

  • Atomic tasks are realistic, one-step scenario-based requests specified with enough detail for deterministic function arguments, without naming tools or functions.The prompt limits task descriptions to 30 words and disallows ambiguous or vague information.Source: §A.2 · p. 15

A.3 COMPOSITIONAL TASK CONSTRUCTION

  • Compositional tasks combine dependent sequential steps or independent parallel tasks whose outputs jointly feed a subsequent step, with a filter checking task-breakdown consistency.The parallel-then-sequential strategy requires the final task to use both earlier outputs; the filter checks whether proposed subtasks can complete the compositional task.Source: §A.3 · pp. 15–17

A.4 FUNCTION GENERATION

  • Generated functions cover each subtask and compose sequentially, using succinct, comprehensible, generalized API names and descriptions rather than task-specific designs.The requirements state that APIs should be reusable for future tasks and describe functions from a generic perspective.Source: §A.4 · p. 18
  • Functions should be granular and independent, with consistent parameters and returns; subtasks may require zero, one, or multiple functions, and LLM-solvable logic needs none.Sequential functions should have outputs that align with or form part of the next function’s inputs.Source: §A.4 · p. 18

A.5 MULTI-AGENT

  • The user agent sustains human-like conversation, gives an initial detailed task without prescribing steps or tools, and encloses its turns in <human> tags.It actively explores the assistant’s ability to solve a complex task through multiple function calls.Source: §A.5 · pp. 19–20
  • The assistant agent decomposes tasks, follows an adaptable plan, checks observations and tool outputs for errors, and formats calls and observations with prescribed tags.The tool agent simulates function execution and returns results in JSON within <func_return> tags.Source: §A.5 · pp. 20–22

B DATA COLLECTION · B.1 EXAMPLE OF COLLECTED DATA

The collected example shows an assistant handling a compositional device-to-server request by retrieving current readings, sending a timestamped status update, and logging the results. It illustrates planning, parallel independent calls, and sequential completion of dependent actions.

B DATA COLLECTION

  • The collected trajectory combines a timestamped status message from DeviceA to ServerB with temperature and humidity retrieval and database logging.The example reports temperature 22.5 and humidity 45.3 in the logged data.Source: §B.1 · pp. 25–27

B.1 EXAMPLE OF COLLECTED DATA

  • The user requests a status update from DeviceA to ServerB containing the current timestamp and a database log with current temperature and humidity readings.The scenario supplies the device, server, and desired contents of both actions.Source: §B.1 · p. 25
  • The assistant plans to fetch the timestamp, send the message, retrieve temperature and humidity, then log the data, running independent retrieval steps in parallel.The plan delays database logging until the required message and sensor data are available.Source: §B.1 · p. 25
  • The retrieved results report timestamp 2023-10-03T14:22:00Z, temperature 22.5, and humidity 45.3, with no cited passage attributing the readings to DeviceA.Source: §B.1 · pp. 25–26
  • The assistant’s send_message_to_server call sends “Status update at 2023-10-03T14:22:00Z” from DeviceA to ServerB, and the recorded result says the message was delivered successfully.Source: §B.1 · p. 26
  • The assistant logs the message, timestamp, temperature, and humidity on ServerB, receives a successful log result, and tells the user the task is complete.The final response reports temperature 22.5C and humidity 45.3%.Source: §B.1 · p. 27

B.2 PARALLEL FUNCTION CALLING

BUTTON distinguishes parallel from sequential function calling through system-prompt instructions and uses compositional-task trajectories to expose models to both behaviors. The authors state that parallel-calling accuracy is not their focus and leave it for future work.

B.2 PARALLEL FUNCTION CALLING

  • The prompts differ only in the final Function Call instruction: one permits parallel calls only for independent functions, while the other requires one call at a time and waiting for its response.The function-definition list is supplied in JSON format as {func list}.Source: §B.2 · p. 28
  • Trajectory generation uses a 50% prompt assignment to expose compositional tasks to both parallel and non-parallel calling behaviors, including tasks whose subtasks cannot be parallelized.Tasks with parallelizable subtasks receive the disabling prompt half the time and the enabling prompt half the time; tasks whose subtasks cannot run in parallel receive the enabling prompt half the time. Parallel-calling accuracy is not the focus of this work and is left for future work.Source: §B.2 · p. 28

B.3 FUNCTION DISTRIBUTION

Figure 4 shows the distribution of generated BUTTONInstruct functions, indicating diversity and a distribution consistent with daily tasks.

B.3 FUNCTION DISTRIBUTION

  • The sunburst chart shows diverse generated functions consistent with daily tasks, with its inner and outer circles representing the first and second words in function names.Figure 4 presents the distribution of generated functions in BUTTONInstruct.Source: Figure 4 · p. 29, §B.3 · p. 29

C EXPERIMENTS · C.1 EVALUATION METRICS

GTA and Tool-Query metrics follow their original methodologies with modifications: GTA omits questions requiring Mathpix, while Tool-Query accepts answers found within model outputs. The changes apply to baselines as well, supporting fair comparisons.

C.1 EVALUATION METRICS

  • GTA and Tool-Query metric calculations follow their original papers’ methodologies with slight modifications.The section describes these adjustments as applying to all evaluated models, including baselines.Source: §C.1 · pp. 29–30
  • GTA end-to-end evaluation uses the Mathpix API for MathOCR, so questions invoking that subscription-required API were excluded.This avoided evaluating those questions during numerous pilot experiments and analyses.Source: §C.1 · p. 29
  • 209 GTA test samples remained after excluding questions that invoke Mathpix in end-to-end mode.Source: §C.1 · pp. 29–30
  • Tool-Query scoring checks whether the ground-truth answer appears in the model output, replacing the original exact-match strategy.This captures successful answers that include additional wording beyond the labeled answer.Source: §C.1 · p. 30
  • The metric modifications apply to baselines and other models alike, ensuring comparisons remain fair.Source: §C.1 · p. 30

C.2 IMPLEMENTATION · C.3 PROMPTS FOR ABLATION STUDY

This appendix section documents a shared training recipe and ablation prompts for bottom-up task construction and top-down trajectory generation, including task constraints and simulated human–assistant–tool interactions. It also illustrates a multi-turn trajectory that retrieves and reasons over book and author facts before giving a final answer.

C.2 IMPLEMENTATION

  • Training used 4×8 H800 GPUs, learning rate 2e-5, batch size 64, cosine decay, response-only loss, packing, and flash attention; smaller models ran five epochs and larger models two.Llama3-8B and Qwen2-7B trained for five epochs; Llama3-70B and Qwen2-72B trained for two.Source: §C.2 · p. 30

C.3 PROMPTS FOR ABLATION STUDY

  • The w/o Bottom-Up prompt directly generates a compositional task from a scenario, requiring specific, deterministic descriptions of at most 30 words and no more than seven steps.It prohibits mentioning tools, proposing solutions, or relying on unstated personal information or prior knowledge.Source: §C.3 · p. 30
  • The w/o Top-Down prompts generate multi-turn trajectories by simulating a human who supplies task details without solution steps, an assistant that uses tools, and tools that return JSON results.The assistant’s format includes observations, thoughts, and function calls; it checks tool outputs for errors and provides a final answer when the task is complete.Source: §C.3 · pp. 31–32
  • The parallel w/o Top-Down variant permits multiple function calls in parallel when the functions are independent.The appendix distinguishes this prompt from the non-parallel trajectory-generation prompt.Source: §C.3 · pp. 33–34
  • The trajectory retrieves The Great Gatsby’s publication date and identifies F. Scott Fitzgerald as its author.The example uses get_book_publish_date and get_book_author.Source: §C.3 · p. 36
  • After retrieving Fitzgerald’s birthdate, the assistant concludes he was 28 when The Great Gatsby was published.It calls get_author_info, then gives the conclusion as its final answer.Source: §C.3 · p. 36
  • The prompt requires generating the trajectory in the shown format without extra explanation or comments.Source: §C.3 · p. 36

C.4 CASES ON PARALLEL CALLING

Two Tool-Query cases show Llama3-70B-BUTTON using parallel function calls to reduce turns in multi-turn tasks. In case 2, parallel calling completes the task in 6 steps rather than 12.

C.4 CASES ON PARALLEL CALLING

  • In case 2, parallel calling completes the task in 6 steps, versus 12 without it because two steps each expand to four.The count includes the final answer.Source: §C.4 · pp. 36–37
  • In case 1, the assistant checks Lazhar Labiod’s collaborator counts in parallel and identifies Mohamed Nadif, with 2 co-authored papers, as the top collaborator.The other collaborator, Chakib Fettal, has 1 co-authored paper.Source: §C.4 · pp. 36–38
  • The second case asks for the combined citations of four papers co-authored by Florian Kirchbuchner and Fadi Boutros.The outlined procedure checks their collaboration and sums the citation counts of the resulting papers.Source: §C.4 · pp. 38–39

Related papers

Loading 2410.12952v2…