PaperlayerView exact revision on arXiv

Source-linked AI summary

Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks

Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Ikbal, Sachin Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Meza Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim Roukos, Luis Lastras, Pavan Kapanipathi

arXiv:2407.00121v1cs.LGcs.AIcs.CL

TL;DR

LLMs must perform accurately in reasoning and planning and in function calling, including identifying, calling, and interacting with tools and APIs in external environments. The paper instruction-tunes GRANITE-20B-FUNCTIONCALLING through multi-task learning on seven granular function-calling tasks. GRANITE-20B-FUNCTIONCALLING provides the best performance among open models on BFCL and ranks fourth overall.

  • Problem

    LLMs must perform accurately in reasoning and planning and in function calling, including identifying, calling, and interacting with tools and APIs in external environments.

    Source: §1 · p. 2

  • Method

    The paper instruction-tunes GRANITE-20B-FUNCTIONCALLING through multi-task learning on seven granular function-calling tasks.

    Source: §1 · pp. 2–3

  • Results

    GRANITE-20B-FUNCTIONCALLING provides the best performance among open models on BFCL and ranks fourth overall.

    Source: §1 · p. 3, §5.3.1 · p. 11

  • Takeaways & Limitations

    The model shows strong generalizability across multiple out-of-domain function-calling datasets and tasks.

    Source: §1 · p. 3, §6 · p. 14

    BFCL evaluation has concerns about language-specific Java and JavaScript syntax and brittleness from REST API availability and call limits.

    Source: §5.3.1 · p. 12

Abstract

from arXiv · show

Large language models (LLMs) have recently shown tremendous promise in serving as the backbone to agentic systems, as demonstrated by their performance in multi-faceted, challenging benchmarks like SWE-Bench and Agent-Bench. However, to realize the true potential of LLMs as autonomous agents, they must learn to identify, call, and interact with external tools and application program interfaces (APIs) to complete complex tasks. These tasks together are termed function calling. Endowing LLMs with function calling abilities leads to a myriad of advantages, such as access to current and domain-specific information in databases and knowledge sources, and the ability to outsource tasks that can be reliably performed by tools, e.g., a Python interpreter or calculator. While there has been significant progress in function calling with LLMs, there is still a dearth of open models that perform on par with proprietary LLMs like GPT, Claude, and Gemini. Therefore, in this work, we introduce the GRANITE-20B-FUNCTIONCALLING model under an Apache 2.0 license. The model is trained using a multi-task training approach on seven fundamental tasks encompassed in function calling, those being Nested Function Calling, Function Chaining, Parallel Functions, Function Name Detection, Parameter-Value Pair Detection, Next-Best Function, and Response Generation. We present a comprehensive evaluation on multiple out-of-domain datasets comparing GRANITE-20B-FUNCTIONCALLING to more than 15 other best proprietary and open models. GRANITE-20B-FUNCTIONCALLING provides the best performance among all open models on the Berkeley Function Calling Leaderboard and fourth overall. As a result of the diverse tasks and datasets used for training our model, we show that GRANITE-20B-FUNCTIONCALLING has better generalizability on multiple tasks in seven different evaluation datasets.

1 Introduction

Function calling is presented as essential for autonomous LLM agents because it enables identifying, calling, and interacting with external tools and APIs. The paper introduces GRANITE-20B-FUNCTIONCALLING, trained on granular function-calling tasks and evaluated for open-model performance and generalization.

  • Function calling lets LLM agents access current external information and outsource computation, retrieval, and other tasks prone to hallucination.Source: §1 · p. 2
  • Existing function-calling models are limited by weak out-of-domain generalization, insufficient granular-task handling, and limited openness.Source: §1 · p. 2
  • GRANITE-20B-FUNCTIONCALLING is instruction-tuned with a multi-task approach covering function detection, parameter detection, sequencing, parallel functions, and API calling.Source: §1 · pp. 2–3
  • GRANITE-20B-FUNCTIONCALLING is on par with the best open model on BFCL and ranks fourth overall, while performing as well as or better than Meta-Llama-3-70B-Instruct despite having 20B versus 70B parameters.Source: §1 · p. 3

2 Related Work

Prior work improves LLM task performance through instruction tuning and enables function calling through prompting, direct function-call training, and specialized models. The paper positions GRANITE-20B-FUNCTIONCALLING as an open model designed to perform competitively across multiple function-calling tasks.

2.1 Instruction Tuning

  • Instruction tuning reformulates diverse NLP datasets as instruction-following tasks and fine-tunes an LLM on the resulting data.Source: §2.1 · p. 3
  • Smaller function-calling models are motivated by the need for efficient performance on local devices.Source: §2.1 · p. 4

2.2 Function Calling by LLMs

  • Function calling extends LLM applications to information retrieval, mathematical calculation, internet use, task orchestration, and programming.Source: §2.2 · p. 4
  • Function-calling systems include prompting methods, models trained to output calls directly, pretrained models, and specialized fine-tuned models.Source: §2.2 · p. 4
  • GRANITE-20B-FUNCTIONCALLING provides the best or comparable performance among open models across multiple tasks.Source: §2.2 · p. 4

3 Multi-Task Training Data

The training pipeline unifies heterogeneous API data and reuses it across seven function-calling tasks through multi-task instruction tuning. It covers complex multi-function behavior, foundational detection tasks, and natural-language response generation.

3 Multi-Task Training Data

  • API-BLEND supplies about 160K examples from five datasets, while Glaive-V2 is added to form a diverse multi-task training mixture.Source: §3 · p. 4, §3.1 · p. 5
  • The training setup reuses the same data in different formats and instructions across six function-calling subtasks plus Response Generation as a seventh task.Source: §3 · p. 5

3.1 Data Unification

  • All APIs, tools, and functions are converted into JSON representations to provide a language-independent, parsable, and widely used format.Source: §3.1 · p. 5

3.2 High-Level Function Calling Tasks

  • Nested Function Calling feeds one function’s output into the next function’s input, whereas Function Chaining calls multiple functions sequentially without requiring nesting.Source: §3.2 · p. 6
  • Parallel Functions calls the same function multiple times with different parameter-value pairs extracted from the query.Source: §3.2 · p. 6

3.3 Low-Level Function Calling Tasks

  • Low-level tasks isolate function-name selection, next-best-function prediction, and parameter-value pair detection as simpler foundations for high-level calling.Source: §3.3 · p. 7

3.4 Response Generation

  • Response Generation produces natural-language replies for conversation, clarification, or synthesis of function-call outputs.Source: §3.4 · p. 8

4 Instruct Tuning

The model is trained on a weighted mixture of task-specific function-calling datasets, with task instructions embedded in the examples. It is then fine-tuned using QLoRA on the resulting multi-task mixture.

4.1 Training Data Mixture Creation

  • Function Chaining and Next-Best Function receive mixture weights of 3:5, with internal dataset allocations of 2:3 and 2:1, respectively.The configuration divides the Function Chaining portion between SeqSGD and Glaive-V2, and the Next-Best Function portion between SeqTopV2 and SeqSNIPS.Source: §4.1 · p. 8
  • Training examples are embedded with task-specific instructions, using placeholders for the function library, user query, and user-agent conversation.The function library is represented by the <|function_call_library|> tag and the {API_SPEC_INSTRUCTION} placeholder.Source: §4.1 · pp. 8–9

4.2 Training

  • The model is trained with QLoRA on a 142K-example multi-task mixture for three epochs using eight A100_80GB GPUs.The training uses rank 8, alpha 32, dropout 0.1, a 5e-5 learning rate, ApexFusedAdam, and a linear learning-rate scheduler.Source: §4.2 · p. 9

5 Experimental Setup and Evaluation

The evaluation uses diverse out-of-domain datasets and task-specific metrics to compare GRANITE-20B-FUNCTIONCALLING with open and proprietary models. Across leaderboard, academic, response-generation, and extended-context analyses, the model shows strong performance while retaining documented evaluation and context-length caveats.

5.1 Datasets

  • The evaluation spans seven out-of-domain datasets, while BFCL uses AST summary, execution summary, relevance, and overall accuracy metrics and other datasets use F1, sequence, generation, and hallucination measures.The datasets include BFCL, ToolBench, ToolLLM, RestGPT, API-Bank, ToolAlpaca, and NexusRaven API Evaluation.Source: §5.1 · p. 10, §5.2 · p. 11

5.3.1 BFCL Leaderboard Evaluation Results

  • Fourth overall and highest among open-license models on BFCL’s overall accuracy ranking, GRANITE-20B-FUNCTIONCALLING is also among the smallest models evaluated.Models ranked above it are significantly larger.Source: §5.3.1 · pp. 11–12, Table 4 · p. 12
  • BFCL Java, JavaScript, and REST API categories have evaluation concerns because of language-specific syntax and brittleness from API availability and call limits.The authors also could not verify whether some out-of-domain datasets appeared in other models’ training data.Source: §5.1 · p. 10, §5.3.1 · p. 12

5.3.2 Function Calling Academic Benchmarks

  • 8% higher function-name F1, 7% higher LCS, and 11% higher Exact Match distinguish GRANITE-20B-FUNCTIONCALLING on ToolLLM and RestGPT function-name detection.These datasets evaluate function-name detection from natural-language utterances and also support sequencing metrics.Source: §5.3.2 · p. 12, Table 5 · p. 12
  • 0.87 function-name F1 places GRANITE-20B-FUNCTIONCALLING second to Cohere Command-R’s 0.88, while argument F1 is lower at 0.59 versus 0.62 for Command-R.These are averages across API-Bank, ToolBench, and ToolAlpaca evaluations conducted zero-shot and out of domain.Source: §5.3.2 · pp. 12–13, Table 6 · p. 13
  • Less than 0.1 hallucination rate accompanies the highest average function-name detection performance across the evaluated non-BFCL datasets.The comparison uses average F1 for function-name detection and defines hallucinations as predicted APIs absent from the provided function library.Source: §5.2 · p. 11, §5.3.2 · p. 13, Figure 3 · p. 14

5.3.3 Response Generation

  • GRANITE-20B-FUNCTIONCALLING ranks close second to Meta-Llama-3-70B-Instruct on API-Bank response generation, with a 1–5% performance difference across BertScore, ROUGE-L, and BLEU.Both models significantly outperform the other evaluated models, including function-calling-specific models.Source: §5.3.3 · p. 13, Table 7 · p. 13

5.3.4 Further Improvements

  • Function libraries can reach 94 functions, forcing truncation of argument type, required-field, and optional-field specifications under the 8192-token context limit.The authors propose testing complete specifications with longer-context Granite models.Source: §5.3.4 · p. 14

6 Conclusion

The paper introduces GRANITE-20B-FUNCTIONCALLING, an Apache 2.0 open model trained with multi-task learning on granular function-calling tasks. Across multiple out-of-domain datasets, it performs best among open-license models and matches or sometimes exceeds larger proprietary models.

6 Conclusion

  • GRANITE-20B-FUNCTIONCALLING achieves the best performance among models with open licenses across multiple out-of-domain datasets.It also shows on-par or better performance than multiple proprietary models with much larger sizes.Source: §6 · p. 14
  • The model is instruction-tuned with multi-task data covering granular tasks such as function detection, parameter detection, sequencing, and next-best-function prediction.Source: §6 · p. 14
  • GRANITE-20B-FUNCTIONCALLING is released as a capable open function-calling model under the Apache 2 license.Source: §6 · p. 14

Related papers

Loading 2407.00121v1…