PaperlayerView exact revision on arXiv

Source-linked AI summary

Hammer: Robust Function-Calling for On-Device Language Models via Function Masking

Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, Jun Wang, Weinan Zhang

arXiv:2410.04587v2cs.LGcs.AIcs.SE

TL;DR

Function-calling models can be misled by function and parameter naming conventions, causing performance to vary across benchmarks. Hammer tackles this with irrelevance-augmented data and function masking for lightweight on-device models, achieving robust generalization and strong benchmark performance. The paper reports that Hammer-7B outperforms many larger open models and competes with GPT-4 and GPT-4o on BFCL v2.

  • Problem

    Existing function-calling models show inconsistent performance across benchmarks, with naming conventions contributing to poor generalization across task environments.

    Source: §1 · p. 2

  • Method

    Hammer combines a 7,500-instance irrelevance-augmented dataset with function masking that shifts model attention from names toward descriptions.

    Source: §1 · p. 2, §4.1 · p. 6

  • Results

    Hammer demonstrates robust generalization across benchmarks; Hammer-7B outperforms many larger open-source models and competes with GPT-4 and GPT-4o on BFCL v2.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Hammer provides lightweight models and open-source training resources aimed at robust on-device function calling across diverse benchmarks.

    Source: §1 · p. 2

Abstract

from arXiv · show

Large language models have demonstrated impressive value in performing as autonomous agents when equipped with external tools and API calls. Nonetheless, effectively harnessing their potential for executing complex tasks crucially relies on enhancements in their function calling capabilities. This paper identifies a critical gap in existing function calling models, where performance varies significantly across benchmarks, often due to being misled by specific naming conventions. To address such an issue, we introduce Hammer, a novel family of foundation models specifically engineered for on-device function calling. Hammer employs an augmented dataset that enhances models' sensitivity to irrelevant functions and incorporates function masking techniques to minimize misleading. Our empirical evaluations reveal that Hammer not only outperforms larger models but also demonstrates robust generalization across diverse benchmarks, achieving sota results. Our open source contributions include a specialized dataset for irrelevance detection, a tuning framework for enhanced generalization, and the Hammer models, establishing a new standard for function calling performance.

1 Introduction

Hammer addresses inconsistent function-calling performance across benchmarks by reducing reliance on misleading names and strengthening irrelevance detection. The lightweight models, dataset, and tuning framework target robust generalization for on-device use.

Motivation

  • Existing function-calling models can vary substantially across benchmarks, motivating research into robustness and generalization.Table 1 illustrates this inconsistency, including xLAM-7B-fc’s strong results on most benchmarks but lowest average after declines on two.Source: §1 · p. 2, Table 1 · p. 2
  • Misleading function and parameter naming conventions contribute to benchmark-dependent performance because models favor naming styles resembling their training data.This can cause models to generalize poorly when evaluation benchmarks use different naming conventions.Source: §1 · p. 2

Approach

  • Hammer combines function masking with a 7,500-instance irrelevance-augmented dataset to improve description-based function selection and intent matching.Function masking shifts attention away from names toward function and parameter descriptions.Source: §1 · p. 2

Results

  • Hammer demonstrates robust generalization across diverse benchmarks and outperforms many larger open-source models while competing with GPT-4 and GPT-4o on BFCL v2.The evaluations include API-Bank, Tool-Alpaca, Seal-Tools, and Nexus Raven API Evaluation.Source: §1 · p. 2, Table 2 · p. 3, Table 3 · p. 3

Contributions

  • The work releases a function-masking tuning framework and a specialized irrelevance-detection dataset in addition to the Hammer models.The framework and dataset are intended to support robust generalization and awareness of irrelevant candidate functions.Source: §1 · p. 2

Approach

  • Hammer is a family of lightweight models fine-tuned for on-device function-calling tasks.The contribution includes models such as Hammer-7B, alongside the associated training resources.Source: §1 · p. 2

2 Related Works

Related work develops function-calling agents, evaluation datasets, benchmarks, and tuning methods. These efforts span autonomous tool use, diverse API tasks, and techniques for improving selection, orchestration, and deployment efficiency.

LLMs as Agents for Function-Calling

  • Prior work studies LLMs as autonomous agents that perform complex tasks through function calling and tool usage [18] [19].Models and systems such as Granite-20B-FunctionCalling [4], APIGen [12], and ToolACE [20] report strong benchmark performance.Source: §2 · p. 3
  • Function-calling research also examines parallel calls, process vulnerabilities, and benchmarks for diverse function-use abilities [22].These directions broaden evaluation and system capabilities beyond basic single-call execution.Source: §2 · p. 3

Datasets and Benchmarks for Function-Calling

  • Datasets and benchmarks evaluate API/tool detection, tool retrieval, planning, sequencing, slot filling, and API calling in real-world scenarios involving tool-augmented LLMs.Examples include BLEND [23], API-Bank [6] with 2,138 distinct APIs, 1,888 dialogues, and 4,149 API calls, APIGen [12], and Seal-Tools [7].Source: §2 · p. 4

Tuning Techniques for Function-Calling Models

  • Tuning approaches include multi-task learning, LoRA, negative samples, retrieval-augmented examples, DAG comparison, supervised fine-tuning, and direct preference optimization [4] [8] [24] [25] [26].These methods target function selection, orchestration accuracy, and deployment of smaller models.Source: §2 · p. 4

3 Problem Statement and Analysis

Function-calling models can generalize poorly because compact function and parameter names mislead selection and argument generation across contexts. Masking these names exposes this overreliance, while Hammer shows a smaller performance drop under arbitrary naming.

3 Problem Statement and Analysis

  • Function-calling requires selecting the appropriate candidate and supplying correct arguments, or returning an empty list when no candidate matches the user’s intent.Source: §3 · p. 4

3.1 Misleadingness by Function Name and Parameter Name

  • Compact function and parameter names can be ambiguous, causing models to infer incorrect functionality or parameter usage when descriptions and contexts differ.Source: §3.1 · pp. 4–5
  • Naming-style changes such as CamelCase versus snake_case can reduce lightweight models’ confidence and robustness across testing environments.Source: §3.1 · p. 5

3.2 The Impact of Excessive Focus on the Naming

  • Masking function and parameter names caused xLAM-1B-fc’s performance to drop significantly on Seal-Tools despite descriptions retaining the necessary information.Source: §3.2 · p. 5, Figure 2 · p. 5
  • Under the same obfuscation setting, Hammer experienced a much smaller performance drop, indicating greater reliance on function descriptions than on compact names.Source: §3.2 · p. 5

4 Methodology

Hammer’s methodology reduces misleading name-based cues during training and adds irrelevance examples so models learn descriptions and abstain when no candidate matches. The framework combines function masking with a 7,500-example irrelevance-augmented dataset.

4.1 Function Masking

  • The masking framework directs attention toward descriptions, which provide more detailed and flexible information for generalization across naming conventions.Source: §4.1 · p. 6
  • Function masking replaces candidate function and parameter names with random strings, randomizes default values into descriptions, and updates labels to match the masked candidates.Source: §4.1 · p. 6, Figure 3 · p. 6

4.2 Irrelevance-Augmented Dataset

  • Fine-tuning on function-selection data can improve function execution while impairing irrelevance detection, causing inappropriate calls when no valid option exists.Source: §4.2 · p. 7
  • The irrelevance-augmented dataset adds 7,500 examples whose correct functions are excluded and labels replaced with empty lists, training models to abstain when candidates are unsuitable.Source: §4.2 · p. 7

5 Evaluation

Hammer is evaluated on diverse out-of-domain function-calling benchmarks using syntactic, executable, and API-match metrics. It shows strong cross-benchmark performance, task-complexity advantages, transfer across base models, and trade-offs governed by masking and augmentation ratios.

5.1 Experimental Setup

  • BFCL evaluates AST syntax and executable function correctness, while other benchmarks use F1 scores for exact API-name and parameter matches across diverse task settings.Source: §5.1 · p. 7

5.2 Overall Performance on Various Benchmarks

  • Hammer-7B ranks second overall on BFCL behind GPT-4, while the Hammer series maintains remarkably stable performance across benchmarks unrelated to xlam-function-calling-60k.Source: §5.2 · p. 7

5.3 Detailed Performance on Different Types of Function Calling

  • Hammer-7B surpasses GPT-4 on Executable Function Evaluation and achieves state-of-the-art results for both metrics on the complex Parallel Multiple task.Source: §5.3 · p. 8, Table 4 · p. 8

5.4 Ablation on Different Base Models

  • Fine-tuned Hammer variants improve over vanilla Qwen and Deepseek-Coder models, and deepseek-coder-Hammer significantly surpasses xLAM despite sharing its Deepseek-Coder-Instruct base.Source: §5.4 · pp. 8–9, Table 5 · p. 9

5.5 Ablation on Different Masking Ratio

  • Larger masking ratios can slow same-task learning on Seal-Tools but improve cross-scenario generalization on API-Bank by reducing overfitting to training data.Source: §5.5 · pp. 9–10, Figure 5 · p. 9

5.6 Ablation on Different Proportions of Irrelevance-Augmented Data

  • Irrelevance detection and function-calling performance vary inversely with augmentation proportion, while approximately 10% irrelevance-augmented data gives optimal overall performance in these settings.Source: §5.6 · p. 10, Figure 6 · p. 10

6 Conclusion

The paper identifies inconsistent function-calling performance across benchmarks and presents Hammer as a robust solution using augmented data and function masking.

  • Hammer addresses benchmark inconsistency by combining an augmented dataset with function masking to improve generalization across diverse benchmarks.The conclusion frames this approach as a practical solution for real-world function-calling applications.Source: §6 · p. 10

A Extra Experimental Results

This appendix provides the full BFCL leaderboard evaluation as of September 20, 2024.

  • Table 6 reports the full BFCL leaderboard evaluation as of 09/20/2024.Source: Table 6 · p. 14

B Different Types of Function-calling Tasks

The paper distinguishes function-calling tasks by how many APIs or calls are available and whether the request is answerable. It also presents AST evaluation on BFCL and illustrates these styles in Figure 7.

Different Types of Function-calling Tasks

C Example Input to Models with Function Masking

The masking examples show models receiving tool definitions, task instructions, and a weather query, then producing JSON-formatted function calls with selected arguments.

Example Input to Models with Function Masking

Related papers

Loading 2410.04587v2…