PaperlayerView exact revision on arXiv

Source-linked AI summary

ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario

Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, Jie Tang

arXiv:2501.10132v1cs.CL

TL;DR

Evaluating complex function calling in LLMs remains challenging. ComplexFuncBench and ComplexEval provide a benchmark and automatic evaluation framework, and experiments find parameter-value errors and model-specific weaknesses across scenarios.

  • Problem

    Evaluating LLM function calling, especially complex calls, remains challenging, despite APIs offering more accurate and up-to-date outputs.

    Source: §1 · p. 1

  • Method

    ComplexFuncBench provides 1,000 samples of multi-step, constrained function calling across five scenarios, alongside ComplexEval, an automatic multidimensional-matching evaluation framework.

    Source: §1 · p. 2, §6 · p. 8

  • Results

    Experiments find that a significant portion of errors are parameter-value errors, while models show scenario-specific weaknesses.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The detailed experimental analysis is intended to guide future optimization of function-calling capabilities.

    Source: §1 · p. 2

Abstract

from arXiv · show

Enhancing large language models (LLMs) with real-time APIs can help generate more accurate and up-to-date responses. However, evaluating the function calling abilities of LLMs in real-world scenarios remains under-explored due to the complexity of data collection and evaluation. In this work, we introduce ComplexFuncBench, a benchmark for complex function calling across five real-world scenarios. Compared to existing benchmarks, ComplexFuncBench encompasses multi-step and constrained function calling, which requires long-parameter filing, parameter value reasoning, and 128k long context. Additionally, we propose an automatic framework, ComplexEval, for quantitatively evaluating complex function calling tasks. Through comprehensive experiments, we demonstrate the deficiencies of state-of-the-art LLMs in function calling and suggest future directions for optimizing these capabilities. The data and code are available at \url{https://github.com/THUDM/ComplexFuncBench}.

1 Introduction

ComplexFuncBench targets function calling that requires multi-step planning, constraints, implicit parameter reasoning, long parameter values, and 128k context. The paper introduces a 1,000-sample benchmark and ComplexEval, an automatic framework for evaluating complex calls.

  • Existing evaluations miss key aspects of complex calls: prompt-based tests omit parameter correctness, execution-state tests ignore intermediate steps, and rule-based matching suits only simple scenarios.Complex-call evaluation also faces costly annotation, ambiguous valid paths from API responses, and parameter variations that defeat exact matching.Source: §1 · pp. 1–2
  • ComplexFuncBench is a 1,000-sample benchmark of complex function calling, manually annotated across five real-world domains, with multi-step calls, user constraints, implicit parameter reasoning, long values, and 128k context.Functions were collected from Booking.com; GPT-4o generated 1,000 preliminary queries and function-calling paths, then senior annotators selected and labeled 100 complex samples as templates.Source: §1 · p. 2
  • The three-stage annotation process creates a shortest valid call path, removes API-response ambiguity, and expands 100 manually labeled templates into 1,000 samples.The initial GPT-4o-generated queries and paths reduce human annotation costs.Source: §1 · p. 2
  • ComplexEval combines rule-, response-, and LLM-based equivalence matching, uses API errors to assess self-correction, and evaluates final-response completeness and correctness.Its multidimensional matching is designed to handle equivalent calls with differently expressed parameter values.Source: §1 · p. 2
  • A significant portion of errors stem from parameter values, and models show specific weaknesses in particular scenarios.The authors say their analysis is intended to guide future optimization of function calling capabilities.Source: §1 · p. 2

2 ComplexFuncBench

ComplexFuncBench is built from manually curated real-time APIs and human-annotated function-calling paths, then expanded into a 1,000-sample benchmark spanning five domains and multi-step calls.

2.1 Data Collection

  • The candidate function set contains 43 real-time Booking.com APIs across five domains, with manually corrected and annotated descriptions to address incomplete or incorrect website descriptions.The domains are Hotel, Flight, Attraction, Car Rental, and Taxi.Source: §2.1 · p. 3

2.1.1 Stage 1: Coarse Generation

  • GPT-4o generated and automatically annotated 1,000 preliminary samples, but errors in parameter values and call order required human refinement.The preliminary examples were generated from randomly sampled functions and queries spanning different domains.Source: §2.1.1 · p. 3

2.1.2 Stage 2: Fine-grained Annotation

  • Annotators selected 100 samples with relatively complete paths, rewrote queries with details, corrected calls and parameters, and labeled the shortest valid call path.Corrections included adding missing or removing unnecessary calls and fixing parameter type, value, hallucination, and omission errors.Source: §2.1.2 · pp. 3–4
  • Annotators removed overlapping functions and ambiguous API responses, quality-checked all 100 samples until error-free, and retained single- and cross-domain paths.For subsequent calls, they retained only one relevant location ID when an API response returned multiple IDs for the same destination.Source: §2.1.2 · p. 4

2.1.3 Stage 3: Generalization

  • Generalization expanded each of the 100 templates to 10 samples sharing its call path but varying parameter values, with junior annotators filling parameters and removing ambiguous responses.The expansion increased the dataset from 100 to 1,000 samples and required less annotation effort than fine-grained annotation.Source: §2.1.3 · p. 4

2.2 Data Composition

  • The 1,000 samples comprise 600 single-domain cases (150 each for Hotels, Flights, Car Rental, and Attraction) and 400 cross-domain cases; averages are 3.26 steps and 5.07 calls per sample.Taxi, which has only two functions, is used only in cross-domain samples.Source: §2.2 · p. 4

3 ComplexEval: Automatic Evaluation

ComplexEval quantitatively evaluates models’ complex function-calling ability and response generation. It assesses calls against gold calls and separately scores generated responses for completeness and correctness.

3.1 Function Calling Evaluation

  • Given a query and available functions, the model generates calls until its final response; format checks validate function availability, required parameters, and parameter types, returning specific errors that enable self-correction.Source: §3.1 · pp. 4–5
  • Correctly formatted calls are embedded and Hungarian-mapped, then matched by exact call, identical API results, or GPT-4o judgments of semantic equivalence, such as “New York” and “NY” (Xiao et al., 2023).Function names and parameters are concatenated for embedding with bgelarge-en-v1.5; cosine similarity is used to map predicted calls to gold calls.Source: §3.1 · p. 5
  • Success Rate measures the proportion of samples that complete the task, while Call Accuracy measures the proportion of correct function calls.Call Accuracy is calculated using the number of samples, total calls per sample, and correct calls per sample.Source: §3.1 · pp. 5–6

3.2 Model Response Evaluation

  • GPT-4o scores response completeness and correctness from 0 to 2 and averages each score across samples.Completeness measures how fully a response addresses user requirements; correctness measures answer accuracy and alignment with API responses.Source: §3.2 · p. 6

4 Experiments

Experiments evaluate 12 function-calling models with 128k context and analyze their call success, response quality, error patterns, and calling steps. Across these tests, models show difficulty with parameter-value reasoning and long-context extraction.

4.1 Settings

  • The evaluation covered 12 models from six institutions, all selected for 128k context length, including closed- and open-source models.The set included GPT-4o, GPT-4-Turbo, Claude 3.5 Sonnet and Haiku, GLM-4-Long, Llama-3.1 models from 8B to 405B, Qwen2.5 7B and 72B, and Mistral Large 2.Source: §4.1 · p. 6

4.2 Main Results

  • GPT-4o achieved 70% and 82% success rates in Hotel and Attraction, while Claude-3.5-Sonnet handled cross-domain calls better and GLM-4-Long excelled on long Flight and Car Rental parameters.Hotel and Attraction require inferring call order from function descriptions; Flight and Car Rental parameters approach 600 characters.Source: §4.2 · p. 6
  • Claude-3.5-Sonnet led closed-source response quality with completeness and correctness scores of 1.84 and 1.85, while Qwen2.5-72B led open-source models at 1.80 and 1.75.Despite GPT-4o’s high call success rate, it had the lowest completeness score among closed-source models at 1.66.Source: §4.2 · p. 6
  • 61.0%, 60.5%, and 57.1% were the overall success rates for Claude-3.5-Sonnet, GPT-4o, and GLM-4-Long; Qwen2.5-72B led open-source models at 40.1%.Llama3.1 models showed disordered call sequences and severe parameter hallucinations; GLM-4-9B, the best model under 10B parameters, achieved 8.4%.Source: §4.2 · p. 6, Table 2 · p. 7

4.3.1 Error Analysis

  • Value errors were common across the four analyzed models, reaching 78.8% for Qwen2.5-72B; early stopping reached 19.7% for Claude-3.5-Sonnet and 21.0% for GPT-4o.Errors were classified as func_error (wrong or invalid function), param_missing (omitted crucial parameters), hallucination (random parameters), value_error (incorrect parameter values), and stop_early (ending before all required calls). GLM-4-Long had an 11.0% param_missing rate. The analysis focused on the top three closed-source models and leading open-source model.Source: §4.3.1 · pp. 6–7, §4.3 · p. 6, Figure 4 · p. 7
  • Filter and legs parameters had relatively high error rates: GLM-4-Long failed on filter more than 40%, and Qwen2.5-72B failed on legs 38.1%.Claude-3.5-Sonnet retrieved the wrong token parameter on constrained queries, while it and GPT-4o often retrieved the wrong slug from API responses; filter errors involve inferring call order, and legs require gathering information from the query and API response.Source: §4.3.1 · p. 7, Figure 5 · p. 8

4.3.2 Function Calling Steps Analysis

  • Claude-3.5-Sonnet and Qwen2.5-72B averaged 4.47 and 4.28 steps beyond the shortest calling steps, while GLM-4-Long averaged only 0.13 extra steps.Models often exceeded the annotated shortest steps by 1–3, and more complex tasks could require over 10 steps; GPT-4o also tended to complete tasks with fewer steps.Source: §4.3.2 · pp. 7–8, Figure 6 · p. 8

5 Related Work

Prior work equips LLMs with function-calling capabilities through native support or specialized fine-tuning, while evaluation methods leave gaps in parameter correctness and complex-call assessment.

Function Calling with LLMs

Function Calling Evaluation

  • LLM-based tool-planning evaluations produce natural-language thoughts rather than formatted calls, so they cannot assess parameter correctness (Qiao et al., 2024; Nexusflow.ai, 2023).Source: §5 · p. 8
  • Rule-based matching methods, including AST analysis, assess call precision but cover only single-function calls (Wang et al., 2024; Chen et al., 2024b).Source: §5 · p. 8
  • State-based evaluations assess final system states without intermediate call procedures, making results random; this work introduces a benchmark for more complex function-calling settings (Qin et al., 2023; Lu et al., 2024).Source: §5 · p. 8

6 Conclusion

ComplexFuncBench introduces a 1,000-sample benchmark spanning five realistic scenarios, alongside ComplexEval, an automatic framework for evaluating complex function calling. The benchmark targets multi-step and constrained calling, and experiments show deficiencies in current state-of-the-art models.

  • ComplexFuncBench contains 1,000 samples across five realistic scenarios for complex function calling.Source: §6 · p. 8
  • The benchmark tests multi-step and constrained calling, including parameter value reasoning, long-context parameter extraction, and long parameter value filling.Source: §6 · p. 8
  • ComplexEval is an automatic framework for evaluating complex function calling, and experiments reveal deficiencies in current state-of-the-art function calling models.Source: §6 · p. 8

A Data Annotation · A.1 An Example of Annotation Process

The annotation section provides a detailed example of the annotation process in Table 5.

A.1 An Example of Annotation Process

  • Table 5 gives a detailed example of the annotation process.Source: §A.1 · p. 10

A.2 Data Statistics · A.3 Prompt for Query Generation. · A.4 Prompt for Query Generalization

These appendix subsections identify a data-statistics table and figures containing the query-generation and query-generalization prompts.

A.2 Data Statistics

  • Table 3 presents the data statistics for ComplexFuncBench.The supplied passage identifies the table but does not provide its values.Source: §A.2 · p. 10, Table 3 · p. 10

A.3 Prompt for Query Generation.

A.4 Prompt for Query Generalization

  • Figure 9 shows the prompt for query generalization at the generalization stage.Source: §A.4 · p. 10

B Automatic Evaluation · B.1 Experiment Setting

The experiment standardizes generation settings across models, using greedy sampling and a 2048-token maximum; close-source models are allowed to choose tools automatically.

B.1 Experiment Setting

  • All models use the same generation parameters for a fair comparison of complex function calling.Source: §B.1 · p. 10
  • Generation uses greedy sampling, with the maximum generation length set to 2048 tokens.Source: §B.1 · p. 10
  • For close-source models, tool_choice is set to “auto” so the model decides which tool to choose.Source: §B.1 · p. 10

B.2 Golden Function Call Updating … Prompt for Completeness Evaluation

These sections describe ComplexFuncBench’s golden-call updating, query construction and generalization, and prompts and examples for function-call evaluation. They also specify constraints on generated and generalized queries and document examples of parameter types and long API responses.

B.2 Golden Function Call Updating

  • At each step, ComplexEval compares predicted and golden call lists, uses matched calls’ annotated API responses, and adds the next-step golden calls.The appendix illustrates updating with an annotated shortest function-call path spanning three steps.Source: §B.2 · p. 10, Figure 7 · p. 10

B.3 Prompt for LLM-based Match

  • The B.3 subsection identifies Figure 10 as the prompt for LLM-based matching.The passage directs readers to the figure for the prompt.Source: §B.3 · p. 10, Figure 10 · p. 14

B.4 Prompt for Model Response Evaluation

C Parameter Type Examples

Prompt for Query Generation

  • The query-generation prompt asks for real-world tasks using diverse API combinations, without naming the API, and permits repeated or parallel calls while limiting number arguments to fewer than five.The prompt also supplies API usage examples.Source: Prompt for Query Generation · p. 12, Figure 8 · p. 12

Prompt for Query Generalization

  • The generalization prompt allows specified quantities, dates, distances, times, locations, and enum values to change while preserving hotel filters and sorting, review sorting, car-rental components, child policies, user reviews, and payment methods.Replacement rules include dates from November 1 to December 30, 2024, and distances from 10 to 50 kilometers in multiples of 10.Source: Prompt for Query Generalization · p. 13, Figure 9 · p. 13

Prompt for LLM-based Match

  • The appendix includes a dedicated prompt for LLM-based matching in Figure 10.The figure caption identifies its contents as the prompt for LLM-based match.Source: Figure 10 · p. 14

Prompt for Completeness Evaluation

  • The annotation example notes that its displayed API response is abbreviated, while the actual response can reach a context length of 128k.The example is presented in Table 5.Source: Table 5 · p. 17

Related papers

Loading 2501.10132v1…