PaperlayerView exact revision on arXiv

Source-linked AI summary

ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario

Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, Jie Tang

arXiv:2501.10132v1cs.CL

TL;DR

Complex function calling remains difficult to evaluate in realistic settings because it combines multi-step reasoning, constraints, implicit parameters, and long contexts. The paper introduces ComplexFuncBench and ComplexEval to benchmark and assess these capabilities, finding substantial weaknesses and parameter-value errors across current models.

  • Problem

    Evaluation of real-world complex function calling remains underexplored despite the need to assess parameter correctness and intermediate calling behavior.

  • Method

    The paper builds a 1,000-sample benchmark across five realistic scenarios and introduces ComplexEval with multidimensional matching for complex function calls.

  • Results

    Current state-of-the-art function-calling models show deficiencies, with parameter-value errors accounting for a significant portion of failures and domain-specific weaknesses appearing across models.

  • Takeaways & Limitations

    ComplexFuncBench provides a realistic basis for analyzing and improving foundation-model function-calling capabilities.

Abstract

from arXiv · show

Enhancing large language models (LLMs) with real-time APIs can help generate more accurate and up-to-date responses. However, evaluating the function calling abilities of LLMs in real-world scenarios remains under-explored due to the complexity of data collection and evaluation. In this work, we introduce ComplexFuncBench, a benchmark for complex function calling across five real-world scenarios. Compared to existing benchmarks, ComplexFuncBench encompasses multi-step and constrained function calling, which requires long-parameter filing, parameter value reasoning, and 128k long context. Additionally, we propose an automatic framework, ComplexEval, for quantitatively evaluating complex function calling tasks. Through comprehensive experiments, we demonstrate the deficiencies of state-of-the-art LLMs in function calling and suggest future directions for optimizing these capabilities. The data and code are available at \url{https://github.com/THUDM/ComplexFuncBench}.

1 Introduction

ComplexFuncBench targets underexplored real-world function calling, where models must reason over multi-step constraints, implicit parameters, and long contexts. It combines a benchmark with ComplexEval to expose current models’ weaknesses and support future optimization.

  • Real-world function calling can improve LLM responses with more accurate and up-to-date information by integrating external tools and APIs.
  • Existing evaluations incompletely assess complex calling because they overlook parameter correctness, intermediate execution steps, or scenarios beyond simple rule-based matching.
  • Complex function calling involves multi-step calls, user constraints, implicit parameter reasoning, long parameter values, and 128k-token contexts.
  • Experiments find that parameter-value errors form a significant share of failures, while the analysis is intended to guide future function-calling optimization.
  • ComplexFuncBench contains 1,000 samples of multi-step, constrained function calling requiring long-context reasoning, while ComplexEval provides multidimensional automatic evaluation.

2 ComplexFuncBench

ComplexFuncBench is constructed from real-time Booking.com APIs through staged generation, human correction, disambiguation, and generalization. The resulting benchmark contains 1,000 carefully checked samples spanning five domains and single- and cross-domain tasks.

  • The candidate function set contains 43 corrected real-time Booking.com APIs across Hotel, Flight, Attraction, Car Rental, and Taxi domains.
  • Data collection uses coarse generation, fine-grained annotation, and generalization to reduce human-annotation cost while preserving complex calling paths.
  • Fine-grained Annotation: Human correction rewrites queries, arranges the shortest complete function-call path, and fixes parameter errors before evaluation.
  • Fine-grained Annotation: Disambiguation removes overlapping functions and ambiguous API responses so each sample has an unambiguous function and parameter sequence.
  • Generalization: Quality checks produced 100 error-free template paths, which were expanded tenfold into 1,000 samples with 600 single-domain and 400 cross-domain cases.

3 ComplexEval: Automatic Evaluation

ComplexEval evaluates complex function calling by iteratively checking generated calls, matching them against golden calls, and feeding API responses back into the dialogue. It combines call-level metrics with GPT-4o-based assessment of final-response completeness and correctness.

  • ComplexEval continually generates next-step function calls from the user query and available functions until the model produces a final response.
  • At each step, the evaluator compares predicted calls with a golden call list after format checking, Hungarian mapping, and multi-dimensional matching.The format check verifies function availability, required parameters, and parameter types.
  • Function-call equivalence combines exact matching, identical-response matching, and GPT-4o-based semantic matching for equivalent parameter meanings.For example, New York and NY can be treated as equivalent parameter values by LLM-based matching.
  • Correctly matched calls receive their annotated golden API responses, while invalid calls receive a general error and format failures receive a specific error message.These responses support iterative model interaction and evaluation of self-correction.
  • Success Rate measures the proportion of samples that complete the task, while Call Accuracy measures the proportion of correct function calls.Call Accuracy aggregates correct calls c_i over total calls n_i across N samples.
  • GPT-4o separately scores final responses for completeness and correctness on 0–2 scales, with averages reported across samples.Completeness concerns fulfillment of user requirements, whereas correctness concerns agreement with the API response and dialogue history.

4 Experiments

Experiments evaluate 12 long-context function-calling models on complex tasks, revealing strong differences across model families, domains, response quality, and error types. Parameter-value mistakes and inefficient or incomplete calling remain prominent weaknesses.

  • Main Results: Claude-3.5-Sonnet, GPT-4o, and GLM-4-Long achieve overall success rates of 61.0%, 60.5%, and 57.1%, respectively.Qwen2.5-72B leads open-source models at 40.1%, while GLM-4-9B reaches 8.4%.
  • Main Results: Model strengths vary by domain: GPT-4o reaches 70% in Hotel and 82% in Attraction, Claude-3.5-Sonnet handles cross-domain calls better, and GLM-4-Long excels with long parameters.Flights and Car Rental tasks contain parameter values approaching 600 characters.
  • Response Evaluation: Claude-3.5-Sonnet leads closed-source response evaluation with completeness and correctness scores of 1.84 and 1.85, respectively.GPT-4o has the lowest completeness score among closed-source models at 1.66, while Qwen2.5-72B leads open-source response quality.
  • Error Analysis: Value errors dominate model failures, reaching 78.8% for Qwen2.5-72B, while early stopping reaches 19.7% for Claude-3.5-Sonnet and 21.0% for GPT-4o.GLM-4-Long has an 11.0% missing-parameter rate; function errors also reflect weak understanding of function documentation.
  • Parameter Analysis: Filter and legs parameters produce relatively high error rates because they require order inference or combining information from user queries and API responses.GLM-4-Long fails more than 40% on filter, while Qwen2.5-72B fails 38.1% on legs.
  • Calling Steps Analysis: Predicted calling sequences often exceed the shortest paths, although GLM-4-Long averages only 0.13 extra steps.Claude-3.5-Sonnet and Qwen2.5-72B average 4.47 and 4.28 extra steps, respectively.

5 Related Work

Prior work evaluates tool planning, execution outcomes, or function-call precision, but these approaches incompletely assess parameter correctness and intermediate multi-step behavior.

  • Function Calling Evaluation: Prompt-based methods assess tool planning but do not evaluate whether calling parameters are correct.Other approaches evaluate only the final execution state and ignore the correctness of intermediate calls.
  • Function Calling Evaluation: Rule-based matching methods calculate function-call precision but remain limited to simple scenarios and cannot fully address complex calls.These limitations motivate more comprehensive evaluation of complex function calling.

6 Conclusion

The paper introduces ComplexFuncBench and ComplexEval to evaluate complex function calling across realistic, multi-step settings. Experiments expose deficiencies in current models and provide a benchmark for future development.

  • Contributions: ComplexFuncBench contains 1,000 samples from five realistic scenarios, while ComplexEval automatically evaluates complex function calling.The benchmark includes constrained multi-step calls, parameter-value reasoning, long-context extraction, and long parameter filling.
  • Conclusion: The authors report that comprehensive experiments demonstrate deficiencies in current state-of-the-art function-calling models.They position ComplexFuncBench as a resource for guiding future development of foundation-model function-calling capabilities.

A Data Annotation

The cited passage only indicates that Table 5 contains a detailed annotation example.

  • Data Annotation: Table 5 provides a detailed example of the annotation process.

A.2 Data Statistics

This section identifies the benchmark’s data-statistics table and the prompts used for query generation and generalization.

  • Table 3 presents the data statistics for ComplexFuncBench.
  • Figure 8 contains the prompt for query generation.
  • Figure 9 contains the prompt for query generalization.

B.1 Experiment Setting

The experiments use identical generation settings across models, including greedy sampling, a 2048-token maximum, and automatic tool choice for closed-source models.

  • All models use the same generation parameters for fair comparison.Generation uses greedy sampling, with the maximum generation length set to 2048 tokens.
  • Closed-source models use tool_choice = “auto” so the model selects the tool.

B.2 Golden Function Call Updating

Golden function call updating evaluates predicted calls against annotated calls and extends the golden path when a predicted call matches under the multi-dimensional criteria. The surrounding prompts define query generation, parameter variation, call equivalence, and evaluation procedures.

  • Golden Function Call Updating: At step t, the model predicts a list of n function calls, while the golden list contains m annotated calls.
  • Golden Function Call Updating: Each predicted call is evaluated for correctness against the corresponding golden function call list.
  • Golden Function Call Updating: When a predicted call matches under multi-dimensional matching, its matched golden API response is assigned to the prediction and the next-step golden calls are appended.Figure 7 illustrates this golden-list update for t = 0.
  • Prompt Design: The evaluation prompts cover LLM-based matching, completeness evaluation, and correctness evaluation.
  • Prompt Design: Table 4 gives examples of parameter types, while Table 5 notes that actual API responses can reach a context length of 128k.
  • Prompt Design: The query-generation prompt asks an assistant to propose real-world tasks and solve them with related APIs, recording the query, API list, and reasoning.
  • Prompt Design: Query generalization replaces specified quantities, dates, distances, coordinates, and related values within prescribed ranges or categories.
  • Prompt Design: The function-call comparison prompt treats calls as equivalent when parameter values have different but semantically matching expressions.Examples include equivalent expressions across languages and formats.
Loading 2501.10132v1…