Source-linked AI summary
HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios
Jun Wang, Jiamu Zhou, Muning Wen, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Cheng Jin, Xihuai Wang, Weinan Zhang, Qiuying Peng, Jun Wang
TL;DR
Multi-turn function-calling evaluation is challenging because user behavior and intent can vary across interactions. HammerBench builds a fine-grained benchmark from mobile-app tasks and anonymized user logs, covering diverse interaction patterns and conversational snapshots. Its evaluations highlight challenges with argument shifts, external information, and parameter-name errors.
Problem
Complex, variable user behavior makes it challenging to evaluate LLM function calling in multi-turn interactions.
Source: §1 · p. 1Method
Results
Experiments highlight challenges in tracking argument shifts and handling external information, while Progress Rate and PHR/PMR reveal areas for improvement in hallucination resilience and evolving-intent handling.
Source: §6 · pp. 8–9Takeaways & Limitations
The benchmark’s fine-grained metrics expose model issues such as hallucinations and parameter-name errors, offering insights for optimizing robust function-calling LLMs in mobile-assistant scenarios.
Source: §1 · p. 2Because HammerBench uses fixed, pre-generated evaluation trajectories, it covers a less comprehensive range of trajectories than dynamic generation and does not assess model self-exploration.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Evaluating the performance of LLMs in multi-turn human-agent interactions presents significant challenges, particularly due to the complexity and variability of user behavior. In this paper, we introduce HammerBench, a novel benchmark framework for assessing LLMs' function-calling capabilities in real-world, multi-turn dialogues. HammerBench simulates diverse mobile assistant use cases, incorporating imperfect instructions, dynamic question-answer trajectories, intent and argument shifts, and the indirect use of external information through pronouns. To construct this benchmark, we curate a comprehensive dataset derived from popular mobile app functionalities and anonymized user logs, complemented by a cost-effective data generation pipeline leveraging open-source models. HammerBench is further augmented with fine-grained interaction snapshots and metrics, enabling detailed evaluation of function-calling performance across individual conversational turns. We demonstrate the effectiveness of HammerBench by evaluating several leading LLMs and uncovering key performance trends. Our experiments reveal that different types of parameter name errors are a significant source of failure across different interaction scenarios, highlighting critical areas for further improvement in LLM robustness for mobile assistant applications.
1 Introduction
HammerBench addresses the difficulty of evaluating function calling in complex, variable multi-turn interactions with a fine-grained benchmark built around mobile-app tasks and real-user logs. Its evaluation highlights challenges in argument shifts and external information, as well as parameter-name errors and hallucinations.
1 Introduction
- Multi-turn tasks such as ticket purchasing require agents to gather context across interactions, while uncertainty in user intent and behavior makes correct function calls difficult.Source: §1 · p. 1
- Existing benchmarks struggle to evaluate the intertwined dynamics of conversations and function calls, and the paper identifies a lack of fine-grained benchmarks based on real-world scenarios.Source: §1 · p. 1
- HammerBench is a fine-grained, multi-turn function-calling benchmark curated from popular mobile-app functions and queries derived from anonymized user logs.Source: §1 · p. 2
- The benchmark covers imperfect instructions, varied question-answer trajectories, intent and argument shifts, and indirect references or vague descriptions, with fine-grained metrics for conversational snapshots.Its metrics include Function Name Accuracy, Parameter Name Hallucination and Missing Rate, Progress Rate, and Success Rate.Source: §1 · p. 2
- Evaluation of 10 LLMs highlights challenges in tracking argument shifts and handling external information, while fine-grained metrics expose hallucinations and parameter-name errors.The authors present these findings as insights for optimizing robust function-calling LLMs in mobile-assistant scenarios.Source: §1 · p. 2
2 Related Works
Prior dialogue evaluation decomposes open-ended conversations into sub-tasks, while function-calling benchmarks span single-turn and multi-turn settings. The paper positions HammerBench as a targeted evaluation system for fine-grained multi-turn function-calling challenges.
2.1 Dialogue System Evaluation
- Dialogue evaluation moved from topic classifiers to LLM-based assessment, with MT-Bench and MT-Eval decomposing multi-turn dialogue into dimensions or sub-tasks.Function-calling dialogue adds the challenge of evaluating dynamics beyond the function call itself.Source: §2.1 · p. 2
2.2 Function Calling Benchmark
- Single-turn function-calling benchmarks use LLM-generated, manually labeled, or self-instruct data, but struggle to capture real-world interaction diversity and assess function-calling capabilities fully.Source: §2.2 · pp. 2–3
- Existing multi-turn benchmarks use curated data or evaluate interactive tools and broader multi-turn capabilities, but gaps remain, especially for argument shifts.Source: §2.2 · p. 3
- General agent-evaluation systems do not specifically target multi-turn function-calling challenges; HammerBench focuses on fine-grained argument-shift and external-information evaluation.Source: §2.2 · p. 3
3 Principles of HammerBench
HammerBench is organized around authenticity, scenario diversity, and evaluation granularity. Its design draws on mobile-app functions and anonymized user logs, tests varied behaviors and contexts, and tracks both task-level performance and multi-turn progress.
3 Principles of HammerBench
- Authenticity is pursued by supplementing long-tail user-log scenarios, abstracting APIs into user-friendly tools, and curating functions and anonymized queries from real apps.Source: §3 · p. 3
- Scenario diversity spans tool domains, simple and complex queries, vague and detailed instructions, and single- and multi-turn interactions.The four multi-turn categories are imperfect instructions, varied question-answer trajectories, intent and argument shifts, and external individual information.Source: §3 · p. 3
- Granular metrics assess function and argument performance, missing information, progress toward task completion, and final success.The listed measures include Function Name Accuracy, Parameter Name Hallucination and Missing Rate, Argument Accuracy, Progress Rate, and Success Rate.Source: §3 · p. 4
4 HammerBench
HammerBench constructs mobile function-calling datasets through tool and API development, validation, and refinement, then extends imperfect instructions into multi-turn interaction scenarios. Its scenarios include varied question-answer trajectories and changes in intent or arguments.
4.1 Test Data Construction
- The dataset workflow comprises toolset collection, API generation, validation, and manual refinement.Figure 1 presents the construction pipeline stages.Source: §4.1 · p. 4
- APIs are generated from popular mobile-app functionalities grouped into 60 functional categories, with human oversight for simplicity, usability, and minimal redundancy.Manual review selects 1,063 APIs and prioritizes completeness and clarity.Source: §4.1 · p. 4
4.1.1 Multi-Turn Interactions with Imperfect Instructions
- The imperfect-instruction workflow checks generated function calls across zero-shot and one-shot settings using Rouge-L and semantic alignment, followed by manual inspection, yielding 3,240 instances.The query-generation workflow reflects that 76% of user-log queries contain fewer than 10 tokens.Source: §4.1.1 · pp. 4–5
4.1.2 Diverse Question-Answer Trajectories
- Four question-answer trajectories vary whether the model asks one or multiple questions and whether the user responds once or multiple times, including a case with incomplete answers.Randomly combining questions and answers produces 2,310 instances.Source: §4.1.2 · p. 5
4.1.3 Intent and Argument Shifts
- Intent-shift instances test recognition of changed goals, while argument-shift cases model slot overriding and API repurposing.The dataset includes 1,098 intent shifts, 1,462 slot-overriding instances, and 1,066 API-repurposing instances.Source: §4.1.3 · pp. 5–6
4.2 Function Calling Snapshots
Function Calling Snapshots produce a formatted call at every turn, allowing missing information to be identified separately from follow-up question generation. The section also defines fine-grained metrics for evaluating individual calls and multi-turn progress.
4.2 Function Calling Snapshots
- Snapshots generate a formatted function call at each turn, then use rule-based comparison with the function definition to identify missing information.This separates missing-data identification from question generation and supports focused analysis of missing parameters and hallucinations.Source: §4.2 · p. 6
- Function Calling Snapshots outperformed Learning to Ask by 16% in task success rate in a human evaluation of 100 OOD conversations.The comparison fine-tuned Qwen2.5-7B on both paradigms.Source: §4.2 · p. 6
4.3 Fine-Grained Evaluation Metrics
- PHR tracks incorrect parameter names, while PMR tracks missing parameters, distinguishing two kinds of parameter-prediction error.Together, they measure the accuracy and completeness of parameter predictions.Source: §4.3 · p. 7
- PR measures the proportion of correct turns, while SR measures overall function-call accuracy across all turns in a conversation.PR is defined using the number of correct turns and total turns; higher SR indicates better final performance.Source: §4.3 · p. 7
- The metrics cover individual function-call accuracy and multi-turn performance, including function-name accuracy, parameter errors, progress, and success.The framework includes Acc., Func. Acc., PHR, PMR, SR, and PR.Source: §4.3 · pp. 6–7
5 Experiment
HammerBench evaluates LLM function calling across single-turn and multi-turn tasks with 10 models. Results identify model strengths that vary by task and expose challenges in tracking changing arguments, handling external information, and predicting parameters with incomplete context.
5.1 Experimental Setups
- The experiments benchmark 10 commonly used or function-calling-specific LLMs and analyze their performance on HammerBench.The setup lists models including GPT-4o, Claude3.5-sonnet, and open-source models.Source: §5 · p. 7, §5.1 · p. 7
5.2 Results Analysis of Evaluation Types
- Qwen2.5-72B-Instruct, GPT-4o, and Claude3.5-sonnet are among the top performers, while other models show strengths or weaknesses that depend on task type.The reported overall results cover single-turn and multi-turn evaluation types.Source: §5.2 · p. 7
- LLMs with stronger single-turn irrelevance detection tend to perform better on intent shifts.Intent shifts test whether models recognize changing user intentions.Source: §5.2 · p. 7
- Models often mishandle repeated slot changes, and even GPT-4o sometimes fails to maintain updated values across turns after tool execution.Slot Overriding reveals slow adaptation by many open-source LLMs; Multiple Slot Values highlights inconsistent management of values across turns.Source: §5.2 · p. 7
- Indirect references such as pronouns and contextual information can introduce noise and lower slot-filling accuracy.The External Information task identifies this vulnerability in function calling.Source: §5.2 · p. 7
5.3 Results Analysis of Evaluation Metrics
- Initial-snapshot PHR is higher than final-snapshot PHR, indicating more parameter hallucinations with incomplete context.The pattern is reported across LLMs, and parameter-name hallucinations correlate positively with PHR.Source: Table 6 · p. 8, §5.3 · p. 8
6 Conclusion
HammerBench evaluates multi-turn function calling through turn-level snapshots and tests challenges including argument shifts and external information. The experiments identify difficulties with these interactions and with hallucinations and evolving user intents.
6 Conclusion
- HammerBench assesses imperfect instructions, varied question-answer trajectories, intent and argument shifts, and queries using external individual information.It decomposes multi-turn slot-filling interactions into function-calling snapshots and generates external-information data to evaluate pronoun and anaphora resolution.Source: §6 · p. 8
- LLMs face challenges handling argument shifts and external information, including tracking dynamic slot updates.Progress Rate and PHR/PMR reveal areas for improvement in resilience to hallucinations and evolving user intents.Source: §6 · pp. 8–9
- Progress Rate and PHR/PMR identify areas for improvement in model resilience to hallucinations and evolving user intents.The paper concludes that further advances are needed for LLM robustness in real-world applications.Source: §6 · pp. 8–9
Limitations
HammerBench evaluates multi-turn scenarios using fixed, pre-generated datasets, limiting its coverage of dynamic real-world behavior. Its synthesized data can differ from actual user queries, and exhaustive query coverage is infeasible.
Limitations
- Fixed, pre-generated evaluation datasets limit coverage of dynamic real-world behaviors and do not assess a model’s ability to self-explore.Compared with real-time generation of dynamic trajectories, HammerBench covers a less comprehensive scope of trajectories.Source: Limitations · p. 9
- Automatically synthesized scenarios can differ from actual user queries, and exhaustively capturing all possible queries is infeasible.Fixed evaluation trajectories make evaluation more convenient and controllable but cover only part of real user interactions.Source: Limitations · p. 9
A Analysis of Benchmark Dataset
HammerBench’s dataset spans single- and multi-turn mobile-assistant tasks, with tool and conversation statistics and quality checks. The examples illustrate single-turn inputs and a multi-turn interaction that draws on external information.
A Analysis of Benchmark Dataset
- The dataset has 60 functional categories, 1,063 tools, and 6,531 queries; it includes four single-turn datasets with four types each and eight multi-turn datasets across three types.Figure 3 depicts tool parameter-count and conversation turn-count distributions, as well as the multi-turn data types.Source: A Analysis of Benchmark Dataset · p. 11, Figure 3 · p. 11
- Quality checks include manual correction of failed LLM checks, random human rechecks of 100 instances, and manual review of 1,098 multi-turn instances; data availability exceeds 95%.The manual review checked that the multi-turn instances followed the logical flow of everyday conversations.Source: A Analysis of Benchmark Dataset · p. 11
B.1 Single-turn
- The single-turn examples contrast a complete request, a request with missing arguments, and a request whose plate number comes from external information.The examples show the same traffic-violation task with differing levels and sources of user-provided context.Source: §B.1 · p. 12
B.2 Multi Turn
- The multi-turn example gathers city, sharing method, recipient, plate number, and time across dialogue turns, using external information for the plate number.The interaction ends with a function call containing the accumulated arguments.Source: §B.2 · p. 12
C Examples of Model Output Errors
The examples show three distinct errors: filling unspecified parameters, supplying an unanswered parameter after a follow-up question, and treating a mentioned source as a requested sharing method.
C.1 Imperfect instruction
- For an imperfect instruction with no parameter details, the model fills parameter slots instead of leaving them unspecified.The example request asks only to find a mail, while the function call includes a keyword and max-results value.Source: §C.1 · p. 13
C.2 Diverse Q&A trajectories
- After asking for an ID number that the user does not provide, the model still fills the passenger ID slot.The user provides only the passenger’s name in the example.Source: §C.2 · p. 13
C.3 External Individual Information
- The model assigns Facebook as the sharing medium even though the user only says the time is mentioned in a Facebook message.The example’s requested taxi-ride sharing medium is not specified by the user.Source: §C.3 · p. 13
C.4 Argument shifts
This material describes argument-shift examples and prompts and procedures used to build and evaluate HammerBench’s tools, queries, and multi-turn interactions.
C.4 Argument shifts
- When users override a slot or provide a new destination, models may fail to update the value or retain the original departure parameter.The examples illustrate failures on both an explicitly corrected value and a later destination change.Source: §C.4 · p. 13
D Prompts of Function Calling
- The function-calling prompt requires JSON tool calls, preserves requested time formats, and directs the model to leave unmentioned parameters empty rather than ask follow-up questions.It also instructs the model to reject requests when no tool is suitable.Source: D Prompts of Function Calling · p. 14
E.1 Tool Generation
- Generated toolkits target realistic, diverse, cohesive, and complete tasks, with human review of generated APIs and 1,063 final selections.The toolkit instructions also specify compactness and requirements for stored data access, operation status, and resource identifiers.Source: §E.1 · p. 14
E.2 Query Generation
- Query generation selects commonly used parameters and produces natural, varied requests that include the selected parameter values.The instructions separately ask the model to choose parameters suited to daily habits.Source: §E.2 · p. 17, §E.3 · p. 18
E.4 LLM Check Prompt
- The LLM checker accepts semantically equivalent outputs but rejects mismatched parameter counts and outputs containing unsupported parameters.The prompt also treats some time-format differences as equivalent when they express the same time.Source: §E.4 · pp. 18–19
E.5 Irrelevant tools selection
- A separate relevance check asks whether a tool’s description and parameters can complete the user’s query..Source: §E.5 · p. 19
E.6 Generation with external individual information
- External-information generation rewrites indirect references and simulates messages containing entity values, enabling queries that refer to information outside the explicit request.The generation instructions require simulated messages to contain the entity and avoid referential terms such as “that” or “this.”Source: §E.6 · pp. 19–20
E.7 Multi-turn Prompt
- For missing parameters, the multi-turn prompt asks concise questions only about missing inputs and excludes parameters already present.It instructs the agent to ask about multiple missing parameters together.Source: §E.7 · p. 20
F Extra Experiment Details
These experiments examine snapshot-level performance across dialogue trajectories, slot changes, and external information. They report challenges with slot overriding and pronoun-based inputs, as well as parameter-name hallucinations on imperfect instructions.
Imperfect instructions and parameter-name hallucinations
- The first snapshot has higher PHR than the last across LLMs, and PHR is positively correlated with parameter-name hallucinations; correct parameter names typically coincide with Other Error Rate≈0%.The reported findings emphasize the importance of precise initial instructions.Source: F Extra Experiment Details · p. 21
Single-turn results and experiment details
- For single-turn data, imperfect instructions have much higher PHR than other dataset types, while external individual information has lower end-to-end accuracy.The experiments use datasets generated by open-source LLMs; evaluating GPT-4o costs approximately $75 per evaluation.Source: F Extra Experiment Details · p. 21, Table 8 · p. 22
Conversation success across Q&A trajectories
- Function-calling progress tends to be higher in mQmA and sQmA trajectories, where users provide multiple slot values at once and answer model questions faster.Different Q&A trajectories produce varying disturbances to overall success rates.Source: F Extra Experiment Details · p. 21, Table 12 · p. 25
Argument shifts and external information
- Open-source models are generally slow to detect slot overrides, while GPT-4o and Claude 3.5 Sonnet handle them more effectively; external noise raises PMR and can leave crucial arguments unfilled.The table covers snapshot evaluations at slot overriding and pronoun-based external-information turns.Source: F Extra Experiment Details · p. 21, Table 13 · p. 25