Instruction-Following Evaluation in Function Calling for Large Language Models

1 INTRODUCTION

• Formatting requirements in JSON-schema parameter descriptions are a critical part of function calling, and overlooking them can lead to invalid calls and downstream agent failures.Examples include requiring a capitalized string or an ISO 8601 date.Source: §1 · p. 1
• Existing function-calling benchmarks often assess functional correctness or API selection but overlook whether arguments follow required formats.The paper describes this omission as leaving a crucial dimension of agent robustness under-evaluated.Source: §1 · p. 2
• IFEval-FC evaluates verifiable format instructions embedded in JSON-schema parameter descriptions using automated checks for reproducible evaluation.The benchmark is inspired by IFEval and avoids reliance on human or LLM-as-a-Judge evaluation.Source: §1 · p. 2

2 VERIFIABLE INSTRUCTIONS

• IFEval-FC evaluates precise formatting constraints embedded in JSON-schema parameter descriptions, extending function-calling benchmarks beyond functional correctness and API selection.The approach adapts verifiable instructions from IFEval.Source: §2 · p. 2

2.1 INSTRUCTION CATEGORIES AND TYPES

• The benchmark organizes 19 instruction types into seven categories, adapting many from IFEval and introducing types such as Cyrillic Greek and Python List Format.Table 1 gives examples of format-instruction categories encoded in parameter descriptions.Source: §2.1 · p. 2, Table 1 · p. 3

2.2.1 FUNCTIONS

• Functions combine BFCL schemas enhanced with constraints and GPT-5-generated schemas across 80 domains, with instructions injected into a suitable string parameter.Generated functions include a free-form natural-language parameter without an enum or format field; BFCL functions use a randomly chosen string parameter without those fields.Source: §2.2.1 · p. 2

2.2.2 USER QUERY GENERATION

• GPT-5 generated five conversational queries per function, supplying required parameter values without embedded formats so the model must transform values to meet constraints.Queries are based on the original functions, before format instructions are embedded.Source: §2.2.2 · pp. 2–3
• Tasks were retained only when the function was necessary but not uniformly too difficult or too easy across five queries; an LLM ensemble supported validation.Because Claude Opus 4.1 models more often refused function calls, evaluation added a system message requiring function invocation without clarification.Source: §2.2.2 · p. 3

2.3 INSTRUCTION DIFFICULTY AND FILTERING

• Instruction types that yielded 90–100% accuracy across models, including trivial case constraints, were excluded to preserve the benchmark’s discriminative power.Source: §2.3 · p. 4

2.4 IFEVAL-FC METRICS

• The metric assigns 1 when a response follows a verifiable instruction and 0 otherwise, and the authors evaluated models using this score.Source: §2.4 · p. 4
• Figure 1 presents overall performance across format constraints, with higher scores indicating better adherence.Source: Figure 1 · p. 4, §2.4 · p. 4

3 DISCUSSION AND FUTURE WORK

• No evaluated model exceeded 80% accuracy, although the latest models performed significantly better than their predecessors.The authors conclude that precise instruction following in function calling remains an open problem for LLMs.Source: §3 · p. 4
• The current setup offers only one function, so it does not test selecting the correct function from alternatives; adding that challenge may lower observed performance.Future work also proposes expanding the function set and selecting more difficult samples.Source: §3 · p. 4
• The benchmark currently focuses on English-language function calls; future iterations could add multilingual support to assess cross-lingual instruction following.Source: §3 · p. 4

• Table 2 presents evaluation results for GigaChat models.Source: Table 2 · p. 6
• Table 3 presents evaluation results for Anthropic models.Source: Table 3 · p. 6
• Table 4 presents evaluation results for OpenAI models.Source: Table 4 · p. 7