AGENTIF: Benchmarking Instruction Following of Large Language Models in Agentic Scenarios

1 Introduction

• AGENTIF targets the underexplored problem of following long, complex instructions in real-world agentic scenarios.Such instructions include extended system prompts, detailed tool specifications, condition constraints, and example constraints.Source: §1 · pp. 1–2
• Figure 1 summarizes AGENTIF's instruction-length distribution and representative-model success rates across constraint dimensions.The figure links the benchmark's extended instructions with model performance across different constraint dimensions.Source: Figure 1 · p. 2, §1 · p. 2
• AGENTIF collects 50 agentic tasks and 707 manually annotated instructions, whose constraints receive code, LLM, or hybrid evaluation methods.The benchmark distinguishes formatting, semantic, and tool constraints, as well as vanilla, condition, and example representations.Source: §1 · p. 2
• Fewer than 30% of instructions are followed perfectly by the best-performing model, while condition and tool constraints introduce particularly difficult challenges.The authors also report that performance declines as instruction length increases and identify meta constraints as a further failure mode.Source: §1 · pp. 2–3

2 Related Work

• Existing benchmarks commonly evaluate short, often synthetic instructions, leaving a gap from realistic agentic scenarios.IFEval, for example, averages 45 words, while later work expands constraint types, multilinguality, multi-turn settings, and code generation.Source: §2 · p. 3
• SysBench and AgentOrca approach agentic instruction following but remain limited by short prompts, absent tool usage, or narrow focus on function invocation.These limitations leave complex system prompts and diverse constraint types insufficiently covered.Source: §2 · p. 3
• AGENTIF fills this gap with real-world industrial and open-source agentic data, comprehensive annotations, and instructions averaging about 1,700 tokens and 14 constraints.The benchmark is presented as a challenging evaluation setting for current LLMs.Source: §2 · p. 3

3.1 Constraint Taxonomy

• AGENTIF defines constraints by both evaluation aspect and presentation form, covering formatting, semantic, tool, vanilla, conditional, and example constraints.Tool constraints capture agent-specific requirements such as parameter types, restricted tool sets, and avoiding internet access; conditional constraints apply only when specified conditions are met.Source: §3.1 · p. 4

3.2 Dataset Construction

• The benchmark is built through instruction collection, constraint extraction, and evaluation-design stages, with human review throughout the annotation process.The extraction process segments instructions into semantic blocks, validates constraints across blocks, and manually checks completeness and consistency.Source: §3.2 · pp. 4–5

3.3 Data Statistics

• AGENTIF contains 707 instructions averaging 1,723 words and 11.9 constraints, with substantial coverage of real-world agentic tasks and diverse constraint types.The instructions come from 40 open-source agents and 10 industrial workflows; the resulting annotation contains 8,415 constraints.Source: §3.2 · p. 5, §3.3 · p. 6

3.2 Dataset Construction

• Each constraint receives code, LLM, or hybrid evaluation according to its verification needs, with conditional constraints checked for activation before evaluation.Code evaluation handles deterministic properties, LLM evaluation handles open-ended semantics, and hybrid evaluation combines segment identification with code validation.Source: §3.2 · p. 5

3.3 Data Statistics

• AGENTIF is designed to be realistic, substantially longer than prior benchmarks, and complex, with many instructions exceeding 3,000 words.Its constraints include tool, condition, and example constraints, reflecting structures found in real agentic scenarios.Source: §3.3 · pp. 5–6

3.4 Evaluation Protocol

• The evaluation protocol reports constraint success rate (CSR) for satisfied individual constraints and instruction success rate (ISR) for instructions whose constraints are all satisfied.For each instruction, CSR aggregates constraint-level satisfaction, whereas ISR requires complete satisfaction across the instruction.Source: §3.4 · p. 6

4.1 Experimental Setup

• AGENTIF evaluates diverse advanced LLMs at temperature 0, including non-thinking, thinking, and instruction-following models, using only final responses for reasoning models.The experiments cover models such as GPT-4o, o1-mini, DeepSeek-R1, Crab, and Conifer.Source: §4.1 · p. 6

4.2 Main Results

• 27.2 ISR is the highest reported perfect-instruction score, while o1-mini reaches a CSR of 59.8 and GPT-4o drops from 87.0 on IFEval to 58.5 on AGENTIF.These results show that existing models remain far from perfectly following constraints in agentic scenarios.Source: §4.2 · p. 7
• Condition and tool constraints are especially difficult: models must determine whether conditional requirements trigger and must correctly invoke specified tools, names, and parameters.Models perform relatively better on example constraints, suggesting that in-context examples can support requirement inference and imitation.Source: §4.2 · p. 7
• Thinking models generally outperform other evaluated models, whereas academically instruction-tuned models perform worse, indicating a role for reasoning and test-time scaling.The paper suggests exploring stronger training approaches, including reinforcement learning, for instruction following.Source: §4.2 · p. 7

4.3 Error Analysis

• Condition-constraint errors include incorrect condition checks and failures to follow triggered requirements, with incorrect checking accounting for above 30% of errors.The controlled experiment removes conditional components to distinguish checking failures from general constraint-following failures.Source: §4.3 · p. 7, Figure 4 · p. 8
• Tool-constraint errors are dominated by disallowed tool usage and omission of required tools, alongside incorrect tool names and parameters.Thinking models more frequently neglect required tools, possibly because they rely more on internal knowledge.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 Analysis of Instruction Length and Constraint Quantity

• Model performance generally declines as instruction length or constraint count increases, and ISR is nearly 0 for all models above 6,000 words.The authors suggest decomposing overly long tasks into shorter instructions and collecting long-instruction post-training data.Source: §4.4 · pp. 8–9, Figure 5 · p. 8

4.5 Analysis of Meta Constraints

• Meta constraints appear in approximately 25% of instructions, with constraint selection producing the lowest model success rates.The three categories are constraint selection, constraint detailing, and constraint prioritization; conflicts with original constraints may explain selection failures.Source: §4.5 · p. 9, Figure 6 · p. 9

• AGENTIF comprises 707 instructions across 50 real-world agentic applications, averaging 1,717 tokens and 11.9 constraints per instruction.Source: §5 · p. 9
• Current models generally perform poorly on AGENTIF, and the best model perfectly follows fewer than 30% of instructions.Source: §5 · p. 9
• Condition and tool constraints introduce new challenges for existing LLMs.Source: §5 · p. 9
• Performance degrades as instruction length increases, indicating that longer agentic instructions are more difficult to follow.Source: §5 · p. 9
• The authors encourage further research to enhance instruction-following capabilities in agentic scenarios.Source: §5 · p. 9

• Semi-automated construction still requires substantial manual verification, limiting direct generalization to large-scale data generation.Source: A Limitations · p. 13
• AGENTIF contains only Chinese and English instructions, limiting broader multilingual coverage.Source: A Limitations · p. 13
• Experiments use zero-shot evaluation and do not explore prompt engineering techniques such as in-context demonstrations.Source: A Limitations · p. 13

B Ethical Considerations

• AGENTIF uses licensed agent sources, annotator-created queries, anonymization, safety checks, and informed, fairly paid human annotation.Source: B Ethical Considerations · p. 13

C.1 Constraint Type

• Constraints are classified as formatting, semantic, or tool constraints, covering output structure, intended meaning, and computational or external resource use.Source: §C.1 · pp. 13–14

C.2 Constraint Presentation Type

• Constraint presentation is categorized as vanilla, condition, or example, depending on whether requirements are unconditional, triggered by conditions, or implied by demonstrations.Source: §C.2 · p. 14

• Figure 7 presents constraint distributions across semantic, formatting, and tool categories, with vanilla, condition, and example presentation types.Source: Figure 7 · p. 14
• Semantic constraints and vanilla activation are the most prevalent categories shown in the dataset distribution.Source: Figure 7 · p. 14
• Templates cover instruction collection, constraint annotation, conditional-check generation, and three evaluation types.Source: §D.1 · p. 14

• Graduate students from computer science participated in annotation and were compensated according to pre-agreed salaries and workloads.Source: §D.2 · p. 14
• Annotation employment was formalized through contracts and conducted in compliance with local regulations.Source: §D.2 · p. 14
• Three rounds of sampled review and feedback iteratively refined and finalized the annotation results.Source: §D.2 · p. 14

E Experimental Details

• The evaluation covers thinking, non-thinking, instruction-tuned, and distilled language models, including models from OpenAI, Qwen, DeepSeek, Anthropic, Meta, and Mistral.Source: E Experimental Details · p. 15
• Temperature is fixed at 0, with a 32,000-token generation limit or each model’s shorter context limit.Source: E Experimental Details · p. 15
• Each evaluation round costs approximately $20 using gpt-4o-2024-11-20 as the evaluator.Source: E Experimental Details · p. 15
• Appendix tables document prompts for constraint extraction, classification, conditional decomposition, verification, and evaluation generation.Source: Table 4 · p. 16, Table 5 · p. 17, Table 6 · p. 18

NeurIPS Paper Checklist

• The checklist asks whether the paper’s claims match its results, contributions, assumptions, and intended scope.Source: NeurIPS Paper Checklist · p. 21

2. Limitations

• Authors are expected to disclose strong assumptions, robustness to violations, and the scope of claims, especially when results use few datasets or runs.Source: §2 · p. 21
• Limitations reporting should address factors affecting performance, computational efficiency and scaling, and applicable privacy or fairness concerns.Source: §2 · p. 21
• The checklist records that the paper has no theoretical assumptions and proofs, and reports experimental-result reproducibility as addressed in Appendix E.Source: §2 · p. 22
• Experimental reproducibility requires sufficient information about settings, compute resources, and procedures, whether or not code and data are released.Source: §2 · pp. 22–24
• Responsible release guidance covers societal harms, misuse risks, safeguards, asset licensing, documentation, and human-subject protections.Source: §2 · pp. 25–26