CCTU: A Benchmark for Tool Use under Complex Constraints

• Constrained tool use requires accurate function calling [20], reliable instruction following [8], and self-refinement [9], while practical deployments impose latency, tool-access-frequency [19], and response-formatting constraints [17].The benchmark’s validation module performs step-level verification throughout multi-turn interactions.Source: §1 · p. 1, Figure 1 · p. 2
• Prior evaluations examine tool selection and invocation [21] [19] [28] [33] [34], instruction compliance [16; 37], or self-refinement [9] [23] separately, leaving integrated constrained tool use underexplored.A model may follow constraints yet fail to distinguish tools, or select tools correctly but fail to adhere consistently during interactions.Source: §1 · pp. 1–2
• CCTU comprises 200 test cases spanning 12 constraint categories across four dimensions, averaging seven constraint types and prompts exceeding 4,700 tokens, with verified annotations and step-level validation.Its executable validation module checks compliance across multi-turn interactions and provides feedback when violations occur.Source: §1 · p. 2, Figure 1 · p. 2
• The best-performing model achieves less than 20% task completion under strict adherence, most models fall below 15%, and models violate constraints in over 50% of cases, especially in resource and response dimensions.LLMs also struggle to self-refine after receiving detailed constraint-violation feedback.Source: §1 · p. 2

Evaluations for Tool Use

• Tool-use evaluations span diverse scenarios and increasingly complex multi-hop and parallel tasks, but mostly measure eventual query-solving rather than constrained action trajectories.The work instead focuses on planning tool-use trajectories under specified restrictions and analyzing how constraint types affect performance.Source: §2 · p. 2

Evaluations for Instruction Following

• Instruction-following evaluations progressed from template-based constraints to longer, more complex instructions and LLM-as-a-judge methods [16] [31].Early studies generated simple constrained instructions; later approaches increased instruction length and complexity.Source: §2 · pp. 2–3
• Agentic instruction-following studies [17] mainly check explicit constraints in static instructions, whereas this work validates compliance step by step during multi-turn model-environment interactions.The executable constraint validation module conducts step-level checks throughout those interactions.Source: §2 · p. 3

3.1 Constraint Taxonomy

• The taxonomy organizes 12 representative constraints into resource, behavior, toolset, and response dimensions.It covers interaction and tool-call limits; sequential and parallel dependencies and parallel-call limits; permitted tools and parameters, required parameters, and parameter types; and response length, format, and content.Source: §3.1 · pp. 3–4

3.2 Benchmark Construction

• 200 cases were built from FTRL’s four subquery-structure categories (50 each) using trajectory-guided constraint expansion, consistency filtering, task contexts, step validators, and two-annotator checks.The validation module checks outputs after each step, gives detailed violation feedback for revision, and uses executable code generated for each instance’s constraints.Source: §3.2 · pp. 4–5

3.3 Data Analysis

• 28 domains are represented, and each case combines 4–12 constraint types (7 on average), with resource and toolset constraints in every instance.Behavior constraints are less common because they depend on specific scenario structures, while constraints from the other three dimensions appear in most cases.Source: Figure 2 · p. 5, §3.3 · pp. 5–6, Figure 4 · p. 6
• 4,754 tokens is the average instance length, with most cases spanning 3,000–7,000 tokens; multi-turn interaction further increases context length.Lengths are measured with the Qwen3 tokenizer and include tool descriptions.Source: Figure 5 · p. 6, §3.3 · p. 6
• Locally executable tools and annotated responses enable subquery-level scoring, while executable validators check stepwise constraint compliance and support evaluation of self-refinement.Together, these components enable evaluation of problem-solving progress, constraint adherence, and revision after violations.Source: §3.3 · p. 6

4.1 Models

• The evaluation covers Claude Opus 4.6, DeepSeek-V3.2 [4], Gemini 3 Pro, GPT-5.1, GPT-5.2, Kimi 2.5 [25], OpenAI o3, Qwen3.5-Plus, and Seed-2.0-Pro [22], each in thinking and non-thinking modes.Source: §4.1 · p. 6

4.2 Metrics

• SR requires solving all subqueries while satisfying constraints, allowing later refinement, whereas PSR requires solving all subqueries without any constraint violations.A constraint initially violated but subsequently refined is considered soft-satisfied.Source: §4.2 · pp. 6–7

4.3 Implementation Details

• Models use official API interfaces with default inference hyperparameters apart from thinking-mode toggles; results average three independent runs and include standard deviations.Source: §4.3 · p. 7

5.1 Main Results

• Below 20%, every model’s PSR remained (most below 15%); GPT-5.2 achieved the highest PSR, while Claude Opus 4.6 achieved the highest SR.GPT-5.2 violated fewer constraints but struggled to self-refine after errors; Claude Opus 4.6 made more violations but corrected behavior more effectively.Source: §5.1 · pp. 7–8
• GPT-5.2’s PSR was 14.67% lower in parallel multi-hop tasks than in single-hop tasks.In thinking mode, its PSR exceeded Kimi K2.5’s by more than 10%.Source: §5.1 · p. 7
• Seed-2.0-Pro’s PSR was 4.83% higher and its SR 2.16% higher in thinking mode, but Claude Opus 4.6 and Kimi K2.5 sometimes overthought and altered correct parameters.Those parameter changes caused errors to persist and occurred less often in non-thinking mode.Source: §5.1 · p. 7

5.2 Constraint Violation Analysis

• More than 50% of instances involved constraint violations for every model, with DeepSeek-V3.2 reaching 86.83%; violations were especially frequent in resource and response constraints.Tool-call-count restrictions were a common resource error, while response errors often came from omitting required content when summarizing tool outputs.Source: Figure 6 · p. 8, §5.2 · p. 8
• Thinking mode shifted error patterns inconsistently, while hallucinated tools, omitted required parameters, and incorrect parameter types still caused tool invocation failures.For GPT-5.1, thinking reduced violations of specific tool-call-count constraints but increased violations of overall tool-call-count limits; the authors suggest extra deliberation can prompt exploratory calls.Source: §5.2 · p. 8

5.3 LLM Self-Refinement Analysis

• 65.36%, Claude Opus 4.6’s correction rate was highest; other models remained below 60%, and OpenAI o3 reached 18.57%.The validation module supplied detailed feedback, yet some models failed to correct all Toolset errors, with Behavior and Response errors also remaining difficult.Source: Figure 7 · p. 9, §5.3 · p. 9
• Claude Opus 4.6, DeepSeek-V3.2, GPT-5.2, and Qwen3.5-Plus had higher correction rates in thinking mode, while other models showed no improvement or declined.The authors attribute some declines to models doubling down on incorrect approaches during extended reasoning.Source: §5.3 · p. 9
• 52.77%, DeepSeek-V3.2’s overall correction rate in thinking mode did not prevent relatively low overall performance.The paper attributes this to more constraint violations and weaker function-calling; Claude Opus 4.6’s stronger function-calling and self-refinement supported higher SR.Source: §5.3 · p. 9

6 Conclusion

• CCTU comprises 200 curated test cases grounded in a four-dimensional, twelve-category constraint taxonomy.It targets tool use under complex constraints.Source: §6 · p. 9
• An executable constraint-validation module enables step-level compliance checking during multi-turn interactions.Source: §6 · p. 9
• Evaluation of nine state-of-the-art LLMs in thinking and non-thinking modes highlights CCTU's challenges and differences in instruction-following and self-refinement capabilities.Source: §6 · p. 9

A Limitations

• CCTU’s 12-constraint taxonomy across four dimensions cannot capture every constraint category found in real-world production environments.The authors propose expanding the taxonomy and constructing more challenging data in future work.Source: A Limitations · p. 13
• Because CCTU is built from the single source FTRL, its data distribution reflects that dataset’s characteristics, though domain diversity partially mitigates the limitation.The authors note that FTRL covers all combinations of subqueries and that CCTU spans diverse domains.Source: A Limitations · p. 13
• CCTU contains 200 test cases because its source dataset, FTRL, provides 200 instances.Each case involves an average of seven constraint types and prompts exceeding 4,700 tokens; the authors propose an automated pipeline for expansion to other datasets and conduct three independent repeated experiments to improve evaluation reliability.Source: A Limitations · p. 13

B Example of Constructed Data

• Each CCTU sample combines a constrained system prompt, user query, tool schemas and implementations, validation codes, scenario category, expected answer, and unresolved tool-execution set.These are the benchmark’s listed core components.Source: B Example of Constructed Data · p. 13

Tools / Code Implementations

• The example’s Tools use JSON function schemas, while Code Implementations provide executable Python behavior for those functions.The excerpt shows schemas for two tools and notes that additional tool schemas and implementations are omitted.Source: B Example of Constructed Data · p. 13, Tools · pp. 15–16, Code Implementations · p. 16

Constraint Validation Codes / Others

• Constraint Validation Codes implement step-level checks, and the Others entry identifies a parallel multi-hop scenario with answer “a” and its required execution trajectory.The displayed trajectory runs from identifying Plato through locating the Liberty Bell and retrieving the dates 1876 and 1752.Source: B Example of Constructed Data · p. 13, Constraint Validation Codes · p. 16, Others · p. 16

System Prompt

• The System Prompt makes tool outputs the sole source of truth and requires the agent to manage an autonomous plan–action–observation–reflection loop.The sample frames the agent as an autonomous historical-temporal analysis system and prohibits reliance on internal knowledge.Source: System Prompt · p. 14

1. Complex Task Decomposition

• For complex objectives, the agent must decompose the task into atomic subtasks and create a roadmap before acting.This requirement is stated as the sample’s task-decomposition instruction.Source: §1 · p. 14

2. Dependency Analysis and Parallel Execution

• The agent must sequence dependent tasks and batch independent tool calls for simultaneous execution.The sample requires explicit analysis of dependencies between planned subtasks.Source: §2 · p. 14

3. Precision in Tool Selection

• The sample limits execution to 20 rounds, requires schema-compliant tool selection and ordered calls, and directs retries after tool errors.Its protocol also specifies thought, action, reflection, and final-answer phases, and requires stopping at the round limit.Source: §3 · pp. 14–15

User Query

• The User Query asks which is older: a monument linked through a philosopher, reform movement, politician, and province, or the telephone’s invention.The prompt also instructs the agent to obtain information through tools and retry corrected calls after failures.Source: User Query · p. 15

C Pipeline for Benchmark Construction

• For each task, the pipeline extracts unsolved subqueries, toolset, and scenario, then samples model trajectories until all subqueries are solved.The completed reference trajectory is stored for later constraint expansion.Source: C Pipeline for Benchmark Construction · p. 17
• Compatible non-toolset constraints are independently injected with probability 0.5, then revised until the task and constraints pass consistency verification.Source: C Pipeline for Benchmark Construction · p. 17
• The pipeline adds a generated task context and generates validation code for each constraint before adding the instance to the dataset.Source: C Pipeline for Benchmark Construction · p. 17

D.1 Overview

• Over 70% of the 200 instances required no manual intervention, and modification rounds did not exceed two.The reported code-modification share was slightly higher than the instruction share; the authors attribute this to using Qwen3-32B, which was not specifically designed for code generation.Source: Table 3 · p. 18, §D.1 · p. 18

D Details of Human Annotation · D.1 Overview

• Data and validation code undergo iterative annotation and revision until two consecutive annotators independently agree they pass inspection.For data, annotators review and revise instances when needed; code receives a parallel review and revision process.Source: C Pipeline for Benchmark Construction · p. 17, §D.1 · p. 18
• Six computer-science graduate students annotated independently, completing all annotations within three weeks.Source: §D.1 · p. 18

D.2 Identified Issues during Data Verification

• Data verification grouped issues into conflicts among constraints, mismatches with scenario structure, and conflicts between constraints and tools.These categories frame the manual corrections described below.Source: §D.2 · p. 18
• A JSON-only requirement conflicted with requiring the response to end in a period, so annotators required a JSON object within the response instead.This retained the composite-constraint test while resolving the incompatibility.Source: §D.2 · p. 18, Figure 8 · p. 19
• For a single-hop task, annotators changed a minimum of three interaction rounds to a maximum so the task remained logically consistent and solvable.The minimum had arisen from LLM-referenced trajectories containing trial-and-error steps.Source: §D.2 · p. 18, Figure 9 · p. 20
• When answering required two specification-tool calls but the constraint allowed one, annotators raised the permitted maximum from one invocation to two.The task queried specifications for two distinct devices.Source: §D.2 · p. 18, Figure 10 · p. 20

D.3 Identified Issues during Code Verification

• Code verification identified overly strict evaluation criteria and redundant logic across constraint handlers as its two main issue types.The corrections aimed to prevent unfair evaluation and duplicate penalization.Source: §D.3 · p. 18
• For Markdown validation, annotators changed the pass condition from requiring a heading, list, and emphasis to requiring any one of those elements.This relaxation was intended to prevent false negatives and make formatting evaluation fairer.Source: §D.3 · p. 19, Figure 11 · p. 21
• Annotators removed word-count checks from the JSON format validator because they belonged to response-length validation, preserving modularity and avoiding duplicate penalties.The LLM-generated format checker had combined logic for distinct constraints.Source: §D.3 · p. 19, Figure 12 · p. 22

E Case Studies

• Several models, particularly Claude Opus 4.6 and Kimi K2.5, do not benefit from thinking mode, largely because of overthinking.The case studies contrast their performance across thinking and non-thinking modes.Source: E Case Studies · p. 23
• Claude Opus 4.6 changes multiple parameters at once, while Kimi K2.5 infers alternative locations instead of testing the countries named in the query, causing both to fail in thinking mode.In non-thinking mode, Claude adjusts parameters after a failed query, and Kimi sequentially tests Israel and Palestine to retrieve the answer.Source: E Case Studies · p. 23, Figure 13 · p. 24, Figure 14 · p. 25
• Seed-2.0-Pro ignores feedback to make parallel calls to specified tools, assuming a prior call makes another invocation unnecessary.The analysis suggests extended reasoning can increase confidence in an initial incorrect solution and prevent behavior adjustment from feedback.Source: E Case Studies · p. 23, Figure 15 · p. 26

Prompt Template for Controlled Constraint Expansion

• The controlled-expansion prompt asks an expert constraint generator to analyze a user query and toolset and produce rigorous, testable constraints for an agent.Figure 16 presents the prompt template for controlled constraint expansion.Source: Prompt Template for Controlled… · p. 27, Figure 16 · p. 28

Target Agent Profile

• The target agent is assumed to obtain answers through tools, recover from errors by correcting and retrying calls, and invoke multiple tools in parallel.The main directive uses the original question, available toolset, and correct trajectory list as inputs.Source: Target Agent Profile · p. 27

Data Generation Requirements

• The generation requirements add a selected constraint, make it explicit in the revised question, and keep the correct trajectory feasible without prescribing its exact steps or tools.They also require nonconflicting constraints, references only to available tools, specific content, natural wording, and adherence to the response format.Source: Data Generation Requirements · p. 27, Notes · p. 27

Notes

• The appendix presents templates for LLM-based filtering, two-stage task-context integration, and validation-code generation for interaction rounds.These templates are shown in Figures 17–20.Source: Figure 17 · p. 28, Figure 18 · p. 29, Figure 19 · p. 30
• Additional validation-code generation templates address tool-call counts, specific tool-call counts, sequential and parallel dependencies, and parallel-call counts.These templates are shown in Figures 21–25.Source: Figure 21 · p. 32, Figure 22 · p. 33, Figure 23 · p. 34
• Validation-code snippets explicitly enforce available-tool and parameter constraints, required parameters, and parameter types through recursive schema validation.These snippets appear in Figures 26–28.Source: Figure 26 · p. 37, Figure 27 · p. 37, Figure 28 · p. 37
• The appendix also provides a validation-code generation prompt for length constraints and a separate prompt for format and content constraints.These prompts appear in Figures 29 and 30.Source: Figure 29 · p. 38, Figure 30 · p. 39