Source-linked AI summary
CCTU: A Benchmark for Tool Use under Complex Constraints
Junjie Ye, Guoqiang Zhang, Wenjie Fu, Tao Gui, Qi Zhang, Xuanjing Huang
TL;DR
LLMs must use tools while satisfying complex constraints, but dedicated evaluations of this integrated capability have been lacking. CCTU addresses the gap with a taxonomy-based benchmark and executable step-level validation, finding that strict task completion remains below 20% for every model and that the benchmark’s coverage is bounded by its taxonomy and single data source.
Problem
Existing evaluations do not systematically measure integrated tool use, instruction following, and self-refinement under complex constraints.
Method
CCTU combines 12 constraint categories across four dimensions, 200 curated test cases, and executable step-level validation during multi-turn interactions.
Results
No evaluated model exceeds a 20% task completion rate under strict constraint adherence, while models violate constraints in over 50% of cases.
Takeaways & Limitations
CCTU exposes substantial differences and limitations in current models’ instruction-following and self-refinement capabilities under constrained tool use.
Takeaways & Limitations
The taxonomy omits some real-world constraint categories, and the benchmark’s single-source construction may limit coverage of tool-use scenarios.
Abstract
from arXiv · showhide
Solving problems through tool use under explicit constraints constitutes a highly challenging yet unavoidable scenario for large language models (LLMs), requiring capabilities such as function calling, instruction following, and self-refinement. However, progress has been hindered by the absence of dedicated evaluations. To address this, we introduce CCTU, a benchmark for evaluating LLM tool use under complex constraints. CCTU is grounded in a taxonomy of 12 constraint categories spanning four dimensions (i.e., resource, behavior, toolset, and response). The benchmark comprises 200 carefully curated and challenging test cases across diverse tool-use scenarios, each involving an average of seven constraint types and an average prompt length exceeding 4,700 tokens. To enable reliable evaluation, we develop an executable constraint validation module that performs step-level validation and enforces compliance during multi-turn interactions between models and their environments. We evaluate nine state-of-the-art LLMs in both thinking and non-thinking modes. Results indicate that when strict adherence to all constraints is required, no model achieves a task completion rate above 20%. Further analysis reveals that models violate constraints in over 50% of cases, particularly in the resource and response dimensions. Moreover, LLMs demonstrate limited capacity for self-refinement even after receiving detailed feedback on constraint violations, highlighting a critical bottleneck in the development of robust tool-use agents. To facilitate future research, we release the data and code.
1 Introduction
Constrained tool use requires models to coordinate tool calling, instruction following, and self-refinement throughout dynamic interactions, yet existing evaluations do not assess these capabilities together. CCTU addresses this gap with a complex-constraint benchmark and executable validation, revealing severe performance limitations.
- Motivation: Constrained tool use combines accurate tool selection, persistent constraint adherence, and adaptation during multi-turn interactions.Practical deployments impose latency, tool-access, and response-format constraints.
- Research Gap: Existing benchmarks evaluate tool use, instruction following, and self-refinement largely in isolation, leaving their integrated performance underexplored.A model may invoke tools correctly yet violate constraints, or follow instructions while misidentifying tool roles.
- CCTU: CCTU organizes 12 constraint categories across resource, behavior, toolset, and response dimensions and curates 200 challenging test cases.Cases average seven constraint types and exceed 4,700 tokens in average prompt length.
- CCTU: The benchmark uses an executable module for step-level validation and compliance enforcement during multi-turn model-environment interactions.The module provides feedback when violations occur.
- Findings: Under strict adherence requirements, the best model completes fewer than 20% of tasks, while models violate constraints in over 50% of cases.Violations are especially common in resource and response dimensions, and self-refinement after detailed feedback remains limited.
2 Related Work
Prior work evaluates tool use and instruction following across increasingly complex settings, but benchmark coverage remains fragmented. CCTU extends this landscape by targeting integrated constrained tool-use performance with executable process-level validation.
- Tool-Use Evaluation: Tool-use evaluations span diverse interactions and increasingly complex multi-hop and parallel scenarios, but often focus on final query success.Prior work provides limited control over the intermediate process.
- Instruction Following: Instruction-following benchmarks progress from template-based constrained instructions to longer, more complex prompts and agentic settings.Many evaluate whether static responses violate explicit constraints.
- CCTU’s Position: CCTU differs by evaluating constrained tool use through executable step-level compliance checks during interactions.This addresses the gap between isolated capability evaluations and integrated constrained execution.
3 CCTU
CCTU constructs a diverse, long-context benchmark around a four-dimensional taxonomy and validates constraints throughout tool-use execution. Its data combine varied domains, multi-constraint cases, and executable quality checks.
- Constraint Taxonomy: The taxonomy identifies 12 representative constraints organized into resource, behavior, toolset, and response dimensions.These dimensions structure the construction of diverse and challenging tool-use cases.
- Benchmark Construction: The construction pipeline comprises prompt sourcing, automated constraint integration, executable step-level validation, and manual quality control.The source dataset supplies diverse subquery structures, while manual review checks conflicts and logical inconsistencies.
- Data Characteristics: CCTU spans 28 domains, including specialized fields such as politics and sports and everyday domains such as culture and tourism.The source data cover single-hop, parallel single-hop, multi-hop, and parallel multi-hop subquery relationships.
- Data Characteristics: Most instances contain 3,000 to 7,000 tokens, with an average length of 4,754 tokens measured using the Qwen3 tokenizer.Tool descriptions are included in tokenization, and multi-turn execution further increases effective context requirements.
- Evaluation Reliability: The framework supports precise evaluation through locally executable tools, annotated tool responses, and code-based compliance checks at each interaction.This enables assessment of both subquery resolution and constraint adherence.
4 Experimental Setup
The evaluation tests nine top-tier models in thinking and non-thinking modes using metrics that distinguish constraint-tolerant solving from perfect compliance. It reports means across repeated runs under default API settings.
- Models: Nine top-tier LLMs are evaluated separately in thinking and non-thinking modes.The models include Claude Opus 4.6, DeepSeek-V3.2, Gemini 3 Pro, GPT-5.1, GPT-5.2, Kimi 2.5, OpenAI o3, Qwen3.5-Plus, and Seed-2.0-Pro.
- Metrics: Solve Rate measures whether all subqueries are solved while all constraints are satisfied, allowing soft satisfaction after refinement.Soft satisfaction denotes an initially violated constraint that is subsequently refined.
- Metrics: Perfect Solve Rate measures whether all subqueries are solved without any constraint violations.PSR is stricter than SR because it excludes soft-satisfied constraints.
- Evaluation Protocol: Models use official APIs with default inference hyperparameters, except for thinking-mode toggling, and results average three independent runs.The study reports means and standard deviations to mitigate sampling variability.
5 Experiments
CCTU reveals that current LLMs struggle with constrained tool use, with violations concentrated in resource and response constraints and self-refinement remaining inconsistent.
- Main Results: PSR remains below 20% for all models, with most below 15%; GPT-5.2 is 14.67% lower on parallel multi-hop than single-hop tasks.In thinking mode, GPT-5.2’s PSR exceeds Kimi K2.5’s by more than 10%.
- Main Results: Thinking mode usually improves performance, but overthinking can cause models to rewrite correct parameters and preserve errors.Seed-2.0-Pro improves by 4.83% in PSR and 2.16% in SR, whereas Claude Opus 4.6 and Kimi K2.5 show exceptions.
- Main Results: High SR does not guarantee high PSR: GPT-5.2 follows instructions better but self-refines less effectively, whereas Claude Opus 4.6 shows the opposite strategy.The results indicate that both instruction following and self-refinement contribute to strong constrained tool-use performance.
- Constraint Violation Analysis: More than 50% of instances contain constraint violations for every model, reaching 86.83% for DeepSeek-V3.2, especially in resource and response dimensions.Resource errors commonly involve tool-call counts, while response errors involve omitting required content when summarizing tool outputs.
- Constraint Violation Analysis: Thinking mode shifts error patterns unpredictably: it can reduce specific tool-call-count violations while increasing overall invocation violations.The authors attribute this pattern to more careful allocation planning combined with additional exploratory calls.
- LLM Self-Refinement Analysis: Claude Opus 4.6 reaches the highest correction rate at 65.36%, while OpenAI o3 reaches 18.57% despite detailed feedback.Correction remains limited in behavior and response dimensions, and toolset correction is not perfect for some models.
- LLM Self-Refinement Analysis: Thinking mode improves self-refinement for some models but lowers correction rates for others that may double down on incorrect approaches.Higher correction rates are reported for Claude Opus 4.6, DeepSeek-V3.2, GPT-5.2, and Qwen3.5-Plus.
- LLM Self-Refinement Analysis: A 52.77% correction rate for DeepSeek-V3.2 does not yield high overall performance because constraint violations and weak function calling remain substantial.Claude Opus 4.6 combines strong function calling with stronger self-refinement, resulting in higher SR.
6 Conclusion
CCTU benchmarks tool use under complex constraints through a structured taxonomy, curated cases, and executable compliance checking. Evaluations reveal substantial challenges for current LLMs and differences in instruction-following and self-refinement abilities.
- Conclusion: CCTU uses a four-dimensional, twelve-category constraint taxonomy and contains 200 curated test cases.An executable validation module performs step-level compliance checking during multi-turn interactions.
- Conclusion: Evaluations of nine state-of-the-art LLMs in thinking and non-thinking modes reveal challenges and differences in instruction following and self-refinement.The benchmark is intended to provide insights for advancing LLM development.
A Limitations
CCTU’s limitations concern coverage: its taxonomy omits some real-world constraint categories, its data comes from one source, and its benchmark contains only 200 cases.
- The 12-category taxonomy does not cover every constraint category encountered in real-world production environments.The authors propose adding further constraint types and more challenging data in future work.
- CCTU is built from a single data source, so its distribution may not cover all possible tool-use scenarios.The source covers all combinations of subqueries and diverse domains, partially mitigating this limitation.
- 200 test cases limit benchmark scale because CCTU inherits the size of its original dataset.Each case averages seven constraint types and prompts exceeding 4,700 tokens, and an automated generation pipeline supports future expansion.
B Example of Constructed Data
The constructed-data example combines a complex historical question, many tools, executable implementations, and strict operational constraints governing tool-dependent execution.
- Test-case structure: Each test case includes a system prompt, user query, tool schemas, executable tool implementations, and constraint-validation code.These components define the agent’s role, task, available functions, feedback-generating tools, and executable checks.
- Operational constraints: The example prohibits answering from internal knowledge and requires all information to come from tool executions.Unverifiable information must be treated as unavailable.
- Operational constraints: Independent subtasks must be executed in parallel, while dependent subtasks must be executed sequentially.The workflow requires dependency analysis before tool invocation.
- Resource constraints: The agent must operate within a maximum of 20 interaction rounds and stop immediately when the limit is reached.At the limit, it must provide the best final answer using only accumulated information.
- Self-correction: The example requires retries after tool errors, including diagnosis, plan correction, and continued execution until the subtask succeeds.The corrective plan may modify parameters, search terms, or tool selection.
- Tool and response constraints: Tool calls must follow schemas and constraints, including a maximum of one invocation for philosopher_concept_identifier.The final response must use only tool observations and end with a period.
C Pipeline for Benchmark Construction
The benchmark-construction pipeline generates reference trajectories, injects compatible constraints, validates instances and code, and applies iterative human quality control.
- Pipeline overview: The pipeline starts from an initial dataset and taxonomy, then produces a constrained dataset with validation code.It processes each instance through trajectory generation, constraint expansion, filtering, integration, validation-code generation, and verification.
- Trajectory generation: Reference trajectories are generated by solving all unsolved subqueries before constraints are added.The trajectory records the tool-use path for each original instance.
- Constraint integration: Constraints are injected selectively when compatible with the scenario, using model-conditioned expansion across constraint types.Toolset constraints are excluded from this expansion step.
- Constraint integration: LLM-based filtering verifies each constrained instance and revises constraints when conflicts are detected.The constrained task context is then combined with the original instance and selected constraints.
- Validation-code generation: Validation code is generated for every constraint and added to the dataset alongside each constrained instance.The code is subsequently reviewed during data verification.
- Quality control: Human verification continues until two consecutive annotators agree that both data and code pass inspection.Six computer-science graduate students worked independently, and all annotations were completed within three weeks.
- Quality control: More than 70% of data required no manual intervention, and modification rounds did not exceed two.Code required slightly more modification than instructions because Qwen3-32B was not designed specifically for code generation.
D.2 Identified Issues during Data Verification
Data verification identifies conflicts among constraints, mismatches with scenario structure, and conflicts with tools; code verification finds overly strict and redundant validation logic.
- Data verification: Constraint conflicts can require revising incompatible output requirements, such as valid JSON combined with ending the response with a period.The revised requirement asks that the response contain a JSON object rather than making the entire output JSON.
- Data verification: A single-hop scenario with an initially required minimum of three interaction rounds was corrected by converting that lower bound into an upper bound.The change preserves logical consistency and solvability despite trial-and-error reference trajectories.
- Data verification: A tool-call limit was increased from one to two when answering required retrieving specifications for two distinct devices.The adjustment aligned the constraint with the operational requirements of the example.
- Code verification: Validation-code review addresses overly strict criteria and redundant logic across constraint handlers.These issues are corrected through manual annotation.
- Code verification: A Markdown check was relaxed so that any one of a heading, list, or emphasis element counted as success.The change prevents false negatives caused by requiring all three elements simultaneously.
- Code verification: Word-count checks were removed from a JSON-format validator when they duplicated a separate response-length constraint.The correction preserves modularity and prevents duplicate penalization.
E Case Studies
The case studies examine why thinking-mode performance can stagnate or worsen, focusing on overthinking and increased confidence in incorrect solutions.
- Thinking-mode behavior: Thinking-mode gains are not universal: Claude Opus 4.6 and Kimi K2.5 particularly do not benefit from switching from non-thinking mode.The paper attributes this behavior largely to overthinking, illustrated in Figures 13 and 14.
- Self-refinement: Overthinking can cause models to become more confident in initially incorrect solutions, limiting behavioral adjustment after feedback.Seed-2.0-Pro is presented as a case study of this pattern in Figure 15.
- Case-study coverage: Figures 13–15 provide model-specific case studies of overthinking and confidence in incorrect solutions.Figures 13 and 14 concern Claude Opus 4.6 and Kimi K2.5, while Figure 15 concerns Seed-2.0-Pro.
F Prompts for Benchmark Construction
The benchmark-construction prompts guide constraint expansion, task-context integration, and executable validation-code generation through staged templates and structured output requirements.
- Pipeline overview: Qwen3-32B assists data construction, with prompt templates spanning Figures 16–30.The templates cover constraint expansion, filtering, context integration, and validation-code generation.
- Target Agent Profile: The target agent is tool-driven, resilient to failed calls, and capable of parallel execution.The profile requires obtaining information through tools, correcting failed parameters, and invoking multiple tools in one action.
- Main Directive: Constraint expansion selects specific new constraints from a categorized list and incorporates them into an updated constraint list and question.The prompts use the original question, toolset, and correct trajectory while requiring explicit natural-language constraints.
- Data Generation Requirements: Generated constraints must preserve valid solution paths, avoid overfitting to one trajectory, and avoid mandating specific tools or parameter values.These requirements preserve strategic autonomy while maintaining compatibility with the example trajectory.
- Notes: The generated outputs include structured reasoning, updated constraints, a new specific constraint, and a question incorporating the new constraint.The required response format is explicitly specified in the construction prompts.
- Notes: Task-context integration first creates neutral background and then synthesizes a specialized agent system prompt from a generic template and task context.Stage I targets concise contextualization without solutions, while Stage II performs prompt synthesis.
- Notes: Specific tool-call-count prompts extract independent maximum call limits for each specified tool and return them in JSON format.The constraint prohibits further use of a tool after its individual limit is reached.
- Notes: Validation-code prompts cover sequential dependencies, parallel dependencies, parallel-call counts, available tools, parameters, length, and format or content.The appendix includes corresponding templates and validation snippets, including recursive schema validation for parameter types.