PaperlayerView exact revision on arXiv

Source-linked AI summary

Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints

Fangzheng Li, Aimin Zhang, Chen Lv

arXiv:2606.25605v1cs.CL

TL;DR

The interaction between tool calling and structured-output constraints is insufficiently understood because they are often evaluated separately. This paper uses controlled experiments and mechanism analysis to study their joint deployment, reproducing tool suppression and showing that Transparent Two-Pass Execution restores tool invocation while preserving structured output.

  • Problem

    The interaction between tool calling and structured output under joint deployment has received limited attention, while evaluations typically assess each capability separately.

  • Method

    The paper conducts controlled experiments across models and settings, analyzes grammar-based token masking, and evaluates a two-pass inference-time mitigation strategy.

  • Results

    When tool calling and schema constraints are activated simultaneously, tool execution disappears despite successful schema generation, while both remain functional when assessed independently.

  • Takeaways & Limitations

    Evaluating tool use and structured output independently may not fully capture reliability in Agent systems operating under simultaneous constraints.

  • Takeaways & Limitations

    The proposed mitigation relies on external orchestration and does not eliminate the underlying mechanism responsible for suppression.

Abstract

from arXiv · show

Tool Calling and Structured Output are two core capabilities of modern Agent systems, yet their interaction under joint deployment conditions remains insufficiently understood. This paper reports a reproducible phenomenon observed in a production Agent system: when Tool Calling and JSON Schema constraints are simultaneously enabled, multiple open-weight models cease invoking tools despite maintaining high schema compliance. We refer to this behavior as Tool Suppression. Through controlled experiments across multiple model families and deployment settings, we consistently reproduce Tool Suppression under joint constraints, while tool execution and schema compliance remain functional when evaluated independently. Further analysis reveals that JSON Schema constraints are compiled into grammar-based token masks, causing tool-call tokens to become unreachable during decoding. This provides an implementation-level explanation for the observed behavior. To interpret the phenomenon, we formulate the Constraint Priority Inversion (CPI) hypothesis, which suggests that schema satisfaction may dominate action-selection behavior under multiple simultaneous constraints. We present CPI as a behavioral hypothesis consistent with the observed evidence rather than a verified internal mechanism. To mitigate the problem, we propose Transparent Two-Pass Execution, an inference-time strategy that decouples tool execution from schema-constrained response generation. Experimental results show that this approach restores tool invocation while preserving structured output guarantees without requiring model retraining. These findings suggest that evaluating tool use and structured output separately may overlook important reliability issues in production Agent systems. Code, data, and docs will be released at https://github.com/Fzsama/Constrain-Tax-26-06.git.

1 Introduction

This introduction identifies a production failure in which jointly enabling Tool Calling and Structured Output suppresses tool invocation despite schema compliance. The paper investigates its mechanism, formulates CPI as a behavioral hypothesis, and proposes Transparent Two-Pass Execution and a suppression taxonomy.

  • Motivation: Production Agents increasingly require tool acquisition followed by schema-compliant responses, yet evaluations usually assess these capabilities separately rather than under joint deployment.The paper highlights that production workflows activate both capabilities simultaneously, while their interaction remains insufficiently understood.
  • Problem Observation: Under joint constraints, the Agent repeatedly failed to invoke tools while producing schema-compliant responses; disabling Structured Output restored tool use with other conditions unchanged.The observation persisted across repeated trials despite unchanged model weights, prompts, tool definitions, and task requirements.
  • Contributions: The paper defines this reproducible failure pattern across multiple open-weight models as Tool Suppression.Tool Suppression is characterized through controlled experiments in which tools cease being invoked when Tool Calling and Structured Output are simultaneously enabled.
  • Contributions: Inference-stack tracing localizes the phenomenon to grammar-based constrained decoding, where JSON Schema token masks render tool-call tokens unreachable.The paper presents this as an implementation-level explanation for the observed suppression phenomenon.
  • Contributions: The paper introduces CPI as a behavioral hypothesis, proposes Transparent Two-Pass Execution, and develops a TS-A to TS-E taxonomy of suppression patterns.Transparent Two-Pass Execution separates tool execution from schema-constrained generation and restores tool invocation while preserving structured output guarantees.

2 Background

Tool Calling enables Agents to use external systems through structured function requests, while Structured Output constrains responses for reliable programmatic consumption. This paper examines how these capabilities interact, showing that format constraints can propagate into action selection and suppress tool execution despite compliant outputs.

  • Tool Calling: Tool Calling lets language models access external resources for real-time information, environmental perception, and external execution.Tool-augmented Agents generate structured function call requests rather than relying solely on parametric knowledge.
  • Tool Calling: Agent tool use typically involves task understanding, tool planning, tool selection, and tool execution.Models must determine whether external information is needed, generate correct actions, and use returned results for subsequent reasoning.
  • Research Gap: Existing research has largely studied tool capability and structured-output effects separately, leaving their interaction in Agent execution systematically understudied.Prior work primarily examined tool correctness or answer accuracy, with less attention to whether format constraints alter tool execution.
  • Structured Output: Structured Output constrains responses through predefined schemas, commonly producing JSON objects, function parameters, or grammar-conforming text.Implementation approaches include API-layer schema constraints and decoding-layer grammar restrictions.
  • Tool Suppression: Tool Suppression occurs when structured-output constraints propagate from response generation to execution, causing tool execution to be skipped while outputs remain format-compliant.The paper frames this as Cross-Stage Interference and proposes Constraint Priority Inversion as a possible explanation mechanism.

3 Problem Definition

This section defines Tool Suppression as the systematic omission of tool execution under joint constraints despite preserved tool capability, and frames it as an Agent-level manifestation of the Constraint Tax. It introduces TIR and SR to measure tool execution and constraint-induced suppression, respectively, while distinguishing observable suppression patterns across deployment settings.

  • Tool Suppression Definition: Tool Suppression occurs when models that can identify genuine tool requirements and normally call tools systematically omit execution under joint constraints.The phenomenon requires a genuine tool requirement, normal tool-calling capability without joint constraints, and ultimate execution omission.
  • Tool Suppression Definition: Tool Suppression differs from Tool Incapability because the model retains tool-calling capability but fails to activate it under specific constraint conditions.Tool Incapability involves failing to identify tool requirements or generate valid calls, whereas suppression involves capability that is not activated.
  • Behavioral Manifestation: Tool Suppression is a selective failure in which models understand tasks, identify information gaps, and generate final responses while skipping the tool-execution stage.The degraded capability is Agent action execution rather than answer quality, making suppression a behavior-level manifestation of the Constraint Tax.
  • Measurement: TIR measures tool execution behavior by counting requests that successfully initiate at least one valid tool call, regardless of returned correctness or final-answer quality.Higher TIR values indicate greater inclination to execute tools, and TIR ∈[0, 1].
  • Measurement: SR compares constrained and baseline TIR to quantify suppression while reducing the influence of models’ inherent tool-calling tendencies.Higher SR indicates stronger suppression, and the section uses SR as a standardized indicator for cross-model comparison and constraint-effect analysis.
  • Suppression Taxonomy: The taxonomy includes omission, compensatory content generation, explicit recognition without execution, tool-free hallucination, and failure even when tool use is required.TS-C separates recognizing tool requirements from executing tools, while TS-E shows that suppression may persist under tool choice="required" settings.

4 Experimental Setup

The experiments isolate Tool Suppression by comparing tool-only, joint-constraint, and schema-only conditions across standardized tasks and diverse models. They measure tool execution and structured-output outcomes while controlling prompts, tasks, and recording artifacts.

  • Scope: The study focuses on tool execution under simultaneous Tool Calling and Structured Output constraints, rather than general reasoning quality.
  • Experimental Conditions: The controlled design compares T1 tool-only, T2 joint constraints, and T3 schema-only conditions with identical prompts, tools, parsing, and tasks.Only the tools and response format API parameters vary across conditions.
  • Tasks: Evaluation tasks sample five business-oriented categories and require external information unavailable from model parameters alone.The categories include Buyer Background Analysis, Market Intelligence Search, Product Knowledge Retrieval, and Compliance and Risk Investigation.
  • Models: Seven model instances span 20B to 397B scales, MoE and Hybrid architectures, local and cloud deployments, and one closed-source reference model.
  • Metrics: The core metrics are Tool Invocation Rate (TIR), Suppression Rate (SR), JSON Compliance Rate (JCR), Average Tool Calls per Session (ATC), and End-to-End Success Rate (ESR).Each model runs 5 independent test rounds per condition with consistent tasks and fixed random-sampling parameters.
  • Validation: No tool-call delta events occurred in reported suppression cases, ruling out parser failures or logging artifacts as the explanation.

5 Empirical Findings

Across open-weight models, tool invocation succeeds without schemas but disappears when Tool Calling and Structured Output are enabled together, while schema compliance remains functional independently. Ablations rule out schema complexity, prompting, fine-tuning, and serving-stack differences, while grammar-constrained decoding makes tool-call tokens unreachable.

  • Cross-condition findings: Under T1, Tool Invocation Rate reached 100%, whereas under T2 no valid tool call events were observed in any evaluated session.T1 enabled tools without schemas; T2 enabled both tools and schemas, with unchanged task requirements and tool definitions.
  • Cross-condition findings: Under T3, models generally maintained high schema compliance, showing that tool execution and schema compliance function independently but fail jointly under T2.The failure emerged only when both constraints coexisted, rather than from either capability in isolation.
  • Cross-condition findings: All tested open-weight models exhibited complete T2 suppression across model families, parameter scales, deployment environments, and inference stacks, unlike GPT-5.4-mini.Within the evaluated model set, suppression was reproduced across diverse configurations rather than appearing as an isolated model defect.
  • Ablation findings: Tool Invocation Rate under T2 remained at 0% across simple, medium, and production schemas, with no threshold effect from schema complexity.Even relatively lightweight schema constraints were sufficient to trigger suppression.
  • Ablation findings: No tool call events were generated under T2 even when API-level tool choice="required", indicating suppression persisted beyond task interpretation or weak prompting.Results remained consistent across optional use, explicit system instructions, explicit user instructions, and API-level enforcement.
  • Ablation findings: Fine-tuning did not remove suppression, and SGLang and vLLM showed identical T2 tool-execution failure despite minor schema-compliance differences.The results indicate that conventional post-training and framework-specific parser or scheduling explanations are insufficient.
  • Mechanistic interpretation: Under grammar-constrained decoding, tool-call tokens become unreachable by design, establishing an inference-stack basis for the absence of T2 tool calls.This implementation-level cause does not preclude additional model-level behavioral factors.

6 Mechanism Analysis and Behavioral Interpretation

Grammar-based token masking provides the primary implementation-level explanation for Tool Suppression by making tool-call tokens unreachable under JSON Schema constraints, while behavioral patterns such as TS-C suggest a separate model-layer phenomenon. The CPI hypothesis offers a cautious interpretation of that model-layer behavior, but its internal-mechanism status remains unresolved.

  • Decoding Mechanism: Grammar-based token masking applies at every decoding step, assigning −∞ to tokens disallowed by the current JSON FSM state.For Qwen-family XML-style tool calls, the < character is never permitted in JSON states, making tool-call tokens unreachable throughout generation.
  • Decoding Mechanism: Because masking occurs after logits are produced and before sampling, model-weight optimization cannot overcome token-level exclusion.This explains why evaluated SFT and GRPO variants failed to eliminate suppression: the mask has no gradient path back to the weights.
  • Two-Layer Interpretation: Models sometimes output need search: true without tool calls, showing that tool requirements can be acknowledged even when execution is impossible.This TS-C pattern indicates that grammar-level exclusion explains missing calls but does not fully explain all observed behavior.
  • Two-Layer Interpretation: The two-layer interpretation separates decoding-level token exclusion from model-level recognition of tool requirements expressed in natural language or structured fields.The decoding layer explains absent tool calls, while the model layer explains TS-C patterns.
  • Behavioral Interpretation: CPI proposes that models may prioritize schema satisfaction when Tool Calling and Structured Output constraints coexist, but it remains a behavioral hypothesis rather than a verified internal mechanism.The observed evidence is consistent with CPI, although expressed tool awareness may instead reflect task representations independent of decoding constraints or training-distribution artifacts.

7 Mitigation Through Transparent Two-Pass Execution

Transparent Two-Pass Execution decouples tool execution from schema-constrained generation by invoking tools first and applying grammar constraints only during a second pass. It restores tool invocation and task completion while preserving JSON compliance, but adds inference overhead and remains an engineering workaround rather than a definitive solution.

  • Mechanism: The strategy avoids suppression by completing tool execution before grammar-based token masking makes tool-call tokens unreachable.It changes only the order in which constraints are applied and does not modify model parameters, decoding algorithms, or training procedures.
  • Method: Transparent Two-Pass Execution uses two sequential inference stages: unconstrained tool execution followed by schema-compliant response generation.The flow is User Request → Pass 1 Tool Execution → Tool Results Collection → Pass 2 Structured Output Generation → Final Response.
  • Results: Tool Invocation Rate increased from 0% to 100%, while JSON Compliance Rate remained unchanged and End-to-End Success Rate improved from complete task failure to successful completion.Restored tool execution enabled use of real external information and substantially reduced simulated retrieval and tool-free hallucination.
  • Overhead: The mitigation adds approximately one inference round plus tool execution time and increases token consumption because the second pass includes collected tool outputs.In the evaluated production workflow, the additional cost was considered acceptable because the suppressed baseline could not successfully complete the target tasks.
  • Limitations: Transparent Two-Pass Execution remains an engineering mitigation that relies on external orchestration and does not eliminate the underlying suppression mechanism.Its benefits beyond tool-calling scenarios involving workflow execution, MCP ecosystems, or multi-agent collaboration remain an open question.

8 Discussion

The findings show that independently strong tool use and structured-output performance can fail under joint constraints, motivating dedicated evaluation and orchestration. Conclusions remain limited to the evaluated models and tasks, while CPI and broader applicability require further validation.

  • Evaluation implications: Independent capability measurements may miss failures that emerge when Tool Calling and Structured Output constraints are activated simultaneously.Models executed tools under T1 and complied with schemas under T3, yet failed tool execution under T2.
  • Evaluation implications: Future evaluation suites should include explicit Tool Calling + Structured Output conditions rather than relying solely on independent capability measurements.Joint-constraint scenarios can expose behaviors that separate evaluations do not reveal.
  • Engineering implications: Grammar-based token masking mediates the interaction between structured outputs and tool execution, making explicit orchestration important for production reliability.Transparent Two-Pass Execution demonstrates one effective strategy by separating tool execution from grammar-constrained generation.
  • Model differences: All evaluated open-weight models exhibited complete suppression under T2, whereas GPT-5.4-mini maintained stable tool execution behavior.The study identifies a concrete grammar-masking root cause for the open-weight cases but does not establish a universal open-weight versus closed-source distinction.
  • Limitations: The conclusions apply to the finite evaluated model set and relatively small benchmark, not to all large language models.The study included multiple task categories but remained substantially smaller than large-scale academic evaluation suites.
  • Limitations: CPI remains a behavioral hypothesis, and broader effects across MCP, multi-agent, workflow, and computer-use architectures remain open questions.The study observes consistent external behavior without directly observing model internals or providing theoretical guarantees.

9 Future Research Directions

Future research should evaluate joint constraints directly, improve inference-framework support and transparency, investigate unresolved intent-without-action behavior, and test whether token exclusion affects agent behaviors beyond tool calling. These directions aim to quantify suppression, clarify its mechanisms, and determine its broader scope across models, formats, frameworks, and agent architectures.

  • Joint-Constraint Benchmarks: Dedicated benchmarks should evaluate Tool Calling + Structured Output jointly across schema complexities, tool-call formats, frameworks, and reliability metrics.Current benchmarks usually assess these capabilities separately, although suppression emerges when they operate simultaneously.
  • Joint-Constraint Benchmarks: Such benchmarks could quantify suppression prevalence and enable comparisons of mitigation strategies across model families and agent architectures.
  • Inference Frameworks: Inference frameworks could permit interleaved tool-call tokens within JSON Schema grammars and provide transparent feedback when masking prevents tool calls.Grammar-based token masking is identified as a concrete mechanism underlying Tool Suppression, while current masking is silent and debugging requires source-code tracing.
  • Inference Frameworks: Future studies should test compatibility among grammar constraints and XML, function-call, and MCP formats.
  • Unresolved Mechanisms: Research should investigate whether TS-C reflects independent tool recognition, learned intent-without-execution behavior, or output-distribution interaction with the mask.Proposed methods include activation analysis, controlled training experiments, and systematic grammar-constraint variation.
  • Beyond Tool Calling: Token-level exclusion may also affect workflow control tokens, multi-agent communication markers, and computer-use action sequences beyond tool calling.Future evaluations should examine structured control flows, specialized multi-agent formats, MCP action tags, and computer-use agents to assess whether this is a broader constraint-driven exclusion phenomenon.

10 Conclusion

The study identifies reproducible Tool Suppression when Tool Calling and Structured Output constraints are enabled together, tracing it to grammar-based token masking that makes tool-call tokens unreachable. It also proposes CPI as a behavioral hypothesis and Transparent Two-Pass Execution as an inference-time mitigation that restores tool invocation while preserving structured output guarantees.

  • Findings: Tool Suppression occurs under simultaneous Tool Calling and Structured Output constraints, although tool execution and schema compliance remain functional when evaluated independently.The failure persists despite unchanged tool definitions and task requirements, alongside successful schema generation.
  • Findings: Controlled experiments across models, scales, deployments, frameworks, schemas, invocation strategies, and post-training variants characterize the phenomenon and test alternative explanations.The results do not readily support model size, framework implementation, deployment configuration, schema complexity, prompt enforcement, or conventional instruction tuning as sufficient explanations.
  • Mechanism: Grammar-based token masking in constrained decoding renders tool-call tokens unreachable, localizing the concrete root cause to the inference-framework layer.Because this mechanism is independent of model weights, SFT and GRPO do not eliminate suppression.
  • Interpretation: Constraint Priority Inversion proposes that schema satisfaction may dominate action selection under joint constraints, but remains a behavioral hypothesis rather than a verified internal mechanism.CPI is presented as an interpretation consistent with the observed suppression pattern.
  • Mitigation: Transparent Two-Pass Execution separates tool execution from schema-constrained response generation and restores tool invocation while preserving structured output guarantees without retraining.The mitigation operates at inference time and does not require modifying model weights or training procedures.
  • Implications: Independent evaluations of tool use and structured output may miss reliability problems in Agent systems operating under multiple simultaneous constraints.The findings motivate further investigation of token-level exclusion mechanisms and interactions between inference-framework constraints and model behavior.

A Test Case Design and Tool/Schema Definitions

This appendix documents the test cases, tool and schema definitions, and experimental configurations used in the main experiments to support reproducibility and clarify the conditions of observed Tool Suppression.

  • Test Case Design: The appendix details the test case design used in the main experiments.These materials support reproducibility.
  • Tool and Schema Definitions: It specifies the tool definitions and schema specifications used in the experiments.The documentation clarifies the exact experimental conditions.
  • Experimental Configurations: It records the experimental configurations under which Tool Suppression was observed.The appendix is intended to clarify those conditions.

A.1 Controlled Experimental Design

The controlled design tested each evaluated model independently under three conditions, summarized in Table 13 as the T1/T2/T3 experimental conditions.

  • Experimental Conditions: Each evaluated model was tested independently under three conditions.The experimental configuration is summarized in Table 13.
  • Experimental Conditions: Table 13 summarizes the T1/T2/T3 experimental conditions.

A.2 Standardized Test Script Protocol … A.7 Experimental Environment

The study used standardized, repeated cross-model tests spanning diverse tasks, tool sets, schemas, inference frameworks, and experimental configurations. Tool-call detection combined API-level and content-level checks, which agreed on zero calls in over 200 T2 queries.

  • A.3.1 Standard Cross-Model Test Task: The standard evaluation used a fixed foreign-trade inquiry prompt requiring websearch, knowledge-base retrieval, and analysis across nine models with five repetitions per condition.The prompt specified a US lighting importer, LED strip lights, 2,000 reels, an FOB quote, and UL listing.
  • A.3.2 Extended Task Diversity: Qwen3.6-35B-A3B was additionally tested on 200+ queries spanning 10 company profiles and 8 compliance markets, with identical T2 TIR = 0% results.Synthetic coverage also included 200 Tool Mandatory prompts, 30 seed scenarios, and 6,000 Scale Synthesis prompts.
  • A.4.1 Production Environment Tool Set; A.4.2 Cross-Model Test Tool Set: The production Agent used websearch, knowledge_base, and fetchurl, whereas standard nine-model tests retained only websearch and knowledge_base with identical {‘‘query’’: string} signatures.The production tools supported web search, foreign-trade knowledge-base queries, and webpage retrieval.
  • A.5.1 Cross-Model Test Schema (4-field); A.5.2 Minimal Schema (3-field): Standard cross-model tests used a strict four-field inquiry_analysis JSON schema requiring buyer_background, product_analysis, recommendations, and key_findings.Fine-tuning ablations and GRPO instead used a strict three-field company_info schema requiring company_name, company_info, and compliance_notes.
  • A.5.3 Production-Grade Schema: The production-grade strict inquiry_analysis schema represented outputs as blocks and tracked tool_dependency fields including required, tools_used, claims, and reason.Its block variants included text and card objects, with additionalProperties allowed at the top level.
  • A.6.1 SGLang Server Configuration; A.6.2 vLLM Server Configuration: The primary deployment used SGLang on 2x A800 80GB GPUs, while vLLM was used for framework-independence validation with tensor parallelism set to 2.SGLang configured Qwen3 reasoning and qwen3_coder tool-call parsers; both configurations used 0.85 memory utilization.
  • A.6.3 Tool Call Detection Mechanism; A.7 Experimental Environment: Dual detection parsed streaming API tool_calls and scanned generated content for <tool call> XML tags; both methods agreed on zero tool calls in over 200 T2 queries.This agreement was used to rule out parser-level false negatives, and the experimental environment configuration is reported in Table 14.
Loading 2606.25605v1…