PaperlayerView exact revision on arXiv

Source-linked AI summary

Repair, Not Improvement: Decomposing Constrained Decoding in Tool-Call Abstention

Janghoon Lee

arXiv:2608.13959v1cs.CL

TL;DR

Constrained decoding may undermine the abstention decision that safe tool-calling routers must preserve, a case prior work did not measure. This paper separates stopping from token masking and finds that constraints cost abstention accuracy—up to 29.5 points—while mainly repairing unreadable outputs rather than improving judgement.

  • Problem

    Prior evaluations measure reasoning, schema validity, self-correction, and invocation, but not whether models abstain when no tool applies.

  • Method

    The study compares three byte-identical decoding conditions to separate the grammar’s stopping effect from its token-masking effect.

  • Results

    −29.5 points was the worst abstention contrast, while repaired abstentions overwhelmingly contained no readable answer and none reflected scorer-refused judgements.

  • Takeaways & Limitations

    Report stop-token and grammar effects separately, and do not expect constrained decoding to improve tool-use judgement.

  • Takeaways & Limitations

    Because every benchmark row places the gold tool first, tool-selection accuracy alone cannot establish that models selected tools by reading the query.

Abstract

from arXiv · show

Function calling is what the recent accounting of constrained generation explicitly sets aside: it finds the decoder's contribution small for format constraints, then warns in its Section 7 against extrapolating where a constraint encodes a correctness requirement, and names function calling as one. Tool abstention is that case at its sharpest: an enum leaves the wording of an answer alone and narrows the set of answers there are, and declining to call anything is the first it drops. We measure the excluded case. Three conditions over one byte-identical prompt separate a grammar's two jobs: it fixes where generation stops as well as which tokens may be emitted. We evaluate open-weight models from 0.6B to 4B on matched English and Korean items, so the language comparison is made within item. Against an unconstrained decoder, prior work's contrast is negative on abstention in four of six cells with intervals excluding zero, worst -29.5 points, and positive with an interval excluding zero in none. The total is a sum with opposite signs: on the smallest model in Korean the stop token costs -20.0, the enum returns +19.5, and the two leave -0.5. What it recovers is form: of 698 abstentions repaired, 545 had no readable answer and 0 were judgements the scorer refused. On tool-needed items it is positive throughout; abstention leads because it is the preregistered measure, and the pooled number being kinder to the intervention makes moving to it worse rather than better. Both preregistered language claims fail.

1 Introduction

This paper measures function-calling abstention, a correctness-sensitive case that prior constrained-decoding work explicitly excludes, by separating termination control from token masking. It finds that enum constraints can repair malformed abstentions even while their total effect is negative, motivating matched-language evaluation and component-level analysis.

  • Motivation: Tool routers must both select applicable tools and decline when none applies, because indiscriminate calling can invoke an inappropriate API.Abstention is presented as the safety-critical routing function.
  • Prior work: Prior work attributes about −3.9 points to JSON-format prompting and about −1.6 points to enforcing that instruction during decoding, but excludes function calling because constraints may encode correctness.This paper treats that exclusion as a scope boundary to measure rather than extrapolate across.
  • Method: Three conditions over one byte-identical prompt separate unconstrained generation, stop-controlled generation, and enum-constrained generation.The design distinguishes the enum’s lexical masking effect from the stop token’s termination effect and reports their sum separately.
  • Results: ∆total is negative with an interval excluding zero in four of six abstention cells, reaches −29.5 points at worst, and is positive with an interval excluding zero in none.The cited contrast aligns with prior work because it combines lexical masking and termination against a baseline with neither.
  • Results: Of 698 repaired abstentions, 545 had no readable answer beforehand and 0 were judgements rejected only by the scorer’s ordering rule.The paper therefore characterizes the constraint’s benefit as repair rather than improvement, while noting that it also shifts decisions toward naming a tool.

2 Related Work

Prior work shows that constrained decoding can impose measurable performance and sampling costs, but largely omits tool abstention, multilingual evaluation, and small-model settings. This paper addresses that intersection by decomposing enum-constrained tool calling into prompt and decoder effects across matched English and Korean items.

  • Constraint costs: Five prior studies measure constraint costs across reasoning accuracy, schema validity, self-correction, and tool invocation, but none tests whether models still abstain when abstention is correct.An enum over tool names can remove abstention, a behavior deployed routers need to preserve.
  • Constraint costs: Mean −3.9 points arise from format instructions to the prompt versus mean −1.6 points from decoder enforcement, making enforcement appear nearly free in prior accounting.That accounting explicitly excludes function calling where constraints encode correctness requirements rather than presentation preferences.
  • Abstention and multilingual gaps: Lee et al. (2026b) report that an enum raises tool-selection accuracy but lowers accuracy on items whose correct answer is abstention, using an English-only measurement.This result reaches abstention from the perspective of adapting tool schemas to small models rather than measuring the broader constraint tax.
  • Abstention and multilingual gaps: Multilingual tool-calling work addresses parameter-language mismatches or trains large models, while evaluated datasets and systems lack an abstention axis and leave the sub-4B inference-time setting underexplored.Luo et al. evaluate Chinese, Hindi, and Igbo without no-call items; Garg et al. adapt a 111B model for Korean tool calling.
  • This paper’s contribution: This paper measures tool abstention under an enum in two languages on small models, separating decoder and prompt contributions with a third condition that changes output length without changing the grammar.Its design differs from work where a mask makes tool-call tokens unreachable: abstention remains an explicit option in the answer set.

3 Experimental Setup

The experiment pairs matched abstention and tool-needed items, then separates stopping effects from grammar effects using three decoding conditions over byte-controlled English and Korean prompts. The registered evaluation covers three Qwen3 models, while data limitations restrict the decomposition to six cells.

  • Sampling: 200 paired positions yield 200 abstention and 200 tool-needed items with matched queries, lengths, and vocabulary, without filtering for difficulty.Both halves of each sampled pair enter evaluation, preserving within-item matching across task types.
  • Decoding design: The same prompt supports contract-free, contract-short, and constrained decoding, isolating ∆mask from stopping cost ∆length and their combined ∆total.The constrained decoder allows the ten candidate names plus None; ∆mask compares against a format-controlled baseline, whereas ∆total compares against an unconstrained decoder.
  • Registered evaluation: 36,000 requests constitute the registered analysis: three Qwen3 models, two languages, three conditions, five seeds, and 400 items at temperature 0.The registered design is 3×2×3×5×400 = 36,000 requests, and confirmatory claims rest on it.
  • Data limitations: Only three of sixteen models have all three conditions in both languages, so ∆length and ∆total are unavailable for the other thirteen.The three-contrast decomposition therefore rests on six cells because contract-free was collected fully only for the three registered models.
  • Language manipulation: English and Korean prompts differ only in the translated user query, with both query occurrences replaced and every other byte asserted identical after re-parsing.Tool names, descriptions, instructions, and few-shot examples remain in English, representing a non-English user driving a language-invariant interface.

4 Results

The results separate grammar-induced stopping from enum masking: against the unconstrained baseline, abstention performance is mostly harmed, while the enum can repair losses caused by the stop token without improving judgement. The repair account is supported by simultaneous gains on tool-needed items, whereas the preregistered abstention measure—not the more favourable pooled metric—remains primary.

  • Decomposition: −20.0 points from stopping and +19.5 from enum masking leave −0.5 points for the smallest model in Korean, showing recovery of formatting rather than improvement beyond the starting point.The combined effect is read as the enum giving back what format control had taken away, not manufacturing better judgement.
  • Repair versus improvement: +32.5 points in tool-selection accuracy accompanies +19.5 points in abstention accuracy in the clearest cell, ruling against a degenerate always-abstain explanation.Across tool-needed items, the constraint raises accuracy in six of six cells with intervals excluding zero and lowers it in none.
  • Alternative explanations: +0.5 points is the average shift toward naming the first candidate on abstention items, with at most +3.0 in any cell; the direct off-first test is unavailable because it is 0 of 200 items.The gold tool is first in every row, so ordering limits the test of position bias, though the observed pull is small.
  • Confirmatory results: Against the unconstrained, unstopped baseline, abstention ∆total is −29.5 to +1.5 points, negative with intervals excluding zero in four of six cells and positive in none.Against the stop-token baseline, ∆mask is −24.0 to +19.5 and positive in two cells; ∆length is negative in four cells and positive in none.
  • Primary measure: +7.7 points is the pooled mean against −29.5 at worst on abstention alone, but abstention remains the preregistered headline because switching metrics after seeing the result is impermissible.The pooled column is negative in three of six cells, while the abstention column is negative in five; both are reported, but the registered measure leads.

5 Analysis

The analysis preregisters paired within-item comparisons and a language model testing whether abstention differences are explained by masked probability mass. It also treats masked_mass as a lower bound, examines model-size effects secondarily, and labels post hoc analyses exploratory.

  • Preregistered comparisons: The preregistered test is a single within-item comparison on abstention items.Contrasts are paired rather than compared across independent draws.
  • Uncertainty estimation: 10,000 percentile-bootstrap replicates jointly resample evaluation items and decoding seeds.Resampling both levels captures seed-driven accuracy variation, especially for smaller models.
  • Language analysis: The second registered analysis tests whether language effects remain after adding a language term to a model containing masked_mass.An interval covering zero supports mediation through mass; an interval excluding zero indicates residual language effects.
  • Mass measure: masked_mass is a lower bound because it sums only a truncated distribution head, with Table 7 quantifying head coverage.The coverage makes the bound interpretable.
  • Model-size analysis: The registered secondary observation examines whether the abstention penalty grows as models shrink by plotting ∆abstain against parameter count.The relationship is plotted separately for each language.
  • Exploratory analyses: Post hoc analyses are reported in §H as exploratory and carry no confirmatory weight.They do not affect the registered test.

6 Conclusion

Constrained decoding reduces abstention accuracy because its stopping and token constraints pull in opposite directions, while its gains repair unreadable outputs rather than improve judgement. The conclusion recommends reporting these effects separately and treating grammar as a form-repair mechanism, not a decision-improvement mechanism.

  • Decomposition: The grammar’s stopping rule and token restrictions affect abstention in opposite directions and can nearly cancel, so they should be reported separately.The third condition isolates these two components.
  • What the grammar buys: Grammar repairs mostly turn outputs with no readable answer into readable ones, without releasing correct decisions or improving judgement.The intervention moves the decision boundary toward acting when measured against free generation.
  • Limitations: The cell-level regression cannot distinguish the proposed account from headroom, the decomposition uses six cells, and independently varying format failure would be needed to settle the explanation.Contract-free data were collected in full for only three models.
  • Practical implication: For practice, report the stop token and grammar separately because a single result can conceal both a large loss and a large recovery.Grammar-induced readability is useful, but the conclusion does not support expecting better decisions.

Limitations

The study’s conclusions are limited by benchmark ordering, incomplete condition coverage, and identification and measurement constraints. External validity is also restricted by translated Korean queries, missing argument fields, and reliance on one inference stack and grammar backend.

  • Benchmark design: The gold tool is first in every released Subtask1 row, so tool-selection accuracy cannot by itself evidence query-based selection.The study’s data cannot test the relevant non-first-gold subset; a companion permutation note reports lower accuracy when the gold is first.
  • Coverage: Only three models have all three conditions, leaving the paper’s decomposition untestable for the other thirteen models.Those thirteen models support only the single contrast collected for all models, and exploratory analyses are identified as such.
  • Identification: At cell level, the two-term account is indistinguishable from baseline headroom because unreadable and readable error coefficients cannot be told apart.The item-level transition counts are not subject to this objection, but the cell-level regression does not support the proposed reading.
  • Experimental design: Distractors are disjoint across paired conditions and abstention prompts include two None examples, so within-item contrasts do not make Table 3’s columns directly comparable.The pairing controls query difficulty but not distractor difficulty; identical prompts difference out priming while leaving the columns unlike.
  • Measurement: Mention-based abstention scoring can match ordinary words inside justifications, while first-token top-k masked_mass is only a lower bound on constraint pressure.The released collision report quantifies residual ambiguity, and first-token measurement misses later constraint pressure.
  • External validity: Korean queries are translated and reviewed rather than natively authored, while argument mismatch and cross-stack behavior remain unmeasured.Only user queries are translated; MetaTool Task2 lacks argument fields, and results use one inference stack and grammar backend.

Ethics Statement

The study uses public benchmark data without human subjects or personal-data processing, and releases full per-request records to support verification of its safety-relevant claim.

  • Ethics Statement: The evaluation uses the public MetaTool benchmark, released under the MIT license.
  • Ethics Statement: No human subjects were involved, and no personal data was processed.
  • Ethics Statement: The release includes full per-request records so the safety-relevant claim can be independently checked.The claim concerns whether a common deployment technique degrades models’ ability to decline to act.

B Item construction … E Scoring rules

The benchmark uses uniformly sampled, byte-verified English–Korean prompt pairs with a documented translation and review process. It preserves lookup-key names, holds the execution interface fixed, and applies explicit candidate-restricted matching rules.

  • B Item construction: 200 selected row positions are uniformly sampled without replacement, with item identities, selection seed, source-file hashes, and exclusions recorded.Only translation drift permits exclusion, and it is reported with per-item reasons.
  • B Item construction: Korean prompts replace only the two user-query occurrences, while the harness asserts byte-identity for the preamble, tool list, few-shot block, and completion cue.Each produced prompt is re-parsed before evaluation.
  • C Which names stay in English: 37 of 200 item pairs required name decisions, using English for lookup keys and Korean for ordinary geography, languages, and nationalities.The affected items and reviewer rulings are released.
  • D Translation protocol: Keeping tool names and descriptions in English while translating only the query preserves the deployed execution-interface condition.Tool names, descriptions, instructions, and few-shot examples remain unchanged when the user’s language changes.
  • D Translation protocol: The used Korean draft was authored to released instructions after machine translation was discarded for name, role, question-form, script, and language errors.Structural checks cover English tool names, unexpected characters, and question status; machine drafts also receive round-trip bag-of-words token F1 checks.
  • D Translation protocol: All 200 pairs undergo native-speaker review, and Korean evaluation is blocked until the human-created review record is present.Reviewers may correct, accept, or exclude items, with per-item statuses released.
  • D Translation protocol: A 50-item natively authored Korean schema is released to address translationese concerns, but this work does not fill it.The native-Korean set is available as an empty schema rather than as evaluated data.
  • E Scoring rules: Scoring is case-insensitive, limited to each item’s ten candidates, boundary-aware, and longest-match resolving, with 20 hand-labelled tests fixing edge cases.The rules prevent substring matches and avoid counting shorter names inside longer overlapping names.

F Why the primary metric is paired · G The chat template and the first token

The paper replaces a preregistered cross-language difference-of-differences with a paired analysis that isolates whether constraints break abstentions one language survives. It also shows that chat templates alter first-token behavior, motivating raw-completion evaluation while acknowledging its off-label costs.

  • F Why the primary metric is paired: The preregistered absolute difference-of-differences was replaced before confirmatory data, with the change and rationale recorded in PREREG.md.The retired metric remains reported so readers can assess whether metric choice mattered.
  • F Why the primary metric is paired: Absolute differences and retained fractions can rank languages oppositely when their baselines differ, because abstention accuracy is bounded and the difference-of-differences embeds those baselines.The retention ratio can also become misleading near a floor; the paired statistic instead conditions on absolute levels.
  • F Why the primary metric is paired: Pairing conditions on items both languages answered correctly unconstrained, making both baselines 100% and targeting cross-language differences in constraint breakage.Items lost in both languages cancel in the paired comparison.
  • F Why the primary metric is paired: The retention ratio divides constrained abstention accuracy by contract-short accuracy, not contract-free accuracy.It is informative away from the floor but can exaggerate small movements from near-zero baselines.
  • G The chat template and the first token: Raw completion prompts place the next token after User query and tool:, whereas chat templates tend to elicit tool: None and a justification before the answer.That first token does not begin any enum member, so the grammar removes the entire first position.
  • G The chat template and the first token: The masked first-token mass is higher under the chat template in all six ladder-model language cells, with every interval excluding zero.The two larger models saturate, while the smallest does not; candidate-name-starting tokens also fall under the template in every cell.
  • G The chat template and the first token: The confirmatory design uses completions because chat-path first-token distributions vary too little for the mechanism regression, especially where saturation matters most.This choice is off-label because the checkpoints’ model cards document chat usage rather than text completion.
  • G The chat template and the first token: Under the chat template, the same models produce fewer unreadable outputs and the two larger models abstain more accurately than on the raw-completion path.The paper reports this as the cost of measuring the deployed completion configuration, where constraint and template expectations collide at the first token.

H Exploratory analyses

The exploratory analyses were added after the registered results and carry no confirmatory weight or revision of the §4 verdicts. They describe how the produced numbers are arranged across three additional tables.

  • Exploratory analyses: The analyses were added after seeing the registered results and do not revise the verdicts in §4.The section explicitly states that these analyses carry no confirmatory weight.
  • Exploratory analyses: Table 14 quantifies how the masked_mass coefficient changes when a language term enters the model and decomposes language’s total effect on flipping.The decomposition separates the portion carried by masked mass from the portion that is not.
  • Exploratory analyses: Table 15 models recovery as a dependent variable and adds item contract-short difficulty as a covariate to the flip model.Recovery is defined as an abstention repaired rather than broken by the constraint.
  • Exploratory analyses: Table 16 reports the flip rate of each language.

H.1 The effect against the level it is taken from · I Request outcomes and the execution environment

The constraint’s benefit follows baseline headroom rather than a fixed format cost: both baseline-error components predict the contrast, while model-tier sensitivity intervals still cover zero. Request failures are excluded from accuracy but reported, and aggregated results require a single execution environment.

  • H.1 The effect against the level it is taken from: The effect is bounded by the unconstrained abstention level, so low-baseline cells have less to lose and high-baseline cells less to gain.Table 17 regresses the masking effect on that level, with one point per model-language cell.
  • H.1 The effect against the level it is taken from: The decomposition uses sixteen models across two languages, while the preregistered regression is restricted to three named models and withholds its coefficients.The larger set includes released-registry models with complete primary-contrast condition pairs, selected for licence, size, and family coverage.
  • H.1 The effect against the level it is taken from: Every tier-dropping subset has an interval covering zero, so the verdict does not resolve model-composition sensitivity.Table 18 refits the analysis after removing whole tiers.
  • H.1 The effect against the level it is taken from: Both halves of baseline error predict the contrast, their coefficients are indistinguishable, and together they explain no more than baseline performance alone.The error is split into items with no readable answer and readable but incorrect answers; these components sum to one minus baseline by construction.
  • I Request outcomes and the execution environment: Failed requests are excluded from accuracy rather than counted as model errors, but their counts are reported to make the exclusion visible.Outputs containing reasoning blocks are scored wrong rather than treated as infrastructure failures.
  • I Request outcomes and the execution environment: Results aggregated across many hosts require a single execution-environment row, because multiple environment columns would confound environment differences with seed variance.Table 21 records the execution environment as a precondition for sound aggregation.

J Schema-translation control

An exploratory condition translates Korean tool descriptions while retaining English tool names, separating non-English user-query effects from non-English schema effects. It is outside the confirmatory design.

  • J Schema-translation control: The release adds an exploratory condition in which tool descriptions are translated into Korean while tool names remain English identifiers.This condition is not part of the confirmatory design.
  • J Schema-translation control: The condition separates the effect of a non-English user query from the effect of a non-English schema.
  • J Schema-translation control: Intervals resample cells and one seed per cell, but the exploratory cells share a benchmark and sometimes a family, understating dependence.The intervals exclude zero only when bolded, and the zero crossing is −a/b.
Loading 2608.13959v1…